Confronting the Learning Crisis Lessons from World Bank Support for Basic Education, 2012–22 An Independent Evaluation © 2024 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: World Bank. 2024. Confronting the Learning Crisis: Lessons from World Bank Support for Basic Education, 2012–22. Independent Evaluation Group. Washington, DC: World Bank. COVER PHOTO Adapted from Shutterstock/Zurijeta and Shutterstock/wavebreakmedia EDITING AND PRODUCTION Amanda O’Brien Valentina Such GRAPHIC DESIGN Rafaela Sarinho Vonda Witley This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Confronting the Learning Crisis Lessons from World Bank Support for Basic Education, 2012–22 An Independent Evaluation October 10, 2024 Contents Abbreviations v Acknowledgments vi Overview vii Management Response xxi Report to the Board from the Committee on Development Effectiveness xxv  ackground���������������������������������������������������������������������������������������������������������������� 1 1 B Who Is Affected? 6 A Complex Problem 7 World Bank Approach to Learning in Basic Education 9 Evaluation Methods 13 Evaluation Questions 13 Limitations 14 The World Bank’s Global and Regional Analytics and Programs and Global 2  Partnerships������������������������������������������������������������������������������������������������������������18 Increasing Global Knowledge and Building Awareness 19 Sharing Global and Regional Knowledge 23 Convening and Partnership 25 Feedback Process to Strengthen Systems, Teaching, and Learning 29 The World Bank’s Approach to Basic Education and Learning Outcomes at the 3  Country Level����������������������������������������������������������������������������������������������������������33 Overall Financing for Basic Education 35 The World Bank Basic Education Portfolio 36 World Bank Analysis toward Learning for All at the Country Level 40 Effective Support for Policy Reform toward Learning for All 44 World Bank Inputs and Responses to Address Learning for All at the Country Level 49 Monitoring and Evaluation 68 World Bank Engagement with Development Partners and Other Stakeholders at the Country Level 73 ii 4 Conclusions and Recommendations----------------------------------------------------------------------82 Measurement 85 Equity 86 Support for Teaching Career Framework 87 Support for Capacity Development across All Levels of the System 88 Looking Ahead: Contributions to Learning 89 Recommendations 92 Bibliography����������������������������������������������������������������������������������������������������������������95 Boxes Key Concepts in Learning Measurement Box 1.1.  4 What Is Systems Thinking? Box 1.2.  8 Influence of the World Bank’s Regional Advisory Services and Analytics Box 2.1.  on Country Clients in Developing Projects or Making Policy Reforms 24 Learning Poverty and Its Effect on Global and Country Stakeholders Box 2.2.  toward Action on the Learning Crisis 26 Regional Partnerships’ Help with Identification of Issues Contributing to Box 2.3.  Low Learning 29 The Foundational Learning Compact Lacking Outcome Measures  Box 2.4.  31 The Project Ratings System Box 3.1.  38 Ethiopia: What Is Not Going Well and Why Box 3.2.  43 Examples of World Bank Support for Inclusion in Pakistan and Box 3.3.  Sierra Leone  57 Examples of World Bank Support for Children with Disabilities in Box 3.4.  Four Case Countries 59 Case Study Countries Facing Large Challenges from Box 3.5.  Population Growth 63 What Are Structured Pedagogical Approaches? Box 3.6.  64 Examples of World Bank Support for Data in Four Case Study Box 3.7.  Countries in the Africa Region 71 Global Partnership for Education Added Value for the World Bank Box 3.8.  at the Country Level 74 iii Figures Share of Students Achieving at Least Minimum Proficiency Levels Figure 1.1.  in Reading and Mathematics by Country Income Level 3 Conceptual Framework for Support of Quality Education Figure 1.2.  15 Basic Education Project Approval during Fiscal Years 2012–22 Figure 3.1.  37 Regional Distribution of All Projects and COVID-19 Support  Figure 3.2.  39 World Bank Lending and Grants for Basic Education during Figure 3.3.  Fiscal Years 2012–22 40 Levels of Financial and Political Commitment to Learning for All Figure 3.4.  in Brazil, Kenya, and Viet Nam 46 World Bank Project Support of Inputs in Areas Related to Figure 3.5.  Basic Education 51 World Bank Projects Targeting Equity-Related Issues Figure 3.6.  56 Projects Targeting Specific Groups with Indicators on the Figure 3.7.  Target Group 58 Tables Challenges of Most Concern in Project Appraisal Documents Table 3.1.  42 Capacity Building in Some Case Study Countries and Table 3.2.  Associated Results 53 Minimum Learning Proficiency in Reading and Mathematics at the Table 3.3.  End of Primary and Lower Secondary Education in Case Study Countries 67 Basic Education Quality–Related Indicators in Latest Country Table 3.4.  Partnership Frameworks in Case Study Countries 69 Appendix Appendix A. Evaluation Methods 112 iv Abbreviations ASA advisory services and analytics CPF Country Partnership Framework EMIS education management information systems FY fiscal year GP Global Practice GPE Global Partnership for Education GPG global public goods IEG Independent Evaluation Group PAD Project Appraisal Document PDO project development objective READ Russia Education Aid for Development SABER Systems Approach for Better Education Results SDG Sustainable Development Goal TTL task team leader UNESCO United Nations Educational, Scientific, and Cultural Organization UNICEF United Nations Children’s Fund WDR World Development Report All dollar amounts are US dollars unless otherwise indicated. Independent Evaluation Group World Bank Group    v Acknowledgments This evaluation was prepared by an Independent Evaluation Group team led by Susan Caceres, under the guidance and supervision of Galina Sotirova (manager, Corporate and Human Development Unit), Oscar Calvo-Gonzalez (director, Human Development and Economic Management Department, 2019–23), and Theo David Thomas (director, Human Development and Economic Management Department, 2023–present) and under the overall di- rection of Alison Evans (Director-General, Evaluation, 2019–23) and Sabine Bernabè (Director-General, Evaluation, 2023–present). The evaluation team consisted of Anthony Tyrrell, Xiaoxiao Peng, Mariana Branco, Jeffery Marshall, William B. Hurlbut, Dafni Skalidou, Sengphet Lattanavong, Disha Zaidi, Denise A. Vaillancourt, Judith Hahn Gaubatz, Sian Williams, Brian Gregory Allen, Rosen Asenov, and Julia Monteath de Franca. Yezena Yimer provided superb administrative support at all stages of the evaluation. Romina Paula Cicerello developed the graphics for the evaluation. The team was assisted in conducting the case studies by Hai Thi Thanh Pham, Irene Wangari Kinuthia, Arnaud Gotoraye, Henok Fasil Telila, Boimourod Bobodjanov, Maaheen Shah, Abijit Sharma, Naif Talal Mustafa, and Mark Vandi Bawoh. The initial stages of the evaluation benefited from foundational research, consultation with World Bank task team leaders, Confronting the Learning Crisis Acknowledgments and methodology advice from Estelle Raimondo. The team benefited from comments by members of Independent Evaluation Group staff, especially Diana Goldemberg, Rasmus Heltberg, and Estelle Raimondo, whose feedback helped strengthen the report. The report was peer-reviewed by Barbara Bruns (visiting fellow at the Center for Global Development and adjunct professor at Georgetown University), Paula Malan (senior education adviser for the Ministry for Foreign Affairs of Finland), and Michelle Kaffenberger (deputy director of research, Blavatnik School of Government, University of Oxford). vi Overview Key Messages The World Bank has helped build awareness, shared knowledge, and convened global stakeholders around a commitment to quality basic education. Its data and analytics and the World Development Report 2018 have drawn attention to the learning crisis and helped define learning poverty, which has encouraged stakeholder buy-in. The World Bank is well positioned to help address the learning crisis. Its relationship with governments and its role as the largest external education funder mean it can, with committed clients, help reform key aspects of the education system needed for inclusive learning. Overall, support could be more strategically focused and institutional incentives better aligned to deliver learning outcomes and take advantage of opportunities to reform education systems. Better monitoring—with enhanced disaggregation of data—would provide a feedback loop for adaptive management that is currently missing. World Bank financing for basic education delivers inputs into education systems and typically tracks outputs, with few operations assessing changes in systems, teaching, and learning. For example, 48 out of 77 projects with project development objectives that address improving learning have learning outcome data, and only 22 out of 188 operations with on-the-job training for teachers tracked the impact of the training on teachers’ practices. Country analytics focus on specific aspects of basic education systems (that is, what is not working rather than why systems are failing children). This makes it difficult to influence and implement changes related to the intertwined challenges that result in learning poverty. vii  Reference to marginalized groups and the level of analytic focus on broader equity-related issues increased over the evaluation period in Country Partnership Frameworks and Systematic Country Diagnostics, as well as in Project Appraisal Documents, where targeting of such groups has also increased; however, monitoring and disaggregated reporting of results predominantly focus on gender and not on other groups. The World Bank responded at the country and global levels to the learning losses arising from the COVID-19 pandemic, with significant additional funding and strengthened partnerships, new working coalitions, and alliances, which are of strategic value. World Bank support was a broad-based emergency response focused on remote learning and on reopening schools. viii   Tackling Low Levels of Learning Outcomes— A Complex and Costly Development Challenge Learning losses associated with school shutdowns implemented in response to the COVID-19 pandemic reinforced the international community’s atten- tion to the long-standing issue of the learning crisis phenomenon framed by the World Development Report (WDR) 2018 (World Bank 2018b). In 2019, before the COVID-19 pandemic, learning poverty—the share of children younger than 10 years of age who have not achieved minimum reading profi- ciency, as adjusted by the proportion of children who are out of school—was at 91 percent in low-income countries, compared with 9 percent in high-income countries (World Bank, UNESCO, et al. 2022). Since the pandemic, the situa- tion has worsened. Among the children most failed by their education systems are those already disadvantaged by poverty, location, ethnicity, gender, disability, and other factors. The capacity of those children to engage in and benefit from basic education is often stymied by factors that also affect the population at large, such as lack of investment in infrastructure and a shortage of qualified teach- ers. However, these children also often face a combination of barriers: distance from school, physical access to buildings and related facilities, the availability of adaptive equipment, language barriers, and so on. Learning for all is a much more difficult and expensive pursuit than access for Independent Evaluation Group World Bank Group    ix all. The latter, with an emphasis on more infrastructure and more teachers, is a simpler proposition (although literature reports may be open to clien- telism, patronage, and corruption [Grindle 2004; Kingdon et al. 2014]). The former requires greater discipline within the education system and its service delivery, potential loss of power for actors, and greater levels of account- ability throughout the system. Improving learning for all involves enhanced teacher quality, alignment of curriculum and textbooks with local contexts and local language and culture, and investment in education equity to ensure equality of opportunity (rather than equal access) for diverse student cohorts. The Global Education Monitoring Report estimates an annual financing gap of $97 billion during 2023–30 in 79 low-income and lower-middle-income countries to achieve the Sustainable Development Goal 4 targets established in 2012, which focus on the reduction of inequality in access to and quality of education (UNESCO 2023). This gap reflects the existing scale of the chal- lenge, which is expected to grow as a result of the projected demographic profile of many of the worst-affected countries. About This Evaluation This evaluation assesses the World Bank’s contribution to improving learning outcomes in basic education—defined as primary and lower secondary education—over the 2012–22 decade. It pays particular attention to the extent to which the World Bank has adopted a systems approach to its support for basic education as advocated in Learning for All: Investing in People’s Knowledge and Skills to Promote Development—World Bank Group Education Strategy 2020 and as reinforced since the publication of the WDR 2018 (World Bank 2011, 2018b). It is designed to identify lessons and present recommendations to inform any future education sector strategy. A conceptual framework informed the design of the methodological approaches and data analysis employed. The evaluation draws on portfolio and document analyses, interviews, country case studies, literature, and secondary data analysis. The starting point for this assessment was to develop and test a conceptual framework that captures the complexity of basic education systems and the system characteristics required to deliver quality education for all. The framework was aligned with the WDR 2018 and was used to assess the extent to which the World Bank response to the learning crisis—at global, regional, and country levels—can be characterized as a systems-based approach (that is, to what extent the World Bank’s approach has taken account of material and Confronting the Learning Crisis Overview intangible elements of basic education systems and the relationships between them, the structuring of basic education systems and subsystems, the systems’ formal and informal functions, and feedback loops that influence behavior within systems). The framework also identifies points at which the World Bank can, and typically does, intervene in basic education systems—strategizing, planning, implementing, and monitoring and evaluation—to provide support through policy dialogue, knowledge, financing, and other means. x The World Bank Has Highlighted the Crisis in Learning The World Bank’s knowledge and analytic work, partnerships, and global initiatives have contributed to global knowledge and awareness and have encouraged action by country clients to support improvements in system alignment and capacity, teaching, and measurement of learning. With the Commitment to Action, launched at the United Nations Secretary-General’s September 2022 Transforming Education Summit, the World Bank and partners advanced support for foundational learning. Since then, partners of the Global Coalition for Foundational Learning have encouraged more countries to sign the Commitment to Action.1 World Bank vice presidents have played a key role by encouraging ministers of education and finance in client countries to improve learning for all. For example, a high-level meeting in Latin America in March 2023 convened many regional partners and was followed by collaboration with the Inter-American Dialogue, the Inter-American Development Bank, and other partners to raise awareness of the region’s learning crisis (World Bank 2023a). In addition, the Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación at the United Nations Educational, Scientific, and Cultural Organization ensured regular data collection for calculating learning poverty through regional standardized tests (Colombia, Ministry of Education 2023). The World Bank’s high-quality data and analytics and the WDR 2018 have drawn attention to the learning crisis and addressed improvement in quality Independent Evaluation Group World Bank Group    xi and learning. The WDR 2018 renewed attention to the political barriers to progress and the need for more effort in client countries to measure learning and help identify system failures. Valuable global and regional contributions have supported assessing levels of education policy development (Systems Approach for Better Education Results [SABER]) and producing comprehen- sive regional reports (Facing Forward: Schooling for Learning in Africa [Bashir et al. 2018]). The World Bank provided dialogue and knowledge dissemination for clients who expressed a commitment to implement policies and programs to reform teacher status, evaluation, and remuneration informed by evi- dence and tailored to the local context (Great Teachers: How to Raise Student Learning in Latin America and the Caribbean [Bruns and Luque 2014]). The evaluation observes that influence with country clients is enhanced where there is support for follow-up and the ability to link global and regional ana- lytics to country programs. However, a supply-demand tension is often found with the more specific, country-level utility of global public goods and global knowledge. Increasing global public goods is difficult, particularly when gov- ernments are asked to finance data collection. The World Bank is the largest source of external financing for the education sector in low- and middle-income countries, although funding for the sector— and for basic education—is a small part of its overall lending. Commitments to the education sector in fiscal year 2022, from preprimary to tertiary levels, were just under 5 percent of total commitments. Primary education, the core of basic education, attracted about 1 percent of World Bank lending, whereas only about 1 percent of all official development assistance goes to education at any level, with less than half of that going to basic education. The portfolio for this evaluation consists of 236 basic education operations (or portions of them) approved during the evaluation period (fiscal years 2012–22), with a total commitment value of $25 billion. The portfolio, by number of projects, is concentrated in the Africa Region (44 percent), with 16 percent of projects each in the Latin America and the Caribbean and South Asia Regions. Eighty operations in the portfolio (34 percent) address the effects of COVID-19 on basic education systems. The most regularly sup- ported inputs across the entire portfolio are government-level management, in-service teacher training, and school management, which are supported in 88 percent, 80 percent, and 75 percent of projects, respectively. This lev- Confronting the Learning Crisis Overview el of concentration across the portfolio and similar patterns in case studies suggest a lack of nuanced response to the differing contexts and basic edu- cation systems within which the World Bank works, without a similar focus on the fundamental causes of context-specific education failure. Our analysis suggests that the types of input likely to be supported by the World Bank remained consistent across the evaluation period and across the primary instrument types—that is, investment project financing and Programs-for- Results. The success of operations is typically measured in outputs: teachers trained, textbooks in classrooms, and schools built. Although these are valid indicators for education access, they are ineffective for measuring changes to xii systems, teaching, and learning. Measuring Learning Outcomes Is Critical to Informing Policy Makers Learning poverty is an easily understood concept that has gained global stakeholder buy-in. The World Bank and multiple partners have supported ambitious targets to motivate global and country stakeholders toward collec- tive action and alignment on a single target and message. Reducing learning poverty was added to the World Bank’s corporate targets at the 2023 Annual Meetings of the World Bank Group and the International Monetary Fund, replacing the previous indicator of human capital (measured as students reached). The increased attention to outcomes is welcome; however, without further attention by the World Bank and partners to address the lack of under- lying data (to calculate learning poverty), the indicator will be unable to fill its critical global and corporate monitoring function. In every Region, learning poverty data are lacking for two or more countries. This indicator is represent- ed in 58 out of the 91 countries in the portfolio. Progress has been made in supporting systems to measure learning from national assessments in 44 countries and from subnational assessments in 6 countries, which will provide data to assess progress on Sustainable Development Goal target 4.1. Data remain particularly scarce at the lower secondary level, with not quite a third of portfolio countries (27 out of 91) reporting data on the proportion of students in lower secondary education achieving at least a minimum proficiency level in reading, compared with Independent Evaluation Group World Bank Group    xiii half (47 out of 91) of them providing data for end of primary reading. All leading education systems have a learning assessment function in place—a prerequisite to focusing on improvements in learning. Nearly two-thirds of portfolio operations supported learning assessment, learning surveys, ca- pacity building, and dissemination activities. In most cases, the assessments covered grades 6 and lower (92 percent) and evaluated reading (98 percent) and mathematics (84 percent). The SABER Learning Assessment Platform and the Russia Education Aid for Development program have provided numerous tools, reports, and global public goods designed to improve global knowledge. For example, SABER–Student Assessment tools have been applied in about 60 countries, resulting in two regional reports and seven country reports, as well as case studies of lessons learned from learning assessments and learning standards. In addition, the Russia Education Aid for Development program and the SABER Learning Assessment Platform provide technical assistance to a few countries to improve student learning assessments. In recent years, this sup- port has been more strategically focused on measurement in primary grades. Nevertheless, the World Bank needs to focus more consistently on learning outcomes at both the country and the project levels. Monitoring and eval- uation of improved learning outcomes is specified in one-third of project development objectives in the basic education portfolio. The focus on im- proved learning outcomes in project development objectives is higher in Africa (43 percent of projects). Improving the quality of education is an objec- tive in 62 percent of operations that typically focus on improvements to the learning environment (such as enhanced infrastructure, textbooks, or teach- ers trained)—often prerequisite conditions to improving learning. Of the 77 projects with project development objectives that improve learning, 48 have outcome indicators. Analysis of Independent Evaluation Group ratings shows that operations with learning indicators receive lower ratings than those with- out, although this is significant only at the 0.1 level. The lower frequency in measurement of learning compared with outputs of the learning environment is due, in part, to internal incentives that do not encourage country teams or task team leaders to set more ambitious objectives and indicators, which are more challenging to achieve than access-related objectives. The lack of ade- quate measurement data was also a factor in lower project ratings (Bedasso and Sandefur 2024). The same authors note that task team leaders may ac- count for a significant portion of the variations in type of project activity, suggesting that institutional incentives are needed to ensure that task team leaders support the World Bank’s strategic aim—learning for all—but will not Confronting the Learning Crisis Overview feel pressure to deliver uniformly high ratings. The measurement of learning outcomes at the country program level is lim- ited. The efficacy of education management information systems remains weak in many client countries despite World Bank operational support. Due attention is not always paid to interactions and interdependencies between the technical (for example, hardware and software) and human components (that is, capacity) within basic education systems that are necessary to sup- port a joined-up, functioning model. The World Bank’s SABER–Education Management Information Systems assessment tool is not “internalized” within the Education Global Practice’s current analytic support for systems xiv strengthening in client countries. Capacity Development Is Needed across All Levels of Basic Education Systems The World Bank has put less emphasis on supporting lower levels of basic education systems, such as at the provincial and the district levels. This has negative implications for policy implementation because success depends on actors in the lower levels of the system who are responsible for ensuring fidelity to policy reform and implementation. Inputs at lower levels of basic education systems, where they exist, are often designed to ensure effective World Bank project delivery through, for example, training in financial man- agement and procurement without transfer of capacity to the broader system outside of the project boundaries. A High-Quality Teaching Career Framework Is an Essential Part of the System World Bank analysis recognizes that quality teaching is essential to improv- ing learning outcomes and that teaching quality is affected by many factors, including funding, recruitment, monitoring, and motivation. The primary response to the challenges associated with quality teaching is support for on-the-job training (in 188 projects). Such training can be an important input, particularly where there is a cadre of well-qualified teachers and where the training builds on existing knowledge, expertise, and competence. Yet Independent Evaluation Group World Bank Group    xv few operations assess the efficacy of training. Only 22 out of 118 operations in the portfolio that supported on-the-job training systematically tracked the impact of that training on teachers’ practices. Typically, the World Bank monitors participation in training. Projects that provide follow-up support to enhance the effectiveness of training do not generally assess the efficacy of the follow-up support provided (assessed by 38 out of 110 operations). Other challenges, such as recruitment, initial training, placement, and retention, were addressed to a lesser extent. There is also a need to expand the evidence base of the efficacy of the World Bank’s Teach and Coach tools to improve teaching practices and student learning. What is needed is consistent exam- ination of whether the interventions and inputs the World Bank finances are having a positive impact on systems, teaching, and learning. Equity and Inclusion Are a Concern in Many Systems Despite increased attention to equity and inclusion in documentation, few projects produce disaggregated data related to equity other than for gender, making it difficult to assess the extent to which target groups have been reached or had their needs addressed. World Bank operations planned to support gender-targeted activities, which almost exclusively target girls, account for 67 percent of all projects, with an even higher rate among proj- ects in Africa. Equity issues other than gender are targeted to a lesser extent: 51 percent of projects target children in rural, remote, or nomadic areas; 39 percent support activities related to the educational needs of children with disabilities. Almost all projects addressing gender disparity have indicators with gender disaggregation; only about 30 percent of projects that include other groups in their targeting, such as children with disabilities or those who are rural residents or out of school, have indicators that capture the progress of those target groups. Global-level advisory services and analytics document inequities in learning for various marginalized groups. Clients would benefit from support in and knowledge about the additional challenges faced by children with disabilities and the learning adaptations they may need for the delivery of education, as this has received modest attention. To maximize its contribution to ad- dressing the learning crisis, the World Bank could provide context-specific evidence and promote equity considerations being fully built into education system planning, implementation, and monitoring. Confronting the Learning Crisis Overview Key Evaluation Conclusions The evaluation concludes that where the World Bank has a willing and com- mitted partner, it has been able to better focus on key policy reforms and to lay foundations for a learning-oriented system. Where countries are in tune with, or at least in deliberate, sustained pursuit of certain conditions—teaching and career progression, measurement of learning, financing to achieve equity of learning, and meritocracy in hiring at all levels of the system—World Bank support can be effective. Brazil, Kenya, and Viet Nam all demonstrated strong xvi political and financial commitment in support of learning for all, combined with a strong equity focus (and significant contextual differences). This is evident in clear implementation actions to improve quality, equality, and learning, as well as a commitment to communicate learning data and establish clear goals for learning improvement. These countries have also allocated ed- ucational resources to prioritize primary and foundational learning, consistent with leading global education systems. In such a context, the World Bank has been able to deploy its knowledge, technical assistance, policy dialogue, and financing more effectively in support of reforms that contribute to improvement in learning outcomes. Aspects of the analysis of context and engagement could have been strengthened in each case, but because the World Bank is working with de- termined and focused partners, it has been able to engage with an enhanced understanding of context and apply its resources to leverage points in sup- port of more effective system reform. The evaluation concludes that the World Bank is well placed to lead in delivering a more strategic response to the learning crisis and shifting to an outcome orientation. The World Bank typically has well-developed re- lationships with client governments that can be used to support reform in favor of learning for all. It also has strong research and analytic capabilities, and as the largest provider of development aid to education, it occupies a strong and influential position in relation to other development partners. Independent Evaluation Group World Bank Group    xvii These comparative advantages can reorient dialogue to emphasize changes in systems to improve learning, which require tracking learning outcomes and improving the quality of teaching. To support an outcome orientation at the country level requires detailed theories of change that define the path- ways from enhancements of preservice institutions, teacher recruitment, and teacher monitoring to intermediate outcomes and how those outcomes are to result in improved pedagogical practices in classrooms that increase student learning. This shift will require incentives and signals to staff and Country Management Units that more ambitious objectives and measure- ments of learning outcomes in primary grades 3 and 6, with disaggregated data, are needed to improve accountability by clients and the institution for improved learning for all. A stronger contribution would entail the adoption of a contextualized, sys- tems-based approach that gives more attention to political commitment, public funding, and the education system’s capacity to deliver learning for all. Such an approach would recognize the unique political, social, cultur- al, and economic characteristics of individual basic education systems and facilitate the design and implementation of tailored responses, consistent with the call by the WDR 2018 to identify and address failures in systems for learning. Country case studies found several weaknesses in the World Bank approach. Documentation from the portfolio and case studies suggests that support to basic education has taken a more uniform, less nuanced approach. For example, documentation rarely emphasized the potential impacts of dy- namic interaction among multiple, potentially powerful stakeholders on the achievement of desired outcomes. The case studies also found no assess- ments of the alignment and capacity of the basic education delivery system, especially for actors in the lower levels of the system on whom fidelity to policy reform and implementation success depends. Analysis undertaken by the World Bank should take account of the level of political will in support of inclusive education reform, the level of financial commitment in support of reform, and the extent of capacity within and across the system. Such assess- ment may lead the World Bank to prioritize lending for basic education in some countries and to prioritize dialogue and capacity building in others. Finally, the evaluation concludes that in many countries, World Bank sup- port for basic education lacks intensity and continuity in the face of a Confronting the Learning Crisis Overview particularly challenging problem that requires alignment between many dynamic actors and components in complex systems. Solving the problem requires sequenced engagement leading to incremental reform of systems, measurement of learning, establishment of a teaching career framework, and improvement in learning for all. The relatively limited level of engagement in many instances makes it difficult to provide the traction needed to sup- port the systemic reform necessary to improve learning outcomes. Given the scale and depth of the learning crisis, the limited resources the World Bank and other development partners have to address this challenge need to be strategically deployed. xviii More comprehensive and scalable approaches to addressing the learning crisis will also require a much greater level of collaboration among develop- ment partners. COVID-19 has spurred the emergence of greater collective urgency and innovation among partners, including the World Bank, partic- ularly at the global level. At the country level, however, although partners communicate and cooperate, true collaboration is much more limited and is undermined by the absence of a widely shared understanding of the factors contributing to system failure. A common analysis and understanding could support the co-pursuit of quality education and learning outcomes and the co-pursuit of reform of the teaching career framework and measurement of learning because no leading education system has succeeded without these two critical aspects. Recommendations Develop country-specific education engagement plans that include systems- based enhancements to the teaching framework to improve learning out- comes. These plans should be informed by a comprehensive systems analysis of the constraints to implementation of a career framework—teacher re- cruitment, training, development, motivation, and evaluation—as learning outcomes require capable and motivated teachers. Understanding the un- derlying issues, such as political will, system capacity, funding, and political economy obstacles and opportunities, will involve eliciting feedback from key Independent Evaluation Group World Bank Group    xix stakeholders at all levels and compiling existing and new analysis to guide the development of a medium-term engagement process ideally anchored, where appropriate, within a pillar of the Country Partnership Framework. Sufficient data would be needed to inform adaptive management decisions related to corrective actions and learning during implementation to address the underlying constraints to sustainably improve systems and track interme- diate outcomes. Implementation could be measured in Country Partnership Frameworks, supported by analytics and projects, with intermediate out- comes related to the performance of the teaching career framework, rather than just the completion of activities. The success of the recommendation can also be measured in lessons that inform a scaling up of approaches from the World Bank’s engagement. Collaborate with global and country partners to close the data gaps on learn- ing outcomes (aligned with Sustainable Development Goal target 4.1) and to track progress in ending learning poverty. This would be demonstrated by showing an increase in the number of countries with (i) education projects and Country Partnership Frameworks that include indicators for learning improvements in grades 3 and 6, which may require more ambitious project goals and indicators; (ii) improvement in national educational assessment capabilities and systems for data collection and decision-making; and (iii) participation in cross-national assessments for better data comparability. A focus on those countries that lack quality national assessments and have not been part of international or regional assessments in the last five years is particularly needed. Confronting the Learning Crisis Overview 1  More than two dozen low- and middle-income countries, as well as high-income countries and organizations, have signed the Commitment to Action. See https://www.worldbank.org/ xx en/topic/education/brief/commitment-to-action-on-foundational-learning. Management Response Management of the World Bank thanks the Independent Evaluation Group for the report Confronting the Learning Crisis: Lessons from World Bank Support for Basic Education, 2012–22. The evaluation assesses the World Bank’s contri- bution to improving learning outcomes in basic education between 2012 and 2022 and offers numerous insights into the effectiveness of the World Bank approach. The learning crisis poses a challenge to countries’ efforts to build human capital and reach the Sustainable Development Goals. Addressing the learning crisis is central to achieving the World Bank’s mission. Management thanks the Independent Evaluation Group team for the timely and relevant analysis and continued collaboration. World Bank Management Response Overall Management welcomes the overall conclusion that the World Bank has helped convene global stakeholders to address the learning crisis and remains well placed to advance this agenda. Management welcomes the evaluation’s recognition of the World Bank’s leading role in building awareness, sharing knowledge, and making foundational learning a global priority. The report Independent Evaluation Group World Bank Group    xxi finds that the World Bank has worked strategically and synergistically with partners to build global buy-in for the concept and measurement of learning poverty and to unify global messaging on the learning crisis. Management also welcomes the conclusions that the World Bank has contributed to im- proving learning outcomes between 2012 and 2022 through financing, knowledge, data, and partnership, and that it continues to be well positioned to carry this agenda forward, given its role as the largest external funder of education and the strong relationships country teams have forged with gov- ernments. Management is also pleased with the recognition of how World Bank adviso- ry services and analytics, including the 2018 World Development Report, have drawn attention to the learning crisis and how to address it. The report high- lights that the World Bank has produced a wealth of relevant, high-quality knowledge products, including the 2018 World Development Report and the subsequent development of the learning poverty metric, which have both shone a spotlight on the learning crisis. The report notes that the World Bank has provided substantial advisory and analytic work, including global public goods, on education-system strengthening, capacity development, learning measurement, and use of data to inform education policy and teaching prac- tices. It also notes the increase in advisory services and analytics that focus on issues related to equity during the evaluation period. Management appre- ciates the acknowledgment of impactful publications tailored to the regional context, such as Facing Forward: Schooling for Learning in Africa and Great Teachers: How to Raise Student Learning in Latin America and the Caribbean. Management acknowledges that more could be done to strengthen capacity building at the lower levels of the education system. Management welcomes the report’s finding that 69 percent of projects focus on the challenge of “weak education system: governance, accountability, and institutional over- sight” (table 3.1), meaning they are aimed at strengthening system capacity. Capacity building is part and parcel of World Bank operations and is a central part of the technical assistance that World Bank teams provide alongside lending. The report notes that the World Bank has put less emphasis on sup- porting lower levels of basic education systems, such as at the provincial and the district levels. While most projects include efforts to strengthen capacity Confronting the Learning Crisis Management Response at the school level by providing on-the-job support to teachers (80 percent of projects), management notes that additional efforts to build capacity at all levels of the system are needed. Management notes the report’s assertion that World Bank operations focused on outputs rather than outcomes and is taking steps toward a more strate- gic and results-oriented approach. The World Bank has taken steps toward larger and more results-based operations, with a higher potential for systemic impact. Management takes note of the report’s finding that monitoring and evaluation of improved learning outcomes are specified in only one-third of project development objectives in the basic education portfolio. A new more outcome-oriented education indicator is included in the new World Bank Group Scorecard, which tracks students supported with better educa- tion, and this will be cascade to operations. The data from the Scorecard can xxii help inform strategic discussions about the World Bank education approach, including how the World Bank can do more to support clients to reduce learning poverty in places where it is high, and for those who need it most. Management welcomes the finding that while the share of country engage- ment products referencing marginalized groups increased during the period, monitoring and disaggregated reporting of results predominantly focused on gender. There is scope to enhance disaggregation of results to be able to track and improve outcomes for marginalized groups along other dimensions, such as disability inclusion. Recommendations Management agrees with the first recommendation to develop country-spe- cific education engagement plans with system-based improvements to the teaching framework to improve learning outcomes. Management agrees that diagnosing and addressing country-specific barriers to learning for all, including political economy barriers, is critical to build systems that better serve all its students, in line with 2018 World Development Report’s call to identify and address system failures that inhibit learning. Management agrees that the performance of the teacher career framework could be anchored in and tracked through Country Partnership Frameworks, where the lack of such a framework is a binding constraint on learning. Management agrees with the need for a more holistic, system-based approach to improving teaching, rath- Independent Evaluation Group World Bank Group    xxiii er than one that focuses primarily on on-the-job training for teachers. The recent World Bank report Making Teacher Policy Work takes such an approach, and the World Bank is now expanding its work on other areas concerning teachers, such as support for preservice teacher education, and will produce a global report on the topic. Management also agrees that operations that support teacher training should systematically track the effectiveness of that training. Management agrees with the second recommendation to close the data gaps on learning outcomes and to track progress in ending learning poverty with its partners. Improving availability and use of learning data is a workstream where the World Bank has made progress with partners, including through the Learning Data Compact, a coalition to end the learning data crisis. The World Bank has developed a range of analytic tools and guides to support countries in the development of their national assessment systems, as not- ed in the report. The World Bank’s new Accelerating Learning Measurement for Action program, launched together with the United Kingdom’s Foreign, Commonwealth and Development Office, builds on this progress in coordina- tion with Learning Data Compact partners to provide technical and financial support to priority countries. Management notes that institutionalizing high-quality measurement requires sustained long-term engagement and that countries are not always willing to participate in international assess- ments or link their national assessments to the Global Proficiency Framework to allow for Sustainable Development Goal monitoring. Management is expanding its in-house capacity to support the goal of more extensive learn- ing measurement and action, by improving staff learning and deepening its expertise on foundational learning and learning measurement through recruitment of additional staff. On learning measurement in early primary, the World Bank recently released a joint statement with partners advocating for better reporting on Sustainable Development Goal 4.1.1a, and it continues to work at the country level to increase data availability and use. Management will also work to ensure that more projects with learning-focused project development objectives track learning outcome data, noting that the report shows that 48 of the 77 projects with project development objectives that Confronting the Learning Crisis Management Response address improving learning track learning outcome data. Because a lack of baseline learning data is what constrains many projects from tracking learn- ing outcomes, Accelerating Learning Measurement for Action and other World Bank efforts to expand data availability will help with this goal. Overall, the new Scorecard is revitalizing the measurement agenda across the Bank Group. xxiv Report to the Board from the Committee on Development Effectiveness The Committee on Development Effectiveness met to consider the Independent Evaluation Group (IEG) report Confronting the Learning Crisis: Lessons from World Bank Support for Basic Education, 2012–22, and the World Bank draft management response. The committee welcomed the evaluation and commended IEG for a com- prehensive and timely assessment. They expressed support for the report’s findings and IEG recommendations. These call for the World Bank to (i) develop country-specific education engagement plans that promote conti- nuity in World Bank support, (ii) include systems-based improvements in the teaching framework to improve learning outcomes, (iii) collaborate with global and country partners to close the data gaps on learning outcomes, and (iv) track progress toward ending learning poverty. While Executive Directors recognized the World Bank’s efforts in the education sector, they noted the increasing learning poverty rate in low- and middle-income countries and re- iterated that the institution is well placed to lead delivery of a more strategic Independent Evaluation Group World Bank Group    xxv and accelerated response to the learning crisis. Members urged management to take a more holistic systems-based approach to address the issue of basic education, including focusing on resilient infrastruc- ture and the digital access divide. Members encouraged management to support the development of digital literacy as part of its human capital programs in Africa. They also underscored the importance of focusing on the nexus be- tween education and jobs and economic transformation and on strengthening capacity building, including at the provincial or district levels for successful im- plementation of education reforms. In line with IEG’s findings, members urged management to move beyond delivering and monitoring of outputs to tracking and measuring outcomes, such as improvements in learning poverty, and asked about management’s plans to incentivize staff to promote the measurement and targeting of outcomes versus outputs in World Bank engagements. 1 | Background Highlights The COVID-19 pandemic resulted in severe learning losses and underscored the crucial role of basic education for children and families beyond learning, such as socialization, nutrition, and social-emotional well-being. Those losses also highlighted longer- term global learning deficits. The roots of the learning crisis lie in a historical focus on basic schooling access without commensurate investment in quality and in approaches to quality improvements that focus on inputs and outputs—books, curriculums, teachers, management capacity— applied to complex systems subject to social, political, cultural, structural, logistical, and institutional crosscurrents that require thorough analysis and a sophisticated, strategic systems approach. The children most often failed by their education systems are those already disadvantaged by poverty, location, ethnicity, gender, or disability. Improved learning outcomes for all are more difficult to motivate and more costly to achieve than improvements in access. 1  A recent World Bank study reports that governments in low- and middle-income countries closed schools during the COVID-19 pandemic for an average of 37 weeks—the equivalent of one year of schooling (Schady et al. 2023). The study calculates that “1.3 billion children…missed at least half a year of school, 960 million missed at least a full year, and 711 million missed a year and a half or more” (Schady et al. 2023, 62). Every month of school closure represented more than a month of learning loss. The time students spent studying also fell, even where remote learning was available: “In Ghana, Liberia, and Sierra Leone, the time spent learning per day declined by 71 percent, or six hours per day, compared with time spent when schools were open. In Kenya, the decrease was 61 percent, and the average student spent only two and a half to three hours per day on learning- related activities” (Schady et al. 2023, 64). Moreover, access to remote tech- nology was not uniform within countries, leaving those most at risk even more vulnerable to learning loss. These severe learning losses, if not ad- dressed, will have consequences for the long-term economic prospects of the children affected and for their countries (Rodriguez et al. 2020). The learning losses because of COVID-19 school shutdowns only extend- ed and deepened a long-standing challenge—low learning outcomes and persistent learning poverty are common in the basic education systems of low- and middle-income countries. The World Development Report (WDR) 2018 found that students were losing an average of one to six years of schooling as a result of low educational quality (World Bank 2018b). The result is that 53 percent of all 10-year-old children in low- and middle-income countries experienced learning poverty. Furthermore, learning poverty increased to Confronting the Learning Crisis Chapter 1 57 percent by 2019. In low-income countries, learning poverty was 91 percent by 2019, compared with 9 percent in high-income countries (World Bank, UNESCO, et al. 2022). That was the situation even before the learning loss- es arising from the school shutdowns during the COVID-19 pandemic. (See box 1.1 for definitions of learning poverty, reading proficiency, and learning loss.) Figure 1.1 highlights the divide between high-income and low-income countries in achieving minimum proficiency levels, which will be used to measure progress toward Sustainable Development Goal (SDG) indicator 4.1.1 in reading and mathematics with the most recent United Nations Educational, Scientific, and Cultural Organization (UNESCO) Institute for Statistics data. 2  hare of Students Achieving at Least Minimum Proficiency Figure 1.1. S Levels in Reading and Mathematics by Country Income Level Mathematics Reading at the Mathematics at Reading at the at the end of end of lower the end of lower end of primary primary secondary secondary education education education education High 86 65 75 71 income Upper-middle 63 43 50 43 income Lower-middle income 34 22 38 31 Low income 31 18 13 14 Source: United Nations Educational, Scientific, and Cultural Organization Institute for Statistics, September 2023. Learning is important both for individual well-being and for economic devel- opment. Filmer et al. (2018) note that studies of learning and skills among adults have found learning effects on individual earnings, health, financial behavior, social mobility, and economic growth. Hanushek’s research con- firms that the growth of countries and growth rates of nations are closely linked to the skills of their population. The implication of the research is the importance of raising basic skill levels in countries (particularly those farthest Independent Evaluation Group World Bank Group    3 behind in skills) to enable their participation in a modern productive econo- my (Hanushek 2022; Hanushek and Woessmann 2008). The roots of low learning quality in basic education (primary through lower secondary school) lie in part in approaches from international development organizations that focus on inputs and outputs rather than outcomes.1 For example, the Asian Development Bank report on lessons learned from its work in the education sector indicates that policy reforms that are lacking attention to political realities and understanding of country contexts are bound to fail (ADB 2013). A report on bilateral support to primary educa- tion by the UK Department for International Development (replaced by the Foreign, Commonwealth & Development Office) states that the emphasis on enrollments “in part reflects how governments and donors collectively have interpreted Millennium Development Goals for Education” (Comptroller and Auditor General 2010, 7). The report also specifies that the Department for International Development’s approach was evolving toward a focus on edu- cation quality. The Aga Khan Foundation notes that, in consequence of the focus on enrollments, “the very gains in school access have exacerbated the quality issue” (Aga Khan Foundation 2010, 4). More recently, documents from the US government and the UK Foreign, Commonwealth & Development Office, among others, acknowledge a narrow focus on increasing enrollments in pursuing the education targets for the Millennium Development Goals, which came at the cost of attention to education quality.  ey Concepts in Learning Measurement Box 1.1. K International metrics for education have evolved over the past 25 years. With the Millennium Development Goals, attention was focused on measuring education access. Learning outcomes, which are more challenging to measure, were less common and less developed. In 2012, the Learning Metrics Task Force, convened by the United Nations Educational, Scientific, and Cultural Organization Institute for Statistics and the Center for Universal Education at the Brookings Institution, issued six recommendations to improve learning measurement, supporting efforts to shift global focus and investment from universal access to universal access plus learning. The adoption of the Sustainable Development Goals in September 2015 committed countries and their development partners to improving learning outcomes for all and posed challenges for the measurement and tracking of learning. Sustainable Confronting the Learning Crisis Chapter 1 Development Goal target 4.1 aimed to ensure that by 2030, all girls and boys would complete free, equitable, and quality primary and secondary education, leading to effective learning outcomes. Linked to target 4.1, Sustainable Development Goal indicator 4.1.1 focused on the proportion of children and young people (in the second and third grades, at the end of primary education, and at the end of lower secondary education) achieving at least a minimum proficiency level in reading and mathematics, with information disaggregated by gender. (continued) 4  ey Concepts in Learning Measurement (cont.) Box 1.1. K The development of the learning poverty indicator by the World Bank and the United Nations Educational, Scientific, and Cultural Organization joined school deprivation with learning deprivation in a single measure. Learning deprivation in this indicator is based on reading proficiency, which is expected to be accomplished by the end of primary education and is a proxy for other foundational learning. Hence, children unable to read and understand a simple text by age 10 years are described as “learning poor.” Conceptually, “learning poverty” begins with the share of children who have not achieved minimum reading proficiency (as measured in schools) and is adjusted by the proportion of children who are out of school. Reading proficiency is measured by the ability to read at grade level. By age 10 years, or by the end of primary education at the latest, a child should be able to read a simple text, fluently, in their own language. In rural India in 2016, only half of fifth-grade students could fluently read text at the level of the second-grade curriculum, which included sentences (in the local language) such as “It was the month of rains” and “There were black clouds in the sky.” Learning loss—the loss of knowledge and skills or the reversal of academic progress— can occur with any extended break or disruption in schooling, including seasonal breaks, frequent changes in school, dislocation as a result of social strife, and school closures. Learning loss can affect literacy, numeracy, and social and emotional development and can increase learning poverty by increasing either or both learning deprivation and Independent Evaluation Group World Bank Group    5 schooling deprivation. Hence, “during the COVID-19 pandemic, after lengthy school closures and remote instruction that was less efficient than learning in schools and was provided with unequal access, the learning poverty rate could reach as high as 70 [percent; from a base of 57 percent]” (World Bank, Bill & Melinda Gates Foundation, et al. 2022, 4). The largest losses were in Africa, where second-grade students in South Africa suffered up to 70 percent learning loss and fourth-grade students in Malawi lost two years of learning. Sources: Azevedo et al. 2021; World Bank 2018b, 2021d; World Bank, Bill & Melinda Gates Foundation, et al. 2022. Who Is Affected? The children most often failed by their education systems are those al- ready disadvantaged by poverty, location, ethnicity, gender, or disability. Understanding the barriers to learning and discrimination has become increasingly sophisticated in the literature on education strategy and policy. Analysis has moved beyond metrics of economic status (with gender disag- gregation) to include social identity and a broader equality agenda, while also noting how discriminatory barriers overlap and compound, such as the multiple barriers faced by a girl from a poor family living in a rural area who is a member of an ethnic minority. Yet poverty is at the root of most exclu- sion from school and from learning (UNICEF 2020). As many as 44 percent of girls and 34 percent of boys from the poorest quintile never attend any school or drop out before completing primary education. There is a serious equity problem in the distribution of public financing, such that the poorest children—who face compounding barriers including location, disability, or ethnic origin—are failed further (UNICEF 2020). Children in crisis situations are often failed by their education systems. Novelli et al. (2014) found that recognition of the importance of education is less pronounced in the agenda-setting process in fragile contexts and else- where compared with both the humanitarian aid and security sectors. UNESCO and the United Nations Children’s Fund (UNICEF) highlight the situation of children in conflict-affected countries, who account for just 20 percent of the world’s children of primary school age but for 50 percent of those who are out of school, and note that entrenched gender roles often determine whether a Confronting the Learning Crisis Chapter 1 child enrolls and stays in school (UNESCO UIS 2015). The report emphasizes that education is often delivered in a language that children neither speak nor understand, which is a barrier to learning for many. UNICEF (2020) indi- cates the importance of focusing education funding on children during and after emergencies—at present, only 2.6 percent of humanitarian funds go to education. Only half of refugee children go to primary school, and less than a quarter are in secondary school. Children in conflict-affected countries are 30 percent less likely to complete primary school and 50 percent less likely to complete lower secondary education. 6 A Complex Problem The issues involved in improving the quality of basic education are multi- layered, including social, structural, logistical, and institutional matters that require a sophisticated analysis, understanding, and approach. Clearing the obstacles to orienting and aligning basic education toward learning re- quires ensuring that (i) children are prepared to learn, (ii) teachers are well trained and motivated, (iii) learning inputs are available and culturally and grade-level appropriate, and (iv) management and governance of the system have the capacity and authority to coherently integrate the various factors (World Bank 2018b). To address this level of intertwined complexity, the literature (UNICEF 2023; USAID 2022; World Bank 2018b) maintains that development organizations need a deep understanding of context when supporting education reform and that this can be supported by a greater emphasis on systems-based approaches to education. Improved learning outcomes are more difficult to generate and are more costly and complex to achieve than improvements in access. Improving learning for all involves, for example, enhanced teacher quality; alignment of curriculum and textbooks with local contexts reflecting local language and culture; investment in education equity to ensure equality of opportunity (rather than, simply, access) for diverse students with reference to socio- economic status, gender, disability, and geographic location; and effective assessment systems to allow for the monitoring of progress. Grindle (2004) concludes that “from a political perspective,” access reforms are easier than Independent Evaluation Group World Bank Group    7 quality reforms. This is because the latter can result in lost jobs and dimin- ished control over decision-making or resources, such as budgets and people, and can introduce new pressures and expectations for students, teachers, management, and oversight (Kingdon et al. 2014). Clientelism, patronage, and corruption are the most intense political forces pushing states to expand education access by building more schools or hiring more teachers without an equal emphasis on improving education quality. The literature suggests that this is why “most [country] education policies are to do with expanding access to education, and providing inputs to schools, which require expenditure” (Kingdon et al. 2014, 35).2 Negotiating both the general complexity of education reform and opposition- al factors of the political economy may be better done using a systems-based approach. The WDR 2018 advocated addressing the political economy barriers at all levels of the system to improve learning outcomes. Magrath, Aslam, and Johnson (2019) posit that the change in focus from access (output) to learning (outcome) has shifted the emphasis away from individual interventions and programs to entire education systems. Faul and Savage (2023) state that a systems-thinking lens would emphasize the importance of locally led, non- standardized, and context-responsive education to enable all children to learn (and to love learning). However, Faul and Savage (2023, 16) find that “systems thinking in international education remains contentious because it refuses to tout a single, one-size-fits-all solution; that can never be adequate to the complexity of the learning and equity crises that learners face.” Instead, it recognizes the complexity of education systems and encourages use of an ap- proach focused on short dynamic feedback loops to understand elements and functions within the system (box 1.2).  hat Is Systems Thinking? Box 1.2. W Broadly, systems thinking enables a more holistic consideration of the following: » System elements—both material (teachers and schools) and intangible (beliefs and information) » The relationships between those elements and subsystems » The structuring of the system and subsystems within it Confronting the Learning Crisis Chapter 1 » The functions of the system—both formal (stated) and informal (in practice) » The positive and negative feedback loops and influence pathways in the system The World Development Report 2018 observed poor alignment of education systems with learning goals because of technical complexities, pursuit of conflicting goals, and limited policy implementation capacity in government agencies responsible for learning. System incoherence occurred across learning objectives and responsibilities, information metrics, finance, and incentives. Sources: Faul and Savage 2023; World Bank 2018b. 8 World Bank Approach to Learning in Basic Education For more than two decades, the pursuit of enhanced learning outcomes has featured in World Bank literature, especially since the publication of its 2011–20 strategy, Learning for All: Investing in People’s Knowledge and Skills to Promote Development—World Bank Group Education Strategy 2020 (World Bank 2011). Previous education sector strategies and documents have also em- phasized the need to measure student learning and achievement as an input to helping country clients achieve educational outcomes (World Bank 1999, 2005). The 2011–20 strategy explicitly made learning a priority. Subsequently, the WDR 2018 marked an important shift in the World Bank approach to edu- cation and highlighted the learning crisis (World Bank 2018b). It emphasized the need for context-specific solutions, especially those developed by the country client. Countries need support, the WDR 2018 suggested, to correct poor service delivery and address system-level technical and political chal- lenges that allow low-quality schooling to persist. Following the themes of the WDR 2018, the World Bank described its most recent approach to the learning crisis in Ending Learning Poverty: What Will It Take? (World Bank 2019b). The report focused on the difficult challenge of eliminating learning poverty, noting the inadequacy of continuing with business as usual. Realizing the Future of Learning: From Learning Poverty to Learning for Everyone, Everywhere elaborated on the response, taking on the added challenges posed by the global pandemic (World Bank 2020a). In these Independent Evaluation Group World Bank Group    9 formal statements of intent, the World Bank set out to strengthen its efforts to confront learning poverty and to influence the focus on learning poverty at the global level by setting an operational global learning target to cut the learning poverty rate by at least half before 2030 and by introducing three key pillars of work: (i) a literacy policy package, (ii) a refreshed approach to strengthen entire education systems, and (iii) an ambitious measurement and research agenda. These three pillars aim to support countries to improve the human capital outcomes of their people. In Realizing Education’s Promise: A World Bank Retrospective, which looks back at the World Bank’s approach, the World Bank strategy is further re- fined to address five pillars (World Bank 2023c). The document articulates an approach to the education sector focused on policy actions that are needed to accelerate learning and that characterize the way many successful systems operate. The five pillars—learners, teachers, learning resources, schools, and system management—are fundamental to a well-functioning school system.3 Measurement of learning is central to the World Bank’s approach, as has been consistently emphasized by the Learning for All strategy and the WDR 2018 (World Bank 2011, 2018b), requiring support for national assessment capacity building to help countries develop reliable, timely statistics about student learning. The Independent Evaluation Group (IEG) has contributed to learning by the World Bank through several evaluations relevant to basic education. From Schooling Access to Learning Outcomes: An Unfinished Agenda—An Evaluation of World Bank Support to Primary Education called for strengthening the capacity of countries to track learning outcomes, ensuring the disaggrega- tion of learning data across different income and social groups (World Bank 2006). It also recommended working with development partners to reorient their emphasis on completion of primary education to focus on learning out- comes for all. This would require the establishment of learning achievement indicators, baselines, and targets (both intermediate and outcome targets) to support learning outcomes, and technical and financial support to countries to set up systems for conducting repeated learning assessments, comparable over time and capable of tracking outcomes separately for disadvantaged groups. The evaluation found sector management capacity a common weak- ness in primary education projects and suggested that better organizational capacity assessments at the outset and better capacity-building programs Confronting the Learning Crisis Chapter 1 might have helped reduce the problem. The IEG portfolio review of educa- tion operations concluded that poor people may be the last to benefit from the World Bank’s investments, that stronger monitoring and evaluation was needed in government programs supported by the World Bank, and that the factors contributing to variation in results needed to be better understood (World Bank 2010).4 A 2019 IEG evaluation of preservice and in-service train- ing interventions found that World Bank engagement in training teachers before entry into the profession (preservice training) has been limited and has prioritized coursework, with less emphasis on other drivers of quality, such as screening, practicum, and quality assurance (World Bank 2019c). 10 Instead, the World Bank has relied heavily on continued training during employment (in-service training) to address shortcomings in preservice training through support to programs for both underqualified and qualified teachers. Evaluation Objective and Scope The evaluation assesses the extent to which the World Bank has supported efforts to improve learning outcomes in basic education over the past decade, fiscal year (FY)12–22. It pays particular attention to the extent to which the World Bank has adopted a systems approach to its support for basic education as advocated in Learning for All: Investing in People’s Knowledge and Skills to Promote Development—World Bank Group Education Strategy 2020 and as rein- forced since the publication of the WDR 2018. The evaluation also responds to the increased urgency of the learning crisis, a priority for the Board of Executive Directors, which has been exacerbated by the COVID-19 pandemic. Thus, the evaluation has a secondary focus on support provided to address critical challenges to education delivery and the exacerbation of learning loss associated with the COVID-19 pandemic. It offers lessons and recom- mendations to inform the next education sector strategy and the further development of the World Bank’s approach to this persistent development challenge. The evaluation examines World Bank inputs, including financial support, Independent Evaluation Group World Bank Group    11 knowledge, policy dialogue, and strategic partnerships. The evaluation covers all operations approved during FY12–22, as detailed in appendix A. For case studies, it also includes relevant projects supported by other Global Practices (GPs) directly supporting basic education, such as cash transfer projects under the Social Protection and Jobs GP or development policy operations from noneducation GPs that incentivize education policy reforms or actions. The evaluation considers inputs to global knowledge and initiatives, such as teachers or learning assessment, as well as regional and country-level knowl- edge (the latter via country cases), based on a sample of knowledge products published during the FY12–22 period. In addition, the evaluation examines policy dialogue and the extent to which the World Bank engages with clients on learning outcomes aiming to maximize positive change and strategical- ly leverage its points of influence. Support for data measurement and its analysis can be critical in informing policy dialogue and in leveraging political and administrative support in favor of reform and may require interconnect- edness across sectors. Evaluation Approach and Methods IEG developed a conceptual framework based on the literature reviewed, including the most recent World Bank education strategy documents, and on high-level interviews to guide this evaluation’s data collection and analysis. The framework sets out the characteristics of a basic education system that seeks to deliver quality education for all (figure 1.2). The findings presented in chapters 2 and 3 are linked to broad areas of the framework. The framework recognizes the necessary contextual conditions to support learning for all. These conditions include contextual factors, such as political commitment to education for all, capacity within the system to deliver on that commitment, and the level of funding available to support basic educa- tion for all. More recently, the context shifted with the COVID-19 pandemic, intensifying the learning crisis. The framework presents the process by which the World Bank can support basic education systems to achieve learning for all. The process begins with systems analysis to fully understand the context and to identify binding constraints and understand why systems fail to improve learning outcomes. It also requires engagement with the full range of stakeholders, beyond the views of ministry officials, to arrive at clarity of intent (that is, access or Confronting the Learning Crisis Chapter 1 learning) and political commitment for learning for all. With that clarity, the World Bank can offer country responses that draw on tailored combinations of its global, regional, and country resources. Support is necessary to ensure sufficient ongoing financial support; basic education reform is a lengthy process and requires sustained government and social commitment. Policies suitable to realize the agreed vision and support for the development of im- plementation capacity and technical expertise and management are required, as are measurement and assessment that deliver feedback for the ongoing calibration of policy and practice subject to the country context and the posi- tioning of the country in the process toward realizing learning for all. 12 Evaluation Methods The evaluation used a mixture of methods with a multilevel design, includ- ing case-based analysis, portfolio analysis of lending and advisory services and analytics (ASA), and key informant interviews. These core methods were supported by literature reviews related to global knowledge, the political economy of education, development partner understanding of the learning crisis and their response to it, and the characteristics of high-performing ed- ucation systems. The evaluation also undertook analysis of secondary data for case countries, which related to population growth and learning assessment via regional and global assessments. The evaluation questions were applied to relevant evaluation components to answer the overarching evaluation question. Appendix A describes the eval- uation methodology in more detail. IEG met with Education GP management and task team leaders (TTLs) to ensure that the evaluation methods were likely to produce learning and evidence useful to ongoing efforts to improve learning outcomes. IEG also shared findings with a group of TTLs to ensure balanced interpretation in analysis and reporting. Evaluation Questions To provide insights for a new education strategy, the evaluation questions and scope were designed through a consultative process with key Education GP Independent Evaluation Group World Bank Group    13 staff and management. The overarching evaluation question is as follows: How has World Bank support for basic education contributed to the achieve- ment of enhanced learning outcomes since the Learning for All strategy, and what can be learned from those efforts to inform support to the learning recovery from the COVID-19 pandemic? To respond to this question, the evaluation answers the following subquestions: » How effective has World Bank support for basic education (FY12–22) been in addressing the binding constraints that hinder the achievement of enhanced learning outcomes in client countries? » To what extent and how effectively has the World Bank: » Collaborated with country and global partners to support education quality and enhanced learning outcomes? » Used feedback from evidence and experience to inform its work to support improved education quality and learning outcomes for all? » How well prepared is the World Bank to address additional challenges to education systems that have arisen because of the impact of the COVID-19 pandemic? Limitations The evaluation team found low-quality data or limited availability of data in some countries (particularly in fragile and conflict-affected countries or those with low institutional capacity), which limited the specificity and precision of the analysis. Mitigation strategies included ensuring that data collection was context driven, collaborating with experienced local consultants to facilitate data collection, working closely with the Country Management Units and leveraging the support of the Education GP, engaging with as many relevant stakeholders as possible to ensure as broad a perspective as possible, and working with and analyzing existing (secondary) data sets to provide robust coverage of any quantitative data available. Some structural evaluation choices have a bearing on the nature of this evaluation. First, IEG limited its field travel to reduce its carbon footprint. Evaluators interviewed stakeholders via videoconferencing, and they were paired with experienced local evaluators. The pair worked closely with the Confronting the Learning Crisis Chapter 1 Education GP and Country Management Unit in each of the 10 countries selected to ensure interactions, albeit remotely, and produced the needed information. To the extent possible, the evaluation engaged with teachers through representative organizations, such as trade unions. The evaluation scope excludes early childhood development. The evaluation recognizes that the World Bank is engaged in a continuum of interrelated support at this critical stage of life; however, to focus the evaluation, the scope was limited to basic education and education systems to ensure robust findings.5 14  onceptual Framework for Support of Quality Education Figure 1.2. C PLANNING IMPLEMENTATION M&E SUPPORTING Systems Coalition of Strategic Political and societal Adequate Learning recovery stakeholders    analysis                                            CONDITIONS                   partnership    commitment  financing  post COVID-19   World Bank: convening,  World Bank: financing with World Bank: data (for example, learning strategic partnerships, outcome orientation, policy poverty, capacity building, knowledge, and capacity building   dialogue convening, and and global-level initiatives such as READ partnerships and the Learning Assessment Platform) Potential Stakeholders Intervene Stakeholder Students World Bank: Curriculum engagement and policy assessment, coalition of actors Parents dialogue education TOWARD management Communities Financing Data Clarity of information QUALITY system, capacity, INTERMEDIATE Civil society organizations intent and Assessment EDUCATION strategy Mobilize connectivity, and OUTCOMES Teachers domestic infrastructure Measurement World Bank: data, resources and knowledge School management Teacher training, Research Improvements in Donors textbooks, school alignment of systems FOR ALL Subnational administration management, Evaluation and system capacity Education ministry Policy design teaching Adaptation Systems analysis materials, Improvements in Other aspects of government information and teaching communication International actors technology, FOR ALL conditional cash Improvements in Employers transfers, and measurement of school feeding learning Inclusive of poverty, gender, disability, ethnicity, refugee, and so on ACTIVITIES Build political commitment to Create and share Build partnerships to Increase the availability of Develop client Establish feedback reduce learning crisis and improve global knowledge improve learning for all disaggregated learning capacity process systems and measurement of data at country, regional, learning and teaching and global levels Source: Independent Evaluation Group. Note: M&E = monitoring and evaluation; READ = Russia Education Aid for Development. Independent Evaluation Group World Bank Group    15 1  This report uses the United Nations Educational, Scientific, and Cultural Organization International Standard Classification of Education definition of basic education, which includes level 1 (primary education) and level 2 (lower secondary education). For more information, see UNESCO UIS (2012). 2  Hickey, Hossain, and Jackman (2019) observed that in more competitive political contexts, such as Bangladesh and Ghana, there was evidence of the partisan allocation of jobs and other rents at different levels of the education system, whereas access to free education was considered important to maintaining the rural electoral base. In Uganda, the persistence of an officially “fee-free” policy reflected ruling party fears that introducing cost-sharing would undermine rural political support, even though cost-sharing was believed necessary to raise existing very low educational standards. 3  The approach identifies specific elements required for each of the five pillars: » For learners, these are quality childcare, nutrition, early stimulation, and early childhood education. » For teachers, these are meritocratic profession, effective human resource function of the education ministry, and continuous school-based professional development. » For learning resources, these are a simple, effective curriculum; books and supportive technology; coaching and structured pedagogy; and a policy action that all students are taught at the right level. » For schools, these are elimination of all types of violence and discrimination in schools, access to and participation in learning for students with disabilities, and universal access in built and virtual environments. Confronting the Learning Crisis Chapter 1 » For system management, these are enhancement of implementation capacity from schools to central ministries, career track for school leaders, clear mandates and accountability, measured learning, and merit-based professional bureaucracy. 4  Critically, the evaluation found that many country programs did not assess the strength of political forces acting both in support of and against the change agenda. Instead, program designers widely presumed that decisions would be based on rational planning and technical merits, although research has shown that officials lack appreciation for the scale of the learning problem. Risk assessments for the many projects reviewed rarely mentioned politically motivated threats to project implementation and success. Thus, no mitigation 16 strategies were formulated. 5  The World Bank has increased its lending to early childhood education. Although its efforts have focused on access (Bedasso and Sandefur 2024), international evidence suggests that interventions such as early stimulation services for at-risk children and support for their parents have reduced intergenerational poverty (Gertler et al. 2014; Schweinhart 2007; Walker et al. 2005, 2006, 2011). Longer-term results from 31 years after receiving the services highlight the importance of serving disadvantaged children (Walker et al. 2017). Independent Evaluation Group World Bank Group    17 2|T  he World Bank’s Global and Regional Analytics and Programs and Global Partnerships Highlights In its work at the global and regional levels, the World Bank has recognized the depth of the learning crisis and responded with substantial advisory and analytic work that has provided global public goods related to education system strengthening and capacity development, to measurement of learning and use of data to inform education policy, and to teaching practices. The World Bank’s regional and global analytic products have diagnosed inequality in access and learning for a variety of marginalized groups. Attention to addressing the adaptations to education delivery to ensure learning for children with disabilities has been modest. The World Bank has worked strategically and synergistically with partners to convene and build global buy-in for the learning poverty concept, with ambitious goals and targets, helping to focus partners on foundational learning and unified communications on a common message, and has worked with partners to produce a volume of global public goods to respond to the challenges introduced by the COVID-19 pandemic. The World Bank has not systematically evaluated the country-level influence of this work, which creates a critical feedback loop in ensuring that the global and regional products are appropriately suited to country contexts and are creating improvements in education systems, measurement of learning, and teaching practices. 18   The World Bank’s knowledge and analytic work, partnerships, and global initiatives, as noted in the conceptual framework, can be important con- tributors to building global knowledge and encouraging action by country clients to support improvements in system alignment and capacity, teaching, and measurement of learning. This chapter, consistent with the conceptual framework, covers the World Bank’s production of global public goods (GPG) and programs to build knowledge and awareness, as well as its actions to translate the GPG into improvements in system alignment and capacity, measurement of learning, and teaching practices by governments. The World Bank has created a wealth of relevant, high-quality knowledge products. The knowledge portfolio analyzed for this evaluation consists of all 145 ASA products that address basic education, of which 88 are ASA and 57 are impact evaluations classified by the World Bank as global and regional products.1 Equity was a theme analyzed to assess whether the ASA would help countries ensure that children and youth from disadvantaged groups bene- fit from equitable access to a quality education, consistent with the equity and inclusion aims of the Systems Approach for Better Education Results (SABER). IEG also reviewed a sample of programs and initiatives financed by trust funds (see appendix A), which did not cover all the disadvantaged groups noted in the World Bank ASA. Increasing Global Knowledge and Building Awareness Independent Evaluation Group World Bank Group    19 The World Bank produced a wealth of knowledge products throughout the evaluated period related to student assessment systems to guide middle- and low-income countries in developing such systems. A conceptual framework created key indicator areas for tracking the development of an effective assessment system, with questionnaires and rubrics to collect and evaluate data on each of the assessment types: classroom assessment, examinations, national learning assessment, and international assessment. SABER–Student Assessment tools were applied in nearly 60 countries, resulting in two re- gional reports, seven country reports, and case studies of lessons learned from learning assessments and learning standards. Among the many useful, high-quality products that built global knowledge on learning measure- ment were the following: Primer on Large-Scale Assessments of Educational Achievement (Clarke and Luna-Bazaldua 2021); Map of Country Participation in Regional and International Large-Scale Assessments—Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación, the Programme for the Analysis of Education Systems, the Pacific Islands Literacy and Numeracy Assessment, the Progress in International Reading Literacy Study, the Programme for International Student Assessment, the Southern and Eastern Africa Consortium for Monitoring Educational Quality, the Southeast Asia Primary Learning Metrics, and the Trends in International Mathematics and Science Study; learning standards with questionnaires and rubrics to collect and evaluate data on content, process, and performance standards in language arts, mathematics, and science; and the online course Student Assessment for Policymakers and Practitioners. This analytic work and the programs and initiatives built global knowledge. A major initiative, SABER, launched in 2011 and continued with various iterations through to its completion in 2020. SABER produced a significant amount of knowledge work focused on specific education policies. Its initial purpose was to help identify and reach consensus on the education poli- cies and programs most likely to create quality learning environments and improve student performance, especially among the disadvantaged, thus filling an important knowledge gap. One hundred countries applied SABER, selecting from among 13 areas of interest (such as school finance, equity and inclusion, and workforce development) with assistance from the World Bank or other development partners to analyze and benchmark their policies and institutions.2 Confronting the Learning Crisis Chapter 2 By design, SABER produced comparative data and knowledge on education policies and institutions such that countries could seek to strengthen their systems to support the pursuit of learning for all. For example, the eight pol- icy areas in the SABER–Teachers policy framework address the stock and flow of teachers and were used to diagnose and benchmark countries’ policies to inform government clients and other interested stakeholders and to support dialogue with education ministries and inform project design (World Bank 2018b).3 This aspect of SABER was applied in 36 countries and was used by partner organizations, such as by UNESCO for its Global Education Monitoring Report (UNESCO 2016, 2017). However, it only measured policy intent at the 20 central level and did not assess what was implemented. This tendency to con- centrate on the central policy level without strong supplemental attention to implementation and delivery across the system is a feature of the approach at the country level (as discussed in chapter 3). Several development partners used SABER tools to support global research and their country work, and SABER also informed World Bank opera- tions (World Bank 2018b). For example, SABER–Education Management Information Systems (EMIS) focused on data as a critical element for the education system and on the use of data in decisions by policy makers, school staff, parents, and students. Improved learning requires a functioning EMIS embedded in a strong enabling environment, supported by a well-defined policy framework and organizational structure, with sufficient infrastruc- ture capacity, human resources, and budget—aspects emphasized in the SABER–EMIS framework. SABER–EMIS was implemented in six countries and developed and disseminated additional knowledge products. In addition, an EMIS was a component in numerous World Bank investment projects. Typically, EMIS counted the number of schools, teachers, and students and did not integrate data from the school census or on finances, human resources, learning assessments, or infrastructure. Consistent with the eval- uation framework, the success of the EMIS relies on buy-in and collaboration between stakeholders within the education ministry and other relevant agen- cies. With the evolution in analytic support, EMIS is no longer a feature of the Independent Evaluation Group World Bank Group    21 programs and initiatives in the Education GP, even though it is foundational for education systems to influence performance, policy making, planning, and monitoring of results and learning. The absence of a focus on EMIS leaves a gap in the analytic support designed to benefit client countries that still have weaknesses, as further described in chapter 3. With experience and feedback, SABER’s analytic approach evolved toward systems strengthening and alignment. In time, the focus on policy intent di- minished, leading the World Bank to convene an Education Systems Technical Advisory Board in 2017 to gather views on how to enhance the next phase. Based on that feedback, SABER 2.0 evolved to begin developing a framework for measuring and analyzing service delivery at the school level. This effort still left a gap in assessing capacity and coherence at all levels of the education system. The idea was to use SABER tools to provide countries with information to improve their policies and institutions to better meet their education goals; however, scaling up the use of these instruments proved difficult and costly, particularly when governments were asked to finance data collection. Given the complexity of education systems, a focus on feedback loops is necessary to understand context-specific elements and functions within a system. This implies using existing information and data to identify what is needed to understand the political economy and capacity in that context to create sound, sequenced actions to improve systems that align with gov- ernment financial resources. Particularly needed is an understanding of the capacity, functioning, and motivation of various actors and agencies that support a teaching career framework (across the stages of recruitment into teaching, preservice institutions, hiring, professional development, moti- vating, and monitoring) to ensure the system incrementally improves in a manner consistent with higher-performing countries. This information can be used to develop context solutions and a feedback loop so the World Bank can tailor its solutions to a well-understood context. ASA products have increasingly focused on documenting inequalities in access and learning—an important element of learning for all. These knowledge products (23 out of 88 ASA products and 13 out of 57 impact evaluations) refer to disadvantaged groups, identify risk factors, and propose interventions to improve education outcomes. Groups include school drop- outs in Latin America and the Caribbean, refugees and internally displaced Confronting the Learning Crisis Chapter 2 persons in the Middle East and North Africa, and out-of-school children and girls in South Asia. In Latin America and the Caribbean, interventions in- clude youth employment training and remedial classes; in the Middle East and North Africa, interventions include psychosocial support and remedial classes. What is particularly needed is moving beyond documenting the num- ber of children with disabilities to provide guidance on adaptations to deliver education that will ensure learning among this diverse and vulnerable group. This type of context-specific evidence fed into the planning and implementa- tion of education systems would be a way to respond to client need. 22 Sharing Global and Regional Knowledge The World Bank has shared global knowledge predominantly through work- shops, meetings, and policy guidance. The Sixth World Bank Europe and Central Asia Education Conference, supported by Russia Education Aid for Development (READ), helped clients better interpret and communicate learn- ing data with country stakeholders. The team that produced Facing Forward: Schooling for Learning in Africa participated in 10 media events (regional and international); 11 regional or country-level dissemination events with World Bank colleagues and stakeholders; and 3 other events, including with the UNESCO International Institute for Educational Planning and the Norwegian Agency for Development Cooperation ministers of international development. A Country Note prepared for Rwanda distilled the 450-page report into more manageable findings for the client. World Bank documents report that this Country Note helped Rwanda tackle grade repetition. Country Notes were not developed for other countries featured in the report. Box 2.1 summarizes sev- eral notable examples of the World Bank generating evidence, disseminating, and engaging in dialogue about policy actions among government clients. The WDR 2018 benefited from substantial trust fund resources and strong leadership in the GP and Regions to bring further awareness of the need to focus on learning and to align the education system, including the techni- cal, political, and social challenges. Online dissemination of the WDR 2018 is Independent Evaluation Group World Bank Group    23 evident in the quantity of people it reached, as it was the second-most-down- loaded global report in World Bank history. Findings from the WDR 2018 were presented at 100 dissemination events in 54 countries, particularly low- and middle-income countries, according to World Bank documents. Trust funds also allowed the report and messages to be translated into multiple languag- es. Practice managers invited members of the WDR team to discussions with finance and education ministry officials, local civil society, and researchers to spur further political commitment. Regions developed context-specific strate- gic papers that were later discussed with government clients (which is useful to clients). nfluence of the World Bank’s Regional Advisory Services and Box 2.1. I Analytics on Country Clients in Developing Projects or Making Policy Reforms Out-of-School Youth in Sub-Saharan Africa: A Policy Perspective transferred knowledge that enhanced policy dialogue (Inoue et al. 2015). The report brought attention to the needs of 89 million youth (12–24 years of age) who fell through the cracks once they dropped out of school. These youth, many of them living in rural areas, faced a variety of barriers, such as early marriage among girls. The report raised awareness of the problem and offered guidance on how to address it through school retention, educational remediation, and integration of youth into labor markets. The authors of the report disseminated the findings and policy recommendations to government clients in easily digestible form. The report contributed to the development (and financing) of skills development projects in Mali and Niger that focused on in-school and out-of-school youth. This report also resulted in country-specific advisory services and analytics. The Latin America and the Caribbean Region Teacher Quality Launch Conference presented findings from Great Teachers: How to Raise Student Learning in Latin America and the Caribbean to promote South-South knowledge sharing at the political level with regional ministers of education (Bruns and Luque 2014). The report noted that “the low average quality of LAC [Latin America and the Caribbean] teachers is the binding constraint on the region’s education progress” and that three fundamental steps, “recruiting, grooming, and motivating better teachers,” are essential (Bruns and Luque 2014, 2), but the challenge confronting teacher reform is political. With clients who Confronting the Learning Crisis Chapter 2 expressed commitment, further workshops brought together academics, politicians, and technical or other people in government at a seminar in Brazil. The September 2013 seminar sought to build capacity to understand and implement policies and programs that are informed by evidence and tailored to the local context. The study outcomes and dialogue influenced subsequent policy actions to reform teacher status, evaluation, and remuneration in Chile, Peru, and one state of Brazil. However, political changes in Brazil and Peru affected the continuity of the reforms, highlighting the susceptibility of reforms to political influence. Sources: Independent Evaluation Group interviews and analysis of other World Bank documents; Bruns and Luque 2014; Inoue et al. 2015. 24 Among the factors that contributed to the World Bank’s influence with country clients were support for follow-up and the ability to link global and regional analytics to country programs. The World Bank provided resources to support staff time not only for dissemination but also for follow-up dialogue to build client support and commitment. In some cases, the evidence was transformed into analytic pieces that were more accessible to country clients. A negative incentive to task teams is that any time spent on influence was not accounted for within the World Bank’s time recording system, according to interviews. Processes that linked global and regional analytic support with country programs, support, and implementation realities were also key factors in in- fluence. Country clients informed IEG that they valued knowledge tailored to their context, which was similarly found by IEG (World Bank 2013a). Clients who use the World Bank’s ASA find it most effective when it customizes “best practice to local conditions” and formulates “actionable recommendations that fit local administrative and political economy constraints” (World Bank 2013a, 65). Interviews noted tension between the current supply-driven mod- el, which provides evidence, tools, advice, measures, and analyses developed by the Education GP, and the level of interest, demand, and uptake by coun- tries and operational TTLs that support client governments. Convening and Partnership Independent Evaluation Group World Bank Group    25 The World Bank has worked strategically and synergistically with partners to build global buy-in for learning poverty, which has unified partners in their global message. Through its partnerships, the World Bank has contributed to two important supporting conditions in the conceptual framework. It has built international commitment to improve learning and helped create a coalition of development partners to address the learning crisis. The learning poverty initiative was particularly important in these achievements (box 2.2). The Foundational Learning Compact also holds promise for improving the availability of essential national assessment data. The World Bank, the UNESCO Institute for Statistics, UNICEF, and other partners have agreed to the Foundational Learning Compact (launched in 2021) to support coor- dinated efforts to strengthen national assessment systems to ensure that low-income countries have at least one quality measure of learning in two grades and two subjects by 2025 and two measures of learning in two grades and two subjects by 2030. IEG’s review of documents and interviews was not able to identify how many new countries met the goal from the support provided by the Foundational Learning Compact. Going forward, a focus on the 24 countries in the Africa Region that lack a good-quality national assess- ment and have not participated in international or regional assessments in the past five years is particularly needed, given the global importance placed on learning data (Global Coalition for Foundational Learning 2023).  earning Poverty and Its Effect on Global and Country Box 2.2. L Stakeholders toward Action on the Learning Crisis The World Bank and the United Nations Educational, Scientific, and Cultural Organization (UNESCO) Institute for Statistics launched the learning poverty initiative in 2019 supported by trust funds and implemented in coordination with many development partners, especially the UNESCO Institute for Statistics and the United Nations Children’s Fund. The World Bank focused its analytic support, building on a comparative strength, to develop consensus among global partners for the validity of learning poverty.a Learning poverty measures both schooling and learning, focusing on reading because it is a requisite skill for other subjects and a proxy for foundational learning. The Learning Poverty Global Database and code book are accessible to the public.b Thus far, learning poverty data are only disaggregated by gender. Although learning poverty was already codified in one of the measures to evaluate progress on Sustainable Development Goal target 4.1, its simplicity and focus enabled stakeholders to unite on a common message. Data to calculate learning poverty Confronting the Learning Crisis Chapter 2 are missing in some countries, particularly in Eastern and Southern Africa, where only 29 percent of countries have this indicator. The World Bank, the United Nations Children’s Fund, the UNESCO Institute for Statistics, and UNESCO have prepared joint publications about the status of learning. For example, Ending Learning Poverty: What Will It Take? provided global estimates based on the learning poverty of the low levels of learning, and ambitious global and country targets have been set to motivate global and country stakeholders toward collective action (World Bank 2019b). Other partners (the UK Foreign, Commonwealth & Development Office; the United States Agency for International Development; and the Bill & Melinda Gates Foundation) (continued) 26  earning Poverty and Its Effect on Global and Country Box 2.2. L Stakeholders toward Action on the Learning Crisis (cont.) also use learning poverty, which created a single message from global partners to education stakeholders across the globe. At the end of 2020, the World Bank and partners (the United Nations Children’s Fund; the UK Foreign, Commonwealth & Development Office; the United States Agency for International Development; the Bill & Melinda Gates Foundation; and UNESCO) drew attention to the phenomenon of learning loss, which has become the focus of significant public debate in light of the COVID-19 pandemic,c and The State of Global Learning Poverty: 2022 Update showed the worsening of learning poverty associated with pandemic school closures (World Bank, UNESCO, et al. 2022). Sources: World Bank 2019b; World Bank, UNESCO, et al. 2022. Notes: a. The learning poverty indicator is defined as the share of children in countries who are un- able to read and understand a simple text by age 10 years. This indicator brings together schooling and learning indicators. b. The code is in GitHub; running it allows anyone to generate the outputs (https:/ /github.com/ worldbank/LearningPoverty). The output data (Excel file) is in the development data hub (World Bank 2023b). The output country briefs are also public (World Bank 2024a). c. See, for example, Mervosh (2022). The World Bank response to the challenges introduced by the COVID-19 pandemic has been characterized by heightened partnership and the forma- tion of new working coalitions and alliances that were established to address Independent Evaluation Group World Bank Group    27 immediate needs arising from school closures. There are many examples of important partnership-driven work, including regional partnerships (box 2.3) and larger-scale initiatives. Examples of the latter include the joint approach between the World Bank and the Global Partnership for Education (GPE)— technical assistance, data analysis, and financing—in support of learning continuity and building resilience; the joint surveys and framework on re- opening schools developed by the World Bank, UNESCO, and UNICEF; and a joint initiative with the World Health Organization that sought to provide guidance and support to governments on how to safely reopen schools during the pandemic. GPG were developed with GPE financing to provide guidance on school reopening, use of learning assessments in that process, surveys of government education response, the Global Education Recovery Tracker, blogs guiding COVID-19 responses and learning assessments, technological guidance on remote teaching, and methodologies for remote formative assessment of children’s learning. The World Bank joined the UNESCO-led Global Education Coalition—comprising over 200 private sector members, multilateral institutions, nongovernmental organizations, civil society actors, networks and agencies, and international media groups—which is an initia- tive to support countries’ efforts to mitigate the effects of school closures. The World Bank also partnered with the Bill & Melinda Gates Foundation to help countries implement remote learning solutions, providing techni- cal assistance to education ministries and supporting the development of education technology. The World Bank has also worked with HundrED to develop Technology for Teaching—a program to enhance and increase teacher professional development opportunities using technology-based solutions. However, the use and influence of these global products among country cli- ents have not been evaluated. More recently, early implementation of the Accelerator Program, which em- braces the multidimensionality of basic education systems and the related political and other dynamics associated with learning failure in basic edu- cation, suggests some lessons. The program, launched in 2020 by the World Bank and UNICEF (in partnership with the Bill & Melinda Gates Foundation; the UK Foreign, Commonwealth & Development Office; the UNESCO Institute for Statistics; and the United States Agency for International Development), “aims to demonstrate that governments that are dedicated to improving their foundational learning outcomes can achieve results within a few years through focused, evidence-based action, with adequate political Confronting the Learning Crisis Chapter 2 and financial support” (World Bank 2021a). Accelerator responds to a critical challenge in brokering the supply of GPG—both those produced by the World Bank and those produced by other partners—with the priorities, demands, and needs of individual countries. Although the pandemic delayed prog- ress, the initial stage of implementation suggests that more may be needed to support countries to demonstrate results than what is allowed for in the design of Accelerator. Lessons from leading education systems highlight that sustainable improvement to learning requires concerted reform of key aspects of education systems.4 Interviews have identified issues with waning political commitment as contextual conditions change, suggesting the need for more 28 flexibility in how financing is deployed. Interviews also noted the additional time needed to coordinate among partners and clients.  egional Partnerships’ Help with Identification of Issues Box 2.3. R Contributing to Low Learning Nonlending technical assistance provided to Papua New Guinea, Tonga, and Vanuatu introduced an Early Grade Reading Assessment (EGRA). Partners of the World Bank included Fast Track Initiative, the Australian Agency for International Development, and the New Zealand Aid Programme, which together produced baselines of EGRA performance for each country during the 2009 and 2010 school years. They also built local capacity to replicate EGRA in each country and worked with each country and partners to interpret assessment findings and analyze their policy and investment implications. The assessments in Tonga and Vanuatu identified the issues underlying low learning and created an agenda for common action among partners. The crucial issues related to the large share of unqualified teachers, lack of exposure to print material, and classroom instruction not conducted in students’ native language. Partners aligned and coordinated work to support phonics instruction. The analytic work informed policy dialogue that resulted in curricular reforms in the lower primary grade. It also supported training for government staff to design and administer EGRA. Interviewees praised the success of this technical support for its high technical quality, but documents also noted implementation weaknesses by the World Bank related to Independent Evaluation Group World Bank Group    29 timeliness and dissemination of results to the client (which were due to constraints in staff availability to respond). Source: Analysis of World Bank documents and Independent Evaluation Group interviews. Feedback Process to Strengthen Systems, Teaching, and Learning The World Bank has not systematically evaluated the influence and use of ASA products in client countries and therefore lacks a feedback mechanism consistent with the conceptual framework to assess the intermediate out- comes in country clients. The World Bank’s internal reporting of outputs lacks systematic and rigorous evaluation of the influence and impact of its ASA and sometimes lacks outcome measures to support assessment. The World Bank should not be satisfied with tracking the number of countries that implement its tools and should seek evidence of sustainable changes, as was the case with the second phase of READ, which reported on baselines, targets, and achievements of capacity against various performance indicators. The efficacy of tools and programs also needs to be evaluated to ensure that they are re- sulting in improvements in systems, teaching, and measurement of learning. For example, expanding the evidence base of the efficacy of Teach and Coach to improve teaching practices and student learning is needed beyond what currently is reported.5 Reports prepared for donors of trust funds describe delivery of outputs but lack reporting of outcomes achieved with the transmission of knowledge or technical assistance. IEG has previously found that global and regional ASA should be subject to a similar self-evaluation process to financial proj- ects (World Bank 2022b), as this is a shortcoming not just of the Education GP. Despite asking for stories of impact of GPG or partnerships during IEG’s interview, the evaluation was able to capture only examples of dissemination, and few respondents were able to tell IEG what happened as a result. As the framework highlights, the World Bank will need to move beyond its current focus on production and sharing to ensure that the uptake with clients results in actions to improve systems, teaching, and measurement of learning and to ensure increases in the availability of disaggregated learning data. A theory of change can facilitate planning for implementation and monitoring of how client governments use global and regional analytics. This change is aligned Confronting the Learning Crisis Chapter 2 with the World Bank’s Knowledge Compact for Action: Transforming Ideas into Development Impact (World Bank 2024b), which will require stronger monitor- ing to provide a feedback loop to the World Bank (box 2.4). 30  he Foundational Learning Compact Lacking Outcome Box 2.4. T Measures The Foundational Learning Compact trust fund provides financing for various initiatives in the Education Global Practice, including Coach, Teach, the Accelerator Program, and the Learning Compact, which were separately funded by various donors before being housed under a single umbrella mechanism. The Concept Note for the Foundational Learning Compact states that its objective is to enhance global and country efforts to pursue systemic and sustained improvements to early childhood, primary, and secondary education systems to achieve learning for all. The higher- level objectives include reducing learning poverty and increasing learning-adjusted years of schooling. The results framework, built on two pillars (figure B2.4.1), does not feature any outcome indicators or explicitly link the path of the World Bank’s actions to changes in clients—for example, moving beyond tracking the number of countries where policy dialogue is informed by the work of the Foundational Learning Compact. Independent Evaluation Group interviews similarly reported the gap in rigor in the results chain and the lack of outcome measures.  esults Framework for the Foundational Learning Com� Figure B2.4.1. R pact Trust Fund PILLAR 1 PILLAR 2 Measurement Evidence-Based Policies and Systemic Reforms Independent Evaluation Group World Bank Group    31 Tracks number of Indicators include » Accelerator countries with foundational » Number of children benefiting from learning targets developed evidence-based interventions » Countries having implemented a Global » Number of teacher policies or programs Education Policy Dashboard changed or informed by relevant » Countries implementing Teach programs Among other measures Source: Independent Evaluation Group based on Concept Note and interviews. 1  Advisory services and analytics products are coded based on the project identification number and may contain multiple outputs. 2  Between 2010 and 2017, the application of the Systems Approach for Better Education Results 1.0 produced 13 framework papers, 11 domain tools, more than 190 country reports, five global or regional analyses, and more than 50 case studies and background papers. 3  The policy areas in the Teachers policy framework were as follows: setting clear expecta- tions for teachers, attracting the best candidates into teaching, preparing teachers with useful training and experience, matching teachers’ skills with students’ needs, leading teachers with strong principals, monitoring teaching and learning, supporting teachers to improve instruc- tion, and motivating teachers to perform. 4  For example, leading systems select and encourage teachers to grow in their careers (Schleicher 2018). NCEE’s Blueprint for a High-Performing Education System highlights attention to reforms focused on systems (capacity and coherence across levels), measurement of learn- ing, teaching, and focus on equity (NCEE 2021). 5  The World Bank plans for further evaluation but reports that Teach is linked to higher student achievement in language and mathematics. It has also explored the extent to which raters or enumerators contributed to Teach score bias, finding that the scores produced by the class- room observation tool are mostly the product of the aspects of teacher quality measured by each of its items and the teacher performance, rather than the product of enumerator bias. Confronting the Learning Crisis Chapter 2 32 3|T  he World Bank’s Approach to Basic Education and Learning Outcomes at the Country Level Highlights During fiscal years 2012–22, in the 91 countries in which the World Bank supported basic education, it supported a single operation in 38 countries (7 in the Africa Region) and two operations in 22 countries (10 in the Africa Region), excluding emergency COVID-19 operations. In 12 countries, it supported five or more operations. Countries with two to four operations have learning poverty rates ranging from 11 percent to 98 percent, which suggests that support for basic education projects may not always focus on the countries with the lowest outcomes. World Bank analyses tend to address symptoms or parts of iden- tified challenges without addressing the fundamental causes of education failure. The World Bank typically supports similar inputs—government-level management, in-service teacher training, and school management— across most country types. The results of these inputs are measured by outputs rather than by changes in systems, improved teaching, or increased learning, missing a critical feedback loop to demonstrate whether the inputs are effective and contributing to learning. For example, monitoring and evaluation of improved learning outcomes are specified in one-third of project development objectives in the basic education portfolio, and only 22 out of 188 operations with on-the-job training tracked the impact of the training on teachers’ practices. 33  The causes of failure can involve stakeholders at levels other than central government. The World Bank typically focuses its attention at the higher level, giving it a comparative advantage in influencing education policy; however, lower levels of basic education systems, which must implement that policy, get less capacity-building sup- port from the World Bank. Reference to marginalized groups and the level of analytic focus on broader equity-related issues increased over the evaluation period in Country Partnership Frameworks and Systematic Country Diagnostics, as well as in Project Appraisal Documents, where there is also an increase in targeting of such groups. Monitoring and reporting of results on a disaggregated basis predominantly focus on gender and not on other groups, which weakens the ability of policy makers to assess the adequacy of the response and to provide feedback for future policy and planning. In responding to the challenges of COVID-19, the World Bank accel- erated emergency financing through 80 operations (34 percent of the portfolio) to address schools reopening by supporting incentives and inclusion, school health and nutrition, information and commu- nication technology, learning materials, and community participation for remote learning and for reopening schools. 34   This chapter examines World Bank country-level support for basic edu- cation. It draws on evidence generated from a portfolio review analysis and 10 case studies undertaken for the evaluation. It also draws on secondary data analysis and evidence from background papers that researched aspects of the literature on the political economy of education and the approach adopted by other development partners in their support for basic education. The chapter details World Bank financing and its basic education portfolio in the context of overall subsector financing and then analyzes key challenges identified by the World Bank and inputs provided in response. Findings are presented with reference to the evaluation framework set out in chapter 1, with specific focus on systems analysis, funding, teaching, measurement of learning, capacity across all levels of systems, equity, partnership, and political commitment—a necessary condition for more effective World Bank support. Overall Financing for Basic Education Before discussing the World Bank basic education portfolio, it is important to contextualize the broad shape of financing for education, including basic education. Between 2009 and 2019, governments contributed 82 percent to all expenditure on education, households contributed 17 percent,1 and devel- opment assistance accounted for 1 percent (World Bank and UNESCO 2021). Less than half (43 percent) of the development aid contribution to expendi- ture on education goes to basic education. Development assistance accounts Independent Evaluation Group World Bank Group    35 for a greater part of expenditure on education (all levels) in low-income coun- tries than in lower-middle-income countries. The main difference between high- and low-income countries in education investment, the Education Finance Watch 2021 found, stems from differences in the overall size of the public sector, rather than differences in how education is prioritized, and in the equity of its distribution (World Bank and UNESCO 2021). This is an important observation regarding the financing of education reform—a key element of the conceptual framework—showing that, broadly speaking, gov- ernments are prioritizing education spending. The Education Finance Watch 2022 found that, since the onset of the pandemic, overall bilateral aid to ed- ucation had fallen (World Bank and UNESCO 2022), and the Global Education Monitoring Report estimates an annual financing gap of $97 billion during 2023–30 in 79 low- and lower-middle-income countries to achieve SDG 4 targets (UNESCO 2023).2 The World Bank Basic Education Portfolio The World Bank is the single largest source of external financing for the education sector in low- and middle-income countries (World Bank 2023c), although funding for the sector—and for basic education—is a small part of its overall lending. For example, International Bank for Reconstruction and Development and International Development Association commitments to education in FY22, from preprimary to tertiary levels, represented just under 5 percent of all World Bank commitments. The World Bank (2022b) reports that 24 percent of the active portfolio ($23.6 billion) goes to primary education and a further 25 percent to secondary education (with no break- down provided between lower and upper levels of secondary education). Extrapolating in the absence of series data, it is reasonable to conclude that primary education, the core of basic education, attracts about 1 percent of overall World Bank lending.3 Given the criticality and the scale and depth of the crisis in learning, the need to ensure strategic targeting of this limited resource is evident, and, as per the evaluation framework, strategic targeting can be achieved if based on a comprehensive systems-based analysis. During FY12–22, the World Bank approved 236 basic education operations with a total commitment value of $25 billion. About 86 percent of those education operations (203 operations) are supported by investment project financing, with an average commitment of $83 million. In more recent years, the World Bank has introduced the Program-for-Results instrument, with 59 percent of the 29 Programs-for-Results in the portfolio approved since Confronting the Learning Crisis Chapter 3 2020. Case studies also found that disbursement-linked indicators had been introduced to investment project financing in Ethiopia, Kenya, Pakistan, and Sierra Leone as a precursor to the more outright results-based approach under Programs-for-Results. IEG found no differences between investment project financing and Programs-for-Results and the type of intervention and measurement.4 The portfolio featured only four development policy loans.5 The average project size across all instrument types, including support during the COVID-19 crisis, is $106 million, with significant variation in project volume, ranging from a minimum of $0.24 million to a maximum of $1,006 million. South Asia had the largest average of $215 million and median of $123 million, probably reflecting the larger populations in the countries 36 supported, such as Bangladesh, India, and Pakistan. About 46 percent of all operations were approved in a four-year period (FY19–22), with a spike in FY21 (47 projects compared with an annual average of 21 over the entire evaluation period) at the height of the COVID-19 crisis (figure 3.1).  asic Education Project Approval during Fiscal Years 2012–22 Figure 3.1. B 50 47 45 40 35 Projects (no.) 30 25 25 25 22 21 20 20 Average 21 20 17 15 15 12 12 10 5 0 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 Approval FY Source: Independent Evaluation Group portfolio review. Note: FY = fiscal year. The largest share of projects by number (44 percent) is in the Africa Region, followed by 16 percent in both the Latin America and the Caribbean and the South Asia Regions (figure 3.2). During the evaluation period, the number of education operations has increased in countries classified as fragile and Independent Evaluation Group World Bank Group    37 conflict-affected situations, from 23 to 39. This increase is consistent with the World Bank’s priorities (fragile and conflict-affected situations strategy) and reflects support provided during the COVID-19 pandemic. The ratings of education sector operations have steadily risen, based on delivery of outputs, and are high relative to all World Bank operations (box 3.1). Financing is one of the World Bank’s levers of influence in the education sector, particularly in lower-income countries. In Ethiopia, for example, the World Bank brought more resources and development partners into pooled funding support for the government’s program in a context characterized by a significant financing gap between the sector strategy goals and the available resources to address the growing demographic. In Viet Nam, a rapidly devel- oping lower-middle-income country, by contrast, the World Bank’s financing offers less leverage, although it does provide “a seat at the table,” which the World Bank has been able to use to help support the introduction of innova- tions from other contexts. This has included the introduction of curriculum and pedagogical changes to promote critical thinking and support for equity in education.  he Project Ratings System Box 3.1. T Education sector projects—inclusive of projects spanning early childhood develop- ment to tertiary education—have been among the best performing of all World Bank projects (figure B3.1.1). Under its agreement with the World Bank, the Independent Evaluation Group uses an objectives-based approach to rating or validating the self-evaluation of projects undertaken by the World Bank itself. A key rating criterion is the level to which a project has achieved its objectives. However, the achievement of objectives set for World Bank projects in the education sector is typically defined and measured by outputs, such as number of schools built or number of teachers trained, and not by learning outcomes.  erformance of Education Sector Projects Relative to All Figure B3.1.1. P Other World Bank Projects 100 Education 90 80 All Project performance (%) 70 60 50 Confronting the Learning Crisis Chapter 3 40 30 20 10 0 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 Year Source: Independent Evaluation Group Implementation Completion and Results Report database. The basic education portfolio includes 80 operations (34 percent) that in whole or in part address the effects of the COVID-19 pandemic on basic 38 education systems.6 The emergency funding was provided to 48 countries. The average volume for the 80 projects is $121 million, which is greater than the overall average project volume across the evaluation period. The majori- ty of projects in the South Asia Region (n = 21; 57 percent) responded to the COVID-19 crisis (figure 3.2). In the Africa Region, 34 projects (32 percent of projects in the Region) responded to the crisis.  egional Distribution of All Projects and COVID-19 Support Figure 3.2. R 250 200 Projects (no.) 150 100 50 0 South Middle East and Africa East Asia Europe and Latin America All Asia North Africa and Pacific Central Asia and Caribbean Region All projects (no.) All projects (average volume) COVID-19 projects only (no.) COVID-19 projects only (average volume) Source: Independent Evaluation Group portfolio review. The evaluation found that, in many countries, World Bank support for basic education lacks intensity and continuity in the face of what, as detailed in the evaluation framework, is a particularly complex problem. World Bank Independent Evaluation Group World Bank Group    39 support was limited to one or two projects during FY12–22 in more than half of the 91 countries in which it provided basic education support (figure 3.3).7 Excluding COVID-19–specific support, World Bank support for basic educa- tion consisted of a single operation in 38 countries and two operations in 22 countries. In a further 11 countries, support for basic education consisted of a single COVID-19 emergency response operation—that is, the World Bank was not otherwise supporting basic education reform in those countries during the evaluation period.8 Countries supported by a single operation were in all Regions, including 7 countries in Africa (Burundi, Cabo Verde, the Republic of Congo, Guinea, Mozambique, Somalia, and Zimbabwe). Among the countries that had two operations during the evaluation period 9 were 10 countries in the Africa Region, where the crisis in learning is concentrated.10 Twenty coun- tries (22 percent) had four or more operations in support of basic education during FY12–20. In 12 of these countries,11 five or more operations were sup- ported. Countries with two to four operations had moderate to high learning poverty rates, ranging from 11 percent to 98 percent. Overall, and notwith- standing competing priorities, lending limits, and other influencing factors, the relative lack of concentration on basic education suggests that support may not always focus on the countries with the lowest outcomes.12  orld Bank Lending and Grants for Basic Education during Figure 3.3. W Fiscal Years 2012–22 Source: Independent Evaluation Group portfolio review. Note: Color coding represents the number of projects, excluding COVID-19 operations, for fiscal years 2012–22. This map has been cleared by the World Bank Group cartography unit (IBRD48078; July 30, 2024). World Bank Analysis toward Learning for All at Confronting the Learning Crisis Chapter 3 the Country Level The framework for this evaluation emphasizes the need for a comprehensive analysis of basic education systems to correctly orient interventions in favor of reform. The analysis needs to look across the system and to take into account, for example, the political economy of basic education and the perspectives and positions of key stakeholders. It also needs to be both technically sound and contextually sensitive so that the design of interventions can lead to du- rable change and desired outcomes. The evaluation found that World Bank analysis of specific aspects of basic ed- 40 ucation systems—what is or is not working—is technically sound but typically does not sufficiently grapple with why education systems continue to fail children. Analysis of the adequacy of specific elements, such as data systems, financing, and teaching quality, helps to identify specific components that may be working or not working and provides a foundation for specific inter- ventions. Such analysis may miss the idiosyncratic nature of basic education systems that respond to the unique mix of political, social, cultural, religious, and other factors that shape such systems. TTLs interviewed for this evalu- ation claim to be tacitly aware of these nuances. Institutional imperatives, such as the rotation of international TTLs every three to four years, may limit the extent to which it is possible to develop and maintain a deep un- derstanding of context. Both formal and informal mechanisms are required to exchange, document, and curate tacit knowledge regarding the political economy and underlying drivers of education system performance. The case studies identified a need for engagement to be contextualized to address political economy barriers, consistent with the evaluation framework. For example, case studies found that the World Bank predominantly engaged with central government actors, or in Brazil and Pakistan also with state and provincial government, and with development partners. Such an approach ne- glects the possible impacts of dynamic interaction among multiple, potentially powerful stakeholders—local administration, teachers and trade unions, par- ents, nongovernmental organizations, and civil society organizations. These parties can affect the achievement of desired outcomes or the alignment and Independent Evaluation Group World Bank Group    41 capacity of the basic education delivery system, especially the actors in the lower levels of the system on whom fidelity to policy reform and implementa- tion success depends. Analysis of documentation at the country level suggests that a more uniform, less nuanced approach is taken in the provision of support to basic education; see the analysis of challenges identified in Project Appraisal Documents (PADs) and World Bank inputs in this chapter. The focus of project analysis and country documents has not changed during the evaluation period, notwithstanding publication of the WDR 2018 and its advocacy of a more systems-based approach. Across the period, the key challenges identified in the basic education portfolio are weak learning outcomes; learning inequity; inadequate teaching quality; and weak gover- nance, accountability, and institutional oversight (table 3.1).13 For fragile and conflict-affected countries, learning inequity was a slightly higher concern (85 percent), whereas the other challenges were consistent with the overall pattern. Two additional challenges are emphasized for the Africa Region (and with countries affected by fragility, conflict, and violence)—inadequate learning environment and low educational attainment—suggesting deeper challenges at a more basic level in relevant education systems (that is, with matters such as levels of enrollment versus the adequacy of basic infrastruc- ture, or numbers of teachers).  hallenges of Most Concern in Project Appraisal Documents Table 3.1. C 2012–17 2018–22 Incidence Incidence Challenges Rank (%) Rank (%) Weak learning outcomes 1 81 2 78 Learning inequity 2 78 1 81 Inadequate teaching quality 3 74 3 62 Weak education system: 4 69 3 62 governance, accountability, and institutional oversight Source: Independent Evaluation Group portfolio analysis. Failure to use systems-based analysis to give attention to the unique political economy and supporting conditions can have negative effects on the effi- cacy of World Bank operations. As the WDR 2018 argues, understanding the Confronting the Learning Crisis Chapter 3 political economy is necessary to address the reasons education systems fail children. The World Bank has supported a comprehensive set of interventions during FY12–22, but these interventions encountered systemic challenges related to motivation, human resource capacity, adequacy of financial re- sources, and need for deeper coordination across all levels of the education system, as the Ethiopia case shows (box 3.2). The case studies identified in- stances where the World Bank had not anticipated government pushback. For example, in Viet Nam, a high-profile program was dropped in one instance and there were delays to a critical teacher training program in another. In Chad, the World Bank did not adequately appreciate low capacity at the cen- tral level, which resulted in the nondeployment of newly qualified teachers 42 because there was no budget to recruit teachers trained under a project sup- ported by the World Bank.  thiopia: What Is Not Going Well and Why Box 3.2. E The government of Ethiopia is constitutionally and legally committed to maintaining the integrity and capacity of education administration down to the woreda (district) level. The World Bank’s analysis finds key impediments to that mandate, such as feedback mechanisms that do not adequately inform policy makers of what works and what does not, inadequate capacity of internal audit systems and procurement, and inadequate capacity at decentralized levels. Regional governments decide how much of the region’s budget is allocated to education and transferred to each woreda, resulting in regional variability in unit costs. Moreover, public spending on a per-pupil basis favors tertiary education, which serves 3 percent of students, whereas per-pupil spending at earlier-grade bands is low, given that those grades serve 63 percent of students. Other sector challenges include teacher qualifications, dropout rates among students from poor families, and a large share of out-of-school children and youth (approximately one-third). Ethiopia’s large and rapidly growing school-age population exacerbates all these challenges, despite the government allocating 22 percent of its public funding to education. Despite the sound analysis and identification of specific challenges (answering what the problem is) and the comprehensive set of interventions supported by the Independent Evaluation Group World Bank Group    43 World Bank during the decade, interventions have encountered systemic challenges that show why one-dimensional fixes do not work. For example, women teachers were trained to become school leaders, but hiring them and sustaining them as principals (or vice-principals) ran into difficulties: civil service regulations requiring open competition for positions, cultural norms in regions unfamiliar with women in leadership roles, and a work burden that is more demanding than teaching (without a commensurate salary). Other challenges arose for the licensing of teachers and school leaders, which was implemented by testing new and existing teachers to assess their performance and by creating accreditation standards for teacher education institutes and a teacher licensing information system for collecting, managing, and analyzing licensing data. In that instance, the effort was challenged by the lack of incentives for teachers and school leaders to undergo licensing tests because the licensing results (continued)  thiopia: What Is Not Going Well and Why (cont.) Box 3.2. E were not linked to their career development, suggesting that understanding teacher motivation at the outset was needed. The analysis of interviews, documents, and reports suggests that some answers may be related to motivation, human resource capacity, adequacy of financial resources, and need for deeper coordination across all levels of the education system. Resource allocation within the sector and resource coordination between the federal government and regions remain an issue because regions are not required to allocate resources to education, as suggested by the Ministry of Education. Project Implementation Completion and Results Reports consistently noted human resource challenges that affected implementation. Despite training and capacity-building efforts, high staff turnover was related to government remuneration, which is capped at low levels. Source: Independent Evaluation Group analysis of World Bank documents and interviews. Effective Support for Policy Reform toward Learning for All Where the World Bank has a willing and committed partner, it has been able to better focus on key policy reforms and to lay foundations for a learning- oriented system. Allowing for difference in context and the modality of im- plementation, leading education systems in the developed world, such as in Confronting the Learning Crisis Chapter 3 Finland and the Republic of Korea, pay particular attention to key facets of basic education systems, including teaching and career progression, measure- ment of learning, financing to achieve equity of learning, and meritocracy in hiring at all levels of the system. Where countries are in tune with, or at least in deliberate, sustained pursuit of these conditions, World Bank support is most effective. Brazil, Kenya, and Viet Nam have significant differences in political structure, culture, basic education systems, and many other characteristics; however, in all cases, they have strong political and financial commitment in support of learning for all combined with a strong equity focus (figure 3.4). Across these 44 countries, political commitment goes beyond rhetorical statements in the media and sector strategies. Commitment is evident with clear implementa- tion actions to improve quality and learning—even in systems with growing demographics. It is also evident in political commitment to communicate learning data and establish clear goals for learning improvement. Brazil, Kenya, and Viet Nam have also made their political commitment appar- ent by allocating financial resources to prioritize primary and foundational learning, consistent with leading education systems. The evaluation found that in those contexts, the World Bank has been able to deploy its resourc- es—knowledge, technical assistance, policy dialogue, and financing—more effectively in support of reforms that contribute to improvement in learning outcomes, consistent with aspects emphasized by leading education systems. In these countries, the World Bank intervened subject to context and across the leverage points included in the conceptual framework for this evalua- tion—during planning, implementation, and monitoring—based on where the country was in relation to reform efforts in pursuit of learning for all. Aspects of the World Bank’s analysis of context and engagement could have been strengthened in each case, but the examples illustrate broad understanding of context and leverage points in support of more effective system reform. Sustained funding by the government and its partners, including the World Bank, and high parental expectations and high social value for education were key to successful basic education reform in Viet Nam. The reform ef- Independent Evaluation Group World Bank Group    45 fort was buoyed by sustained investment from a growing economy, including significant investment in administrative and teaching capacity, with financ- ing for foundational learning and basic education as a priority.14 As part of its planning, the government requested and absorbed the World Bank’s com- prehensive analytic support, which addressed the quality, economics, and equity of education. World Bank research and evaluation, as well as support for student assessments since the early 2000s, were key to defining needs and monitoring progress. World Bank operational support shifted with government needs, from in- terventions focused on specific features of school quality (Viet Nam Escuela Nueva and School Education Quality Assurance Program) toward more comprehensive national curriculum and teacher training projects (Enhancing Teacher Education Program and Renovation of General Education Project). The latter supported competency-based learning, recognizing that Viet Nam’s education system had achieved most of its goals for foundational learning and basic skills. The competency-based curriculum places heavy demands on teachers; therefore, the emphasis is on teacher professional development— work that remains in progress. The government also sustained, with World Bank support, a focus on equity in education. Ethnic minorities were a major focus in projects, reflected in the choice of provinces and schools in the School Education Quality Assurance Program and Viet Nam Escuela Nueva and in implementation arrangements for training, textbooks, and monitoring in the full-scale projects (Enhancing Teacher Education Program and Renovation of General Education Project). Attention to rural poor people was less pronounced, but World Bank programming clearly recognized the extra challenges in project implementation in rural areas and schools. Children with disabilities also received less attention in general in World Bank projects, with one exception—Quality Improvement of Primary Education for Deaf Children.  evels of Financial and Political Commitment to Learning for Figure 3.4. L All in Brazil, Kenya, and Viet Nam BRAZIL KENYA VIET NAM a. System capacity a High High Medium b. Government spending for education (%) Confronting the Learning Crisis Chapter 3 Tertiary Tertiary Primary 13.1 Tertiary 22.4 Primary 15 Primary 25.5 29.7 36.3 Secondary Secondary Secondary 41.8 39.6 39.4 c. Political commitment to learning for all High High High Source: Independent Evaluation Group case studies. 46 Note: a. Capacity was measured using percentiles among Country Policy and Institutional Assessment data. Although broadly successful, the World Bank’s program of support in Viet Nam faced political challenges. An envisaged scaling up of Viet Nam Escuela Nueva was not achieved, despite technical implementation that was largely successful, because of resistance to change among some parents and teach- ers, and the voluntary expansion beyond project targets encountered issues due to lack of resources. Political pushback also led to delay and eventual cancellation of key features of the Renovation of General Education Project curriculum and textbook reform project and to challenges with the in-service teacher training program (Enhancing Teacher Education Program). In the years before the COVID-19 pandemic, the government of Kenya em- barked on ambitious reforms that sought to improve the quality of education through several approaches: a competency-based curriculum, reforming of professional teacher development, textbook policy, and management prac- tices at the local level. Learning outcomes have been variable since 2016, with significant challenges remaining, particularly in rural areas, but overall the country ranks as a top performer in Eastern and Southern Africa. In this case, the World Bank had little influence in basic education during the early years of the evaluation period because of a strained relationship associated with a failed sectorwide approach. During the evaluation period, the World Bank regained and leveraged its influence through (i) ongoing contact with the government, despite the lending hiatus; (ii) renewed lending with GPE support; and (iii) development of a close working relationship with the gov- Independent Evaluation Group World Bank Group    47 ernment in support of strategy and planning.15 Since recommencement of financial support in 2015, continuous support and a high level of engagement have ensured a high level of World Bank influence. After initially using GPE funding to formally reengage with the basic education subsector, the World Bank has since positioned itself at the heart of basic education reform in Kenya with a continuous, connect- ed string of projects (Primary Education Development Project, Secondary Education Quality Improvement Project, COVID-19 Learning Continuity in Basic Education Project, Primary Education Equity in Learning Program, and future operations already being planned). This level of engagement has allowed the World Bank to occupy a sustained, influential position in policy making for basic education. The World Bank also worked with the government toward a more inclusive, quality-driven education system, taking a systemic perspective in working to enhance the capacity and standing of the ministry and key agencies.16 For example, the World Bank supported training of staff in the psychometric department of the Kenya National Examinations Council in specific technical skills to enable them to improve learning assessment instruments and to provide more informed advice to the ministry on Kenya’s participation in international assessments. Sustained effort was also key to advances in data collection in Brazil, where successive governments supported the most disadvantaged individuals and regions over two decades through measures such as Bolsa Família and the Fund for Maintenance and Development of Basic Education. Brazil has a long history of investing in indicators and monitoring systems, including the Basic Education Assessment System and the national education quality index. These initiatives, with support from the World Bank, have provided valuable measures of learning outcomes, promoting evaluation of the effectiveness of schooling and informing better policies and practices. The data and monitor- ing systems compare well with those of many developed countries and have been augmented by many related initiatives at state and municipality levels (OECD 2021). The World Bank contribution started before the evaluation period, with dialogue and technical assistance to support the development of outcome measurement and with help to equalize federal financial allocations, taking equity into account. During the evaluation period, the World Bank worked with the state of Ceará on results-based management mechanisms, providing Confronting the Learning Crisis Chapter 3 technical support that contributed significantly to improving governance, accountability, institutional capacity, school management, and meritocracy. In Ceará, the World Bank supported the research institute Instituto de Pesquisa e Estratégia Econômica do Ceará to develop an innovative and context-specific, results-based management reform to incentivize mayors to improve the qual- ity of basic education. This contributed to substantial system improvement and alignment of implementation between the central and lower levels of government because results-based management mechanisms are now shaping monetary transfers from the federal level to the states and from the states to the municipalities, contingent on achieving predefined learning outcomes. The analysis of binding constraints was used to define areas supported by the 48 operation—length of school day, inadequate teacher quality, and age-grade distortion. The analysis recognized that race, gender, geography, and socio- economic status affect the probability of accessing quality education, but it did not elaborate on the causes, implications, and possible solutions to effec- tively support students in these groups. Instead, efforts to improve learning have focused on students lagging behind their age-group, regardless of other characteristics that may affect learning outcomes. The World Bank’s analyses in Brazil, recognizing the importance of politi- cal context, show a thorough understanding of the governmental actors at the national and state levels in the education system and the perspectives of private companies and employers. The analysis could have been further enhanced with attention to how the role of municipal political actors, school managers and teachers, students, parents, and their communities are aligned to the goal of improving education quality. What distinguishes these ex- amples is that the aim is not delivery of an intervention or inputs into the system but a clear focus on system reforms toward learning. The above examples illustrate attention to some of the critical aspects of the system—measurement of learning, teaching, equity, and system co- herence and capacity. Given the limited consensus about what works to improve learning (Evans and Popova 2016), a feedback loop is essential to inform intervention choices, as depicted in the evaluation framework. This Independent Evaluation Group World Bank Group    49 means consistent examination of whether the interventions and inputs the World Bank finances are having a positive impact on systems, teaching, and measurement of learning. As discussed in this chapter, patterns across the portfolio and case studies reveal that World Bank interventions predominant- ly support activities and monitor outputs (see figure 1.2), rather than seeking incremental reform of aspects of the education system and monitoring changes in teaching, learning, and systems. World Bank Inputs and Responses to Address Learning for All at the Country Level This section discusses World Bank inputs in support of basic education and learning for all at the country level, largely based on analysis of country need. It details knowledge inputs and the types of inputs financed by World Bank operations and then looks in detail at particular aspects of the response relat- ed to capacity building, equity, partnership, and other factors. Knowledge Input at the Country Level The volume of country-level advisory and analytic World Bank support that focuses on quality basic education, the crisis in learning, or inclusion is lim- ited. Among the 10 case countries, IEG identified 68 education-related pieces approved between FY12 and FY22. This consisted of 34 ASA products, of which less than half across the 10 cases over 10 years more directly addressed basic education and matters related to quality and inclusion, such as work on learning (Ethiopia, Pakistan, and Viet Nam), teachers (Kenya, Sierra Leone, and Tajikistan), refugees (Ethiopia and Kenya), and service delivery (Chad and Pakistan). The remainder of the ASA projects (29) typically covered broad policy areas—human capital, workforce education, education public expen- diture review—within which basic education is relevant but not the featured component. Other country-level analytic and advisory activities included technical assistance (15), such as consultancy support for the development of the education sector strategy in Sierra Leone and workforce development in Viet Nam, as well as economic and sector work (14) that covered subjects ranging from human development (Nepal) to science, technology, and inno- vation (Viet Nam) and an education sector review (Pakistan). Types of Input Supported by World Bank Operations Confronting the Learning Crisis Chapter 3 In response to its analysis of education sectors and client demand, the World Bank supports similar inputs across most country types.17 The most regular- ly supported inputs are government-level management, in-service teacher training, and school management, which are supported across the portfolio in 88 percent, 80 percent, and 75 percent of projects, respectively.18 This level of concentration in the portfolio and similar observations in case studies sug- gest a lack of nuanced response to the differing contexts and basic education systems within which the World Bank works.19 The World Bank also supported other inputs, such as learning materials (69 percent), school infrastructure (62 percent), learning assessment (61 percent), and information and com- munication technology (47 percent), with support for the latter increasing, 50 particularly during the COVID-19 pandemic. In the Africa Region, there was added emphasis on school health and nutrition (found in 61 percent of opera- tions, compared with 34 percent across other Regions), which is one of the few interventions consistently shown to have positive effects on learning in most contexts (Snilstveit et al. 2015). In countries classified as fragile and con- flict-affected situations, slightly more attention is paid to school management, learning materials, community participation, school health and nutrition, preservice teaching, and teacher recruitment, and less attention is paid to learning assessments and information and communication technology than in non-fragile and conflict-affected situations (figure 3.5).  orld Bank Project Support of Inputs in Areas Related to Figure 3.5. W Basic Education (percent) Government-level management 87 89 In-service teacher training 81 79 School management 84 71 Learning materials 74 67 School infrastructure 65 61 Learning assessment 56 62 Incentives and inclusion 61 57 Community participation 69 55 Information and communication technology 29 53 School health and nutrition 56 42 Independent Evaluation Group World Bank Group    51 Preservice teacher training 44 37 Curriculum design 39 32 Teacher recruitment 45 29 Education sector law, policy, and regulation 15 16 Other 5 3 FCS Non-FCS Source: Independent Evaluation Group portfolio analysis. Note: FCS = fragile and conflict-affected situation. Support for Capacity Development World Bank engagement at the central level puts it in a strong position to influ- ence policy development—a key initial component of the evaluation framework. Case studies found that the relationship with central government—ministries and key agencies (such as those involved in curriculum or assessment)—represents a comparative advantage in that it allows for access to policy makers and for influencing the broad trajectory of education policy. As figure 3.5 shows, management training, with an emphasis on central government, has been the most common input in World Bank projects over the evaluation period. However, despite the close relationship that the World Bank typically has with central governments, partners told IEG that the World Bank was less likely to “push” governments on more progressive reform. TTLs referred to the balance that must be struck between access to decision makers and the extent to which policy can be challenged. The World Bank’s relative lack of focus on lower levels of basic education systems has negative implications for policy implementation—an aspect highlighted in the evaluation framework (table 3.2). Management training supported by the World Bank includes significant levels of management training for central government officials. Frequent areas of support across case studies at the central level include EMIS and data, teacher training and curriculum, and student assessment capacity in countries supported by the READ trust fund (Ethiopia, Tajikistan, and Viet Nam). At lower levels of basic education systems, inputs are frequently designed to ensure effective delivery of projects—for example, training in financial management and procurement. Capacity developed in that context is relevant to effective project delivery of outputs and project monitoring but not always transferable to how things are done in the environment or context beyond the project. Confronting the Learning Crisis Chapter 3 The negative effects on enhanced learning outcomes associated with capacity constraints at lower levels of basic education systems were consistently cited in World Bank documentation, PADs, and interviews with stakeholder groups. For example, in the case of Nepal, interviewees recognized the many stake- holders in basic education, particularly given the decentralized approach that is being rolled out and the capacity challenges this raises. The World Bank (and other donors) has provided some support for the development of local government capacity, but that support is not comprehensive or sustained. The FY23–28 Country Partnership Framework (CPF) for Kenya identifies “acute capacity constraints at the local level, ambiguities in financing arrange- ments, and weak vertical coordination” (World Bank 2022a, 12) and sets out 52 to target its support where institutional and implementation capacity is most needed—that is, in relation to equity in learning outcomes rather than pri- mary enrollment (noting that education is not a CPF priority).20 Interviewees in Kenya reported that inequitable distribution of available resources to the local level reinforces disadvantages and capacity imbalances between counties and observed that although Kenya has well-developed policies, the core challenge is more about the practice and quality of implementation. Interviewees also noted that in many countries, a high proportion of public expenditure goes to education; however, because GDP is low, that level of ex- penditure is insufficient to support reform and inclusion—that is, expenditure is dominated by recurring costs (salaries), and what is available to support the development of capacity at lower levels of the system is negligible. One senior World Bank interviewee referred to enduring gaps in capacity as one of the great failures of the collective development community, stating that “every project tries to do capacity building, which usually involves studies or bring- ing in technical support or monitoring and evaluation [but does not] actually do the detailed analysis of how institutions function and what is the political economy. We find out during implementation and try to fix them as we go along.”  apacity Building in Some Case Study Countries and Table 3.2. C Associated Results Independent Evaluation Group World Bank Group    53 Capacity Building for Case Capacity Building for State, Province, or Lower Study Central Level Levels Results Brazil ASA and dialogue Project design and State sector related to learning implementation support; performance outcomes, racial equity, technical assistance with monitoring female empowerment, development of a variety contributed to teacher training, results- of outcome and process better coordination based management; indicators to monitor state and accountability technical assistance sector performance. between system on data to develop levels. evidence-based policies. (continued) Capacity Building for Case Capacity Building for State, Province, or Lower Study Central Level Levels Results Iraq Knowledge sharing, In the Kurdistan regional Not yet assessed. workshop, and government, capacity technical support to building related to student ministry relating to the learning assessment and Iraq National Education private sector engagement. Strategy. Nepal Support for governance, Local government Not yet assessed. fiduciary management, institutional capacity via dis- and technical assistance bursement-linked indicators to, in turn, support data (that is, integrating education systems, reinforced via sector plan activities into an- disbursement-linked nual work plan, budget, and indicators. EMIS policy guidelines with clearly defined roles and re- sponsibilities at various levels of government and schools; providing ownership and role to local governments in the implementation of pro-poor targeted scholarships and proscience scholarships). Tajikistan Technical assistance — Technical assistance to the National Testing to the National Center on formative Testing Center was assessments and for halted by World country-level student Bank before capaci- learning outcome data. ty was fully realized. Technical assistance The Agency on for EMIS to contain a Statistics, under Confronting the Learning Crisis Chapter 3 more comprehensive the president of database, including ag- Tajikistan, adopted gregate fiscal statistics, the EMIS-based in- demographic indicators, dicator and reporting and key macroeco- framework, allow- nomic variables, and ing the Ministry of to cover other levels of Education and local education. education groups to identify gaps and determine future priorities. Source: Independent Evaluation Group. 54 Note: ASA = advisory services and analytics; EMIS = education management information systems. Support for Equity World Bank analysis of the basic education sector in its country clients over the evaluation period shows an increasing level of focus on equity-related issues—an important aspect of the evaluation framework. Country strategies, Systematic Country Diagnostics, and PADs demonstrate sound analysis of barriers to participation and learning related to poverty and gender through- out the evaluation period. The portfolio review analysis and case studies suggest that more recent analysis demonstrates increased attention paid to barriers associated with disability, ethnicity, and displacement. Almost all World Bank projects seek to address equity-related issues and reference various target groups, with a particular emphasis on gender equity. The portfolio review examined project activities and beneficiaries, finding that nearly all (93 percent) identified specific target group(s). Gender- targeted activities, almost uniformly treated with reference to girls, were supported in 67 percent of all projects, with an even higher rate among proj- ects in Africa (82 percent). In addition, the disruption of education systems by the COVID-19 pandemic brought about a renewed or reinforced focus on girls, who, in many cases, were considered less likely to return to school because of pregnancy, early marriage, or other cultural barriers. More recent thinking, including in the World Bank gender strategy for FY16–23,21 recognizes that “weaker learning outcomes and educational achievement may be a limiting Independent Evaluation Group World Bank Group    55 factor for boys and young men” (World Bank 2015, 36), suggesting the need for a more nuanced and contextualized approach to gender than is evident across the portfolio, where only 9 out of 148 projects with gender targeting addressed learning for boys.22 The evaluation found that equity-related issues other than gender are tar- geted to a lesser extent in World Bank projects. Fifty-one percent of projects target children in rural, remote, or nomadic areas, whereas 44 percent sup- port children living in poverty (figure 3.6). Case studies indicated that, in most instances, rural poor people were defined with reference to geography rather than relative levels of income within relevant communities or schools. Furthermore, projects also address, to some extent, the educational needs of children with disabilities (39 percent); those facing ethnic, religious, caste, or linguistic disadvantages (33 percent); and those affected by conflict (16 percent) or who are out of school (15 percent), noting that it is not pos- sible through analysis of PADs to identify the level of funding in support of these groups in projects or the extent to which support was marginal or integral. Box 3.3 provides contrasting examples of the attention to inclusion by the World Bank observed in two case studies.  orld Bank Projects Targeting Equity-Related Issues Figure 3.6. W Gender 67 Rural, remote, 51 or nomadic Poor 44 Disability 39 Ethnicity, religion, caste, and language 33 Displaced, refugee, 16 or conflict-affected Out of school 15 Other 3 0 10 20 30 40 50 60 70 80 Share of projects (%) Source: Independent Evaluation Group portfolio analysis. The evaluation identified progress in recognizing barriers and targeting activities among multiple groups but also found insufficient attention to measuring equity (beyond gender and girls)—an important downstream Confronting the Learning Crisis Chapter 3 aspect of the evaluation framework and learning for all. World Bank proj- ects produce limited equity-related disaggregated data (except for gender), which makes it difficult to assess the extent to which target groups have been reached or their needs addressed. The evaluation analyzed the results frameworks in project documents to find the extent to which the targeting of specific groups is reflected in the indicators. The findings show significant variation across target groups (figure 3.7). Projects addressing gender dispari- ty have indicators with gender disaggregation in nearly all cases (91 percent). Fifty-four percent of projects focusing on conflict-affected children also include corresponding indicators to measure progress. However, for projects that include other groups, such as people with disabilities, rural residents, or 56 out-of-school children, in their targeting, only about 30 percent have indica- tors that capture the progress of those target groups.  xamples of World Bank Support for Inclusion in Pakistan and Box 3.3. E Sierra Leone Pakistan: Girls’ education is the World Bank focus in Pakistan and is addressed both in lending and in advisory services and analytics. For example, Constraints to Girls’ Education (2020) focuses on barriers to girls’ education and assesses the potential impact of the COVID-19 pandemic on girls’ return to education. The Country Partnership Framework and project documents refer to displaced populations, and International Development Association support has been provided for refugees and host communities. The World Bank has also focused on inclusiveness in education related to disability through school infrastructure, teacher development related to disability in Punjab, and an Inclusive Education Policy Landscape Study. Sierra Leone: The World Bank’s involvement in supporting the government’s Free Quality School Education initiative (launched in 2018) has taken a wide-angled approach to inclusion, with a focus on poor people and rural dwellers, but it has also focused on girls and, more recently, on people with disabilities. Certain issues, such as language of instruction, have not yet been addressed, although the World Bank is considering how it can engage in what is regarded as a politically complex issue. The World Bank also championed the National Policy on Radical Inclusion in Schools, Independent Evaluation Group World Bank Group    57 which enables and supports pregnant girls to return to school after COVID-19, unlike the situation that pertained after the Ebola outbreak. The National Policy on Radical Inclusion in Schools also embraces children with disabilities. The Country Partnership Framework notes that social safety net programs are to include disability in targeting criteria to reduce social exclusion and refers to “soft conditions” that will be used to signal to parents the importance of enrolling students on time and keeping them in school. Under the Free Education Project, the World Bank identified inadequate support to children with disabilities and barriers to their inclusion in and benefit from education. The project will address relevant issues by increasing access and improving the learning environment based on a disability analysis undertaken during project preparation. Source: Independent Evaluation Group.  rojects Targeting Specific Groups with Indicators on the Figure 3.7. P Target Group Gender 91 Displaced, refugee, 54 or conflict-affected Disabili ty 34 Poor 32 Rural, remote, 29 or nomadic Ethnicit y, religion, caste, and language 29 Ou t of school 29 0 10 20 30 40 50 60 70 80 90 100 Share of projects (%) Source: Independent Evaluation Group portfolio analysis. Operations in 6 out of 10 case countries included some level of support for children with disabilities, including a requirement for universal design in school-related infrastructure development supported by the World Bank, provision of scholarships for children with disabilities, provision of learn- ing materials, and teacher training (box 3.4). In some cases, the World Bank supported production of enhanced data in children with disabilities. In Kenya, the World Bank supported studies of cost-effective models for the expansion and delivery of primary education to disadvantaged groups, including children with disabilities, and all infrastructure development under World Bank proj- Confronting the Learning Crisis Chapter 3 ects was required to be disability friendly. In Sierra Leone, the GPE supported the Revitalizing Education Development in Sierra Leone project (2014), which supported the development of an inclusive education policy. Project prepa- ration for the Free Education Project (2020) included a disability analysis to inform how the project would support a regular monitoring system of student attendance and learning outcomes among children with disabilities and pro- mote universal design as part of the national school construction strategy. 58  xamples of World Bank Support for Children with Disabilities Box 3.4. E in Four Case Countries Ethiopia: Pilot activities provided braille books and supported grants to help raise awareness of disability issues. The World Bank also financed setting up about 800 resource centers, each of which supported a cluster of schools, and technical assistance to support a special needs education strategy. Nepal: The World Bank supported grants to integrated schools for resource classes for children with disabilities. COVID-19 additional financing covered several activities to support inclusion, such as disability-inclusive content, use of sign interpretation on televised content, and targeting of students with disabilities as a key group under a back-to-school sensitization campaign. Tajikistan: Four percent of the budget for the Global Partnership for Education project (2013) was allocated to inclusive education of children with disabilities. That budget was used to promote campaigns of inclusive education and socialization of children from boarding schools and to provide personnel to support children with disabilities and their teachers. It was also used for school upgrades (pathways, accessible latrines, and ramps), although the case study interviews suggested that maintenance of the improvements had been neglected. Viet Nam: The Quality Improvement of Primary Education for Deaf Children Project is the only World Bank–supported basic education project in Viet Nam that addresses Independent Evaluation Group World Bank Group    59 issues related to children with disabilities. Source: Independent Evaluation Group. As part of the World Bank’s corporate commitment to inclusive development for disability, all education operations aim to be inclusive of disability by 2025. To meet the criteria for a disability tag, a project must include stakeholder engagement during preparation, inclusive design features or one activity to benefit learners with disabilities, and one disability inclusion–related indicator or indicator disaggregated by persons with disabilities. IEG’s review of tagging for citizen engagement and gender identified tension between meeting cor- porate targets and ensuring the quality of engagement (World Bank 2018a). In both cases, IEG cautioned that a corporate commitment to increase tagging in 100 percent of projects, without adequate design, implementation, and moni- toring support, could inadvertently generate a “check-the-box” attitude, to the detriment of quality (World Bank 2018a). Applying IEG’s finding to the disability tag highlights a need to support TTLs with disability expertise so that the World Bank’s support and dialogue integrates disability into clients’ planning and im- plementation of education systems, consistent with leading education systems. Language of instruction was found to affect learning outcomes in 9 out of 10 case study countries, and the World Bank provided some level of support in 7 out of 9 cases without addressing underlying barriers, such as teaching quali- ty and broader capacity issues. The literature,23 including the recent literacy policy package of the World Bank, recognizes a potential barrier to learning in the choice of language used in instruction (Chong Soh, Del Carpio, and Wang 2022; UNESCO 2022), particularly for children from ethnic minority and refugee communities. In the case studies, financial support was used to produce instructional materials and textbooks in additional languages, as in Tajikistan with the GPE grant, in Nepal with the development of textbooks in more than 20 languages for first through eighth grades, and in Kenya with the International Development Association grant from the Window for Host Communities and Refugees. In Viet Nam, ethnic minority children receive additional support from local language assistants (who are bilingual) in schools. Viet Nam has also piloted incorporation of bilingual instruction for children at the primary level (and preschools) with the support of Save the Children and a prior action in a development policy operation supporting policy change. Case studies identified teachers’ knowledge as the predom- Confronting the Learning Crisis Chapter 3 inant implementation barrier. For example, the Ethiopia Education Sector Strategy sets the ambitious goal of every learner becoming multilingual— fluent in their first language, the official language, and English. Although instruction in primary grades is conducted in the native language, teachers are also expected to provide instruction in English because it becomes the medium of instruction in secondary schools, despite limited teaching ca- pacity in English, as noted by local researchers.24 The joint GPE–World Bank report for Tajikistan highlights the negative effect of this lack of capacity on learning—having a teacher who was a native Uzbek speaker was associated with a decrease of about seven words per minute in the school-level oral 60 reading fluency score for students in grades 2 and 4 (World Bank 2019d). Viet Nam has also recognized the need for further changes in preservice and in-service training in relation to the ethnic minority language of instruction. Improving Teaching and Instruction As recognized in World Bank analysis and operational support, the lack of an adequate number of well-qualified teachers in basic education is linked to a broad range of factors, such as funding, recruitment, monitoring, and mo- tivation.25 Case studies highlighted challenges in certain countries (among them Chad, Ethiopia, Kenya, and Sierra Leone) in recruiting and retaining teachers in rural, remote, or conflict-affected areas. Recruitment of teachers was directly addressed in World Bank projects in Brazil, Chad (regularizing the recruitment of inadequately qualified community teachers), Nepal, and Pakistan (recruitment of more women teachers). Retention was addressed in certain World Bank projects through the provision of support for incentives, such as allowances and accommodation, to try to persuade teachers to move to and stay in certain, usually rural, locations. However, a lesson drawn from completion reports is the lack of sustainability once incentives conclude. In countries with growing demographics, the challenge of ensuring quality teaching and education will increase, given the need for qualified teachers, particularly in Africa (box 3.5; Bashir et al. 2018).26 The World Bank’s primary response to the challenges associated with quality Independent Evaluation Group World Bank Group    61 teaching is the provision of support for on-the-job training. In-service train- ing can be an important input in basic education systems, particularly where a cadre of well-qualified teachers exists and where the training is designed to complement and build on existing knowledge, expertise, and competence. Literature highlights the importance of professional development when teachers are given follow-up support to be able to practice and use the knowl- edge and skills that result in changes to the learning experience of students (Evans and Popova 2016). Follow-up support as a single activity to comple- ment the training was found in 40 operations, whereas support through a continuous mechanism offering regular supervision or mentoring was iden- tified in 70 out of 188 operations with on-the-job training. This is a design improvement from what IEG identified (World Bank 2019c); however, few of these operations assess or monitor the fidelity of follow-up support (38 out of 110), as is the case with the Enhancing Teacher Effectiveness in Bihar project in India, which uses school-based assessments to judge the quality of teach- ing and learning through classroom observation and lesson planning. When effective training is delivered, it can improve student test scores (McEwan 2015), particularly for girls (Aslam et al. 2016). The World Bank’s measurement of changes in teaching because of training supported by its operations is limited. A review of key and intermediate per- formance indicators across operations containing teacher training identified output measurements related to the number of teachers trained or the num- ber of teachers trained in particular types of pedagogy (such as psychosocial, reading, mathematics, and gender-sensitive) or the inclusion of particular content in training modules (such as gender-sensitive behavior and climate awareness training). Twenty-two operations (out of 188 with on-the-job training) systematically tracked the impact of training on teachers’ practices; participation in training was monitored in the remaining operations. The as- sessment and evaluation of training is particularly needed because IEG found few operations with the combination of effective characteristics to realistical- ly deliver enhanced learning outcomes (World Bank 2019c). Thus, the World Bank is missing a critical feedback loop to learn from operations and ensure that training is in fact improving the capabilities of the stock of teachers and resulting in better learning outcomes. The World Bank has placed a relatively limited emphasis on preservice training or career framework development compared with in-service training Confronting the Learning Crisis Chapter 3 provision. Preservice training is cited as a planned activity or subcomponent in 91 (39 percent) of all basic education projects approved during FY12–22, whereas on-the-job training is a planned activity in 80 percent of projects. The evaluation confirmed the finding of IEG’s Selected Drivers of Education Quality: Pre- and In-Service Teacher Training that support for in-service train- ing represented a partial response to the much deeper challenge and that the World Bank rarely engaged with the more fundamental and more difficult development challenge associated with the comprehensive strengthen- ing of preservice training institutions (World Bank 2019c). The majority (60 percent) of the 91 projects that cite preservice training are in the Africa Region, where the learning crisis is most acute and where teaching standards 62 are most challenged. When asked about the lack of attention to preservice training, World Bank staff recognized the gap and suggested that the lack of engagement might be associated with the scale and cost of the challenge or the fact that teacher training institutions may be highly political. In many client countries, including the case countries, there is a shortage of qualified teachers, and a sizable proportion of qualified teachers have low levels of education and skill, making it unlikely that preservice institutes can produce quality teachers with the required level of skills. This suggests a need for the World Bank to also address the flow of teachers who enter the system with minimum requirements.27  ase Study Countries Facing Large Challenges from Population Box 3.5. C Growth The analysis of demographic trends in the Independent Evaluation Group’s 10 case study countries using actual (2000–21) and projected (2022–40) population data of 5- to 14-year-olds from the United Nations population data portal shows significant population growth in countries such as Ethiopia and Pakistan and population decline in countries such as Nepal and Viet Nam. Chad is in its own category; its overall projected population growth rate is over 3 percent—the eighth fastest growth rate in the world (the top 20 are in Africa). In raw terms, its population of 5- to 14-year-olds will grow from 2.5 million to 7.5 million in 2000–40. A second set of countries, including Independent Evaluation Group World Bank Group    63 Ethiopia, Iraq, Kenya, Pakistan, Sierra Leone, and Tajikistan, will also experience growth. In this group, Ethiopia, Iraq, and Sierra Leone stand out, with about 200 percent growth in this period (2000–40). Kenya, Pakistan, and Tajikistan are at a lower positive rate, with populations that will increase by 50–75 percent. Interviewees strongly emphasized the scale of the challenge associated with this growth, even without factoring in climate change impacts, fragility, and vulnerability. Source: Independent Evaluation Group analysis of data from the United Nations population data portal. Support for Measurement of Learning During FY12–22, the World Bank has developed a large volume of GPG and capacity-building programs financed by trust funds to support learn- ing measurement.28 Among those programs was an effort to improve client assessment capacity using the READ trust fund in eight countries. In more recent efforts, with READ 2 and the Learning Assessment Platform, the focus has shifted to measuring foundational learning in eight countries (Armenia, Cambodia, India, the Kyrgyz Republic, Mongolia, Nepal, Tajikistan, and Viet Nam), which received support to build capacity in learning assessment, data analysis, and use of data in decision-making. READ 2 also measured indica- tors of client capacity. Box 3.6 describes the use of structured pedagogical approaches in case studies, which are evidence-based interventions to im- prove foundational learning.  hat Are Structured Pedagogical Approaches? Box 3.6. W A recent review by the United Nations Children’s Fund defines structured pedagogy as “a systemic change in educational content and methods, delivered through comprehensive, coordinated [programs] that focus on teaching and learning, with the objective of changing classroom practices to ensure that every child learns” (Chakera, Haffner, and Harrop 2020, 5). The Global Education Evidence Advisory Panel (which is co-hosted by the UK Foreign, Commonwealth & Development Office; the United Nations Children’s Fund Office of Confronting the Learning Crisis Chapter 3 Research–Innocenti; the United States Agency for International Development; and the World Bank) rated structured pedagogy as a “good buy” in its comprehensive review of cost-effective interventions in education. The review cites the potential for “step- by-step” lesson guides that are part of a “multifaceted instructional program” to help improve teaching, especially in contexts where teachers have low levels of capacity. These approaches were used in 5 out of 10 cases, with planned use in another case as part of an upcoming project. Some followed a specific pedagogical approach, such as Teaching at the Right Level, whereas others were customized to the context, including student and teacher materials, teaching training, and follow-up support. (continued) 64  hat Are Structured Pedagogical Approaches? (cont.) Box 3.6. W Some partners were supportive of increasing the use of structured pedagogical approaches by the World Bank to improve foundational learning. There are gains from baseline; however, children, after participating in some of the programs, can remain below the internationally recognized minimum basic levels of literacy. The evidence also highlights the need to ensure that structured pedagogical programs are fully supported and aligned in the education system and context because successful programs incorporated materials that were tailored to their specific context, including delivery in the children’s native language instead of the national language. Less successful structured pedagogy programs may also have been unable to overcome substantial teacher capacity limitations and a lack of resources and other implementation weaknesses (He, Linden, and MacLeod 2007; Kerwin and Thornton 2016; Lucas et al. 2014). Thus, planning for the use of structured pedagogical programs requires understanding of the weaknesses of the system and the reasons they exist in that context. Sources: Independent Evaluation Group case studies and interviews; Chakera, Haffner, and Harrop 2020; Crouch 2020; He, Linden, and MacLeod 2007; Kerwin and Thornton 2016; Lucas et al. 2014; Piper and Dubeck 2021. Nearly two-thirds of portfolio operations supported learning assessment ac- tivities. All leading education systems routinely measure learning because it Independent Evaluation Group World Bank Group    65 is not possible to know that improvement is occurring without regular mon- itoring of student learning data from a national sample across various grade levels. Projects containing learning assessment activities predominantly supported assessment capacity building (80 percent of operations), assess- ments of various types (75 percent), and dissemination and use of learning data (50 percent). Thirteen percent of these operations used learning surveys, which measure learning during the operation but do not constitute a regu- lar system of learning assessment. Nearly all projects in the Middle East and North Africa since 2017 have included assessments, making it the Region with the most assessment-related activities. World Bank financing predominantly supported countries’ own national assessments (in 44 countries) and subnational assessments (in 6 countries). Other types of assessment, such as early-grade reading or mathematics assessments (in 19 countries), international or regional assessments (in 16 countries), or classroom assessments (in 15 countries), were less fre- quently supported. Assessments predominantly covered grades 6 and lower (92 percent of operations supporting assessments) and assessed reading (98 percent) and mathematics (84 percent). More countries financed nation- al assessments and international or regional assessments during the second half of the evaluation period, but the use of early-grade reading or mathe- matics assessments decreased during the second half of the period. Nearly half of these operations have a regular frequency of administration of the assessment, such as every two or three years (47 percent); the remainder are administered twice during the operation (35 percent) or once during the project (23 percent). A national sample is used when national assessments are supported because it is important to know what children across the country have learned. The countries with repeated assessment support continue to support national assessment and have also included plans for participation in international or regional assessment (Angola, the Dominican Republic, Ethiopia, India, Kosovo, Nicaragua, Pakistan, Peru, Rwanda, Moldova, São Tomé and Príncipe, and Sri Lanka), showing government commitment to measure learning with accessible and comparable cross-national data. Learning outcome data remain scarce in low- and middle-income countries, making it possible to partially assess minimum proficiency levels in 4 out of 10 case countries only (table 3.3). Out of 91 portfolio countries, 47 reported data on the proportion of students at the end of primary education achieving at least a minimum proficiency level in reading, whereas only 27 countries Confronting the Learning Crisis Chapter 3 have such data in reading for lower secondary. All the case study countries have some kind of national assessment data, but the extent to which they have a systematic learning assessment function in place varies. In Ethiopia and Kenya, data are dated or not accessible to the public. At one extreme of the assessment function, in Sierra Leone, examinations have been used to determine progression from one level of education to another, but there are limited mechanisms to assess learning as a stepping stone toward identifying areas for curricular modification, teacher training, or overall system im- provement. Brazil stands at the other extreme with a highly evolved national monitoring system augmented with individual state (and even municipali- 66 ty) data based on large-sample assessment. Only 3 of the 10 case countries have data from large-scale international assessments—Brazil (Program for International Student Assessment and the Progress in International Reading Literacy Study), Pakistan (Trends in International Mathematics and Science Study), and Viet Nam (Program for International Student Assessment). Some case study countries also participated in regional assessments, including Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación (Latin America, only Brazil); the Southern and Eastern Africa Consortium for Monitoring Educational Quality (Africa, only Kenya); the Programme for the Analysis of Education Systems (French-speaking Africa, only Chad); and the Southeast Asia Primary Learning Metrics (Southeast Asia, Viet Nam). The World Bank is supporting Iraq, Kenya, and Tajikistan with participation in international assessments in 2023. The Early Grade Reading Assessment, sup- ported by the United States Agency for International Development, and the GPE have provided early reading data in nearly all case study countries. The limited progress over the decade in establishing learning outcome measures is illustrated in table 3.3, including the need to improve learning.  inimum Learning Proficiency in Reading and Mathematics Table 3.3. M at the End of Primary and Lower Secondary Education in Case Study Countries (percent) Mathematics Mathematics Reading at the at the End Reading at the at the End End of Lower of Lower Independent Evaluation Group World Bank Group    67 End of Primary of Primary Secondary Secondary Country Education Education Education Education Brazil 44 29 50 32 Chad 8 2 — — Ethiopia — — — — Iraq — — — — Kenya 47 74 — — Nepal — — — — Pakistan — 8 — — Sierra Leone — — — — (continued) Mathematics Mathematics Reading at the at the End Reading at the at the End End of Lower of Lower End of Primary of Primary Secondary Secondary Country Education Education Education Education Tajikistan — — — — Viet Nam 82 91 90 84 Source: Independent Evaluation Group calculation of United Nations Educational, Scientific, and Cultural Organization Institute for Statistics data from September 2023. Note: Large-scale international assessment programs postponed administration and reporting of assess- ments for the years 2020–2022, which has affected coverage. — = not available. Monitoring and Evaluation Monitoring and evaluation of improved learning outcomes is specified in 33 percent (n = 77) of project development objectives (PDOs) in the basic education portfolio. The focus on improved learning outcomes in PDOs is higher in Africa (43 percent of projects). Improving the quality of education is an objective in 62 percent of operations that typically focus on improvements to the learning environment (such as enhanced infrastructure, textbooks, or teachers trained)—often prerequisite conditions to improve learning. Out of the 77 projects with PDOs that address improving learning outcomes, 48 have outcome indicators measuring learning outcomes. Among the 48 proj- ects, 81 percent focus on primary grades, measured predominantly in reading (83 percent) compared with mathematics (58 percent). These indicators are Confronting the Learning Crisis Chapter 3 drawn from national (31 percent), subnational (25 percent), and other assess- ments (such as early-grade reading or mathematics assessments). The Middle East and North Africa is the only Region where all projects have contained indicators on learning outcomes at the output or outcome levels since 2017. Analysis of IEG ratings shows that operations with learning indicators receive lower ratings than operations with less ambitious objectives, which is signifi- cant only at the 0.1 level. Indicators designed to capture learning outcomes are not typically included in CPFs. For example, indicators related to basic education were included in CPFs for 7 out of 10 case countries. However, as set out in table 3.4, these 68 indicators tend to be general and rarely related to learning outcomes; indica- tors are provided to capture learning outcomes in 2 out of 10 case countries, Ethiopia and Iraq. As with the measurement of project effectiveness and success, the measurement of enhanced learning outcomes at country program level is limited. The implementation of World Bank EMIS interventions is challenged by inef- ficiencies in country systems. The World Bank has provided combinations of support that include interventions to promote the development of an EMIS, tablet-based data collection that feeds into annual censuses, technical assis- tance to support capacity development, and others. However, the efficacy of any input is contingent on many facets of the system being equally developed in parallel. Significant gaps continue to exist in that regard. For example, most country cases found an absence of a culture that supports monitoring and evaluation and found gaps in the human resource capacity required at all levels of the system—from central to local—to ensure an effective data man- agement system. The evaluation also identified the need for “joined-up” data management across the system (despite investment in EMIS), the need for greater and more consistent disaggregation of data for marginalized groups, and quantification of learning losses associated with the COVID-19 pandemic.  asic Education Quality–Related Indicators in Latest Country Table 3.4. B Partnership Frameworks in Case Study Countries Independent Evaluation Group World Bank Group    69 Country Latest CPF Indicator(s) Brazil FY18–23 Objective 1.3—increased effectiveness of service delivery in education—has the following quality-related indicator: reduction of dropout rate in the first year of public secondary schools (baseline [2016]: 9.8%; target [2022]: 7.0%). Chad FY16–20 Objective 3.2—improved access to and quality of education—has the following quality-related indicators, all of which have baseline and target values. Indicator 3.2.2: Additional classrooms built or rehabilitated at the primary level (number). Indicator 3.2.3: Document resource centers created and equipped (number). Indicator 3.2.4: Additional qualified community teachers (number). (continued) Country Latest CPF Indicator(s) Ethiopia FY18–22 Objective 2.4—improved basic education learning outcomes—has the following learning outcome indicator (with baseline and target values defined): students scoring at “below basic proficiency” in English and mathematics (NLA subject scores; percentage). Iraq FY22–26 Objective 2.2—improved education quality, better skills, and economic opportunities for youth and women—has the following quality learning outcome indicators (with baseline and target values defined). Indicator 2.2.1: Percentage of early-grade students who can read with comprehension (by gender). Indicator 2.2.2: Percentage of students completing primary education; percentage of students completing secondary education (by gender and quintile). Kenya Focus on youth skills development. Nepal FY19–23 Objective 3.1—improved equity in access to quality education—has the following quality-related indicator with baseline and target values provided: retention rate of poor students to grade 12 in community schools in selected districts. Pakistan FY15–20 Under Country Partnership Strategy outcome 4.3 (increased school enrollment and adoption of education quality as- sessment), the indicator (with no baseline or target defined) is as follows: annual student achievement tests for grades 5 and 8 implemented in at least three provinces; it showed a positive trend in learning outcomes. Sierra Leone FY21–26 Objective 2.1—improved quality of education and skills development—has the following indicators defined by baseline and target values. Indicator 5: Learning-adjusted years of schooling (by year, gender, and North or South dis- tricts). Indicator 6: Female students who completed junior Confronting the Learning Crisis Chapter 3 secondary school (percent, by North or South districts). Tajikistan FY19–23 Focus on early childhood education at CPF level. Viet Nam FY18–22 Focus on tertiary-level education at CPF level. Source: Independent Evaluation Group. Note: CPF = Country Partnership Framework; FY = fiscal year; NLA = national learning assessment. The World Bank has provided significant support for the development of EMIS, but the systems installed often do not serve their purpose as a result of inattention to interdependencies between various technical components of the system and to limitations in human capacity. The evaluation documented 70 this phenomenon in Chad, Ethiopia, Iraq, Kenya, and Sierra Leone, where EMIS, despite investment over time, is not comprehensive, is not populated with joined-up data, and is not a stable and reliable source for evidence-based policy making. EMIS often operate at a basic level, counting numbers of schools, teachers, and students with little disaggregation of student data and limited linking with other system data or data on learning. This reflects perspectives shared by World Bank staff, who said that EMIS had often been developed to support monitoring of World Bank projects without being more deeply embedded in country systems. The COVID-19 experience also high- lighted what Muñoz-Najar et al. (2021) refer to as the “digital chasm”—a lack of connectivity and equipment to support an effective system. In some contexts, the World Bank has successfully harnessed information and commu- nication technology at the local level to support data collection. For example, the World Bank recently supported the introduction of tablets in Chad and Sierra Leone that were used to gather and transfer more accurate and timely data that are contributing, for example, to the production of better-quality school censuses, replacing paper-based systems. Box 3.7 provides examples of data-related work supported by the World Bank in the four case study coun- tries in Africa.  xamples of World Bank Support for Data in Four Case Study Box 3.7. E Countries in the Africa Region In Chad, in the early years of the evaluation period, the World Bank supported the Independent Evaluation Group World Bank Group    71 development of an education management information system (EMIS) that under- performed and was improved under later projects. According to the Implementation Completion and Results Report for the second of three projects that supported the development of the EMIS, the system is currently being used at both the central and regional levels to systematically gather and input accurate data, although, as in other cases, the level of overall alignment in the system remains incomplete. In Ethiopia, the World Bank designed financing and disbursement-linked indicator targets to improve the timeliness and reliability of school census data and technical (continued)  xamples of World Bank Support for Data in Four Case Study Box 3.7. E Countries in the Africa Region (cont.) assistance on school inspection data. It also supported improved methods and capacity in the testing agency via Russia Education Aid for Development trust fund South-South exchange. Financial support also aided the development of the EMIS and tablet-based data collection for secondary schools, but not yet at primary level. In Kenya, the current Country Partnership Framework expands support to include tablets for teachers for improved teacher skills, performance feedback, and de- ployment. Earlier in the evaluation period, a project sought to address inadequate systematic data to inform planning and implementation through regular collection and dissemination of education statistics and student performance data—collection and publication of EMIS data plus dissemination of student performance. One current project is drawing on the now-established EMIS and is also supporting the Kenya National Examinations Council with data collection, processing, and analysis for the implementation of learning assessments. Another is supporting data management, including categorization (in EMIS) of learners with special needs by nature of disability and gender, and registration of refugee learners and learners in host communities. In Sierra Leone, World Bank support helped introduce tablet-based data collection that provides output to support school-level improvement and that feeds into the annual school census. The World Bank also worked with the Directorate of Policy Planning and Project Development to progressively enhance data collection, verifica- tion, and analysis. In addition, the support assisted in setting up and operationalizing a semiautonomous assessment unit in the Ministry of Basic and Senior Secondary Confronting the Learning Crisis Chapter 3 Education to coordinate design and execution of assessments at lower and upper primary and junior secondary schools. Source: Independent Evaluation Group. 72 World Bank Engagement with Development Partners and Other Stakeholders at the Country Level Working in partnership is essential—the World Bank cannot address the sys- temic challenges and solutions in client countries on its own. The World Bank works with development partners through partnerships and related structures that support basic education. Evidence from case studies suggests that World Bank staff are highly regarded and are perceived to engage diplomatically and transparently with government representatives and other development partners. Partnerships typically involve information exchange and managing coordination regarding the location, timing, and targeting of interventions. Those efforts can involve the education sector overall, either geographically, within the sector (at preprimary, primary, and secondary levels; technical and vocational education and training; or tertiary level), or with reference to target groups. Structures and mechanisms are in place to facilitate partner engage- ment in all countries, including local education groups in countries where the GPE is active. Broadly speaking, stakeholders, including World Bank personnel, are of the view that the GPE has helped reinforce the World Bank’s agenda. Case stud- ies found that the GPE has helped shape relationships between development partners through which they can stay connected with one another and with Independent Evaluation Group World Bank Group    73 local civil society organizations. For example, in Sierra Leone, the GPE has helped bring a more systematic approach to the education sector and has done so without antagonizing key players—a function of the time and effort invest- ed in building ownership of plans. Conversely, in Tajikistan, the relationship between the World Bank and the local education group was found less pro- ductive (in that case, the World Bank was not acting as a grant agent for GPE funding). In Pakistan, the GPE and the World Bank supported the development of education sector plans for Balochistan and Punjab Province, aligned with the National Education Policy Framework. There were also some qualifications regarding the relationship between the World Bank and the GPE (box 3.8). Despite the broadly positive relationships between the World Bank and other development partners active in basic education, the evaluation found limited evidence to suggest that partnership arrangements at the country level are collaborative and strategically focused. Despite forums and mechanisms to facilitate partner engagement, the relationship among development partners is typically characterized by information sharing and coordination, with little emphasis on deeper collaborative approaches designed to achieve common outcomes.29 In almost all instances, the World Bank and other development partners consult with one another on strategy and share information about plans, often seeking to ensure that they are not, as they see it, duplicating effort. In some instances, close coordination and cooperation is evident. For example, in Ethiopia and Nepal, partners operate under the umbrella of a sectorwide approach arrangement in support of government policy. Partners can opt to support various aspects of policy and do so with the fiduciary secu- rity provided by a World Bank–operated sectorwide approach. However, the evaluation found very limited evidence of common analysis and shared un- derstanding among development partners of why learning poverty persists. As per the evaluation framework, a starting point for system orientation toward enhanced learning is clarity of intent based on comprehensive analysis. The absence of such analysis and agreement inhibits deeper levels of collaboration in pursuit of common goals.  lobal Partnership for Education Added Value for the World Box 3.8. G Bank at the Country Level Confronting the Learning Crisis Chapter 3 Global Partnership for Education (GPE) funding has helped reinforce World Bank country strategy ambitions. For example, in the seven case study countries where the World Bank functioned as implementer for GPE operations, the evidence suggests that there is broad strategic alignment between the World Bank and the GPE regard- ing the importance of basic education in development. In those cases, GPE financing that is “shaped” by associated projects tends to reinforce World Bank objectives and planning for the basic education sector. The evaluation found that the working re- lationship between the World Bank and the GPE is positive and adds value to the development of basic education—for example, promoting interaction between devel- opment partners, provision of funding, and support for sector planning. (continued) 74  lobal Partnership for Education Added Value for the World Box 3.8. G Bank at the Country Level (cont.) The case studies also offered some qualifications regarding the relationship between the World Bank and the GPE. The formalized interaction promoted by the GPE has not typically supported higher levels of collaboration among development partners. World Bank task team leaders acknowledged that the GPE helped form important coordi- nating arrangements for the education sector, but collaboration is not intrinsic to these arrangements—”the GPE agenda has meant a lot of partner coordination in the field but not real collaboration.” In Ethiopia, stakeholders were broadly positive about the role of the GPE in supporting the development of the government’s sector plan and for piloting certain actions using additional financing. However, the case study also found that the release of GPE funding to the government at the end of a fiscal year coincided with a period during which a World Bank project was not performing well and staff were in discussions with the government about the merits of meeting agreed targets. Source: Independent Evaluation Group. The evaluation found various types of barriers to more strategic collaboration and interaction with development partners and other stakeholders. Through case studies and interviews, the evaluation identified barriers to World Bank engagement with its natural allies—other development partners—including Independent Evaluation Group World Bank Group    75 different planning cycles and reporting requirements, different emphasis in the education models promoted, and some development partners working in parallel to government systems and others working with government. World Bank staff also cite time pressures and resource constraints—“collaboration takes time”—and indicate that the level of sectoral coherence among devel- opment partners is contingent on the relative strength of the government’s capacity. In Pakistan, there is consensus among stakeholders that there is significant room for improvement in donor harmonization and, from a World Bank perspective, for greater on-the-ground presence. Some TTLs who were interviewed stated that they were too busy running projects to have time to put into building collaboration among partners. In Brazil, Kenya, and Sierra Leone, the World Bank was found to have limited engagement with non- governmental stakeholders, which can be pivotal in the implementation of learning-centered reforms. These findings resonate with the conclusions from a 2021 Organisation for Economic Co-operation and Development working paper, which reported that greater levels of alignment among development partners (in that case to SDG indicators) are challenged by organization- al issues such as “weaknesses in partners’ own results-based management systems; limited focus on the SDGs in coordination mechanisms; lack of synchronization with country-level planning; piecemeal approaches in SDG alignment; or, simply, a lack of demand” (Guerrero-Ruiz, Sachin, and Schnatz 2021, 10). Development partners and local actors interviewed for case stud- ies also noted that the World Bank is unlikely to independently engage with stakeholders that governments do not favor or sanction. Subject to context, this may include, for example, teachers’ unions and certain civil society orga- nizations and nongovernmental organizations. Confronting the Learning Crisis Chapter 3 76 1 Governments are the largest funders of education in all country types, but the relative contribution from households to education spending tends to be greater in poorer countries: in 2018–19, household contribution in high-income countries was 16 percent of the total, compared with 38 percent in low-income countries. 2  The United Nations Educational, Scientific, and Cultural Organization calculations found that achieving these national Sustainable Development Goal 4 targets by 2030 would still involve rapid cost increases, which even optimistic assumptions of domestic revenue mobilization cannot match. We estimate that there will be an annual financing gap of $97 billion on average in the 79 low- and lower-middle-income countries between 2023 and 2030, or 21 percent of the total cost (Murakami 2023). 3  The World Bank is also the largest implementing agency for Global Partnership for Education grants for low-income countries, managing $3.6 billion (57 percent) of the Global Partnership for Education grant portfolio. The Global Partnership for Education is the largest global fund and multistakeholder partnership dedicated to improving education in lower-income and conflict-affected countries. For comparison, the largest shares of overall lending in fiscal year 2021 went to social protection and public administration (both 17 percent), and the smallest share went to information and communication technology (3 percent). 4  The assessment of investment project financing and Programs-for-Results is complicated by the small number of Programs-for-Results (n = 29) compared with investment project financings, which constitute the majority of interventions, and many of the latter use disbursement-linked indicators. Results in Education for All Children studies also noted Independent Evaluation Group World Bank Group    77 that the context specificity and intervention specificity of results-based financing actually incentivize results. 5  The World Bank provided cross-sectoral support for basic education with 27 projects (11 percent) tagged to sectors other than education. Cross-sectoral support provided inputs such as incentive-based conditional cash transfers and school feeding, including support for community-based participation and consultation for school building projects. Cross-sectoral support for basic education was provided in 8 out of 10 case study countries, primarily through investment project financing, with limited use of development policy financing (3 countries) and Program-for-Results (2 countries). Sectors that provided cross-sector support for basic education were Human Development (Health, Nutrition, and Population; Social Protection and Jobs), Sustainable Development (Urban, Disaster Risk Management, Resilience, and Land), and Equitable Growth, Finance, and Institutions (Governance). Nepal, where cross-sectoral support was used for basic education four times during the evaluation period, provides an example of ways that such support was used: » Nutrition Policy Dialogue II (P127709) technical assistance helped the government implement multisectoral approaches to address chronic malnutrition, including activities identified in the government’s multisectoral action plan for nutrition. » Mapping Local Service Delivery (P147730) technical assistance provided a stronger analytic basis for improvements to the institutional framework for delivering public services at the local level. The final report includes a chapter on subnational service delivery in the basic education sector. » Policy Notes for the New Government (P147471) economic and sector work looked to inform the new government’s development strategy and stimulate public debate on policy priorities and short-term opportunities. The chapter “Improving Opportunities for All to Escape from Poverty” discussed challenges in the quality of education and addressed the importance of monitoring learning outcomes. » Pilot Project for Seismic School Safety in Kathmandu (P129177), an investment project financing project, had the objective to reduce Nepal’s risk from earthquakes, especially the vulnerabilities of public schools, by strengthening the government’s capacity to implement school retrofitting programs. 6  A project is classified as COVID-19 relevant if it meets any of the following criteria: (i) it has the “COVID” emergency response code; (ii) it has at least one of the COVID-19 crisis response codes; (iii) it has the pandemic response theme code; or (iv) its project name, project development objective, or components contain the words “Covid” or “Corona.” The project does not have to be approved since fiscal year 2020 because projects could have had an emergency response code or COVID-19 crisis response code assigned during implementation. Confronting the Learning Crisis Chapter 3 The project development objective and components could also have been revised during restructuring. 7  The World Bank supports education projects in sectors other than the basic education subsector in 12 countries, 2 of which are in the Africa Region. The World Bank has no education projects whatsoever in 52 client countries, 7 of which are in the Africa Region. 8  Eleven countries had no basic education operation but had one COVID-19 response operation. 9  Among the 22 countries with two projects, 8 have at least one COVID-19–relevant project. 78 10  Angola, the Central African Republic, Côte d’Ivoire, Guinea-Bissau, Madagascar, Malawi, Mali, Mauritania, Niger, and Zambia. 11  Bangladesh, Djibouti, Ethiopia, The Gambia, Guyana, Haiti, India, Kenya, the Lao People’s Democratic Republic, Nicaragua, Pakistan, and Sierra Leone. 12  There are outliers, such as Burundi, with one operation and a learning poverty rate of 93 percent. Conversely, there are four operations each in Viet Nam and Moldova, although their respective learning poverty rates are less than 2 percent and 11 percent. 13  The evaluation team examined two periods during the evaluation, 2012–17 and 2018–22, to look for any change in emphasis since the publication of the World Development Report 2018: Learning to Realize Education’s Promise, which emphasized the need to take account of the extent to which political forces and technical complexities constantly pull education systems out of alignment with learning (World Bank 2018b). 14  A recent review of the success of education reform in Viet Nam noted “the government’s strong commitment to educational development, supported by high accountability mechanisms; relatively high public spending with a focus on investing in general education, basic inputs, and equity, together with high household investment in education; attracting and supporting qualified teachers; strong investment in preschool education; and strategic use of assessments” (Kataoka et al. 2020, 1). 15  Largely attributed by all stakeholders to the work of a task team leader who is a Kenyan national and who has a very strong understanding of context and culture. On a less positive note, there was no country-specific advisory services and analytics in Independent Evaluation Group World Bank Group    79 16  support of basic education, which, interviewees claimed, could have laid the foundation for further influence in relation to learning for all. For example, advisory services and analytics could have helped the World Bank better support dialogue on equity-related issues concerning girls, disability, refugees, and host communities. 17  A similar pattern of supported interventions is apparent in the analysis of characteristics such as time period (before and after the World Development Report), Region, or country classification (middle income or low income). Among the nine cases where the World Bank provided financing, a similar pattern of interventions was also observed. 18  It is not possible to determine the level of support provided under any of these or other headings as measured in US dollars. The data provided represent the incidence of proposed support within the portfolio only, rather than what was actually executed or its cost. 19  Our analysis suggests that the types of input likely to be supported by the World Bank remained consistent across the evaluation period and across the primary instrument types— that is, investment project financing and Program-for-Results. The Program-for-Results instrument provided greater fiduciary cover rather than greater scope in what is being supported by the World Bank. 20  Interestingly, with reference to this evaluation’s emphasis on a systems-based analysis that is inclusive of a broad base of stakeholders, the Country Partnership Framework for Kenya benefited from stakeholder consultations with approximately 500 participants, who included representatives from Indigenous and historically underserved communities, youth, industry civil society organizations, start-up innovators, development partners, private sector leaders, media outlets, the National Assembly, the Senate, the Council of Governors, and the technical teams of the main candidates in the 2022 elections. 21  The Organisation for Economic Co-operation and Development (2015) reports that Programme for International Student Assessment results have consistently shown that boys are more likely than girls to be overall low achievers—that is, they are more likely than girls to perform below the baseline level of proficiency in mathematics and science but to an even greater extent in reading. 22  More recent thinking suggests the need for a more nuanced approach to targeting and measurement to take account of learning challenges encountered by boys (Buitrago- Hernandez, Levin, and Rodríguez Castelán 2023). 23  Research indicates that using the native language in the classroom enhances classroom participation, decreases attrition, and increases the likelihood of family and community engagement in the child’s learning (UNICEF 2016). Confronting the Learning Crisis Chapter 3 24  Researchers have noted the issue of English language capacity among teachers (Bloor and Tamrat 1996; Negash 2006). 25  For example, United Nations Educational, Scientific, and Cultural Organization Institute for Statistics data show that less than 80 percent of primary teachers in Chad and Sierra Leone are qualified, and teacher qualification levels at lower secondary level are below this rate in Brazil, Chad, Ethiopia, Pakistan, and Sierra Leone, based on United Nations Educational, Scientific, and Cultural Organization Institute for Statistics Sustainable Development Goal tracking data. 80 26  Population growth across Africa, along with improved student progression through basic education, suggests that primary enrollment will increase by 50 percent by 2030. 27  The evaluation recognizes that attracting better candidates is often inhibited by the unattractiveness of teaching, and governments do not want to deter candidates when need and demand are growing (World Bank 2019c), as is the case of growing demographics in Sub-Saharan Africa. The 2019 evaluation found that World Bank operations tried to motivate trainees through support for scholarships and stipends to address scarcity rather than create a new mechanism to bring in candidates with stronger content knowledge. The intertwined challenges of teaching suggest a need for the World Bank to convince client countries to address these challenges in a balanced way, through a career framework and by better linking training with career opportunities (Popova et al. 2018), which is a feature within leading education systems. 28  Eight countries (Angola, Armenia, Ethiopia, the Kyrgyz Republic, Mozambique, Tajikistan, Viet Nam, and Zambia) were provided with technical assistance to establish or strengthen assessment systems or instruments. During the first phase of the Russia Education Aid for Development trust fund support, countries focused on topics of their choice, which did not necessarily include improving basic or foundational data or comparable disaggregated data. Some countries focused on basic education, such as Angola (Early Grade Reading Assessment) and Mozambique (grade 3 classroom assessment), whereas others focused on other levels beyond basic education, such as Tajikistan, which developed a unified university entrance examination. 29  We distinguish between activities such as communication, coordination, and cooperation, Independent Evaluation Group World Bank Group    81 all of which apply to actors working together in pursuit of their own goals, and collaboration as part of which actors work together toward shared goals. 4 | Conclusions and Recommendations The World Bank contributed to learning outcomes at the global, regional, and country levels between FY12 and FY22, doing so by convening stakeholders, raising awareness, increasing available data, and financing operations. These contributions have been broadly positive, developing partnerships and pooling financing to address the learning crisis, developing GPG, and reducing financing and capacity gaps in client countries. The World Bank also shaped global discourse to emphasize learning for all. The WDR 2018 renewed attention on the need to address political barriers and the need for more efforts in client countries to measure learning and identify system failures. The World Bank itself has adopted a strong stance on the importance of good-quality foundational learning, and the Education GP has increased its emphasis on the development of foundational skills: Foundational learning is exactly what it sounds like—the founda- tions of a child’s education. It refers to basic literacy, numeracy, and transferable skills, that are the building blocks for a life of learning. Just as we would not build a house without solid foundations, we cannot expect a child to thrive without solid foundational skills. (Herbert et al. 2021) Confronting the Learning Crisis Chapter 4 Contributions at the global level have included activities to build awareness and convene global stakeholders to build commitment to address low levels of learning. The World Bank’s high-quality data and analytics have addressed improvement in quality and learning and drawn attention to the learning crisis. With the Commitment to Action, launched at the United Nations Secretary-General’s September 2022 Transforming Education Summit, the World Bank and partners advanced support for foundational learning. Since then, partners of the Global Coalition for Foundational Learning have en- couraged more countries to sign the Commitment to Action.1 World Bank vice presidents have played a key role by encouraging ministers of education and finance in client countries to improve learning for all. For example, a 82 high-level meeting in Latin America in March 2023 convened many re- gional partners and was followed by collaboration with the Inter-American Dialogue, the Inter-American Development Bank, and other partners to raise awareness of the region’s learning crisis (World Bank 2023a). In addition, UNESCO’s Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación ensured regular data collection for calculating learning poverty through regional standardized tests (Colombia, Ministry of Education 2023). World Bank contributions at the global and regional levels have developed and shared knowledge. Findings from the WDR 2018 were presented at 100 dissemination events in 54 countries, particularly low- and middle-income countries. The WDR was the second-most-downloaded global report in World Bank history. Practice managers invited members of the WDR team to discussions with finance and education ministry officials, local civil society, and researchers to spur further political commitment. Valuable global and regional contributions have supported assessing levels of education policy development (SABER), producing comprehensive regional reports (Facing Forward: Schooling for Learning in Africa), supporting policy reform through dialogue and knowledge dissemination (Great Teachers: How to Raise Student Learning in Latin America and the Caribbean), and defining learning poverty and gaining buy-in from stakeholders. Contributions at the country level have sought to build awareness and shared knowledge and have addressed challenges associated with learning Independent Evaluation Group World Bank Group    83 for all through activities and interventions, as highlighted in the concep- tual framework. Although few country knowledge products relate to basic education, the World Bank and the GPE have nonetheless contributed to the development of national education strategies in many countries. Those strategies have increasingly focused on learning and have progressive con- tent about inclusion and equity, reflecting commitments under the SDGs. The extent to which strategic ambitions for equity and inclusion are im- plemented varies, however, subject to country factors, such as the political economy of education, available resources, and implementation capacity. The World Bank delivered interventions to education systems in 91 coun- tries during the evaluation period; however, few of these interventions translated into improved teaching, systems, or learning. In most cases, the most commonly supported inputs, such as government-level management, in-service teacher training, and school management, were provided with- out addressing the root causes of learning failure in the education system. Project measurement has emphasized output indicators and has not drawn on other evaluations to assess whether positive changes are occurring in measurement of learning, system capacity, and teaching consistent with intermediate outcomes in the conceptual framework. Without evidence that the supported interventions are effective and contributing to improvements in learning for all, the World Bank is missing important feedback, despite the emphasis on systems-based reforms emphasized in its strategy and in the WDR 2018. The contribution to learning outcomes could have been enhanced if the World Bank had adopted an outcome rather than output orientation at all levels. The World Bank’s focus on activities and outputs is evident in work at the global level, such as the absence of outcome orientation in the results framework for the Foundational Learning Compact trust fund.2 A theory of change for the trust fund would facilitate planning, imple- mentation, and monitoring of how country clients use global and regional analytics. This change is aligned with The Knowledge Compact for Action: Transforming Ideas into Development Impact (World Bank 2024b), which will require stronger monitoring to provide a feedback loop to the World Bank. The lack of outcome orientation at the country and project levels is evident where measurement, and success, is usually articulated in terms of outputs. Although that work is typically of high technical quality, its relevance to Confronting the Learning Crisis Chapter 4 the achievement of enhanced learning outcomes for all is limited because it is not designed with that outcome in mind. Specifically, at the country and project levels, the evolution toward an outcome orientation can be support- ed by a systems approach and through reforms in measurement of learning, teaching career framework, equity, and capacity. This will require enhanced national assessment capacity based on improved monitoring and evaluation functions. More detailed theories of change will need to define the path- ways from enhancements of preservice institutions, teacher recruitment, and teacher monitoring to intermediate outcomes and how those outcomes are to result in improved pedagogical practices in classrooms that increase 84 student learning. Measurement Learning poverty is an easily understood concept that has gained global stakeholder buy-in. The World Bank and multiple partners have supported ambitious targets to motivate global and country stakeholders toward collec- tive action and alignment on a single target and message. Reducing learning poverty was added to the World Bank’s corporate targets at the 2023 Annual Meetings of the World Bank Group and the International Monetary Fund, replacing the previous indicator of human capital (measured as students reached). The increased attention to outcomes is welcome; however, with- out further attention by the World Bank and partners to address the lack of underlying data (to calculate learning poverty), the indicator will be unable to fill its critical global and corporate monitoring function. Data required to calculate learning poverty are unavailable in some countries in all Regions, including many countries where learning poverty is likely extreme. Progress has been made in supporting systems in a few countries to measure learning, which will provide data to assess progress on SDG target 4.1. All leading education systems have a learning assessment function in place—a prerequisite to focusing on improvements in learning. The SABER Learning Assessment Platform and READ provide numerous tools, reports, and GPGs designed to improve global knowledge. In addition, READ and the Learning Assessment Platform provide technical assistance to a few countries to Independent Evaluation Group World Bank Group    85 improve student learning assessments. In recent years, this support has more strategically focused on measurement in primary grades. The World Bank supported 143 operations that financed learning assessment, learning surveys, capacity building, and dissemination activities. In most cases, the assessments covered grades 6 and lower (92 percent) and evaluated reading (98 percent) and mathematics (84 percent). Nevertheless, the World Bank needs to focus more consistently on learning outcomes at both the country and project levels. Of the 77 projects with PDOs that address improving learning outcomes, 48 have outcome indicators mea- suring learning outcomes. The reduced frequency in measurement of learning compared with outputs of the learning environment is due, in part, to internal incentives that do not encourage country teams or TTLs to set more ambi- tious objectives and indicators, which are more challenging to achieve than access-related objectives. Analysis of IEG ratings shows that operations with an objective to improve learning outcomes receive lower ratings than oper- ations without such an objective, which is significant only at the 0.1 level. The lack of adequate measurement data was a factor in lower project ratings (Bedasso and Sandefur 2024). The same authors note that TTLs may account for a significant portion of the variations in type of project activities, sug- gesting a shortcoming in the institutional incentives needed to ensure TTLs support the World Bank’s strategic aim—learning for all—and do not feel pressure to deliver uniformly high ratings. The measurement of enhanced learning outcomes at country program level is also limited, and the efficacy of EMIS remains weak in many client countries despite World Bank operational support. Due attention is not always paid to interactions and interdepen- dencies between technical and human components within basic education systems, which are necessary to support a joined-up, functioning model. In those instances, the EMIS assessment tool is not “internalized” and is there- fore not a stable and reliable source for evidence-based policy making. The measurement of progress in basic education and in tackling the learning crisis requires stronger measurement within the World Bank (beyond the new corporate indicator—learning poverty). Because the international commu- nity emphasizes the need for accountability from individual governments, it follows that the international community and the World Bank, as the primary external funder of education, also need to be accountable for learning out- comes and not just for outputs. Additional country program and project-level measurement is needed that tracks intermediate learning outcomes and Confronting the Learning Crisis Chapter 4 provides evidence that more children are learning. The minimum proficiency level can be used as a foundation for an outcome-oriented World Bank ap- proach in basic education. The planning scenarios for country programs and operations allow sufficient time to measure progress at third and sixth grades and track intermediate outcomes in education systems, including national assessment capacity and a teaching career framework. Equity The World Bank has enhanced its focus on equity-related matters central to ensuring learning for all. The quality of country-level analysis in the 86 World Bank improved during FY12–22. Documentation (PADs and CPFs, for example), which had mostly referred only to gender, is increasingly inclu- sive of groups such as children with disabilities, out-of-school children, and displaced persons. Still, the adequacy of activities supported cannot be quantified because, beyond gender, the quantity and quality of disaggregated monitoring and reporting are limited. For example, the availability of dis- aggregated data in projects ranges from 91 percent for projects addressing gender disparity to 30 percent for projects addressing disability or out-of- school children. Thus, there is progress in recognizing barriers and targeting activities among multiple groups, but there is not enough attention on mea- suring equity (beyond gender and girls). Global-level ASA document inequities in learning for various marginalized groups. Going forward, clients need support in and knowledge about the additional challenges faced by children with disabilities and the learning ad- aptations they may need for the delivery of education because this aspect has received modest attention. To maximize its contribution to addressing the learning crisis, the World Bank will need to lead the way in providing context- specific evidence and ensuring that equity is fully built into education system planning, implementation, and monitoring. Support for Teaching Career Framework World Bank support for teachers and teaching emphasized on-the-job Independent Evaluation Group World Bank Group    87 training (80 percent of projects), with limited monitoring of the training or follow-up support, despite the emphasis on a comprehensive approach in analytic work. Such training can be valuable where a cadre of well-qualified teachers exists and where the training is designed to complement and build on existing knowledge, expertise, and competence (for example, to train teachers on new methodologies or on the rudiments of curricular change). However, only 40 out of 188 operations with on-the-job training included follow-up support as a discrete activity, and 70 included a continuous mechanism offering regular supervision. This is a design improvement from what IEG previously identified (World Bank 2019c); however, few of these operations will assess the follow-up support (38 out of the combined 110 operations). The inadequacy of monitoring of teacher training in operations and ASA means that there is a lack of feedback on the efficacy of interventions. Twenty-two operations (out of 188 with on-the-job training) systematically tracked the impact of the training on teachers’ practices; participation in the training was monitored in the remaining operations. Given the high propor- tion of projects supporting training, the World Bank is missing an important opportunity to learn from operations and ensure that training is in fact im- proving the capabilities of teachers and resulting in better learning outcomes. There is also no evidence yet that Teach and Coach are effective in creating and sustaining improvements in teaching practices and student learning—evidence that country clients will need to continue to pursue such approaches in the absence of trust funds. A more balanced approach to support for a teaching career framework is needed, consistent with the World Bank’s ASA developed over the decade. In many countries where the World Bank supports basic education, includ- ing those most affected by the learning crisis, the quality of initial teacher training is inadequate, qualified teachers are in short supply, the quality of teaching is inadequate, and motivation is an issue. Preservice training was a planned activity or subcomponent in 91 (39 percent) of all basic education projects approved during FY12–22. However, placing so much emphasis on one dimension of quality teaching—on-the-job training (a stock issue)—without significant attention to addressing the quality of the flow of teachers into basic education systems, is inefficient and misaligned with the World Bank ASA over the decade. The World Bank response needs to address the intertwined chal- lenges and political economy barriers to teaching quality in a balanced way by also addressing initial training and through a career framework and better Confronting the Learning Crisis Chapter 4 linking of training with career opportunities (Popova et al. 2018), which is a feature of leading education systems. Support for Capacity Development across All Levels of the System World Bank support for capacity development is focused on the central level, with less emphasis on capacity and delivery throughout the education system. Case studies found that the relationship with central govern- ment—ministries and key agencies (such as those involved in curriculum or assessment)—is a comparative advantage for the World Bank, allowing access 88 to policy makers and influencing the broad trajectory of education policy. World Bank support for dedicated capacity building also tends to focus on the central level. Capacity building at lower levels of the system often supports the delivery of World Bank projects rather than the efficacy of the system itself—that is, the capacity of the local government or administration involved in education delivery. This can result in such interventions as the development of EMIS without ensuring capacity within the system to maximize its use to inform policy. IEG recognizes that the scale of the task—building systemwide capacity—is significant, but it is also necessary for longer-term development. IEG also recognizes that the World Bank cannot do the task alone, but it can, working with government, other stakeholders, and development partners, prioritize it to better support implementation fidelity and overall efficacy of delivery of education and enhanced learning outcomes. Looking Ahead: Contributions to Learning To address the learning crisis beyond 2030, the overall approach to basic education will require changes. Governments and other stakeholders, in- cluding donors, will have to prioritize basic education, guarantee access to quality learning opportunities, measure progress, and implement strategies to ensure that individuals acquire the skills they need to lead fulfilling lives and contribute to the prosperity of their communities and economies. The analysis by Azevedo et al. (2021) concludes that, at historically observed rates of progress, the goal of ensuring that all children can read by 2030 will not Independent Evaluation Group World Bank Group    89 be reached3—an early warning that remaining on the current path will not be good enough. Despite international recognition of the importance of ensuring learning for all (see chapter 3), the level of development assistance for educa- tion and, within that, for basic education, is modest. Countries experiencing significant population growth are challenged by growing demand for greater access to secondary education while also needing to improve learning in basic education. In many countries, World Bank support for basic education lacks intensity and continuity (see table A.4). In such places, the World Bank does not combine projects with other influence points in a sequenced engage- ment that would lead to incremental system reform, learning measurement, a teaching career framework, and improved learning for all. Without consis- tent and longer-term engagement, it is difficult to gain traction to support the systemic reform necessary to improve learning outcomes. If funding by the World Bank and international stakeholders remains constant (despite the growing challenges), lending and nonlending initiatives will need to become increasingly strategic in what they support and in which countries. A core proposition of this evaluation is that support for the reform of basic education systems toward realizing learning for all requires country interven- tions calibrated to a systems analysis of basic education. Thorough analysis recognizes the unique political, social, cultural, and economic characteristics of individual basic education systems and facilitates the design and imple- mentation of tailored responses, consistent with the call in the WDR 2018 to identify and address system failures for learning. Documentation at the coun- try level suggests that support to basic education has taken a more uniform, less nuanced approach. Country case studies found several weaknesses in the World Bank approach. For example, documentation rarely emphasized the potential impacts of dynamic interaction among multiple, potentially powerful stakeholders on the achievement of desired outcomes. The case studies also found no assess- ments of the alignment and capacity of the basic education delivery system, especially for actors in the lower levels of the system on whom fidelity to policy reform and implementation success depends. Analysis undertaken by the World Bank should take account of the level of political will in support of inclusive education reform, the level of financial commitment in support of reform, and the extent of capacity within and across the system. Such assess- ment may lead the World Bank to prioritize its lending for basic education in Confronting the Learning Crisis Chapter 4 some countries and prioritize dialogue and capacity building in others. Where it has a willing and committed partner, the World Bank has contribut- ed to key policy reforms that have created a foundation for learning-oriented systems. The evaluation framework highlights several supporting conditions to implement learning for all. Case studies and literature identify political and social commitment as critical precursors. The evaluation also found that the momentum behind reform at the country level is a function of the finan- cial investment in basic education and capacity to deliver—factors that vary from country to country. Examples from Brazil, Kenya, and Viet Nam show deliberate use of World Bank knowledge, technical assistance, policy dialogue, 90 and financing to reform the education system in a manner that contributes to improvement in learning outcomes. The three countries have also provided adequate financial commitment and demonstrated high political commit- ment toward learning for all with a strong focus on equity. The evaluation concludes that the World Bank is well placed to lead in de- livering a more strategic response to the learning crisis and shifting to an outcome orientation. The World Bank has well-developed relationships with client governments that can be used to support reform in favor of learning for all. In addition, it has strong research and analytic capabilities and is the largest provider of development aid to education, putting it in an influential position in relation to other development partners. These comparative ad- vantages provide leverage that can be used to reorient dialogue and support to focus on critical reforms to education systems—equity, teaching career framework, learning assessment system, and capacity across all levels of sys- tems. A strategic response will require a shift from an output orientation to an outcome orientation. Stronger monitoring and evaluation will be required. Consistently examining whether what the World Bank finances is having a positive impact on systems, teaching, and measurement of learning is needed to provide a feedback loop, as depicted in the evaluation framework. A stronger contribution would require better contextualized World Bank engagement that focuses on political commitment, public funding, and the education system’s capacity to deliver learning for all. Developing countries, regardless of type, are on different basic education reform trajectories. Hence, Independent Evaluation Group World Bank Group    91 a thorough understanding of key factors affecting or potentially affecting reforms is necessary to best engage with and support basic education system reform. This requires going beyond isolated assessments of individual sys- tem pieces that are not functioning. To understand why education systems fail children, it is essential to analyze and understand the key driver in the learning for all reform trajectory—that is, the strength and depth of political commitment in favor of reform. Rhetorical acknowledgment of this find- ing appears in key documents, particularly in the WDR 2018 and in Ending Learning Poverty: What Will It Take? (World Bank 2019b), but it is rarely reflected in CPFs or PADs. The evaluation also found that the momentum behind reform at the country level is a function of the level of investment in basic education and capacity to deliver and that these factors vary from coun- try to country. In that regard, a systems-based analysis would provide a lens through which to design a context-specific engagement strategy and associ- ated interventions.4 There are signs that the World Bank and its partners are moving toward a systems focus, particularly at the global level. The Accelerator Program embraces systems reform among clients that show commitment to achieving results. Early lessons from the program show the need for flexibility as com- mitment wanes or context changes. Interviews conducted for the evaluation highlight that further motivating clients may require more resources than the Accelerator Program provides and may take additional time to coordinate—a constraint repeatedly noted in interviews. Among the factors that contributed to the World Bank’s influence with country clients were support for follow-up and the ability to link analytic support with implementation realities. Given the limitations in the quantum of development aid that goes to basic education, more comprehensive approaches to addressing the learning crisis will also require a much greater level of collaboration among development partners. COVID-19 has spurred the emergence of greater collective urgency and innovation among partners, including the World Bank, particularly at the global level. At the country level, however, although partners communicate and cooperate, true collaboration is much more limited and is undermined by the absence of a widely shared understanding of the factors contributing to system failure. A common analysis and understanding could support the co-pursuit of quality education and learning outcomes and the co-pursuit of reform of teaching career framework and measurement of learning, as no leading education system has succeeded without these two critical aspects. Confronting the Learning Crisis Chapter 4 Recommendations The evaluation makes two recommendations to promote an outcome focus in World Bank support to address key aspects to improve learning poverty at the country level. Develop country-specific education engagement plans that include systems-based enhancements to the teaching framework to improve learning outcomes. These plans should be informed by a comprehensive systems analysis of the constraints to implementation of a career framework—teacher recruitment, 92 training, development, motivation, and evaluation—as learning outcomes require capable and motivated teachers. Understanding the underlying issues, such as political will, system capacity, funding, and political economy obsta- cles and opportunities, will involve eliciting feedback from key stakeholders at all levels and compiling existing and new analysis to guide the develop- ment of a medium-term engagement process ideally anchored, where appropriate, within a pillar of the CPF. Sufficient data would be needed to inform adaptive management decisions related to corrective actions and learning during implementation to address the underlying constraints to sustainably improve systems and track intermediate outcomes. Implementation could be measured in CPFs, supported by analytics and projects, with intermediate outcomes related to the performance of the teaching career framework, rather than just the completion of activities. The success of the recommendation can also be measured in lessons that inform a scaling up of approaches from the World Bank’s engagement. Collaborate with global and country partners to close the data gaps on learn- ing outcomes (aligned with SDG target 4.1) and to track progress in ending learning poverty. This would be demonstrated by showing an increase in the number of countries with (i) education projects and CPFs that include indi- cators for learning improvements in grades 3 and 6, which may require more ambitious project goals and indicators; (ii) improvement in national education- al assessment capabilities and systems for data collection and decision-making; and (iii) participation in cross-national assessments for better data compara- Independent Evaluation Group World Bank Group    93 bility. A focus on those countries that lack quality national assessments and have not been part of international or regional assessments in the last five years is particularly needed. 1  More than two dozen low-income and middle-income countries, as well as high-income coun- tries and organizations, have signed the Commitment to Action. See https://www.worldbank. org/en/topic/education/brief/commitment-to-action-on-foundational-learning. 2  This is a shortcoming of World Bank trust funds in many sectors and not unique to education. 3  Using pre–COVID-19 data and, therefore, working from a more positive baseline, the authors estimate that under a business-as-usual approach, 44 percent of children in 2030 will still be unable to read at age 10 years—the 2015 baseline is 53 percent. In fact, their extrapolations suggest that even if every low- and middle-income country doubled or tripled its historical rate of progress, about 27 percent of children would continue to suffer learning poverty in 2030. 4  For example, in a country with low levels of political commitment to learning for all and lim- itations in funding and capacity, the World Bank might focus on building commitment through policy dialogue, exposing decision makers to successful systems and their benefits, identifying and supporting proreform coalitions of stakeholders, and building knowledge and student learning data. Confronting the Learning Crisis Chapter 4 94 Bibliography Abdul-Hamid, Husein, Namrata Saraogi, and Sarah Mintz. 2017. Lessons Learned from World Bank Education Management Information System Operations: Portfolio Review, 1998–2014. Washington, DC: World Bank. ADB (Asian Development Bank). 2013. Learning Lessons: Education Sector. Manila: ADB. Adelman, Melissa, and Renata Lemos. 2021. Managing for Learning: Measuring and Strengthening Education Management in Latin America and the Caribbean. Washington, DC: World Bank. https://openknowledge.worldbank.org/ handle/10986/35514. Aga Khan Foundation. 2010. Improving Learning Achievement in Early Primary in Low- Income Countries: A Review of the Research. Geneva: Aga Khan Foundation. Asadullah, M. Niaz, Liyanage Devangi H. Perera, and Saizi Xiao. 2020. “Vietnam’s Extraordinary Performance in the PISA Assessment: A Cultural Explanation of an Education Paradox.” Journal of Policy Modeling 42 (5): 913–32. Aslam, Monazza, Shenila Rawal, Geeta Kingdon, Bob Moon, Rukmini Banerji, Sushmita Das, Manjistha Banerji, and Shailendra K. Sharma. 2016. Reforms to Increase Teacher Effectiveness in Developing Countries: Systematic Review. London: Evidence for Policy and Practice Information Centre, Social Science Research Unit, Independent Evaluation Group World Bank Group    95 University College London Institute of Education, University College London. Azevedo, João Pedro. 2020. “Learning Poverty: Measures and Simulations.” Policy Research Working Paper 9446, World Bank, Washington, DC. https:// openknowledge.worldbank.org/handle/10986/34654. Azevedo, João Pedro, Diana Goldemberg, Silvia Montoya, Reema Nayar, Halsey Rogers, Jaime Saavedra, and Brian William Stacy. 2021. “Will Every Child Be Able to Read by 2030? Defining Learning Poverty and Mapping the Dimensions of the Challenge.” Policy Research Working Paper 9588, World Bank, Washington, DC. Barrett, Peter, Alberto Treves, Tigran Shmis, Diego Ambasz, and Maria Ustinova. 2019. The Impact of School Infrastructure on Learning: A Synthesis of the Evidence. Washington, DC: World Bank. https://openknowledge.worldbank.org/ handle/10986/30920. Bashir, Sajitha, Marlaine Lockheed, Elizabeth Ninan, and Jee-Peng Tan. 2018. Facing Forward: Schooling for Learning in Africa. Washington, DC: World Bank. https:// openknowledge.worldbank.org/handle/10986/29377. Bedasso, Biniam, and Justin Sandefur. 2024. “The Evolution of World Bank Lending for Education: 1998–2022.” Working Paper 68L, Center for Global Development, Washington, DC. https://www.cgdev.org/publication/ evolution-world-bank-lending-education-1998-2022. Béteille, Tara, Namrata Tognatta, Michelle Riboud, Shinsaku Nomura, and Yashodhan Ghorpade. 2020. Ready to Learn: Before School, in School, and beyond School in South Asia. Washington, DC: World Bank. https://openknowledge.worldbank.org/ entities/publication/94ddb038-2505-5d9d-b2f0-5517ad61f672. Bloor, Thomas, and Wondwosen Tamrat. 1996. “Issues in Ethiopian Language Policy and Education.” Journal of Multilingual and Multicultural Development 17 (5): 321–38. Bruns, Barbara, David Evans, and Javier Luque. 2012. Achieving World-Class Education in Brazil: The Next Agenda. Washington, DC: World Bank. Bruns, Barbara, Isabel Harbaugh Macdonald, and Ben Ross Schneider. 2019. “The Politics of Quality Reforms and the Challenges for SDGs in Education.” World Development 118 (June): 27–38. Bruns, Barbara, and Javier Luque. 2014. Great Teachers: How to Raise Student Learning in Latin America and the Caribbean. Advance edition. Washington, DC: World Confronting the Learning Crisis Bibliography Bank. https://www.worldbank.org/content/dam/Worldbank/document/LAC/Great_ Teachers-How_to_Raise_Student_Learning-Barbara-Bruns-Advance%20Edition.pdf. Buitrago-Hernandez, Paola, Victoria Levin, and Carlos Rodríguez Castelán. 2023. “Can We Achieve Gender Parity in Education while Leaving Boys Out of School?” Latin America and Caribbean (blog), February 21, 2023. https://blogs.worldbank. org/latinamerica/can-we-achieve-gender-parity-education-while-leaving-boys- out-school. Chakera, Shiraz, Deborah Haffner, and Elizabeth Harrop. 2020. “Structured Pedagogy: For Real-Time Equitable Improvements in Learning Outcomes.” UNICEF Eastern and Southern Africa Working Paper 2020, United Nations Children’s Fund, Nairobi. 96 Chong Soh, Yew, Ximena Del Carpio, and Choon Wang. 2022. “Language of Instruction Matters for Learning Foundational Skills.” Education for Global Development (blog), February 3, 2022. https://blogs.worldbank.org/education/ language-instruction-matters-learning-foundational-skills. Clarke, Marguerite, and Diego Luna-Bazaldua. 2021. Primer on Large-Scale Assessments of Educational Achievement. National Assessments of Educational Achievement Series. Washington, DC: World Bank. Collins, Tom. 2022. “Sierra Leone’s President Defends Large Education Budget as ‘Necessary Risk.’” Guardian, November 23, 2022. https:// www.theguardian.com/global-development/2022/nov/23/ sierra-leones-president-defends-large-education-budget-as-necessary-risk. Colombia, Ministry of Education. 2023. “¿De qué se trata este compromiso?” Government of Colombia, April 26, 2023. https://www.mineducacion.gov.co/ portal/micrositios-preescolar-basica-y-media/Un-Compromiso-de-Accion. Comptroller and Auditor General. 2010. Bilateral Support to Primary Education. London: National Audit Office. Crouch, Luis. 2020. “Systems Implications for Core Instructional Support Lessons from Sobral (Brazil), Puebla (Mexico), and Kenya.” Insight Note, Research on Improving Systems of Education, Oxford. https://riseprogramme.org/publications/systems- implications-core-instructional-support-lessons-sobral-brazil-puebla-mexico. Independent Evaluation Group World Bank Group    97 Datta, Lois-ellin. 1997. “Multimethod Evaluations.” In Evaluation for the 21st Century, edited by Eleanor Chelimsky and William R. Shadish, 344–59. Thousand Oaks: Sage. Evans, David K., and Anna Popova. 2016. “What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews.” World Bank Research Observer 31 (2): 242–70. Fan, Xitao, and Michael Chen. 2001. “Parental Involvement and Students’ Academic Achievement: A Meta-Analysis.” Educational Psychology Review 13 (1): 1–22. Faul, Moira V., and Laura Savage. 2023. “Introduction to Systems Thinking in International Education and Development.” In Systems Thinking in International Education and Development: Unlocking Learning for All?, edited by Moira V. Faul and Laura Savage, 1–25. Cheltenham: Edward Elgar Publishing. https://doi. org/10.4337/9781802205930. Filmer, Deon, Halsey Rogers, Noam Angrist, and Shwetlena Sabarwal. 2018. “Learning-Adjusted Years of Schooling (LAYS): Defining a New Macro Measure of Education.” Policy Research Working Paper 8591, World Bank, Washington, DC. Fullan, Michael. 2015. The New Meaning of Educational Change. 5th ed. New York: Teachers College Press. Gertler, Paul, James Heckman, Rodrigo Pinto, Arianna Zanolini, Christel Vermeersch, Susan Walker, Susan M. Chang, and Sally Grantham-McGregor. 2014. “Labor Market Returns to an Early Childhood Stimulation Intervention in Jamaica.” Science 344 (6187): 998–1001. Global Coalition for Foundational Learning. 2023. “Helping Countries Get Comparable Learning Data.” Education for All (blog), July 20, 2023. https://www. globalpartnership.org/blog/helping-countries-get-comparable-learning-data. Grindle, Merilee S. 2004. “Good Enough Governance: Poverty Reduction and Reform in Developing Countries.” Governance 17 (4): 525–48. Guerrero-Ruiz, Alejandro, Kadambote Sachin, and Julia Schnatz. 2021. “Aligning Development Co-operation to the SDGs in Least Developed Countries: A Case Study of Uganda.” OECD Development Co-operation Working Paper 102, Organisation for Economic Co-operation and Development, Paris. Haddad, Wadi D., Martin Carnoy, Rosemary Rinaldi, and Omporn Regel. 1990. Confronting the Learning Crisis Bibliography “Education and Development: Evidence for New Priorities.” World Bank Discussion Paper WDP-95, World Bank, Washington, DC. http://documents. worldbank.org/curated/en/871111468739527551/Education-and-development- evidence-for-new-priorities. Hanushek, Eric A. 2022. “Eric Hanushek on the Basic Skills Gap.” IMF Podcasts, August 17, 2022, https://www.imf.org/en/News/Podcasts/All-Podcasts/2022/08/17/ eric-hanushek-on-basic-skills. Hanushek, Eric A., and Ludger Woessmann. 2008. “The Role of Cognitive Skills in Economic Development.” Journal of Economic Literature 46 (3): 607–68. 98 He, Fang, Leigh L. Linden, and Margaret MacLeod. 2007. Helping Teach What Teachers Don’t Know: An Assessment of the Pratham English Language Program. New York: Columbia University. Herbert, Alicia, Jaime Saavedra, Leanna Marr, and Robert Jenkins. 2021. “The Urgent Need to Focus on Foundational Skills.” Education for Global Development (blog), November 4, 2021. https://blogs.worldbank.org/education/ urgent-need-focus-foundational-skills. Hesse-Biber, Sharlene, and R. Burke Johnson, eds. 2015. The Oxford Handbook of Multimethod and Mixed Methods Research Inquiry. Oxford: Oxford University Press. Hickey, Sam, Naomi Hossain, and David Jackman. 2019. “Identifying the Political Drivers of Quality Education: A Comparative Analysis.” In The Politics of Education in Developing Countries: From Schooling to Learning, edited by Sam Hickey and Naomi Hossain, 172–96. Oxford: Oxford University Press. Inoue, Keiko, Emanuela di Gropello, Yesim Sayin Taylor, and James Gresham. 2015. Out-of-School Youth in Sub-Saharan Africa: A Policy Perspective. Washington, DC: World Bank. Kataoka, Sachiko, Le Anh Vinh, Sandhya Kitchlu, and Keiko Inoue. 2020. Vietnam’s Human Capital: Education Success and Future Challenges. Washington, DC: World Bank. http://hdl.handle.net/10986/34316. Kerwin, Jason T., and Rebecca L. Thornton. 2016. “Making the Grade: Understanding Independent Evaluation Group World Bank Group    99 What Works for Teaching Literacy in Rural Uganda.” Presented at the Institute for International Economic Policy, Elliott School of International Affairs, George Washington University, Washington, DC, April 28. https://iiep.gwu. edu/2016/04/28/economics-and-political-economy-of-africa. Kingdon, Geeta Gandhi, Angela Little, Monazza Aslam, Shenila Rawal, Terry Moe, Harry Patrinos, Tara Béteille, Rukmini Banerji, Brent Parton, and Shailendra K. Sharma. 2014. A Rigorous Review of the Political Economy of Education Systems in Developing Countries. London: Department for International Development. Le, Anh Vinh, Puyuan Han, Maw Maw Khaing, and Olivia Farrar. 2022. “An Emerging Dragon: Vietnamese Education after Resolution 29.” In Education to Build Back Better: What Can We Learn from Education Reform for a Post-Pandemic World, edited by Fernando M. Reimers, Uche Amaechi, Alysha Banerji, and Margaret Wang, 99–123. Cham: Springer. https://doi.org/10.1007/978-3-030-93951-9_5. Lee, Jessica D., and Octavio Medina. 2019. Results-Based Financing in Education: Learning from What Works. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/915061548222619389/ Results-Based-Financing-in-Education-Learning-from-What-Works. Lucas, Adrienne M., Patrick J. McEwan, Moses Ngware, and Moses Oketch. 2014. “Improving Early-Grade Literacy in East Africa: Experimental Evidence from Kenya and Uganda.” Journal of Policy Analysis and Management 33 (4): 950–76. Magrath, Bronwen, Monazza Aslam, and David Johnson. 2019. “Systems Research in Education: Designs and Methods.” Research in Comparative and International Education 14 (1): 7–29. Majchrzak, Ann, Sirkka L. Jarvenpaa, and Mehdi Bagherzadeh. 2015. “A Review of Interorganizational Collaboration Dynamics.” Journal of Management 41 (5): 1338–60. McEwan, Patrick J. 2015. “Improving Learning in Primary Schools of Developing Countries: A Meta-Analysis of Randomized Experiments.” Review of Educational Research 85 (3): 353–94. Mervosh, Sarah. 2022. “Pandemic Learning Loss.” New York Times, November 28, 2022. https://www.nytimes.com/2022/11/28/briefing/pandemic-learning-loss.html. Muñoz-Najar, Alberto, Alison Gilberto, Amer Hasan, Cristóbal Cobo, João Pedro Confronting the Learning Crisis Bibliography Azevedo, and Maryam Akmal. 2021. Remote Learning during COVID-19: Lessons from Today, Principles for Tomorrow. Washington, DC: World Bank. Murakami, Yuki. 2023. “The Annual Financing Gap to Achieve SDG 4 Targets Is Almost $100 Billion.” GEM Report (blog), April 14, 2023. https://world-education-blog.org/2023/04/14/ the-annual-financing-gap-to-achieve-sdg-4-targets-is-almost-100-billion. Muralidharan, Karthik, and Venkatesh Sundararaman. 2011. “Teacher Performance Pay: Experimental Evidence from India.” Journal of Political Economy 119 (1): 39–77. 100 NCEE (National Center on Education and the Economy). 2021. NCEE’s Blueprint for a High-Performing Education System. Washington, DC: NCEE. Negash, Tekeste. 2006. “Education in Ethiopia: From Crisis to the Brink of Collapse.” Discussion Paper 33, Nordiska Afrikainstitutet, Uppsala. Novelli, Mario, Sean Higgins, Mehmet Ugur, and Oscar Valiente. 2014. The Political Economy of Education Systems in Conflict-Affected Contexts. London: Department for International Development. OECD (Organisation for Economic Co-operation and Development). 2015. The ABC of Gender Equality in Education: Aptitude, Behaviour, Confidence. Paris: OECD. https:// doi.org/10.1787/9789264229945-en. OECD (Organisation for Economic Co-operation and Development). 2021. Education in Brazil: An International Perspective. Paris: OECD. https://doi. org/10.1787/60a667f7-en. OECD (Organisation for Economic Co-operation and Development). 2022. Mending the Education Divide: Getting Strong Teachers to the Schools That Need Them Most. Paris: OECD. https://doi.org/10.1787/92b75874-en. Patton, Michael Quinn. 2015. Qualitative Research and Evaluation Methods. 4th ed. Thousand Oaks: Sage. Piper, Benjamin, and Margaret M. Dubeck. 2021. Structured Pedagogy: Literature Review. Independent Evaluation Group World Bank Group    101 Research Triangle Park: RTI International. https://scienceofteaching.site/wp- content/uploads/2022/11/Structured-Pedagogy-Literature-Review-1.pdf. Popova, Anna, David K. Evans, Mary E. Breeding, and Violeta Arancibia. 2018. “Teacher Professional Development around the World: The Gap between Evidence and Practice.” Policy Research Working Paper 8572, World Bank, Washington, DC. Pritchett, Lant. 2018. “The Risks of Dangerous Dashboards in Basic Education.” CGD Note, Center for Global Development, Washington, DC. https://www.cgdev.org/ publication/risks-dangerous-dashboards-in-basic-education. Psacharopoulos, George, and Harry Anthony Patrinos. 2018. “Returns to Investment in Education: A Decennial Review of the Global Literature.” Education Economics 26 (5): 445–58. RISE (Research on Improving Systems of Education). 2023. The RISE Education Systems Diagnostic. Oxford: RISE. https://riseprogramme.org/tools/ rise-education-systems-diagnostic. Rodriguez, Maria Barron, Cristóbal Cobo, Alberto Muñoz-Najar, and Iñaki Sánchez Ciarrusta. 2020. Remote Learning during the Global School Lockdown: Multi- Country Lessons. Washington, DC: World Bank. http://documents.worldbank.org/ curated/en/668741627975171644/Remote-Learning-During-the-Global-School- Lockdown-Multi-Country-Lessons. Sahlberg, Pasi. 2011. “Finnish Lesson #3: What Can We Learn from Educational Change in Finland?” PasiSahlberg.com (blog). https://pasisahlberg.com/finnish- lesson-3-what-can-we-learn-from-educational-change-in-finland. Schady, Norbert, Alaka Holla, Shwetlena Sabarwal, Joana Silva, and Andres Yi Chang. 2023. Collapse and Recovery: How the COVID-19 Pandemic Eroded Human Capital and What to Do about It. Washington, DC: World Bank. Schleicher, Andreas. 2018. World Class: How to Build a 21st-Century School System. Paris: Organisation for Economic Co-operation and Development. https://doi. org/10.1787/9789264300002-en. Schweinhart, Lawrence J. 2007. “Outcomes of the High/Scope Perry Preschool Study and Michigan School Readiness Program.” Ypsilanti HighScope Educational Research Foundation. Snilstveit, Birte, Jennifer Stevenson, Daniel Phillips, Martina Vojtkova, Emma Confronting the Learning Crisis Bibliography Gallagher, Tanja Schmidt, Hannah Jobse, Maisie Geelen, Maria Grazia Pastorello, and John Eyers. 2015. Interventions for Improving Learning Outcomes and Access to Education in Low- and Middle-Income Countries: A Systematic Review. London: International Initiative for Impact Evaluation. Soliván, Maribel, and Rebecca Winthrop. 2014. “Stealing from the Future: The Urgency of the Global Learning Crisis.” Education Plus Development (blog), February 14, 2014. https://www.brookings.edu/blog/education-plus-development/2014/02/14/ stealing-from-the-future-the-urgency-of-the-global-learning-crisis. Spivack, Marla. 2021. “Applying Systems Thinking to Education: The RISE Systems Framework.” Insight Note, Research on Improving Systems of Education, Oxford. 102 https://riseprogramme.org/publications/applying-systems-thinking-education- rise-systems-framework.html. Thorn, William, and Stéphan Vincent-Lancrin. 2021. Schooling during a Pandemic: The Experience and Outcomes of Schoolchildren during the First Round of COVID-19 Lockdowns. Paris: Organisation for Economic Co-operation and Development. https://doi.org/10.1787/1c78681e-en. UNESCO (United Nations Educational, Scientific, and Cultural Organization). 2014. Education for All Global Monitoring Report, 2013–2014—Teaching and Learning: Achieving Quality for All. Paris: UNESCO. UNESCO (United Nations Educational, Scientific, and Cultural Organization). 2016. Global Education Monitoring Report 2016—Education for People and Planet: Creating Sustainable Futures for All. Paris: UNESCO. UNESCO (United Nations Educational, Scientific, and Cultural Organization). 2017. Global Education Monitoring Report 2017/8—Accountability in Education: Meeting Our Commitments. Paris: UNESCO. UNESCO (United Nations Educational, Scientific, and Cultural Organization). 2022. “Why Mother Language-Based Education Is Essential.” UNESCO, February 18, 2022. https://www.unesco.org/en/articles/why-mother-language-based- education-essential. UNESCO (United Nations Educational, Scientific, and Cultural Organization). 2023. Independent Evaluation Group World Bank Group    103 Global Education Monitoring Report 2023—Technology in Education: A Tool on Whose Terms? Paris: UNESCO. UNESCO UIS (United Nations Educational, Scientific, and Cultural Organization Institute for Statistics). 2012. International Standard Classification of Education: ISCED 2011. Montreal: UNESCO UIS. UNESCO UIS (United Nations Educational, Scientific, and Cultural Organization Institute for Statistics). 2015. Fixing the Broken Promise of Education for All: Findings from the Global Initiative on Out-of-School Children. Montreal: UNESCO UIS. http://uis.unesco.org/sites/default/files/documents/fixing-broken-promise- efa-findings-global-initiative-oosc-education-2015-en.pdf. UNESCO UIS (United Nations Educational, Scientific, and Cultural Organization Institute for Statistics). 2017. “More Than One-Half of Children and Adolescents Are Not Learning Worldwide.” Fact Sheet 46, UNESCO UIS, Montreal. UNESCO UIS (United Nations Educational, Scientific, and Cultural Organization Institute for Statistics). 2019. “New Methodology Shows That 258 Million Children, Adolescents and Youth Are Out of School.” Fact Sheet 56, UNESCO UIS, Montreal. UNICEF (United Nations Children’s Fund). 2016. The Impact of Language Policy and Practice on Children’s Learning: Evidence from Eastern and Southern Africa. New York: UNICEF. https://www.unicef.org/esa/reports/language-and-childrens- learning#full-report. UNICEF (United Nations Children’s Fund). 2020. “Addressing the Learning Crisis: An Urgent Need to Better Finance Education for the Poorest Children.” Brief, UNICEF, New York. https://www.unicef.org/media/63896/file/Addressing-the-learning- crisis-advocacy-brief-2020.pdf. UNICEF (United Nations Children’s Fund). 2023. “Transforming Education with Equitable Financing.” Brief, UNICEF, New York. https://www.unicef.org/ media/133431/file/Transforming%20Education%20with%20Equitable%20 Financing.pdf. USAID (United States Agency for International Development). 2022. Joint Strategic Plan, FY 2022–2026. Washington, DC: USAID. Wales, Joseph, Arran Magee, and Susan Nicolai. 2016. “How Does Political Context Confronting the Learning Crisis Bibliography Shape Education Reforms and Their Success? Lessons from the Development Progress Project.” Dimension Paper 06, Overseas Development Institute, London. Walker, Susan P., Susan M. Chang, Christine A. Powell, and Sally Grantham-McGregor. 2005. “Effects of Early Childhood Psychosocial Stimulation and Nutritional Supplementation on Cognition and Education in Growth-Stunted Jamaican Children: Prospective Cohort Study.” The Lancet 366 (9499): 1804–7. Walker, Susan P., Susan M. Chang, Christine A. Powell, Emily Simonoff, and Sally M. Grantham-McGregor. 2006. “Effects of Psychosocial Stimulation and Dietary Supplementation in Early Childhood on Psychosocial Functioning in Late Adolescence: Follow-up of Randomised Controlled Trial.” British Medical Journal 104 333 (7566): 472–74. Walker, Susan P., Helen Baker-Henningham, Susan M. Chang, Christine A. Powell, Florencia López-Boo, and Sally Grantham-McGregor. 2017. “Implementation of Parenting Interventions through Health Services in Jamaica.” Vulnerable Children and Youth Studies 13 (2): 127–41. Walker, Susan P., Christine Powell, Susan M. Chang, Helen Baker-Henningham, and Sally Grantham-McGregor. 2016. Parenting Programme in Health Centres: What You Do with Baby Really Matters. Kingston, Jamaica: The University of the West Indies. Walker, Susan P., Theodore D. Wachs, Sally Grantham-McGregor, Maureen M. Black, Charles A. Nelson, Sandra L. Huffman, Helen Baker-Henningham, Susan M. Chang, Jena D. Hamadani, Betsy Lozoff, Julie M. Meeks Gardner, Christine A. Powell, Atif Rahman, and Linda Richter. 2011. “Inequality in Early Childhood: Risk and Protective Factors for Early Child Development.” The Lancet 378 (9799): 1325–38. World Bank. 1999. Education Sector Strategy. Washington, DC: World Bank. World Bank. 2005. Education Sector Strategy Update: Achieving Education for All, Broadening Our Perspective, Maximizing Our Effectiveness. Washington, DC: World Bank. World Bank. 2006. From Schooling Access to Learning Outcomes: An Unfinished Agenda—An Evaluation of World Bank Support to Primary Education. Independent Independent Evaluation Group World Bank Group    105 Evaluation Group. Washington, DC: World Bank. World Bank. 2010. World Bank Support to Education Since 2001: A Portfolio Note. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2011. Learning for All: Investing in People’s Knowledge and Skills to Promote Development—World Bank Group Education Strategy 2020. Washington, DC: World Bank. World Bank. 2013a. Knowledge-Based Country Programs: An Evaluation of the World Bank Group Experience. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2013b. Responding to Global Public Bads: Learning from Evaluation of the World Bank Experience with Avian Influenza 2006–13. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2014. “SABER–Education Management Information Systems (EMIS) Rubric.” Working Paper 115680, World Bank, Washington, DC. https://documents.worldbank.org/en/publication/documents-reports/ documentdetail/514011496660115497/saber-education-management- information-systems-emis-rubric. World Bank. 2015. World Bank Group Gender Strategy 2016–2023: Gender Equality, Poverty Reduction, and Inclusive Growth. Washington, DC: World Bank. World Bank. 2017. Crisis Response and Resilience to Systemic Shocks: Lessons from IEG Evaluations. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018a. Engaging Citizens for Better Development Results. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018b. World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank. World Bank. 2019a. “Coach Work Program.” Concept Note, World Bank, Washington, DC. World Bank. 2019b. Ending Learning Poverty: What Will It Take? Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/32553. World Bank. 2019c. Selected Drivers of Education Quality: Pre- and In-Service Teacher Confronting the Learning Crisis Bibliography Training. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019d. Tajikistan: Improving Teaching and Learning Efficiency. Washington, DC: World Bank. World Bank. 2020a. Realizing the Future of Learning: From Learning Poverty to Learning for Everyone, Everywhere. Washington, DC: World Bank. World Bank. 2020b. “Teach 2.0.” Concept Note (revised version), World Bank, Washington, DC. World Bank. 2020c. The Human Capital Index 2020 Update: Human Capital in the Time of COVID-19. Washington, DC: World Bank. 106 World Bank. 2021a. “Accelerator Program.” Brief, July 1, 2021, World Bank, Washington, DC. https://www.worldbank.org/en/topic/education/brief/accelerator-program. World Bank. 2021b. “Coach: Helping Countries Accelerate Learning.” Program Overview Document, World Bank, Washington, DC. https://thedocs.worldbank. org/en/doc/b455af6f955a971fd29157524f9d190a-0140052021/related/Coach- Program-Overview-Document.pdf. World Bank. 2021c. Sierra Leone Public Expenditure Review 2021: Primary and Secondary Education. Washington, DC: World Bank. https://openknowledge.worldbank.org/ server/api/core/bitstreams/c2980ab3-5218-5c92-92d7-b002edad4648/content. World Bank. 2021d. “What Is Learning Poverty?” Brief, April 28, 2021, World Bank, Washington, DC. https://www.worldbank.org/en/topic/education/brief/what-is- learning-poverty. World Bank. 2022a. Kenya—Country Partnership Framework for the Period FY2023– 2028. Washington, DC: World Bank. http://documents.worldbank.org/curated/ en/099205010262216711/BOSIB00be0e8020800a68b02235d729b923. World Bank. 2022b. Results and Performance of the World Bank Group 2022. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2023a. “Latin America and the Caribbean Reiterate Their Commitment to Strengthening and Recovering Basic Learning.” World Bank, March 22, 2023. https://www.worldbank.org/en/news/press-release/2023/03/22/ Independent Evaluation Group World Bank Group    107 america-latina-y-el-caribe-reitera-su-compromiso-con-el-fortalecimiento-y- recuperacion-de-los-aprendizajes-basicos. World Bank. 2023b. “Learning Poverty Global Database: Historical Data and Sub- Components.” Data Catalog, World Bank, Washington, DC. https://datacatalog. worldbank.org/int/search/dataset/0038947. World Bank. 2023c. Realizing Education’s Promise: A World Bank Retrospective. Washington, DC: World Bank. World Bank. 2024a. “Country Learning Poverty Briefs.” Brief, World Bank, Washington, DC. https://www.worldbank.org/en/topic/education/brief/country-learning- poverty-briefs. World Bank. 2024b. The Knowledge Compact for Action: Transforming Ideas into Development Impact. Washington, DC: World Bank. World Bank, Bill & Melinda Gates Foundation, FCDO (Foreign, Commonwealth & Development Office), UNESCO (United Nations Educational, Scientific, and Cultural Organization), UNICEF (United Nations Children’s Fund), and USAID (United States Agency for International Development). 2022. Guide for Learning Recovery and Acceleration: Using the RAPID Framework to Address COVID-19 Learning Losses and Build Forward Better. Washington, DC: World Bank. https:// thedocs.worldbank.org/en/doc/e52f55322528903b27f1b7e61238e416-0200022022/ related/Guide-for-Learning-Recovery-and-Acceleration-06-23.pdf. World Bank and UNESCO (United Nations Educational, Scientific, and Cultural Organization). 2021. Education Finance Watch 2021. Washington, DC: World Bank; Paris: UNESCO. World Bank and UNESCO (United Nations Educational, Scientific, and Cultural Organization). 2022. Education Finance Watch 2022. Washington, DC: World Bank; Paris: UNESCO. https://thedocs.worldbank.org/en/doc/ e52f55322528903b27f1b7e61238e416-0200022022/related/EFW-2022-Jul1.pdf. World Bank, UNESCO (United Nations Educational, Scientific, and Cultural Organization), UNICEF (United Nations Children’s Fund), FCDO (Foreign, Commonwealth & Development Office), USAID (United States Agency for International Development), and Bill & Melinda Gates Foundation. 2022. The State of Global Learning Poverty: 2022 Update. Washington, DC: World Bank. Confronting the Learning Crisis Bibliography Yardley, Lucy. 2008. “Demonstrating Validity in Qualitative Psychology.” In Qualitative Psychology: A Practical Guide to Research Methods, edited by Jonathan A. Smith, 235–51. Thousand Oaks: Sage. 108 APPENDIX Independent Evaluation Group Confronting the Learning Crisis Appendix A. Evaluation Methods Evaluation Scope The analysis for this evaluation covers World Bank support—the International Bank for Reconstruction and Development, the International Development Association, and recipient-managed trust funds—for basic education. For this evaluation, basic education is defined as primary and early secondary education. The scope of this evaluation was defined along four dimensions: subject focus, reference period, global knowledge, and country coverage. » Subject focus: The subject focus is World Bank support for education quality and enhanced learning outcomes in basic education, with an additional focus on support provided to address critical challenges to education delivery and the exacerbation of learning loss associated with the COVID-19 pandemic. » Reference period: The reference period is fiscal years (FY)12–22. The evaluation covers projects approved and a sample of knowledge products published during this period. » Global knowledge: The evaluation focuses on a purposeful sample of strategies, initiatives, programs, and research supported by the Education Global Practice (GP) over the FY12–22 period. This included data, impact evaluations, foundational learning and teachers, education system Confronting the Learning Crisis Appendix A improvement via the Systems Approach for Better Education Results, and student learning assessments at regional, national, and subnational levels. The evaluation assesses how well these efforts are generating knowledge, complementing the efforts of other partners, and building awareness about how to improve quality and learning in basic education and to strengthen education systems. » Country coverage: The countries selected for the analysis were from distinct country types based on fragility, conflict, and violence; low institutional capacity; and moderate institutional capacity, as identified in the World Bank’s response to the World Development Report (WDR) 2018 (World Bank 112 2018, 2019). That document provides a useful framework for how the World Bank has planned to engage with countries and their education systems on a differentiated basis. Evaluation Components The evaluation used a mixture of methods, including case-based analysis, portfolio analysis of lending and advisory services and analytics (ASA), and key informant interviews. These core methods were supported by literature reviews related to global knowledge, the political economy of education, how other development partners have understood and addressed the learning crisis, and the characteristics of high-performing education systems. The evaluation also undertook analysis of secondary data for case countries, which related to population growth and learning assessment via regional and global assessments. Table A.1 lists the evaluation components applied to answer the overarching evaluation question and subquestions. Independent Evaluation Group World Bank Group    113 114 Confronting the Learning Crisis Appendix A  ethods by Evaluation Question and Unit of Analysis Table A.1. M Key Review Secondary Case Informant Sample Data Literature Evaluation Question PRA Studies Interview Global ASA Analysis Reviews How has World Bank support for basic education X X X X X X contributed to the achievement of enhanced learning outcomes since the Learning for All strategy, and what can be learned from those efforts to inform support to the learning recovery from the COVID-19 pandemic? EQ1. How effective has World Bank support for basic X X X X X X education (FY12–22) been in addressing the binding constraints that hinder the achievement of enhanced learning outcomes in client countries? EQ2. To what extent and how effectively has the X X X X World Bank: (i) collaborated with country and global partners to support education quality and enhanced learning outcomes? and (ii) used feedback from evidence and experience to X X X X inform its work to support improved education quality and learning outcomes for all? EQ3. How well prepared is the World Bank to address X X X X X additional challenges to education systems that have arisen because of the impact of the COVID-19 pandemic? Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; EQ = evaluation question; FY = fiscal year; PRA = portfolio review analysis. Ensuring the Validity of Findings The conceptual framework (see figure 1.2) was used to guide the collection and interpretation of data. The framework was developed by the evaluation team and tested in consultation with World Bank staff to ensure its relevance and comprehensiveness. Several background papers were prepared to aid the development of the framework and subsequent protocols for data collection. Background papers were prepared that synthesized findings from systematic literature reviews and the qualitative studies associated with these reviews; political economy literature related to improving education quality; ap- proaches of other international financial institutions in basic education; and review of studies of comprehensive programs, such as Teaching at the Right Level, and characteristics of high-performing education systems. The variety of literature reviewed provided a knowledge foundation to ensure that the evaluation team shared an understanding of issues affecting education quali- ty and the context specificity of education systems. The evaluation then used the framework to ensure multiple levels of triangu- lation using common templates, informed by the framework, when conducting portfolio reviews, case studies, and interviews. The framework governed the collection and interpretation of data within each method—for example, in en- suring commonality of understanding of the intent of questions asked among case authors and in testing the consistency of that understanding in compar- Independent Evaluation Group World Bank Group    115 ing responses across cases. The framework was also used to test and underpin interpretation across methods in a manner that ensures robust triangulation. This appendix elaborates on the objective and function of each evaluation component and the selection and analysis process. Case Studies Objective of case study. Case studies were undertaken to understand the World Bank’s support for basic education in context and answer the evalua- tion questions. Selection criteria and process. Adopting a systematic approach to address- ing this deep and long-standing issue suggests that case study selection should prioritize countries where the World Bank has had active projects during most of the evaluation period. Hence, the evaluation team selected from among those countries in which the World Bank had at least two lending projects (with one or more both before and after FY17) and a minimum of two ASA during the evaluation period. This reduced the pool of choices from 125 to 41 countries distributed across the Regions as follows: Western and Central Africa (12), Eastern and Southern Africa (9), Europe and Central Asia (5), South Asia (5), Latin America and the Caribbean (5), East Asia and Pacific (4), and Middle East and North Africa (1). Cases were selected from among those 41 countries based on relative country capacity and education efficiency. This process also identified country “types” such that learning from the cases could be cross-examined through testing differences and similarities among the types. Capacity was measured using percentiles among Country Policy and Institutional Assessment (CPIA) data from all countries (n = 138) with CPIA data available. The CPIA was highly correlated with other capacity measures; thus, the overall CPIA score was used as shown in table A.2.1  ountry Policy and Institutional Assessment Data Analysis Table A.2. C Data Sets Covariance Significance Data set—all countries with CPIA data CPIA quality of public administration 0.8257 0.000 WGI government effectiveness rank 0.8232 0.000 Confronting the Learning Crisis Appendix A WGI control of corruption rank 0.5959 0.000 Data set—41 selected countries CPIA quality of public administration 0.8625 0.000 WGI government effectiveness rank 0.8443 0.000 WGI control of corruption rank 0.6146 0.000 Source: Independent Evaluation Group. Note: CPIA = Country Policy and Institutional Assessment; WGI = Worldwide Governance Indicator. A key consideration for this evaluation is the World Bank’s approach (through, for example, policy dialogue, convening power, lending, ASA, and partnership) 116 to improving learning outcomes in diverse country contexts in which the World Bank will encounter a wide range of country capacity and management efficiency in education systems. Efficiency was examined to see whether the performance in some countries was high, average, or low on indicators of par- ticipation, given their level of spending. The level of spending was measured in two ways. First, the evaluation team examined the level of relative education spending “effort” captured by indi- cators, such as education spending as a percentage of GDP. The second set of indicators focused on “real” spending per student, measured in constant US dollars per pupil. This analysis multiplied the spending per student (by level) as a percentage of GDP per capita by actual GDP per capita. The core criteria applied to identify case types were as follows: » Capacity: Cases were selected based on the country types identified in recent policy documents (2019, 2020). Three country types—low capacity, medium capacity, and high capacity—were decided with reference to CPIA scores, which are available for all countries. Fragility, conflict, and violence countries (which have varying levels of capacity) were also selected using these three types. » Efficiency: Cases were selected based on the relative efficiency of their educa- tion sector—results attained compared with spending per pupil (high, average, and low). The Independent Evaluation Group (IEG) conducted analysis of Independent Evaluation Group World Bank Group    117 education system performance based on the average (2010–19) of three sets of indicators: net enrollment rate (NER) in primary and secondary education; the harmonized learning outcome measure, created by the World Bank, which puts countries on a single, comparable scale based on overlapping regional and international assessments; and the learning-adjusted years of schooling (LAYS) measure, which combines the first two indicators. For countries for which data are available (31 out of 41 in the sample), preliminary analysis provided the relative level of the education sector—simple regressions for the two NER measures and the LAYS measure were regressed onto the measure of average primary-secondary spending per pupil (as discussed further in this appendix). For the 10 countries without a full set of data, IEG regressed the available LAYS measure onto GDP per capita to develop a proxy measure of efficiency to per- mit further examination of the full sample (41 countries). Table A.3 shows the distribution of the sample case countries with reference to level of country capacity and level of efficiency in the education sector.  ample for Case Selection by Capacity and Efficiency Table A.3. S Capacity Low Efficiency Average Efficiency High Efficiency Low capacity Niger* Cameroon*§ Tajikistan Chad* § Myanmar* Mali*§ Timor-Leste† Ethiopia*§ Sierra Leone§ Gambia, The§ Zambia§ Sudan‡§ Haiti* Angola Lebanon‡ Liberia§ Bangladesh§ Medium capacity Tanzania§ Cambodia§ Nepal§ Senegal Pakistan§ Sri Lanka Guinea Ghana Kyrgyz Republic Nigeria*§ Malawi§ Madagascar§ High capacity India§ Viet Nam Kenya§ Georgia Uganda§ Mexico§ Benin§ Peru§ Uruguay§ Confronting the Learning Crisis Appendix A Moldova§ Armenia** Brazil§ Source: Independent Evaluation Group. Note: Fragility, conflict, and violence categories: * = medium-intensity conflict; ** = high-intensity interna- tional conflict; † = small state; ‡ = high institutional and social fragility; § = emergency response operations with education theme code (COVID-19 projects). Given that learning loss recovery varies based on extent of school closures and mitigation measures taken in each context, the sample ensured that World Bank emergency operations with the education theme code were 118 present in the sample (as noted in table A.3). Finally, IEG consulted with the Education GP to gather views on the final sample selected. On the basis of the feedback provided, IEG selected cases across all Regions, rather than focusing on the Regions with the largest concentration of learning poverty. Figure A.1 and figure A.2 are graphical depictions of the analysis done for the countries considered. In figure A.1, the countries are ordered by the NER for primary education and spending per pupil at the primary level (in real terms, US dollars). Countries below the regression line are less efficient, although this does not always mean that they are spending a lot on primary education. For example, Brazil and Uruguay have achieved near-universal primary NER at a higher cost than Mexico, which makes them less efficient. However, the real concerns are for the countries in the bottom left-hand quadrant. These countries have very low NERs in primary education and are attaining NERs that are much lower than other countries spending a similar amount per pupil. For example, the Central African Republic, Mali, and Niger have very low NERs, but they are spending a similar amount per student. One likely explanation is “overage” inefficiency, where there are many children enrolled in these systems, but they are overage (that is, they have higher grade enroll- ment rates). The high-efficiency countries in figure A.2 are those that have attained NERs of nearly 100 percent but at low levels of spending, among them Georgia, Sierra Leone, and Viet Nam. Independent Evaluation Group World Bank Group    119 Figure A.2 continues with a plot of LAYS versus primary spending per pu- pil. The results in both figures are similar for many countries, but there are notable exceptions (for example, Kenya has low efficiency for NER but high efficiency for learning). Once again, the regression line provides the cutoff for relatively higher and lower efficiency, and it may be possible to break the countries down further based on their levels of spending (for example, high efficiency–high spending versus low efficiency–high spending). The numbers in figure A.1 were obtained from simple regressions for the two NER measures (primary and lower secondary) and the LAYS measure. In each case, all the countries with data were incorporated (about 130 countries), and the outcome was regressed onto the measure of average primary-secondary spending per pupil (figure A.1). Each dependent variable was standardized, and the numbers in figure A.2 represent the residual for each country. For example, Niger’s primary NER was 3.03 standard deviations below the inter- national average when controlling for level of spending per pupil. By contrast, Viet Nam’s LAYS measure was 1.36 standard deviations above the internation- al average when controlling for level of spending on basic education.  rimary Net Enrollment Rates Versus Primary Figure A.1. P Spending per Pupil, 2010–19 Source: Independent Evaluation Group. Note: NER = net enrollment rate; USD = US dollar. Confronting the Learning Crisis Appendix A The team increased the sample to 10 countries (the Approach Paper noted 9 countries [World Bank 2022]), so that enough countries in Africa could be selected (which will support an ongoing engagement IEG has with the Africa Region), while also including 1 country supported only by World Bank ASA. The purposively selected cases are Brazil, Chad, Ethiopia, Iraq, Kenya, Nepal, Pakistan, Sierra Leone, Tajikistan, and Viet Nam. Countries in the sample reflect a minimum threshold of World Bank lending and projects. The sample has a range of institutional characteristics (that is, eight of the nine possible capacity; efficiency cells). The sample also contains other characteristics, such as COVID-19 lending; presence of Global Partnership for Education projects; World Bank accelerator countries; and fragility, conflict, and violence. The 120 final sample contained cases with government commitment to achieving learning outcomes and financial commitment for basic education, such as Brazil, Kenya, Sierra Leone, and Viet Nam, combined with other cases with less financial and government commitment to achieving learning for all in basic education, which may help explain more successful and less successful cases.  earning-Adjusted Years of Schooling Versus Primary Figure A.2. L Spending per Pupil, 2010–19 Independent Evaluation Group World Bank Group    121 Source: Independent Evaluation Group. Note: USD = US dollar. Collection and analysis. The case studies were supported by a protocol informed by background research and the conceptual framework. The ques- tions in the protocol were designed to facilitate comparable answers across cases. The methods team reviewed the protocol, and feedback was applied that improved the clarity of wording and question types to facilitate consis- tent and comparable data across cases. For example, questions requiring an assessment of presence or absence were accompanied by an explanation of the assessment based on the type of evidence required to support the judg- ment. Questions that required assessment were accompanied by a rating scale to give case evaluators a basis for the rating. The first phase of the protocol was used to develop a contextual understand- ing of the constraints to education quality in that country. A wide range of data sources were reviewed: secondary data of official development assistance educational expenditures; population demographics to examine trends in enrollment and implications for teachers; harmonized learning outcome data and international and regional learning assessment data, where available, and other educational indicators; research published by international financial institutions; education sector strategies; identification of influential stake- holders; and other country-relevant studies or media. The second phase of the protocol assessed the World Bank’s support and contribution to improvements in education quality and the education system. The protocol also developed a nested design to examine not only the system but also the specific activities supported by the World Bank. This assessment was based on multiple sources of information, such as World Bank documents, including Systematic Country Diagnostics, Country Partnership Frameworks, and Project Appraisal Documents, and ASA relevant to the education sector (for example, expenditure reviews or technical work on particular aspects such as teachers and teaching or curriculum), as well as interviews with staff, government clients (at the central and decentral levels), and other relevant stakeholders. Interviews were conducted via virtual platforms, except where an in-person interview was needed because of lack of virtual access or pref- erence, in which case the local consultant conducted the interview. This assessment examined all aspects of the World Bank’s support and engage- ment from country strategies, diagnostics, and indicators to project-level Confronting the Learning Crisis Appendix A activities and indicators. The final phase of the case study was an overall assessment of the World Bank’s contribution to improvements in education quality and learning and extent to which it employed a systems approach in its support and whether it learned from experience and evidence. Case study authors worked collabora- tively to complete the assessment to ensure adequate understanding of local context. Several aspects aided the reliability of the data collected. Training on each section of the protocol was provided to case authors. Frequent meetings with 122 case study authors aimed to ensure consistent understanding and application of the questions in the protocols. The analysis team reviewed protocols to ensure that authors answered the intended question with adequate evidence to support responses and assessment. When needed, responses were further strengthened after the initial feedback from the analysis team. The responses to each question from the protocol for each of the cases were summarized into Excel from the original data entry by case study authors in SurveyMonkey and collated for ease of analysis. Cross-case analysis was used to examine patterns and divergence. Then, other sources of evidence from the portfolio, background research, secondary data, and, where relevant, World Bank strategic documents were used to derive initial findings. At the final stage of analysis, findings from the deep dive of global and regional ASA (described in this appendix) and recent global programs were examined in re- lation to case study findings related to knowledge to examine patterns. Within the sample, recent initiatives, such as the Accelerator Program, were delayed when the World Bank shifted its global support to generate new knowledge related to COVID-19 and did not feature strongly in relevant case study inter- views. Similarly, programs during the earlier decade did not feature strongly in interviews (with changes in ministry staff over the decade); evidence of use of ASA was found in documentary review by case study authors—consistent with the findings derived from the deep dive of ASA. Initial findings were shared with case study authors to refine and ensure accuracy. Findings from the anal- ysis also contributed to refinement of the conceptual framework. Portfolio Review Analysis: Financial Operations Independent Evaluation Group World Bank Group    123 The objective of the portfolio analysis was to help evaluate the effectiveness of World Bank support for basic education in addressing the key barriers hindering the achievement of improved learning outcomes in client countries. Furthermore, the review aimed to contribute to an assessment of the World Bank’s readiness to tackle the emerging challenges that education systems face because of the COVID-19 pandemic. The analysis aimed to answer the following questions: » What binding constraints do the World Bank operations seek to address? » Do the objectives of World Bank projects address improving basic education quality and learning outcomes? If so, what type of indicators are used to mea- sure progress toward these objectives? » What types of inputs has the World Bank supported for basic education? What types of indicators are used to measure education quality and learn- ing? What types of indicators are used to measure inputs supporting student learning assessment and in-service training? » For each type of input, are there specific groups targeted by the World Bank operations? » Do the project indicators capture the target groups of the World Bank operations? » What inputs has the World Bank supported to address challenges to the basic education sector caused by the COVID-19 pandemic? » What types of lessons learned have been captured by the Project Appraisal Documents? Portfolio identification. IEG’s identification methodology used the World Bank’s sector and theme codes and relevant World Bank databases, together with a manual review, to systematically capture and categorize the relevant portfolio. The portfolio identification consisted of five steps: 1. Identification of all active or closed lending projects in the Education GP approved since 2012. Confronting the Learning Crisis Appendix A 2. Development and application of exclusion criteria based on a review of project name, project development objective, sector coding, and a list of nonbasic education keywords related to early childhood, upper secondary, tertiary, vocational, and adult education.2 3. Identification of relevant World Bank theme and sector codes (“primary education,” “secondary education,” “public administration—education,” and “other education”). 4. Refinement of portfolio by further review of these documents and ex- cluding projects with sector codes (i) “public administration—education” or “other education,” and (ii) “early childhood education” or “tertiary 124 education” or “workforce development and vocational education” or “adult, basic and continuing education,” but without (iii) “primary educa- tion” and “secondary education.” 5. Manual coding of project documents further refined the portfolio when a small number of irrelevant projects were identified and subsequently eliminated from the portfolio. Additional financing projects without parent projects in the portfolio were removed—that is, additional financing for projects approved before August 2012. During the manual coding of project documents, a small number of unqualified projects were identified and eliminated from the portfolio. This process yielded 236 relevant projects. The portfolio analysis concentrates on World Bank lending projects for basic education approved since FY12 (as of August 2022). In total, 236 projects were identified, consisting of investment project financing, development policy financing, and Program-for-Results projects, including additional financing (figure A.3 and figure A.4). Out of the 236 projects in the portfolio, the median commitment size is $45 million, whereas the average commit- ment size is $106 million.  umber of Projects by Region and Global Partnership for Figure A.3. N Education Financing Independent Evaluation Group World Bank Group    125 Western and Central Africa Eastern and Southern Africa Latin America and Caribbean Region South Asia East Asia and Pacific Europe and Central Asia Middle East and North Africa 0 10 20 30 40 50 60 70 Projects (no.) GPE only GPE cofinancing Non-GPE Source: World Bank Data Explorer; lending project documents. Note: GPE = Global Partnership for Education. Out of the 236 projects identified, 25 (11 percent) were tagged with noned- ucation sectors. As shown in figure A.4, among the 25 multisector projects, social protection, public administration, information and communication technology, and health are the most notable sectors.  oneducation Sector Tagged in Multisector Projects Figure A.4. N Social protection Public administration Sector ICT Health Other 0 5 10 15 20 25 30 35 40 45 Share of projects (%) Source: World Bank Data Explorer. Note: ICT = information and communication technology. Analysis of the continuity and intensity of engagement with basic education in individual countries was also undertaken. Excluding COVID-19–specific support, World Bank support for basic education consisted of a single operation in 38 countries and two operations in 22 countries. In a further 11 countries, support for basic education consisted of a single COVID-19 emergency response operation—that is, the World Bank was not otherwise supporting basic education reform in those countries during the evaluation period (table A.4). All but Sri Lanka had discontinuities in engagement based Confronting the Learning Crisis Appendix A on the number of missing FYs in the data. 126  orld Bank Project Investments Lacked Continuity and Intensity Table A.4. W Number of Projects Volume of Projects (US$, billions) Basic Basic Non- Basic Edu. edu. Non- Basic Edu. edu. FY Learning Country or Economy Total edu. Edu. edu. (%) (%) Total edu. Edu. edu. (%) (%) Missing Poverty Afghanistan 81 75 6 2 7 33 7.45 7.20 0.25 0.15 3.3 59.5 7 93.45 Angola 22 20 2 2 9 100 5.51 5.19 0.32 0.32 5.9 100.0 2 — Argentina 47 45 2 1 4 50 12.47 11.59 0.88 0.24 7.1 27.4 3 59.15 Armenia 61 58 3 2 5 67 1.16 1.07 0.08 0.03 7.3 34.8 8 27.24 Belarus 20 16 4 2 20 50 1.02 0.84 0.18 0.18 17.9 98.2 4 — Burundi 32 30 2 1 6 50 1.32 1.20 0.12 0.04 9.1 33.3 6 95.82 Cabo Verde 28 26 2 1 7 50 0.52 0.49 0.04 0.01 6.8 26.8 7 — Central African Rep. 39 37 2 2 5 100 1.45 1.39 0.06 0.06 3.9 100.0 6 — China 141 136 5 1 4 20 16.94 16.56 0.38 0.12 2.3 31.4 6 18.20 Colombia 54 50 4 1 7 25 13.25 12.53 0.72 0.08 5.4 11.1 10 51.36 Congo, Rep. 33 30 3 1 9 33 1.08 1.01 0.07 0.03 6.5 42.9 4 70.01 Côte d’Ivoire 58 51 7 2 12 29 6.35 5.92 0.44 0.07 6.8 15.8 1 82.62 Croatia 20 19 1 1 5 100 2.05 2.03 0.03 0.03 1.4 100.0 10 4.46 Ecuador 24 22 2 1 8 50 5.83 5.65 0.18 0.13 3.0 70.6 4 65.94 Egypt, Arab Rep. 37 36 1 1 3 100 13.44 12.94 0.50 0.50 3.7 100.0 6 69.57 (continued) Independent Evaluation Group World Bank Group    127 128 Confronting the Learning Crisis Appendix A Number of Projects Volume of Projects (US$, billions) Basic Basic Non- Basic Edu. edu. Non- Basic Edu. edu. FY Learning Country or Economy Total edu. Edu. edu. (%) (%) Total edu. Edu. edu. (%) (%) Missing Poverty El Salvador 14 12 2 1 14 50 1.19 0.87 0.32 0.06 27.2 18.4 3 69.07 Georgia 44 40 4 2 9 50 2.58 2.08 0.50 0.50 19.5 99.8 7 15.29 Guatemala 19 18 1 1 5 100 1.93 1.93 0.00 0.00 0.0 100.0 8 78.52 Guinea 40 36 4 1 10 25 1.47 1.35 0.11 0.04 7.6 34.7 6 82.67 Guinea-Bissau 17 15 2 2 12 100 0.28 0.26 0.02 0.02 6.3 100.0 7 — Indonesia 113 111 2 1 2 50 22.34 22.02 0.32 0.25 1.4 78.2 7 52.82 Iraq 22 20 2 1 9 50 5.00 4.98 0.02 0.01 0.3 66.7 10 — Kazakhstan 21 18 3 1 14 33 4.47 4.41 0.06 0.02 1.3 30.1 5 2.19 Kosovo 27 26 1 1 4 100 0.48 0.48 0.01 0.01 1.4 100.0 5 — Kyrgyz Republic 66 61 5 1 8 20 1.37 1.28 0.09 0.02 6.2 19.2 4 64.49 Lao PDR 59 55 4 2 7 50 1.45 1.31 0.14 0.06 9.4 46.8 3 97.72 Lebanon 34 32 2 2 6 100 2.24 2.01 0.23 0.23 10.2 100.0 3 — Madagascar 65 62 3 2 5 67 5.01 4.74 0.26 0.18 5.2 67.4 6 93.94 Malawi 53 47 6 2 11 33 3.91 3.47 0.44 0.19 11.3 44.1 5 — Maldives 24 21 3 2 13 67 0.33 0.29 0.03 0.01 10.1 38.8 2 — Mali 51 47 4 2 8 50 2.55 2.28 0.27 0.18 10.6 65.5 3 90.44 (continued) Number of Projects Volume of Projects (US$, billions) Basic Basic Non- Basic Edu. edu. Non- Basic Edu. edu. FY Learning Country or Economy Total edu. Edu. edu. (%) (%) Total edu. Edu. edu. (%) (%) Missing Poverty Mauritania 35 32 3 2 9 67 0.95 0.88 0.08 0.06 7.9 85.0 3 94.77 Mexico 41 36 5 2 12 40 8.41 7.21 1.20 0.47 14.3 39.0 6 47.63 Mongolia 45 40 5 1 11 20 1.01 0.97 0.05 0.03 4.5 65.6 2 39.46 Morocco 67 63 4 2 6 50 12.59 11.29 1.30 0.85 10.3 65.4 5 64.90 Mozambique 95 88 7 1 7 14 7.37 6.73 0.63 0.30 8.6 47.2 9 — Niger 59 55 4 2 7 50 5.29 4.71 0.58 0.45 11.0 77.6 3 90.44 North Macedonia 22 20 2 1 9 50 1.28 1.23 0.05 0.03 3.8 51.3 9 40.86 OECS countries 11 10 1 1 9 100 0.19 0.19 0.00 0.00 0.9 100.0 7 — Peru 43 39 4 1 9 25 6.03 5.58 0.45 0.03 7.5 5.6 5 44.44 Philippines 70 69 1 1 1 100 14.13 13.83 0.30 0.30 2.1 100.0 5 90.91 Romania 19 18 1 1 5 100 6.66 6.42 0.24 0.24 3.6 100.0 3 17.86 Somalia 49 47 2 1 4 50 3.04 2.95 0.09 0.04 3.0 43.5 9 — Sri Lanka 45 40 5 2 11 40 4.02 3.58 0.44 0.24 11.0 53.9 0 14.81 Sint Maarten 14 12 2 1 14 50 0.50 0.46 0.03 0.03 6.3 85.6 10 — Tajikistan 60 56 4 1 7 25 1.84 1.73 0.11 0.02 5.8 15.1 6 — Timor-Leste 15 12 3 2 20 67 0.36 0.33 0.03 0.03 8.1 89.8 3 — (continued) Independent Evaluation Group World Bank Group    129 130 Confronting the Learning Crisis Appendix A Number of Projects Volume of Projects (US$, billions) Basic Basic Non- Basic Edu. edu. Non- Basic Edu. edu. FY Learning Country or Economy Total edu. Edu. edu. (%) (%) Total edu. Edu. edu. (%) (%) Missing Poverty Tunisia 46 44 2 1 4 50 5.76 5.62 0.14 0.07 2.4 49.6 6 65.53 Türkiye 56 55 1 1 2 100 15.02 14.86 0.16 0.16 1.1 100.0 8 14.50 Tuvalu 21 20 1 1 5 100 0.23 0.22 0.01 0.01 6.1 100.0 8 — Ukraine 48 46 2 1 4 50 36.80 36.60 0.20 0.00 0.5 0.2 7 27.88 West Bank and Gazaa 72 67 5 1 7 20 1.32 1.27 0.05 0.02 3.7 41.2 10 — Yemen, Rep. 52 49 3 2 6 67 5.82 5.64 0.18 0.18 3.1 98.4 4 — Zambia 42 39 3 2 7 67 2.52 2.12 0.39 0.35 15.6 90.0 6 98.50 Zimbabwe 18 17 1 1 6 100 0.25 0.25 0.00 0.00 0.1 100.0 10 — Source: Independent Evaluation Group. Note: edu. = education; FY = fiscal year; — = not available; OECS = Organisation of Eastern Caribbean States; PDR = People’s Democratic Republic. a. West Bank and Gaza as an “economy.” Coding and analysis. First, the team acquired all available Project Appraisal Documents and project papers for additional financing and emergency proj- ects. The REST API was queried for document URL links on the World Bank’s external site for documents and reports, followed by bulk downloading using Python code. For closed operations, Implementation Completion and Results Reports were downloaded. Second, the team coded and extracted data from the selected projects using coding protocols developed through insights from literature reviews and aligned with evaluation questions. To analyze the content of the documents, the team employed NVivo software and conducted structured coding. IEG adopted a deductive coding approach, and the team developed a list of codes to be applied to each question and identified the relevant sections of the documents to examine during coding. To ensure consistency and reliability in the coding process, multiple coders reviewed one another’s work, provided feedback, and engaged in discussions to ensure intercoder reliability. In addi- tion, the definition of the coding question was further reviewed and refined during the coding process. The coding protocol extracted data related to com- mitments; binding constraints (based on taxonomy); project development objectives; manual review of the results framework to examine the extent to which the indicators measured toward project development objectives and data are disaggregated; inputs and project activities (based on taxonomy); specific groups targeted based on a review of project activities and beneficia- Independent Evaluation Group World Bank Group    131 ries; and inputs supported to address challenges to the basic education sector caused by the COVID-19 pandemic. Implementation Completion and Results Reports were reviewed particularly in relation to efficacy and monitoring and evaluation to examine indicators, results, and evaluations conducted. Third, IEG used a machine learning technique (unsupervised learning with “GPT-3.5-Turbo” model) to identify lessons contained in Project Appraisal Documents. The results were validated through manual review. The lessons learned were as follows. Realism. Streamlining project design with well-defined objectives is essential for successful execution, particularly in settings with limited resources. Adapting education service delivery to suit the unique circumstances of each country and leveraging available assets and infrastructure can significantly improve efficiency. Taking into account local contexts, fostering flexibility, and adopting phased approaches are key factors in ensuring the smooth implementation of projects. Making use of existing administrative agencies for project execution can leverage established systems, leading to maximum benefits. Implementing projects incrementally promotes sustainability, and evaluating institutional capacity helps mitigate risks when scaling up programs. Adaptability and flexibility. Regular assessments and readjustments en- able project teams to stay responsive and adapt to changing circumstances. Monitoring and adjusting strategies based on the evolving situation is crucial for successful project implementation. This flexibility becomes even more critical when operating in postrecovery contexts and low-capacity environ- ments. In addition, it is essential to adapt project implementation strategies to address evolving security risks, geography, and logistics. By remaining flex- ible and providing technical assistance, project teams can effectively adjust results and overcome challenges as they arise. Monitoring and evaluation quality. Establishing robust monitoring and evaluation systems is crucial to effectively track and evaluate project out- comes. By prioritizing the development of a comprehensive monitoring and evaluation system, project teams can ensure timely reporting and data-driven decision-making. This includes strengthening data collection, monitoring, and evaluation systems to support informed decision-making at every stage. It is essential to use credible and independent data for program design, im- Confronting the Learning Crisis Appendix A plementation, and financing decisions. Addressing low-capacity constraints in monitoring and evaluation is critical. Capacity building. To ensure the long-term success of projects, it is crucial to invest in institutional strengthening, capacity building, and technical exper- tise. One key aspect of this is strengthening public financial management systems, which is essential for complete reliance on government systems. In addition, establishing efficient and adequately staffed project coordination units is critical for maintaining continuity, especially in the event of govern- ment turnover. 132 Collaboration between stakeholders and donors. Collaboration between the government and stakeholders is essential for successful implementa- tion because it fosters shared responsibility and enables collective action. Coordination between development partners and the government reduces parallel financing and facilitates joint planning efforts. Joint annual sector reviews promote coordination and quality implementation. Moreover, it is important to prioritize consensus-driven initiatives and involve stakeholders in decision-making processes. This approach creates a sense of ownership and promotes sustainable change within the education system. Country ownership. Country ownership and alignment with government strategies and programs are essential for successful project implementation. Prioritizing ownership and aligning project actions with the government’s sector plan ensures coherence, efficiency, and sustainability. By coordinating with relevant ministries and stakeholders, projects can contribute to national strategies and institutional mechanisms, fostering synergy and maximizing impact. Close cooperation and alignment with government priorities are cru- cial for program success and long-term sustainability. Decentralization and community participation. Decentralization and com- munity engagement play a vital role in transforming the education system. The decentralized provision of education inputs and enhancing procurement capacity at regional and local levels can reduce reliance on central procure- Independent Evaluation Group World Bank Group    133 ment units and ensure efficient resource allocation. Involving communities through management committees empowers them to actively participate in school oversight, leading to improved service delivery and accountabili- ty. Community-based education models, such as establishing classrooms in villages, enhance accessibility and quality. Tailored school grant programs further strengthen local accountability, ensuring that funds are used effec- tively. Engaging stakeholders, including parents, teachers, and communities, fosters a collaborative approach to project implementation and monitoring. Teacher recruitment and training. Incentives for teacher motivation and performance are crucial in attracting and retaining high-quality educators. For example, expanding teacher hardship allowances can help recruit and keep qualified teachers in remote areas. Continuous professional development op- portunities are essential for improving teaching quality. Tailored training and coaching based on individual assessments and subject specializations are more effective than generalized approaches. In-service and preservice teacher training should be tailored to classroom realities while maintaining high quality. Student assessment. A robust assessment system is crucial for gathering and analyzing student learning data. Establishing a fair, valid, and reliable student assessment and examination system that aligns with the curriculum is essential. Implementing competency-based assessments can lead to bet- ter understanding of students’ learning levels and abilities. Moreover, there is a pressing need to modernize learning assessments to meet international standards and strengthen the capacity for conducting national learning as- sessments at the local level. Portfolio Review of a Sample of World Bank Education Global and Regional Analytics and Global Programs Objective. The objective of the global and regional ASA portfolio analysis was to help evaluate the effectiveness of World Bank support for basic education in addressing the key barriers that hinder the achievement of improved learn- ing outcomes in client countries. The evaluation examined the World Bank’s role as convenor and its collaboration with global partners and how well glob- al knowledge and programs are generating knowledge and building awareness about how to improve quality and learning outcomes in basic education and to strengthen education systems. Furthermore, the review aimed to contrib- Confronting the Learning Crisis Appendix A ute to an assessment of the World Bank’s use of feedback and evidence to inform its work and tackle the emerging challenges that education systems face as a result of the COVID-19 pandemic. Selection criteria and process. IEG’s identification methodology used the World Bank’s sector and theme codes and relevant World Bank databases, such as Standard Reports, IEG Datamart, Enterprise Development Catalogue, and Operations Portal, together with a manual review, to systematically 134 capture and categorize the relevant portfolio. The ASA portfolio identification consisted of the following steps: 1. Identify all ASA in the Education GP that were approved since 2012. 2. Conduct a word search of titles to screen for relevance to basic education; this eliminated technical and vocational education and training, skills, early childhood development, and tertiary education. 3. Review objectives and description of ASA in the Operations Portal to elim- inate the ASA not relevant to basic education. 4. Search additional databases and resources to identify impact evaluations financed by the World Bank and the Strategic Impact Evaluation Fund; the World Bank’s website was used to identify recent partnerships, programs, and initiatives. 5. Manually code all the documents; this further reduced the portfolio. The final portfolio examined consisted of 145 products classified by the World Bank as global and regional that address basic education, of which 88 are ASA and 57 are impact evaluations.3 The ASA consisted of a mix of training, advisory work, analytic work, tools and reports, and meetings. The search also yielded 21 partnerships and programs. A purposeful sample of ASA and partnerships and programs was selected for Independent Evaluation Group World Bank Group    135 the deep dive using the following steps. All the identified ASA and partner- ships were categorized as follows: » Three themes—measurement of learning outcomes, teaching and learning, and education system strengthening and alignment—consistent with the WDR 2018 (World Bank 2018). » Two time periods—pre-WDR (2012–17) and post-WDR (2018–22). » Type of ASA—world and regional. This allowed for the selection of a sample based on themes, period (pre- and post-WDR), and global or regional type. ASA conducted under these themes that were implemented to address the challenges to education systems arising from the COVID-19 pandemic were also selected. Multiyear and multifaceted ASA with substantive knowledge contributions, such as the Systems Approach for Better Education Results program and the WDR 2018, were selected. ASA related to financing of education and resource mobiliza- tion for education were eliminated to maintain focus. This resulted in a total sample of 30, with 10 ASA and programs related to measurement of learning, 11 ASA and programs related to teaching and learning, and 7 ASA and pro- grams related to system strengthening and alignment, as well as the WDR (table A.5).  ample of Advisory Services and Analytics and Programs Table A.5. S Pre-WDR Post-WDR Date ASA Date ASA Measurement of learning 2008–15 READ phase 1 June 2019 First education analytics 2016–20 READ phase 2 May 2022 Second education ana- lytics 2009–13 READ Global Instrument October 2019 Learning poverty indi- Development cator 2009–13 EGRA in Pacific June 2021 Learning Data Compact September– 6th Europe and 2022 READ suspended November 2011 Central Asia Education Conference July 2019 Learning Assessment Platform June 2020 National learning assess- Confronting the Learning Crisis Appendix A ments; policy linking tool kits: Gambia, The; Ghana Teaching and learning 2011–12 SABER Teachers Country 2019–ongoing Coach: Helping Countries and Regional Reports Accelerate Learning 2016–18 Facing Forward: Schooling 2020–ongoing Teach: Measuring for Learning in Africa Teaching Practices 2010–14 Latin America and the 2019–ongoing Global Platform for Caribbean Regional Successful Teachers Study (and launch con- ference) (continued) 136 Pre-WDR Post-WDR Date ASA Date ASA 2016–19 SABER Teachers Work 2018–19 Ready to Learn, Ready to Program: Measuring Thrive: South Asia Teacher Effectiveness 2015–22 Results in Education for All Children 2020–22 Global Grant for Innovation and Inclusion during COVID-19 2021–22 Continuous and Accelerated Learning in Response to COVID-19 Education systems 2014–16 Africa Out-of-School 2017–19 SABER Coordination Youth Program report Application 2 (Annual Reports 2018, 2019, 2020) 2015–17 SABER–EMIS 2018–21 Global Education Policy Dashboard Partnership and ASA 2011 EduStats 2020–ongoing Foundational Learning Compact umbrella trust fund 2020 Accelerator Program roundtable event 2019–ongoing Learning Assessment Independent Evaluation Group World Bank Group    137 Platform 2020–ongoing National learning assess- ments; policy lending tool kits: Gambia, The; Ghana Source: Independent Evaluation Group; Bashir et al. 2018. Note: ASA = advisory services and analytics; EduStats = Education Statistics; EGRA = Early Grade Reading Assessment; EMIS = Education Management Information Systems; READ = Russia Education Aid for Development; SABER = Systems Approach for Better Education Results; WDR = World Development Report. Coding and analysis. For the coding of all ASA, information was obtained from the World Bank’s Operations Portal, specifically the products’ artic- ulated objective statement and product description and, if available, the executive summary of the completed product. A deductive approach was used to develop a taxonomy to categorize the three ASA themes and content of the ASA. In addition, coding also captured how equity and inclusion were addressed in the ASA. Basic data were extracted related to cost, dates, coun- try, Region, and product line. Impact evaluations were coded based on a classification of interventions, outcome measures, and attention to equity. For the deep dive of ASA and programs, IEG reviewed all available and rel- evant documents found in the Operations Portal and additional documents provided by key informants in the World Bank to review and code. The coding protocol aligned with the evaluation questions and conceptual framework. Information included objectives, binding constraints, genesis of the ASA, value added by the World Bank, partner collaboration, outputs, dissemination, and lessons. The coders discussed and reviewed one another’s work to ensure that the extraction of information and assessment was conducted similarly across the ASA. The information extracted from documents and interviews was compiled and analyzed qualitatively. The analysis also involved contrast- ing the qualitative findings with key World Bank strategic documents and literature collected for the evaluation. IEG also conducted interviews with task team leaders, practice managers, and partners to supplement the coding and analysis exercise and to ensure accu- racy of interpretation. Interviews and Consultation with Stakeholders There was broad stakeholder engagement over the course of the evaluation. First, the evaluation’s approach, scope, and findings were discussed with 29 Confronting the Learning Crisis Appendix A staff, managers, and directors from the Education GP to gather feedback and further information. Second, interviews (individual and group) were held with task team leaders and other stakeholders, including staff from ministries of education at the central and other levels, during IEG’s virtual case study missions. IEG also interviewed staff and partners to supplement the information found in docu- ments and verify information for the deep dive analysis of global and regional ASA. World Bank staff were extremely responsive to follow-up questions and provided additional documents, as requested. IEG interviewed 296 people for the case studies and deep dive (table A.6). 138 Selection criteria and process. Key informants were identified based on the sample selected for the case studies and deep dive of global and regional ASA and programs. Global leads were also selected to provide overall evaluation feedback, given their pivotal role in the GP. Interview requests were sent to task team leaders, and a follow-up was sent, if needed.  umber of Interviews by Interviewee Category during the Table A.6. N Evaluation External or Internal Interviews Interviewee Category Interviews (no.) Internal Staff and task team leaders 84 Internal Project managers 15 External Government 112 External Development partners 52 External CSOs and INGOs 33 Source: Independent Evaluation Group. Note: CSO = civil society organization; INGO = international nongovernmental organization. Collection and analysis. Key informant interviews were conducted using a structured set of questions. Notes were taken of interviews with World Bank Independent Evaluation Group World Bank Group    139 staff and other stakeholders, which were an input (among other sources of evidence) into answering the questions in the case study protocols and deep dive of global and regional ASA. Thus, the final outputs of the deep dive ASA analysis and case studies were triangulated with all sources of evidence. Quantitative Analysis of Secondary Data Selection and process. The quantitative analysis used the following data sets: » Actual enrollment trend from 2000 through 2021 using United Nations Educational, Scientific, and Cultural Organization (UNESCO) Institute for Statistics data for the 10 case study countries. » Actual and projected population totals for primary and lower secondary age- groups in each country for 2000–21 (actual) and 2022–40 (projected) from United Nations population data. » Gross enrollment ratio (GER) for 2000–21 based on reported enrollments and the United Nations–reported population total for 2006–11. » International assessments in which countries subject to case studies par- ticipated: Brazil (Program for International Student Assessment and the Progress in International Reading Literacy Study), Pakistan (Trends in International Mathematics and Science Study), and Viet Nam (Program for International Student Assessment). » Regional assessments in which countries subject to case studies participated: Laboratorio Latinoamericano de Evaluación de la Calidad de la Educación (Latin America, only Brazil); the Southern and Eastern Africa Consortium for Monitoring Educational Quality (Africa, only Kenya); the Programme for the Analysis of Education Systems (Francophone Africa, only Chad); and the Southeast Asia Primary Learning Metrics (Southeast Asia, Viet Nam). » National assessment data for country case studies conducted between 2012 and 2022. » UNESCO Institute for Statistics data (2017–20). » UNESCO school closure data. » UNESCO–United Nations Children’s Fund–World Bank COVID-19 surveys, Confronting the Learning Crisis Appendix A rounds 2 and 3. Analysis. For demographic analysis of impact on future enrollment in the 10 case study countries, the calculations for student enrollments followed these steps: 1. Obtain the actual enrollment trend for 2000–21 using data from the UNESCO Institute for Statistics, although even these basic enrollment data are incomplete in some cases. 140 2. Obtain the actual and projected population totals for primary and lower secondary age-groups in each country for 2000–21 (actual) and 2022–40 (projected) from the United Nations population data. 3. Calculate an updated GER for 2000–21 based on reported enrollments and the United Nations–reported population total. The results for this “derived” GER are very similar to the reported GER in all four countries. This step is necessary to be consistent with the GER calculation and trend across the entire 40-year period because the UNESCO Institute for Statistics–reported figures are often based on somewhat different popula- tion numbers than the United Nations numbers. 4. Define three future growth scenarios for each country based on GER (and NER) trends in the past 10 (or 20) years. This is the most subjective aspect of the work as the enrollment ratios can move around a bit, or they are incomplete. In general terms, the three scenarios include a “steady state,” where the most recent GER is maintained for the next 20 years (unlikely); a relatively rapid GER future growth rate (1–3 percent per year); and a slower GER growth rate (0.5–1.5 percent per year). 5. Calculate projected student enrollments by level for 2022–40 for each of the three scenarios by multiplying each GER rate by the projected popu- lation total for that school level by year, and graphically display the three projected enrollment totals for 2022–40 next to the actual enrollment Independent Evaluation Group World Bank Group    141 trend line for 2000–21. For the projection of the number of teachers needed in the case study country based on demographic changes and projection for student enrollment growth, the analysis involved the following: » Calculating the student-teacher ratio (STR) using reported enrollments and teacher numbers for the 2000–21 period. » Calculating updated ratios based on three scenarios created: a steady-state scenario, where the most recent STR is maintained for the next 20 years; a “teacher increase” scenario, where the STR is projected to steadily decline by five students per teacher (on average) and then level off; and a “teacher decrease” scenario, where the STR is allowed to steadily increase up to five more students per teacher and then level off. » For learning outcome data for case study countries, the scores were extract- ed and presented based on period and subject for international and regional assessment. This information was then compared with results from national assessment data related to country learning to assess how consistent this information is with the international and regional data (when applicable) and what kind of trends are notable in the study period (2012–present). The goal for this section was also to identify assessment results in at least gener- ally comparable formats (that is, proficiency levels); however, for this initial review, this standard was not met. Basic statistics were prepared related to out-of-school youth using UNESCO Institute for Statistics data (2017–20) and related to school closure using UNESCO school closure data and UNESCO–United Nations Children’s Fund– World Bank COVID-19 surveys, rounds 2 and 3. Limitations The evaluation has several limitations—among them, limitations associated with remote qualitative data collection, limitations with the generalizability of case data, and limitations associated with structural choices. Limitations posed by remote qualitative data collection. To mitigate this risk, the team worked closely with the Education GP and the Country Confronting the Learning Crisis Appendix A Management Unit in each of the countries selected to engage an experi- enced local consultant to ensure that the team interacted, although remotely, with stakeholders. The Country Management Unit worked with the local consultant to ensure that case study authors interviewed as many relevant stakeholders as possible to gain as broad a perspective as possible. The team supplemented interviews with a range of other data sources—including re- search, media publications, and secondary data—to understand the barriers to learning for all in the context. The evaluation team recognizes that case study findings may not be generalizable because findings may be contextual. The team sought to 142 mitigate this challenge by triangulating findings with global knowledge, literature, and evidence from the portfolio review. The team also applied robust case selection criteria to select comparable cases and avoid selecting unique cases. The team developed and implemented case studies with a com- mon protocol and process to review case studies to foster consistent answers to the question and promote reliable data for analysis. Some structural evaluation choices have a bearing on the nature of this evaluation. First, IEG limited its field travel to reduce its carbon footprint. Evaluators interviewed stakeholders via videoconference platforms and were paired with experienced local evaluators. The pair worked closely with the Education GP and Country Management Unit in each of the 10 countries se- lected to ensure that interactions produced the needed information. Because the evaluation focuses on systems, it did not engage with the school level or with student beneficiaries of education. It engaged with teachers only through representative organizations, such as trade unions. The evaluation scope ex- cludes early childhood development. The evaluation recognizes that the World Bank and its partners are engaged in a continuum of interrelated support at this critical stage of life; however, to focus the evaluation, the scope was limit- ed to basic education and education systems to ensure robust findings. References Bashir, Sajitha, Marlaine Lockheed, Elizabeth Ninan, and Jee-Peng Tan. 2018. Facing Independent Evaluation Group World Bank Group    143 Forward: Schooling for Learning in Africa. Washington, DC: World Bank. https:// openknowledge.worldbank.org/handle/10986/29377. World Bank. 2018. World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank. World Bank. 2019. Ending Learning Poverty: What Will It Take? Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/32553. World Bank. 2022. Confronting the Learning Crisis: Lessons from World Bank Support for Basic Education, 2012–22. Approach Paper. Independent Evaluation Group. Washington, DC: World Bank. 1  The Worldwide Governance Indicators are a research data set summarizing the views on the quality of governance provided by a large number of enterprise, citizen, and expert survey re- spondents in industrial and developing countries that are gathered from a number of survey institutes, think tanks, nongovernmental organizations, international organizations, and pri- vate sector firms and do not reflect the official views of the World Bank, its Executive Directors, or the countries they represent, and are not used by the World Bank to allocate resources. 2  Projects were excluded from the portfolio if they met any of the following conditions: » The project name contained nonbasic education keywords. » The project development objective contained nonbasic education keywords, and the project lacked “primary education” or “secondary education” sector codes. » No relevant sector codes were found among the tagged sectors. 3  Each advisory services and analytics product may include multiple outputs or products from each operational number. Confronting the Learning Crisis Appendix A 144 The World Bank 1818 H Street NW Washington, DC 20433