EDUCATION WORKING PAPER No. 7 | JANUARY 2025 Unlocking Potential: Enhancing Education Quality in the Western Balkans based on Insights from the PIRLS 2021 and PISA 2022 studies Tigran Shmis, Paul Cahu, Lucia Brajkovic, and James Gresham © 2025 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Email: AskEd@worldbank.org Internet: www.worldbank.org/en/topic/education This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy, completeness, or currency of the data included in this work and does not assume responsibility for any errors, omissions, or discrepancies in the information, or liability with respect to the use of or failure to use the information, methods, processes, or conclusions set forth. The boundaries, colors, denominations, links/footnotes and other information shown in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. The citation of works authored by others does not mean the World Bank endorses the views expressed by those authors or the content of their works. Nothing herein shall constitute or be construed or considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Cover design: Marianne Siblini Unlocking Potential: Enhancing Education Quality in the Western Balkans based on Insights from the PIRLS 2021 and PISA 2022 studies Tigran Shmis, Paul Cahu, Lucia Brajkovic, and James Gresham January 2025 Abstract Western Balkan countries are consistent participants in international learning assessments like the Program for International Student Assessment (PISA) and Progress in International Reading Literacy (PIRLS). This publication provides an analysis of the recently published data of the PISA 2022 and PIRLS 2021 studies. This study identified four key areas where the performance of Western Balkan education systems could be improved. They include (a) duration of learning in schools, (b) teaching practices, (c) school preparedness, and (d) use of smartphones in schools induced by COVID-related school closures that serve as a major distractor from learning. Based on the analysis, the publication provides policy options and recommendations on advancing the education systems of Western Balkan countries to improve learning outcomes that are critically important for the future well-being of students and the economic development of these countries. JEL: I21, I24, I25, I28, J24, L86, O1, Q01 Keywords: Education, International Learning Assessment, PISA, PIRLS, Learning Analytics, Western Balkans, Albania, Kosovo, Montenegro, North Macedonia, Serbia. 1 Table of Contents I. Acknowledgments ................................................................................................................... 5 II. Executive Summary ................................................................................................................. 7 III. Introduction: Navigating Challenges and Aspiring for Progress in the Western Balkans ....... 13 IV. PISA Student Results at the Lower Secondary Level: Understanding Underperformance..... 21 A. Low performance in the Western Balkans. ......................................................................... 21 B. Factors behind the performance gap................................................................................... 24 1. The role of inequalities in the performance of WeBa countries. ................................. 24 2. Performance gap in secondary and primary level. ...................................................... 28 3. Limited hours of schooling in WeBa. ........................................................................... 32 4. Gender gaps in student performance in Western Balkan countries. ........................... 35 5. School preparedness in the Western Balkans ............................................................. 36 6. Socio-emotional factors and relation to student outcomes. ....................................... 43 V. Teachers and Teaching Practices: Enhancing Quality and Efficiency ..................................... 45 VI. The roles of School Closures and the Digital Transition in Plummeting Learning Outcomes in the Western Balkans ................................................................................................................... 51 A. The global decline in education performance between 2018 and 2022 and its impacts on WeBa countries. ............................................................................................................................. 51 B. School closures had an adverse impact on scores. ............................................................ 54 C. Overuse of digital devices had a negative effect on students in all WeBa countries. .. 55 VII. Conclusion and Policy Recommendations ............................................................................. 60 References .................................................................................................................................. 63 Annex 1: Technical Appendix ...................................................................................................... 65 2 List of Figures Figure 1.1: WeBa countries achieve improvements in Learning Outcomes but Still Behind the EU Average ....................................................................................................................................... 16 Figure 1.2: Student and School-level Factors that Impact Learning Outcomes in the Western Balkans ........................................................................................................................................ 20 Figure 2.1: Average PISA 2022 Scores by Discipline .................................................................... 21 Figure 2.2: Distribution of 15-year-old Students by Proficiency Levels in Math and Reading in PISA 2022 .................................................................................................................................... 22 Figure 2.3: Gaps between Average Scores and What Might Be Expected Given Socioeconomic and Linguistic Factors in Equivalent Years of Schooling .............................................................. 24 Figure 2.4: A Scatterplot of Standard Deviations of Average Scores Versus the Mean Average Score on PISA 2022 ..................................................................................................................... 25 Figure 2.5: Social Stratification and Determinism in the Western Balkans.................................. 28 Figure 2.6: Gaps in Reading Score Compared with Other Participating Countries after Accounting for Differences in Socioeconomic Factors ................................................................ 29 Figure 2.7: The Performance Gap in Reading for PIRLS 2021 and PISA 2022 After Accounting for Socioeconomic and Linguistic Factors ......................................................................................... 31 Figure 2.8: Metrics Contributing to Quality All Above the Global Average at the primary level . 33 Figure 2.9: Share of Students in PISA 2022 Countries Reporting Having Skipped At Least One Class in the Previous Two Weeks ................................................................................................ 35 Figure 2.10: Gender Performance Gap in WeBa Countries Compared with OECD and non-OECD PISA Participants, PISA 2022 ....................................................................................................... 36 Figure 2.11: Marginal impact of participation in early childhood education programs (daycare) and pre-primary school on reading scores using PIRLS 2021 data in the WeBa and other participating countries. ............................................................................................................... 37 Figure 2.12: difference in PISA scores between students who did or did not participate in some form of ECEC. .............................................................................................................................. 38 Figure 2.13: Relationship between duration of ECEC and PISA results, all participating WeBa countries ..................................................................................................................................... 40 Figure 2.14: Preschool Attendance of Students Sampled in PISA 2022 ....................................... 41 Figure 2.15: Gross Enrollment Rates at the Pre-primary Level, 2010-2022. ................................ 42 Figure 2.16: Good School Preparedness in Reading in WeBa ...................................................... 43 Figure 2.17: Various PISA 2022 Metrics in the Western Balkans and EU Countries .................... 44 Figure 2.18: Socioemotional Skills as Measured in PISA 2022 ..................................................... 45 Figure 3.1: Teaching Practices in the WeBa Countries by Quintiles of Average Socioeconomic Factors at the School Level Expressed in Equivalent Math PISA Points ....................................... 49 Figure 3.2: Disciplinary Climate at the School Level in the WeBa Countries by Quintiles of Socioeconomic Factors Expressed in Equivalent Math PISA Points ............................................. 50 Figure 4.1: Change in Average PISA Scores between 2018 and 2022 and the Part of the Change Not Explained by the Evolution of Socioeconomic Factors ......................................................... 52 Figure 4.2: Evolution of Average PISA Scores between 2018 and 2022 by Percentiles of Scores and Gender ................................................................................................................................. 53 3 Figure 4.3: Evolution of Average PISA Scores between 2018 and 2022 by Quintiles of the Socioeconomic Index .................................................................................................................. 54 Figure 4.4: Students Performance in Mathematics as Related to the Time Spent on Screens During School Hours for Learning and Leisure ............................................................................ 56 Figure 4.5: Impact of the Overuse of Digital Devices by Students on their PISA Scores, WeBa versus OECD Average .................................................................................................................. 57 Figure 0.1: Average PISA score by domain and by number of hours a day spent on digital devices at school for learning or leisure. ................................................................................................. 72 List of Tables Table 1.1: Growth Rate of the Population by Age Group between 2010 and 2030..................... 15 Table 1.2: Participation of WeBa Countries in International Assessments of Student Learning Outcomes .................................................................................................................................... 17 Table 2.1: Social and Cultural Factors and Linguistics in the Western Balkans............................ 23 Table 2.2: Inequalities in Performance Explained by Differences between Schools.................... 26 Table 2.3: Share of Variance in Scores Explained by Differences in Socioeconomic and Linguistic Factors ........................................................................................................................................ 27 Table 2.4: Tardiness and Absenteeism at the Primary Level ....................................................... 33 Table 2.5: Hours of Instruction Per Year and Minutes Per a Typical School Day ......................... 34 Table 2.6: Participation of WeBa countries in different optional questionnaires on socio- emotional skills ........................................................................................................................... 44 Table 3.1: Contribution of Different Kinds of Teaching Practices to the Gap between WeBa Scores and the Average PISA Score in Math................................................................................ 48 Table 4.1: Pandemic Related School Closures in the Western Balkans (number of weeks) ........ 54 4 I. Acknowledgments This publication provides an analysis of the recently published data of the Program for International Student Assessment (PISA) 2022 study for participating Western Balkan countries. It also uses Progress in International Reading Literacy (PIRLS) study results to build a more comprehensive picture of the student outcomes in the Western Balkans. The team who contributed to this publication included Tigran Shmis (Senior Education Specialist), James Gresham (Senior Education Specialist), Paul Cahu (Consultant), and Lucia Brajkovich (Education Specialist). Fiona Mackintosh proofread and edited the report. This study identified four key areas where the performance of Western Balkan education systems could be improved. They include (a) duration of learning in schools, (b) teaching practices, (c) school preparedness, and (d) use of smartphones in schools induced by COVID-related school closures that serve as a major distractor from learning. Based on the analysis, the publication provides policy options and recommendations on advancing the education systems of Western Balkan countries to improve learning outcomes that are critically important for the future well-being of students and the economic development of these countries. The team expresses utmost gratitude for the invaluable comments provided by the peer reviewers, Marguerite Clarke (Senior Education Specialist), Koen Geven (Consultant, Senior Economist), and Katia Herera Sosa (Senior Economist). The team is also immensely grateful for the valuable insights and contributions provided by the global lead for learning assessments, Diego Luna Bazaldua (Senior Education Specialist). The team is thankful for the intellectual leadership and guidance provided during the preparation by Xiaoqing Wu (Country Director for Western Balkans), Rita Almeida (Practice Manager for the 5 Europe and Central Asia Region), Indhira Santos (Lead HD Economist), and Nathalie Lahire (Program Leader for Western Balkans). 6 II. Executive Summary Human capital is a critical determinant of economic development and for Western Balkan (WeBa) countries. According to the World Bank’s Wealth of Nations methodology 1 the WeBa are ten percent behind the OECD countries in terms of the share of human capital. The recent OECD report 2 states that “at the current average growth levels for both the WeBa and the EU, convergence would only be achieved in 2076”. To leapfrog these five decades needed to catch up, Western Balkan countries will need to advance reforms in the areas of human capital development. Education quality remains a cornerstone of economic and human capital development in the Western Balkans. Like many Eastern European countries, those in the Western Balkans are facing demographic decline. The number of children has been declining fast, leading to smaller cohorts of working age in the future to sustain economies. Improving quality and endowing each graduate with more skills will be the main way to accrue more human capital and sustain living standards and human development in the long run. Social returns to education quality are very high. For instance, raising a country’s PISA scores by about 20 to 25 points is equivalent to adding another year of schooling. One of the World Bank studies 3 showed that one additional year of schooling could increase lifelong earnings by 11.5 percent in Serbia, 4.5 percent in Albania, and 4.2 percent in Kosovo. Western Balkan countries actively participate in the OECD’s Program for International Student Assessment (PISA) and other international studies 4. All WeBa countries except Bosnia and Herzegovina participated in the 2022 wave of PISA after previously participating in every round of assessments since 2003. The 2022 PISA round was focused primarily on mathematics. The recent evolution of education quality can be tracked by comparing the scores and contextual indicators of countries from the 2018 and 2022 rounds. This comparison shows that education 1 Wealth Accounting, The World Bank 2022 Link 2 OECD, Economic Convergence Scoreboard for the Western Balkans 2023. Link 3 H. Patrinos, C. Montenegro, Comparable Estimates of Returns to Schooling Around the World (2014) Link 4 Progress in International Reading Literacy Study (PIRLS) and Trends in International Mathematics and Science Study (TIMSS) by IEA. 7 quality had been improving in most WeBa countries prior to the pandemic, albeit only slowly in some. The WeBa results of the PISA 2022 cycle were disappointing, especially for Albania dropping by about 2.5 years of learning. Only a few countries in Europe and Central Asia (ECA) managed to sustain their level of education quality between the two rounds, with Serbia being the only country in WeBa to do this. 5 The downward shift is not specific to WeBa but is a global phenomenon largely resulting from the significant changes in schooling caused by the COVID-19 crisis. Hence, the levels of PISA in WeBa were behind the EU average and ECA average. The best- performing country in this group – Serbia – is in line with the ECA average, where students are still 16 months behind their peers from EU countries. In Kosovo, the lowest performer in WeBa, the gap for students reaches a whopping 4.6 years of schooling (see the figure E1). Figure E.1. PISA 2018 and 2022 scores in Mathematics for WeBa Participating Countries as Opposed to the EU Average 1.3 years of schooling Serbia 440 448 Montenegro 406 430 EU Average 2022 = 472 North Macedonia 389 394 Albania 368 437 4.6 years of schooling Kosovo 355 366 300 320 340 360 380 400 420 440 460 480 Mathematics 2022 Mathematics 2018 Linear (EU Average) Source: PISA 2022, Note: According to OECD PISA, 25 points are equivalent to 1 year of learning Learning inequalities in the WeBa countries are twice lower than in the OECD on average; however, they still need to be addressed. In terms of absolute numbers, the difference in results 5 R. Almeida, C. Avitabile, T. Shmis, 2024. Beyond the learning drop: Why countries in Eastern Europe and Central Asia should act now to avoid a teacher crisis. Link 8 between the top and bottom quintiles in ECA reaches 93 points, while for WeBa, only North Macedonia reaches an 81-point difference, and Kosovo reaches a 45-point difference. The dispersion of educational performance at the national level is in line with the quality of education. School stratification (the extent to which students are clustered in schools according to their socioeconomic level) is also not worse than in other countries that participate in PISA. Socioeconomic background has a larger effect on education performance in WeBa countries than in OECD countries, but this is likely since social indicators are more unequal in the WeBa region overall. In all WeBa countries, students in the most affluent schools are benefiting from better teaching practices. The gap is the largest in Albania and Kosovo, where these unequal conditions are widening the performance gap within the country by more than half a year. If teaching practices were as good in all schools as they are in the most affluent ones, average math scores would be bumped up by between 5 points in Serbia and 12 points in Albania. Therefore, it will help to target students from lower socioeconomic groups when teaching practices are improved in these countries. Young people in WeBa countries have similar levels of socio-emotional skills as their peers in OECD countries as measured in the PISA 2022 study. Therefore, the contribution of such skills to WeBa’s performance gap compared with the OECD is limited. The link between curiosity and academic performance in some WeBa countries may be weaker than in the average PISA participating country. This may indicate that teaching practices in WeBa may not be tapping into the natural enthusiasm of some students, thereby missing out on their energy and interest in the classroom. Students in Albania and Kosovo demonstrated almost twice more higher levels of curiosity as compared to global averages. Tapping on this student's potential with adequate pedagogical effort may help quickly advance those students. Overall, stimulating curiosity and fostering soft skills development will have positive spillover effects on students’ performance in major PISA domains, as well as increase their well-being and boost their career trajectories. This study identified four key areas where the performance of Western Balkan education systems could be improved. They include: (a) duration of learning in schools, (b) teaching practices, (c) school preparedness, and (d) use of smartphones in schools that may be a major distractor from learning. 9 Starting from the primary level, students in WeBa learn from 14 percent (North Macedonia) to 28 percent (Montenegro) less than in the comparator countries. Accumulated over the four years of primary school, this lack of schooling time might explain losses of 0.6 to 0.8 years of schooling, depending on the country. Academic schedules are also shorter in secondary school, according to PISA 2022. Students in Kosovo and Montenegro have 1.5 hours of class less per week than in other PISA countries, which represents 8 percent of the weekly class workload. In North Macedonia, the number of hours is similar to the global average, while more hours of class are provided in Serbia. Given the duration of the school year, as can be inferred from PIRLS, these reduced hours of classroom time might bring down scores by another 6 points. 6 Students in WeBa are skipping class very often, and this has likely had an additional detrimental impact on their performance. Above average skipping accounts for a 11-point gap in math performance in Kosovo and a 6-point gap in Montenegro. The numbers are lower in North Macedonia at about 4 points, while students in Serbia are skipping school no more often than students from other countries participating in PISA. PISA data shows that teaching practices are not less efficient in all WeBa countries than in other PISA participants except for Serbia. These practices include teacher-student relationships, teachers’ support for students, and different types of cognitive activation in mathematics. It is known that the way teachers practice is one of the strongest factors influencing student learning, meaning that what teachers are doing in classrooms is one of the most critical determinants of student success. The total contribution of teaching practices to the math performance gap in Kosovo is about 14 points. This corresponds to the difference in average math scores after accounting for social and structural factors. Most of this gap is due to students not being very familiar with mathematical topics, which is indirectly connected to the content of the curriculum and the extent to which teachers cover it comprehensively. The PISA data show that students in Serbia and Albania are more familiar with various math topics than those from the average OECD country, so content coverage may reduce the performance gap. In all of the WeBa countries except Albania, teachers are less efficient at fostering mathematical reasoning by pushing 6 Which corresponds to 8 percent of the 25 points represented by a single year multiplied by three years of lower secondary education. 10 students to explain their reasoning and to plan ahead for what method to use to solve a problem. In Serbia and Montenegro, teaching practices on that front are more efficient, whereas in North Macedonia and Kosovo, they deprive students of a few points. The least efficient country in the math domain is Albania, where this specific issue costs students about 8 points. Another stark finding from the PISA dataset is that almost no students in WeBa are reaching high levels of proficiency (levels 5 and 6 according to the PISA framework), while in the OECD around 8 percent of students score at these levels. Preschool education impact in WeBa may signal issues in the schooling cycle that won’t sustain the impact of early learning by the age of 15 years. Whereas only 25 percent of OECD PISA students affirmed that they did not receive any pre-schooling, this share was much larger in the WeBa countries, ranging from 39 percent in Montenegro to 77 percent in Serbia. The returns to preschool education are not being sustained throughout the school cycle in WeBa. According to PIRLS, students in WeBa come to school well prepared to read, and by grade 4, children immersed in preschools perform much better than their peers who did not attend preschool. However, the results are not sustained up until the age of 15, except for Serbia and Montenegro. PISA demonstrates that students who attended preschool a decade ago in Albania, Kosovo, and North Macedonia show lower results than their peers who stayed home. This would call for an additional investigation into the school strategies that would preserve and improve quality throughout the learning cycle. PISA data suggests that the intensive use of phones by teens in WeBa have a dramatic negative impact on their performance, driving results down in Albania by almost one year of learning. In 2022, only 3 percent of 15-year-olds did not have their own smartphone both in the OECD and in the Western Balkans. Albania was the only country in WeBa to administer an additional questionnaire on smartphone use by students. The marginal impact on students’ math scores of the amount of screen time that they report is large. The more harmful uses are related to the creation, editing, and sharing of digital content and videos, where one additional hour a day spent on such activities decreases math scores by one point. The impact on reading is about 20 percent greater than for math. This overuse of smartphones by 15-year-olds is much more prevalent in Albania than in OECD countries. This indicator alone explains Albania’s 20-point gap in average 11 scores with OECD countries. In fact, Albania ranks last on this indicator among the PISA participants for which this kind of data was collected. Based on this analysis, several policies are proposed for implementation in WeBa countries The following recommendations were provided to help improve learning outcomes: i) increase the focus on the quality of early childhood education; ii) update curricula and increase the academic load in alignment with other countries; iii) increase the control of and the discipline related to students’ school attendance; iv) increase focus on teachers and teacher professional development; v) focus on studies that help to understand effective classroom practices; vi) develop a tailored policy on the use of smartphones in schools and information campaigns for parents and communities on the best use of digital technologies. 12 III. Introduction: Navigating Challenges and Aspiring for Progress in the Western Balkans 1. The Western Balkans (WeBa) region has traversed a journey of economic and social advancement over the past two decades, making notable progress in human capital development, poverty reduction, and improved living standards. However, despite these strides, the region still faces challenges in fully realizing its human capital potential, lagging other European countries. During the period leading up to the global financial crisis of 2008, most economies in the region experienced robust economic growth and financial sector expansion. Yet the subsequent economic downturn exacerbated existing social, institutional, and environmental challenges, resulting in a decline in their economic performance. Labor market dynamics have been particularly strained, characterized by high nonparticipation rates, although there have been signs of improvement in recent years. 7 Emigration, while alleviating employment pressures and contributing to social welfare through remittances, has also led to brain drain and aging societies. According to World Bank data from 2020, the Human Capital Index for WeBa countries ranges from 56 to 68 percent, indicating that children born in the Western Balkans may be only two-thirds as productive in adulthood as they could be with a complete education and full health, highlighting the need for significant improvements. 2. European Union (EU) accession and alignment with European standards constitute a common aspiration for all WeBa countries with profound implications for policymaking at all levels. Four of the six WeBa countries—Albania, Montenegro, North Macedonia, and Serbia—are candidate countries for EU accession, while Bosnia & Herzegovina (BiH) and Kosovo are potential candidates. WeBa’s geographical proximity to key European markets, coupled with ongoing integration efforts, presents a major opportunity to attract international investment, enhance competitiveness, develop tourism, and fortify the democratization process. The EU enlargement policy has various pillars, including bilateral Stabilization and Association Agreements, trade agreements, financial assistance through the Instrument for Pre-accession Assistance, and regional cooperation. Promoting the convergence of WeBa countries with the EU27 in terms of 7 IMF (2021) “Inclusivity in the Labor Market” - https://www.imf.org/en/Publications/WP/Issues/2021/05/19/Inclusivity-in-the-Labor-Market-50230. 13 socioeconomic and political development lies at the heart of the accession process, with recent initiatives such as the Enlargement Package and the Economic and Investment Plan aimed at spurring long-term economic recovery, promoting a green and digital transition, and fostering regional integration and convergence with the EU. 3. The youth population across the Western Balkans is experiencing a notable decline. Since 2010, there has been a rapid decrease, though current projections indicate a potential stabilization in certain countries before 2030. Table 1 illustrates that the youth population in all Western Balkan (WeBa) countries and across all age groups is expected to shrink by double digits between 2010 and 2030. Consequently, this demographic shift will lead to a substantial reduction in the future labor force. To counteract this, all WeBa countries will need to increase the human capital of the remaining working-age population by extending the duration of education and by enhancing skill development at all levels of schooling. 14 Table 1.1: Growth Rate of the Population by Age Group between 2010 and 2030 0 to 4 7 to 12 13 to 18 Kosovo -37% -49% -34% North -19% -25% -25% Macedonia Montenegro -18% -14% -15% Serbia -12% -10% -18% Albania -22% -37% -44% Source: Computations from UN population division statistics 4. Although enrollment rates are already high at the compulsory level, further improvements will be necessary to offset the demographic decline. While primary-level enrollment is universal only in Montenegro, it surpasses 90 percent in the other countries, but achieving universal coverage at the primary level will not be enough to counter the demographic decline. Although total net enrollment rates at the lower secondary level are close to 100 percent, opportunities are limited to increase enrollment rates. At the upper secondary level, total net enrollment rates hover around 85 percent in Albania, Montenegro, and Serbia, high enough that it will be difficult to increase human capital simply by prolonging the duration of schooling. 5. Realizing the full potential of human capital for the development and EU convergence of WeBa countries requires improving foundational learning outcomes and increasing the contribution of their education systems to economic productivity. While progress in learning outcomes has been made over the years, it has been slow, which makes it challenging for WeBa countries to fully converge into EU. Proficiency levels vary significantly across age cohorts and countries, while learning inequalities persist across different genders, locations, and income groups, underscoring the need for targeted interventions. Challenges persist in ensuring that students master the advanced higher-order skills that are critical for driving economic growth, especially given the evolving nature of work and the demand for technical, cognitive, and socioemotional competencies. Consequently, it is imperative to address these challenges to fulfill the potential for sustained and inclusive growth in the region. 15 Figure 1.1: WeBa countries achieve improvements in Learning Outcomes but Still Behind the EU Average Harmonized Learning Outcome Score Singapore North America Serbia European Union High Income Europe & Central Asia Bosnia and Herzegovina Montenegro Upper Middle Income Albania Lower Middle Income North Macedonia Kosovo Low Income 0 100 200 300 400 500 600 Source: World Bank Data, Note: * HLO is the most recent or computed score of the reliable international assessment data, not including the most recent PIRLS and PISA results. 6. The Western Balkans, like the rest of the world, grappled with the profound impacts of the COVID-19 pandemic on education. The crisis resulted in widespread learning losses among children, a consequence of the extended school closures that impacted 1.4 billion students globally. The most prolonged closures, those that exceeded 200 days by the end of 2021, had repercussions equivalent to nearly one-and-a-half academic years lost. 8 This setback has disproportionately affected the most vulnerable people in developing countries, amplifying the existing challenges of poor educational quality and high inequality. In the Western Balkans, where the pandemic also exposed gaps in household possession of digital infrastructure, the full extent of learning loss has yet to be quantified. However, amidst these challenges, there lies an opportunity for the Western Balkans to address learning loss through innovative measures such as digital education for blended learning, prioritized and strengthened student assessments, targeted learning recovery strategies such as tutoring and teaching at the right level, and the 8 Learning loss: a Covid-19 mass casualty [Link] 16 enhancement of the resilience of education systems to better withstand future shocks 9. The ongoing commitment of the region’s governments to improving learning environments remains crucial in navigating the complexities of the COVID-induced learning crisis in the region. The newly developed education strategy for Europe and Central Asia region aims to address these gaps 10. 7. A wealth of rich and diverse data is readily accessible from prominent international studies such as PISA, TIMSS, and PIRLS, yet their full potential remains largely untapped. All WeBa countries actively participated in the 2021 Progress in International Reading Literacy Study (PIRLS), which assesses the reading skills of students at the end of primary school. Conducted by the International Association for the Evaluation of Educational Achievement (IEA), PIRLS is distinct from the Program for International Student Assessment (PISA), which is organized by the OECD. PIRLS evaluates the literacy of 4th-grade students every five years, providing a basis for comparing the quality of education around the end of the primary level in relation to other economies and over time. The participation of WeBa countries in the latest PIRLS round in 2021 makes it possible to evaluate education quality using the socioeconomic indicators reported by students and parents in the contextual questionnaire. WeBa countries have also consistently participated in PISA since the first round in 2000 when Albania and North Macedonia joined. These valuable datasets present a unique opportunity to delve into the underlying school-level factors contributing to low performance and to uncover successful practices in countries both within and beyond the region. Given this wealth of information, there is an urgent need to conduct new analyses and to formulate policy recommendations aimed at expediting improvements at the school level. By harnessing the insights gleaned from the available data and drawing upon international best practices, tailored interventions can be devised to effectively bolster the human capital potential of the region. Table 1.2: Participation of WeBa Countries in International Assessments of Student Learning Outcomes PISA 2022 TIMSS 2019&2023 PIRLS 2021 9 Arcia, Gustavo; de Hoyos, Rafael; Patrinos, Harry; Sava, Alina; Shmis, Tigran; Teixeira, Janssen. 2021. Learning Recovery after COVID-19 in Europe and Central Asia: Policy and Practice. © World Bank, Washington, DC. [Link] 10 Fasih,Tazeen; Avitabile,Ciro; Kalbfuss,Matthieu; Iqbal,Syedah Aroob. 2024. Investing in People and Enhancing Innovation and Growth through Education in Europe and Central Asia. © World Bank, Washington, DC. [Link] 17 Albania Albania Albania Serbia Serbia Serbia Montenegro Montenegro Montenegro North Macedonia North Macedonia North Macedonia Kosovo Kosovo Kosovo Bosnia & Herzegovina 8. This note aims to conduct an initial analysis of education quality in the Western Balkans, focusing on the lower secondary level. By analyzing student scores on PISA 2022 and PIRLS 2021, the objective is to identify and understand the main factors contributing to the region's underperformance in comparison with OECD countries. The investigation delves into critical aspects such as the length of school days, gender gaps, school preparedness, socioemotional factors, and the impact of socioeconomic conditions on education outcomes. Additionally, the note explores the role of teachers and teaching practices in influencing learning outcomes and proposes strategies to enhance their quality and efficiency. Furthermore, it investigates the recreational use of digital devices in schools and its potential impact on learning. The analysis extends to technical and vocational education and training (TVET) programs, examining enrollment trends and the overall quality of these programs. The ultimate goal is to provide evidence-based insights and policy recommendations that can guide stakeholders in the Western Balkans toward effective measures for improving education quality and for fostering positive educational outcomes in the region. Box 1: Spotlight on Recent Education Initiatives in the Western Balkans supported by international partners In recent years, there has been a surge in transformative education initiatives in the Western Balkans region aimed at improving the quality and increasing the relevance of its education systems. Led by by the governments of WeBA countries in collaboration with organizations like the World Bank, the European Union, UNICEF, ÖAD (Austria’s Agency for Education and Internationalization), many projects have been supported, including one focusing on improving work-based learning. This particular initiative spans Bosnia & Herzegovina (BiH), Serbia, and 18 Montenegro, placing an emphasis on dual education and fostering the harmonization of work procedures across borders. Another significant endeavor is the BRIDGE project under the Erasmus+ framework, which aims to connect Western Balkans partner countries with the European Union and to facilitate knowledge exchange, enhance training networks, and foster a deeper understanding of the EU market among young people in WeBa. The Erasmus+ Connected through Mobility Project is familiarizing Western Balkans countries with international vocational education mobility programs, enhancing their market understanding, and helping them to adapt their regulatory framework. Additionally, initiatives like the EQET SEE regional project, supported by the Austrian Development Agency, target the development of regional occupational standards in agriculture, aligning vocational education with industry needs. Across the region, countries like Montenegro and Serbia have implemented projects supported by funds from the EU’s Instrument for Pre-accession Assistance (IPA) and the World Bank, focusing on integrating key competencies into their curricula, enhancing early childhood education, and aligning their education systems with European standards. North Macedonia is establishing Regional Centers for Vocational Education and Training, while Albania's Skills for Jobs project aims to modernize vocational education through the integration of technology. Bosnia & Herzegovina is also committed to improving the quality of its vocational education and training and increasing its market relevance. Collectively, these diverse projects and reforms underscore the dedication of the Western Balkans countries to building adaptive, high-quality education systems in collaboration with global partners such as the EU and the World Bank, which will ultimately contribute to the long-term socioeconomic development in the region. 9. The note will provide an analysis framed around four major identified areas of concern for student learning outcomes. These areas are presented in Figure 2 and include short school days, outdated and inefficient teaching practices, a lack of school preparedness, and COVID-19- 19 related school closures. These pillars speak about the broader areas of concern substantiated by the data analysis. 10. Study limitations include the lack of follow-up analysis of real school practices and qualitative information. This study is based on rigorously collected data from international assessments. However, it does not go deeper in investigating the real practices in schools, which defines the next step of the work that the World Bank team or country researchers may undertake to better understand the issues highlighted in this report. Figure 1.2: Student and School-level Factors that Impact Learning Outcomes in the Western Balkans •Student absenteeism •Inequality in access to •LImited number of good teaching hours of instruction. •Performance gender gap in favor of girls Outdated Short school teaching days practices Below-average learning outcomes Insufficient Covid-related school school preparedness closures •Socioeconomic •Excessive use of background smartphones and •Lower enrolments social networks •Necessity to sustain ECEC impacts through the schooling cycle 20 IV. PISA Student Results at the Lower Secondary Level: Understanding Underperformance A. Low performance in the Western Balkans. 11. The academic performance of 15-year-olds in the Western Balkans is significantly behind OECD levels. Only in Serbia are student scores higher than the average for all participating countries, but all other Balkan countries lag behind (see Figure 2.1). Literacy scores for Kosovo, Albania, and North Macedonia are significantly below their math and science scores. Average scores for the region are several years 11 behind average OECD levels, which were above 475 points in 2022. Figure 2.1: Average PISA 2022 Scores by Discipline 500 484 475 474 480 Average PISA 2022 scores 460 447 440 440 440 420 406 406 403 400 388 376 379 380 368 355 358 358 358 360 343 340 320 300 Kosovo Albania North Montenegro Serbia EU Macedonia Reading Math Science Source: PISA 2022 12. Most students in Montenegro, North Macedonia, Albania, and Kosovo are below minimum proficiency level by age 12 and unable to perform basic mathematical operations. Serbia is the only Western Balkan country where proficiency levels are somewhat closer to those of OECD countries. About a third of 15-year-olds in Serbia are functionally illiterate, and more than 40 percent of them are below the minimum proficiency threshold in mathematics (see 11 According to the OECD, one year of schooling represent about 20 to 25 points. 12 Below minimum proficiency means that the students are not reaching proficiency beyond Level 2 of PISA. 21 Figure 4). Math and reading proficiencies in Montenegro are higher than in non-OECD participant countries but still very far from OECD levels. In North Macedonia and Kosovo, more than two- thirds of students score below the minimum PISA proficiency levels. In Albania, more than 70 percent of 15-year-olds cannot read properly or solve basic mathematics problems. 13. WeBa countries have a limited number of student achieving the highest performing standards in PISA. In Montenegro, Albania, North Macedonia, and Kosovo, there are virtually no high-performing students who are capable of scoring above level 4. This small share of high- performing students that can fill high-level skill jobs is another roadblock to economic development. In fact, according to PISA statistics, there were fewer than a dozen 15-year-olds in Kosovo who scored at level 4. These acute shortages of current students with good basic skills are likely to affect all sectors of economy and trickle down to affecting teaching workforce. Hanushek, Piopiunik and Wiederhold (2018) have showed indeed that students learning outcomes are better when their teachers have higher cognitive skills. Figure 2.2: Distribution of 15-year-old Students by Proficiency Levels in Math and Reading in PISA 2022 OECD 35% 23% 21% 14% Non-OECD 73% 16% 7% 3% Mathematics Serbia 43% 27% 18% 9% Montenegro 60% 22% 12% 4% North Macedonia 67% 20% 10% 3% Albania 75% 16% 6% Kosovo 86% 11% 3% OECD 25% 25% 26% 17% Non-OECD 63% 22% 11% 3% Serbia 36% 32% 23% 9% Reading Montenegro 53% 26% 16% 4% North Macedonia 74% 21% 5% Albania 75% 19% 5% Kosovo 84% 14% 2% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Source: PISA 2022 22 14. Social and cultural factors tend to be more favorable in some Balkan countries than in the typical country participating in PISA, as measured by the embedded index of economic, social, and cultural status (ESCS 13). Serbia and Montenegro are close to OECD levels on the index, though this is not the case for Kosovo, North Macedonia, and Albania. Moreover, almost all students in the Western Balkans are assessed in the language that they speak at home, which ought to be helpful to them in taking the tests (see Table 2.1). Table 2.1: Social and Cultural Factors and Linguistics in the Western Balkans Share of students not ESCS speaking the test language at home European Union (EU) 13.30% -0.05 OECD 12.0% -0.19 Albania 4.7% -0.74 Kosovo 2.0% -0.34 North Macedonia 6.7% -0.28 Montenegro 3.5% -0.20 Serbia 3.0% -0.20 Source: Computations from PISA 2022 microdata. 15. The performance of WeBa students at the secondary level was much worse than expected. After accounting for the above differences in socioeconomic, cultural, and linguistic factors, 14 the quality of education in the five Balkan countries is much lower than might be expected. The gaps are very large for Kosovo and North Macedonia, almost the equivalent of three years of schooling. The gap is also large in Montenegro, between two and two and a half years of schooling. The gap for Albania and Serbia is more limited but still exceeds one year of schooling, (see Figure 2.3). The performance gaps in mathematics are lower than those in reading and science for all WeBa countries. 13 This index is a composite score built by the OECD from parents’ highest education level and occupation as well as household items. Household items have been combined from the beginning of PISA into an index that is reflective of wealth. 14 Using a simple OLS regression featuring the social index (see technical appendix for details), the social index averaged at the school level as well as country fixed effects. 23 Figure 2.3: Gaps between Average Scores and What Might Be Expected Given Socioeconomic and Linguistic Factors in Equivalent Years of Schooling North Albania Kosovo Macedonia Montenegro Serbia EU OECD linguistic factors expressed in equivalent 0 Gap between scores and what could be -0.1 -0.5 -0.2 -0.2 -0.2 expected given socioeconomic and -0.4 -0.6 -1 -0.7 -0.8 -1.2 years of schooling -1.5 -1.4 -1.4 -1.5 -1.5 -2 -1.7 -1.7 -1.9 -1.9 -2.1 -2.5 -2.3 -2.4 -2.5 -3 -2.7 -2.8 -2.8 -2.9 -3 -3 -3.5 -3.4 -4 Reading Math Science All domains Source: Computations from PISA 2022 database. Note: One additional year of school is assumed to increase scores by 25 points. The net gap is computed by accounting for the effects of socioeconomic levels thanks to a linear hierarchical regression of scores on gender and social indices (both at the individual and school levels) as well as country fixed effects and random school fixed effects. B. Factors behind the performance gap. 1. The role of inequalities in the performance of WeBa countries. 16. The dispersion of education performance at the national level is in line with the quality of education in the WeBa countries. The simpler way to measure the inequalities in learning is to use the standard deviation of scores within a country. The simpler way to measure the average performance level is to use the average score within a country. Across the PISA dataset, there is a positive relationship between these two indicators. This means that the dispersion of scores within a country – the inequalities in learning – are expected to grow as the overall education performance increases. Such a relationship is represented by the dotted regression line on Figure 2.4. In North Macedonia, Montenegro, Albania, and Serbia, the standard distribution of average scores is very close to the regression line in Figure 2.4. Inequalities in Kosovo are nevertheless significantly below what might be expected, given the country’s average score. This is likely to be because most students have very low scores, and most PISA items are probably too hard for the 24 average student. These findings show that education performance is no more unequal in the Western Balkan countries than in a typical nation participating in PISA. Figure 2.4: A Scatterplot of Standard Deviations of Average Scores Versus the Mean Average Score on PISA 2022 110 Standard deviation of average scores, 100 90 PISA 2022 80 MNE MKD SRB 70 ALB 60 KSV 50 300 350 400 450 500 550 Average scores, PISA 2022 Source: Author’s calculations from PISA 2022 microdata. Note: The scores have been averaged over the three domains: reading, math, and science. The figure displays weighted statistics (weighted means and standard deviations using sampling weights). The regression line plots the expected inequality in score given the average level. In a country below the line, inequalities between students are lower than expected. 17. Between-school inequalities are very low in Serbia and North Macedonia but are higher and more in line with non-OECD countries in Kosovo and Montenegro. As can be seen in Table 2.2, about 5 percent of the variance in scores in North Macedonia and Serbia can be explained by between-schools variations. This is two times less than in a typical OECD country. In Kosovo, the share of variance explained by differences between schools is about 15 percent, which lies between the shares in OECD and non-OECD participating countries. In Montenegro, the share of variance explained by school differences is very close to the share in non-OECD countries. Albania is the only country suffering from abnormally large inequalities related to differences between schools. Such differences can be related to social stratification - the fact that students are likely to attend different schools depending on their socioeconomic status - or from inequalities in the conditions prevailing in different schools. For instance, the provision of textbooks, infrastructure 25 or class sizes could vary extensively from one school to another, inducing differences in student’s performance. Table 2.2: Inequalities in Performance Explained by Differences between Schools Albania Kosovo North Montenegro Serbia Non- OECD EU Macedonia OECD Reading 32% 15% 6% 22% 6% 22% 12% 22% Math 32% 14% 5% 21% 5% 22% 12% 22% Science 32% 15% 5% 20% 5% 22% 12% 22% Source: Author’s calculations from PISA 2022 microdata. Note: Share of variance in scores explained by between-school variance. 18. The impact of socioeconomic factors on education performance in Montenegro, Serbia, and North Macedonia is close to what can be seen in the OECD. The impact of socioeconomic factors on education performance are lower in Albania and Kosovo. As can be seen on Table 2.3, whatever the indicator used, the share of variance in scores explained by differences in socioeconomic and linguistic factors are close to OECD levels for Montenegro, Serbia and North Macedonia. In Albania and Kosovo, the social index, which is the more comprehensive measure, explains 31% and 32% respectively of the variance in scores, versus 38% in the OECD. 19. In Montenegro, Serbia, and North Macedonia, larger socioeconomic inequalities are fueling learning inequalities. A high share of the variance explained by socioeconomic factors can be related to two distinct phenomena that can possibly be cumulated. Education quality depends indeed on the socioeconomic conditions of the student and of its peers 15. The mechanisms are complex and multidimensional. We know for instance that better educated mothers are more likely to read to their children from an early age, which is one of the most effective strategies to boost reading. We also know that richer households tend to feed their children better, with meals richer in proteins and fatty acids that are critical for the brain’s development. Thus, learning inequalities can arise from more socioeconomic inequalities between students at the individual level but also from larger differences in the socioeconomic composition of schools (induced by higher school stratification). As we have seen that between- 15 The peer effects are estimated using the average socioeconomic condition at the school level. 26 schools inequalities are low or moderate in these three countries, inequalities in learning must be fueled directly by socioeconomic inequalities (see Table 10 in the annex for details). Table 2.3: Share of Variance in Scores Explained by Differences in Socioeconomic and Linguistic Factors Albania Kosovo North Montenegro Serbia Non- OECD EU Macedonia OECD ESCS, student 10% 7% 16% 12% 15% 14% 17% 19% only ESCS, student 14% 19% 32% 26% 34% 25% 28% 32% and school ESCS and home language (school 14% 20% 33% 27% 34% 27% 29% 33% and student) Social index (both school and 31% 32% 40% 38% 37% 35% 38% 41% student) Source: Author’s calculations from PISA 2022 microdata. Note: The share of variance of the scores is explained using a linear model accounting for gender and socioeconomic factors. The dependent variable is the average score minus the national average and the explanatory variables are indexed at the student level and their average at the school level (when stated). The home language is a binary indicating whether the student speaks the language of instruction at home. The social index is built from the various home possessions, the immigration status, the education, and occupation level of both parents (see annex). This index is different from the OECD’s ESCS. It is built as the best predictor of average PISA scores. The ESCS index was conceived rather as a measurement of wealth. The regressions use weights and are run independently for each country. The OECD and non-OECD column display the average R2 over the countries’ regressions in the group. 20. School stratification is not a concern in the Western Balkans. Everywhere in the world, students tend to be clustered in schools with children from similar socioeconomic backgrounds. To some extent, this is inevitable because schools tend to service a limited area, and neighborhoods tend to be socially stratified. However, other factors such as urbanization can increase school stratification, which can exacerbate performance inequalities and hinder social mobility. In Kosovo, North Macedonia, and Montenegro, social stratification tends to be lower or similar to the level in non-participating countries, regardless of the indicator chosen to measure socioeconomic levels. In Serbia, schools are slightly more stratified according to the PISA ESCS index. However, the social index (built for this analysis from the students’ household possessions and the immigration status, education, and occupation level of both of their parents), which is a much stronger predictor of academic performance, shows that social stratification is much lower 27 than in other PISA-participating countries. These metrics show that school stratification is not fueling social inequalities related to education quality in the Western Balkans (see Figure 2.5). Figure 2.5: Social Stratification and Determinism in the Western Balkans 0.6 0.56 0.55 0.54 0.53 0.52 Correlation between student and 0.5 0.45 0.41 0.44 0.4 0.36 0.35 0.32 0.3 0.26 school index 0.19 0.2 0.16 0.1 0 ESCS Social Source: Computations from the PISA 2022 database. Note: Data are weighted. 2. Performance gap in secondary and primary level. 21. PIRLS 2021 performance of WeBa countries is mixed, with North Macedonia, Serbia, and Kosovo performing worse than expected after accounting for socio-economic factors. While the reading performance of grade 4 students in Serbia, Albania, and Montenegro was above the average for all PIRLS 2021 participating countries, it has been affected by the socioeconomic factors that impact students’ performance. Therefore, after accounting for these socio-economic factors, students’ reading performance in Serbia turned out to be behind what might have been expected (see Figure 2.6). Four-graders were more than one year behind the PIRLS average in Kosovo and more than half a year behind in Serbia and North Macedonia, although primary students in Albania were more than one year and in Montenegro half a year ahead of the average in reading. 28 Figure 2.6: Gaps in Reading Score Compared with Other Participating Countries after Accounting for Differences in Socioeconomic Factors 80 58 2021 with other participating countries 60 Gap in average reading score PIRLS 38 38 40 20 13 12 0 -20 -15 -40 -33 -33 -60 -48 -55 -80 Albania Kosovo North Montenegro Serbia Macedonia Accounting for socioeconomic factors PIRLS scale gap Source: Authors’ calculations from PIRLS 2021 microdata. Note: The socioeconomic index was computed from household items, language spoken at home and parents’ education. The gap net of social factors is computed thanks to a linear model using the social indexes at both the individual and school average as well as gender and country fixed effects. 22. Average scores at the primary and secondary level can be benchmarked but not equated. Reading scores computed by PIRLS in grade 4 and reading scores computed by PISA at the age of 15 are not directly comparable. They are not measuring the same skills and do not use a common scale. A priori, one PIRLS point might not necessarily represent the same thing as a PISA point. It is known, for instance, that as students progress through the grades, the dispersion of scores increases as the effects of family, social, and differences in teaching quality are cemented. At the secondary level, one additional year of schooling represents a smaller part of the variance in performance than at the primary level. Nevertheless, there is a robust statistical relationship between PIRLS and PISA scores 16. Students are accumulating skills throughout their school journey, using the skills that they have already acquired to learn more. In the absence of remedial education, it is likely that students falling behind at the end of the primary would end up being delayed at the end of the lower secondary. Therefore, countries where education quality is lower than expected at the primary level are likely to have lower than expected scores at the end of lower secondary, even if schools are as efficient as in other countries at that level. 16 Patrinos, Harry Anthony; Angrist, Noam. Global Dataset on Education Quality: A Review and Update (2000-2017) (English). Policy Research working paper, no. WPS 8592 Washington, D.C.: World Bank Group. Link 29 A simple OLS regression displayed in the appendix shows that amongst the 45 countries that participated in both the PIRLS 2021 and the PISA 2022 assessment, there is a unitary 17 correspondence between PIRLS and PISA reading scores. 23. The performance gap among 15-year-olds is likely due to shortcomings at the pre- primary and primary levels. The data from PISA and PIRLS studies show that the skills gap among young people varies a lot from one WeBa country to another (see Figure 2.7). In Kosovo, students are losing ground to other participating countries at both the primary and the secondary levels. The performance gaps at the secondary level are indeed twice as large as at the primary level, suggesting that the children are falling further behind. The gaps at the end of lower secondary are about 80 PISA points, which is about three years of schooling. In North Macedonia as well, the education performance gap is twice as large at the secondary than at the primary level. The performance gap in secondary education represents 50 PISA points or two years of schooling. However, 15-year-olds in North Macedonia are still reading better than in Kosovo, perhaps because they accumulated less of a delay at the primary level, as witnessed by the country's PIRLS 2021 scores. 24. In Serbia, the performance gap from the primary level looks different at the lower secondary level. The performance of WeBa students in PIRLS at the end of the primary was about 20 points behind the PIRLS 2021 average. But in PISA at the lower secondary level, Serbia looks about 10 points ahead of the typical participant. Given the initial reading skills of primary students in Serbia as measured by PIRLS, their reading levels at the age of 15 in PISA suggest that they might have progressed slightly faster than the average for lower secondary students across the world (considering differences between PISA and PIRLS assessment). 17 The regression coefficient is not statistically different from one. 30 Figure 2.7: The Performance Gap in Reading for PIRLS 2021 and PISA 2022 After Accounting for Socioeconomic and Linguistic Factors 58 60 20 11 Gap between average PISA 2022 scores and Gap between average PIRLS 2021 scores and socioecomomic conditions (in PIRLS points) socioecomomic conditions (in PISA points 40 0 what could be expected given what could be expected given 20 13 -20 -27 0 -40 -20 -60 -15 -56 -64 -80 -40 -33 -80 -48 -100 -60 Primary gap Secondary gap Source: Author’s calculations from PIRLS 2021 and PISA 2022 microdata. Note: Reading scores are adjusted for socioeconomic and linguistic factors using linear regressions, with country fixed effects. Both the scores and the determinants differ. PIRLS and PISA scores are not comparable and there is no equivalence between the left and right graphs. 25. In Montenegro, the performance gap is likely concentrated at the secondary level. The reading skills of 4th graders in Montenegro, as measured by PIRLS, are indeed slightly better than might be expected. However, the performance at the secondary level is about 30 PISA points behind what might be expected. 26. Albania’s performance gap is also likely to be more pronounced at the secondary level. The country’s academic performance at the secondary level looks very low, but the picture has likely been distorted by the pandemic. The performance of Albania’s students at the primary level was much better than the average for all countries participating in PIRLS 2021. Students were almost 60 PIRLS points ahead in reading at grade 4. However, this did not translate into good results at the beginning of the upper secondary level. At the age of 15, Albanian youths perform much worse than what might have been expected, given their socioeconomic conditions. There could be several varied reasons for this paradox. The first one is that PIRLS 2021 and PISA 2022 did not test the same cohort. The cohort that was in grade 4 in 2021 will not reach 15 years old (the age at which PISA tests students) until 2026. It is possible that significant progress in improving primary education in Albania during the 2010s may not yet have been picked up by 31 PISA. Albania participated in TIMSS in 2019, and in that year, primary students’ scores in math were close to the global average and not much ahead as they were in PIRLS 2021 for reading. This suggests that at least part of the paradox could be explained by a relatively rapid improvement in quality at the primary level. 27. Albania’s PISA performance has plummeted since 2018, with its scores falling by 52 points. This is the second-worst performance, with a 56-point decline. The average reduction in scores was only about 8 points. It would be reasonable to assume that the drop in scores is entirely due to the pandemic and does not reflect the underlying quality of secondary schooling. However, average PISA scores in Albania are significantly below the world’s average. Therefore, all the advantages enjoyed by Albanian students at the primary level in PIRLS 2021 vis-à-vis other participating countries aren’t confirmed by measures of PISA 2022 at the age of 15 (with a caveat of different assessments and different student cohorts). 3. Limited hours of schooling in WeBa. 28. Teaching practices and observable factors at the primary level do not explain why reading performance in the WeBa countries is lower than expected. Teaching practices in all WeBa countries, as reported in PIRLS, are much better than in EU countries. This makes a positive contribution to teaching quality and makes the part of the gap that cannot be explained by observable factors (the structural quality gap) even larger. 29. Both the disciplinary climate and the amount of bullying in schools in the region are also better at the primary level than in the EU countries. This also contributes favorably to students’ reading performance in grade 4 as measured by PIRLS 2021. Moreover, parental involvement in their children’s education is reportedly higher, which is another positive feature of the WeBa countries (see Figure 2.8). This is consistent with the level of students’ school readiness as reported by parents in PIRLS, which is around average for the participating countries. Logically, after accounting for all of these factors, the remaining structural quality gap between WeBa countries and the rest of the world appears even larger. It is equivalent to about half a year of schooling in Montenegro and Serbia, about a year in Albania, and around one and a half years in Kosovo and North Macedonia. 32 Figure 2.8: Metrics Contributing to Quality All Above the Global Average at the primary level Bullying Disciplinary climate Parental stimulation Teaching practices 1.5 Mean index as % of global standard deviation 1.2 1.2 1.0 1.0 1 0.5 0.5 0.4 0.5 0.4 0.3 0.2 0.3 0.1 0.1 0.1 0 0.0 -0.5 -1 -1.0 -1.1 -1.2 -1.5 -1.4-1.4 North Macedonia Montenegro Serbia Kosovo EU Source: Authors’ computations from PIRLS 2021 microdata. The global average is set as zero. 30. Students’ absenteeism and tardiness are serious or moderately serious issues in all WeBa countries from the primary level onwards. Reports from both students and principals consistently show that students are more frequently absent in Kosovo and Albania than in other participating countries (see Table 2.4). This might be contributing to poor outcomes in Kosovo, but the impact on learning in other countries may be more limited. Table 2.4: Tardiness and Absenteeism at the Primary Level Principals reporting Students Absenteeism Tardiness Absenteeism Albania 23% 7% 16% North Macedonia 13% 4% 18% Montenegro 1% 0% 15% Serbia 2% 0% 13% Kosovo 15% 10% 21% EU 10% 9% 10% Other participating countries 13% 9% 18% Source: PIRLS 2021. Note: Share of principals reporting as a serious or moderate issue. Share of students who reported missing school at least once every two weeks. 33 31. Primary schools in the Western Balkans have shorter days than in most countries, which translates into fewer hours of instruction per year. Students in North Macedonia, Albania, and Kosovo spend around 15 percent less time at school than in a typical PIRLS country, while this gap is twice as large in Serbia and Montenegro. It is not possible to directly link these lighter workloads to disappointing performance using PIRLS microdata. 18 However, natural and random experiments have shown – and the COVID lockdown confirmed – that cutting school hours, all other things equal, has a negative impact on performance. Accumulated over the four years of primary school, this lack of schooling time might explain losses of 0.6 to 0.8 years of schooling, depending on the country. The workload's impact typically exceeds the mere number of hours as more teaching time enables a more ambitious curriculum while driving up expectations about what students can achieve. Table 2.5: Hours of Instruction Per Year and Minutes Per a Typical School Day Minutes Hours per Gap with per day year average in % Albania 242 742 -15.4% North Macedonia 255 758 -13.5% Montenegro 212 634 -27.7% Serbia 224 667 -24.0% Kosovo 235 754 -14.0% EU 292 877 0.0% Source: PIRLS 2021. 32. Students in WeBa are skipping class very often, and this has likely had an additional detrimental impact on their performance. Above-average skipping accounts for an 11-point gap in math performance in Kosovo and a 6-point gap in Montenegro. The numbers are lower in North Macedonia at about 4 points, while students in Serbia are skipping school no more often than students from other countries participating in PISA (see Figure 2.9). 18 The number of schools and the within-country variance in workload is not large enough. Moreover, endogeneity biases are plausible since less motivated teaching staff may prefer to work fewer hours. 34 Figure 2.9: Share of Students in PISA 2022 Countries Reporting Having Skipped At Least One Class in the Previous Two Weeks 80% skipped school in the last two weeks 68% 70% Share of pupils reporting having 60% 52% 50% 51% 50% 45% 38% 36% 40% 34% 30% 17% 20% 13% 10% 10% 0% Source: PISA 2022. 33. A lighter academic schedule at the secondary level may also reduce scores. According to the PISA data, the number of hours of class per week in Kosovo and Montenegro is also significantly lower than average. Students in both countries spend about 1.5 hours less in class than the PISA average, which represents 8 percent of the weekly class workload. In North Macedonia, the number of hours is similar to the global average, while more hours of class are provided in Serbia. The data in PISA 2022 are not detailed enough to enable an estimation of the impact of hours on performance. 19 Given the duration of the school year, as can be inferred from PIRLS, these reduced hours of classroom time might bring down scores by up to 6 points. 20 The actual impact is uncertain but is likely to be significant, perhaps even reducing scores by twice as much. 4. Gender gaps in student performance in Western Balkan countries. 34. Gender performance gaps are larger in the Western Balkans than in other countries. On average, girls score significantly higher than boys in reading and science in PISA. The performance gaps in reading are very large, being close to or more than one year of schooling. This gap is 19 There is a likely endogeneity bias as more able students are more likely to sign up for additional subjects. 20 Which corresponds to 8 percent of the 25 points represented by a single year multiplied by three years of lower secondary education. 35 significantly larger than in OECD countries (see Figure 2.10). Gender gaps in mathematics are close to zero, which is similar to the situation in non-OECD countries but not in OECD countries, where boys score significantly higher than girls in this domain. In science, gender gaps in WeBa are also larger than in both non-OECD and OECD participating countries, with girls scoring significantly higher than boys in all WeBa countries, with gaps varying from 4 PISA points in Serbia to 28 PISA points in Albania. The average gender gap in the countries of the region is large, representing more than one year of schooling. 35. These large gender gaps in favor of girls suggest that engagement strategies for attracting to study may need revamping. At the global level, girls’ PISA scores are more consistent and less elastic to changes in school conditions and teaching practices. The large gender gaps in favor of girls may suggest that teaching practices and school culture could be improved to engage boys and girls equally in PISA disciplines. Figure 2.10: Gender Performance Gap in WeBa Countries Compared with OECD and non-OECD PISA Participants, PISA 2022 45 Average difference between girls and boys' 35 25 PISA 2022 scores 15 5 -5 -15 Reading Math Science Source: Authors’ calculations from PISA 2022 microdata. 5. School preparedness in the Western Balkans 36. The story of the impact of preschool education on the learning outcomes of WeBa students is mixed. While WeBa countries show a positive impact of their preschool systems on 36 student reading outcomes in the PIRLS study by 4th grade, PISA results are not so promising for most of the countries except for Serbia and Montenegro. This reinforces the importance of fostering good education throughout the school cycle, as the impact of preschool may fade over years of schooling. 37. The benefits of preschool on reading skills measured at the primary level by PIRLS are positive and similar in the Western Balkans and other PIRLS participating countries. An analysis of the PIRLS 2021 microdata does not exhibit statistically significant differences in the benefits of preschool participation on reading scores measured in grade 4 in the WeBa countries vs. the group of other participants, see Figure 2.11. The PIRLS data allows for disentangling the participation of early education programs (daycare from the age of 1 or 2) to actual preschool programs. This data demonstrates that the likely benefits of these two stages of ECD are different. While preschool brings benefits in terms of reading comprehension, students who attended daycare only score lower, although the difference is very low. It is to be noted that the marginal impacts displayed on Figure 2.11 remain robust to the additional control of early stimulation practices reported by parents. Figure 2.11: Marginal impact of participation in early childhood education programs (daycare) and pre-primary school on reading scores using PIRLS 2021 data in the WeBa and other participating countries. 15 Difference in PIRLS reading points once accounted for gender, home language, 13 11 9 social factors 7 12 5 9.8 3 1 -1 -3 -1.22 -1.5 -5 WeBa Others Day care Pre-primary 37 Source: Authors’ calculations from PIRLS 2021. The PIRLS reading scores are regressed over gender, home language, social factors (individual and school levels), and participation in daycare and pre- primary schools. We use a mixed model with random school effects and country-fixed effects. Two separate models are estimated, one for WeBa countries and one for the other PIRLS participating countries. 38. According to PISA though, the premium associated with participation in ECEC is much lower in the Western Balkans countries, with the exception of Serbia and Montenegro. Contrary to PIRLS, PISA data does not allow disentangling of the various forms and stages of Early Childhood Education and Care (ECEC). The 2022 dataset only records the age at which interviewed students reported having entered ECEC regardless of the form. The OECD computed a derived variable that estimates the number of years that children spend in ECEC. Among PISA participants outside of WeBa and Serbia, students who benefitted from some ECEC scored on average about 17 points higher than those who did not, see Figure 2.12. In the rest of the WeBa countries, however, students who received some ECEC appear to be worse off. Students who did not benefit from ECEC are indeed scoring about 11 points higher. This difference is statistically significant. In Albania, Kosovo, and North Macedonia, students who stayed at home are better off, and the gap is statistically different from zero. Figure 2.12: difference in PISA scores between students who did or did not participate in some form of ECEC. 50 between students who had none vs. 40 Difference in average PISA scores 30 22.3 17.2 17.0 20 10 4.9 some ECEC 0 -10 -4.9 -20 -10.9 -18.3 -30 -20.6 -40 WeBa WeBa excluding Serbia Kosovo North Macedonia PISA countries excluding Montenegro Serbia Albania WeBa Source: Authors’ calculations from PISA 2022. 38 39. According to both PISA and PIRLS data, participation in preschool is beneficial in the Western Balkans. The mixed results at the stage of PISA in Albania, Kosovo, and North Macedonia show that students who attended any preschool are behind their peers who did not attend preschool at all. This apparent negative premium associated with ECEC participation in PISA may be related to intrinsic differences between children who did or did not participate in ECEC in these countries or their learning progression between cycles. However, the fact that these potential biases dwarf the benefits of ECEC underlines that the quality and efficiency of preschool and further stages of schooling may be limited. Although students who did not participate in ECEC in Albania, Kosovo, and North Macedonia look better off, the PISA scores are larger for students with more than one year of ECEC versus less than one year, see Figure 2.13. This is evidence that attending preschool longer is beneficial. 39 Figure 2.13: Relationship between duration of ECEC and PISA results, all participating WeBa countries Albania Kosovo 410 380 400 370 390 360 350 380 340 370 330 360 320 350 310 300 340 Reading Math Science Reading Math Science Montenegro North Macedonia 430 420 420 400 410 380 400 360 390 340 380 320 370 300 Reading Math Science Reading Math Science Serbia 470 460 450 440 430 420 410 Reading Math Science Source: Authors’ calculations from PISA 2022. The y-axis displays the average PISA scores for the pupils grouped according to the duration of their ECEC. 40 40. Most of the 15-year-olds from the Western Balkans who were tested in PISA 2022 did not attend preschool. Whereas only 25 percent of OECD youths tested in PISA affirmed that they did not receive any pre-schooling, this share was much larger in the WeBa countries, ranging from 39 percent in Montenegro to 77 percent in Serbia (see Figure 2.14). Although the returns to preschool might have been lower at that time, this still could explain part of the performance gap. It should be noted that the returns of preschool education are largely diffuse. Students who are ready to start primary school are not only likely to read faster but also to require less time from their teachers, who can focus on the children who did not attend pre-primary education. Figure 2.14: Preschool Attendance of Students Sampled in PISA 2022 OECD 25.8 53.3 20.9 Non-OECD 44.2 42.0 13.9 Albania 52.8 37.0 10.2 Serbia 77.4 9.3 13.4 Montenegro 38.8 49.7 11.5 North Macedonia 51.2 35.6 13.3 Kosovo 63.5 27.3 9.3 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% No preschool Preschool Does not remember Source: Weighted numbers from PISA 2022. 41. The coverage of pre-primary education in WeBa is growing, but it is far from becoming universal. Albania is the country where pre-primary education is the most developed, yet 20 percent of a generation is not enrolled (see Figure 2.15). While there has been a sharp increase in preschool enrollment in Montenegro over the past decade, in North Macedonia, the coverage of pre-primary education is still very limited, below 40 percent, according to UNESCO data. In Serbia, in 2022, at least a third of a generation was not covered. As enrollment rates increase in the WeBa countries, this is likely to contribute to a rise in reading scores in the coming decade. 41 Figure 2.15: Gross Enrollment Rates at the Pre-primary Level, 2010-2022. 90.0% 84.6% 83.9% Gross ernolmnent rate at pre-primary 80.0% 73.6% 70.0% 70.0% 65.5% 58.1% 60.0% 55.0% 52.0% 50.0% level 40.0% 32.3% 33.9% 35.9% 30.0% 25.5% 20.0% 10.0% 0.0% Albania Montenegro North Macedonia Serbia 2010 2015 2022 Source: UNESCO. * Kosovo is not included in the UN statistics. 42. School preparedness is at a good level in WeBa, according to PIRLS 2021. The PIRLS data reveal that students starting primary school in WeBa are as likely to be able to read a word or a short story as those from the EU countries (see Figure 2.16). Children are less advanced in North Macedonia and Montenegro than in the other three WeBa countries, but most kids can already decipher a text prior to entering primary school. Pre-reading skills are very developed in Albania, where almost half of students already know how to read stories before entering primary school. This is also corroborated by the positive impact of ECEC on PIRLS outcomes highlighted above. This school preparedness is also aided by reports in the data by WeBa parents saying that they stimulated their children in early childhood. Based on these data, the level of parental stimulation during early childhood in the five WeBa countries is significantly above the global average. 42 Figure 2.16: Good School Preparedness in Reading in WeBa 90% 78% 77% 80% 73% 70% 66% 65% 58% 60% 50% 45% 44% 45% 40% 34% 32% 29% 30% 20% 10% 0% Can read some words Can read story Albania Montenegro North Macedonia Kosovo Serbia EU Source: Authors’ calculations from PIRLS 2021. 6. Socio-emotional factors and relation to student outcomes. 43. A review of the socioemotional indicators in the PISA 2022 dataset did not yield any obvious explanations for the underperformance of the Western Balkan countries. There are two indicators where WeBa stands out. First, bullying is less of an issue in these countries than elsewhere. This is consistent with what was evident at the primary level in PIRLS 2021, and this should contribute positively to students’ scores. Second, a lot of students are reporting skipping school often (see Figure 2.17) and not only in Kosovo and North Macedonia. In Montenegro and Serbia, 15-year-olds were also more likely than their peers from other participating countries to state that they had skipped school. This is highly likely to hurt their academic performance. Students rate teacher support and teacher-student relationships as below average in Montenegro and Serbia but near average in Kosovo and North Macedonia. The ratings for a sense of belonging, a feeling of safety, and the disciplinary climate are at or above the benchmark of other participating countries. 43 Figure 2.17: Various PISA 2022 Metrics in the Western Balkans and EU Countries 0.6 Metrics are standardized over the OECD population, with zero mean and unitary 0.4 0.2 standard devaition 0 -0.2 -0.4 -0.6 Bulliying Teacher-student Disciplinary climate Skipping school Sense of belonging School risks Repetition rate Safety Teacher support relationships Albania Kosovo North Macedonia Monternegro Serbia EU Source: PISA 2022. 44. Students in the WeBa countries overall do not stand out in terms of their socioemotional skills. Although PISA did not survey students in all countries across all domains (see Table 2.6), the national average for WeBa students in terms of socioemotional skills is very close to the global mean. The only exceptions are Kosovo and Albania, whose students appear to be significantly more curious than those in the rest of the world. Serbia was the country with the largest number of available metrics for 21st-century skills. The data show that Serbian students are less curious and less cooperative than those from other countries, 21 while their level of emotional control, empathy, and perseverance is not statistically different from that of students from other countries around the world. Table 2.6: Participation of WeBa countries in different optional questionnaires on socio- emotional skills Resistance Emotional Curiosity Perseverance to stress control Empathy Assertiveness Albania + - - - - - Kosovo + - - - - - Macedonia + - - - - - Montenegro + + - + - + 21 There is a negative gap between the national average and the global average that is small but statistically significant. 44 Serbia + + + + + + EU + + + + + + Source: PISA 2022 Figure 2.18: Socioemotional Skills as Measured in PISA 2022 Resistance to Emotional Curiosity Perseverance stress control Empathy Assertiveness 0.4 National average as % of global 0.35 0.3 standard deviation 0.25 0.2 0.15 0.1 0.05 0 -0.05 -0.1 Albania Kosovo Macedonia Montenegro Serbia EU Source: PISA 2022. 45. Socio-emotional skills such as curiosity generally favor student performance on the test but less so in the Western Balkans. After accounting for gender, social factors, teaching practices, the disciplinary climate, bullying, and school skipping, students who report having higher levels of curiosity tend to score higher in math, reading, and science. However, the marginal impact of curiosity on PISA scores is significantly lower for students from the Western Balkans, being almost twice as small. One explanation could be that teaching practices are not taking advantage of the curiosity of children in the Western Balkans. Over the whole PISA dataset, math scores tend to be more sensitive to curiosity when math teachers’ practices are more efficient overall. In other words, efficient teachers are also better at tapping into the natural curiosity of their students, but it appears this is not common in WeBa countries. V. Teachers and Teaching Practices: Enhancing Quality and Efficiency 46. Schools in the Western Balkans are well-staffed from a global perspective. The student- to-teacher ratios in WeBa, as recorded in PISA, are either at the same level or lower than in OECD countries, which average less than 13 to 1. These ratios have been decreasing in all WeBa 45 countries except North Macedonia, which reached 13.1 in 2022. Furthermore, there is no negative correlation that can be seen in the PISA dataset between student-to-teacher ratios and performance, all other things being equal. This is likely due to an endogeneity problem, as small and remote schools are more likely to have lower ratios but these students’ performance might be hampered by poverty, a lack of infrastructure, and the difficulty of attracting and retaining high-performing teachers in remote areas. However, in the WeBa region, the share of schools where student-to-teacher ratios are higher than 30 to 1 is very rare, affecting only about 0.3 percent of students in 2022. This is significantly lower than in the average OECD country, where 5.4 percent of students in 2022 were attending schools that were clearly lacking teachers. 47. The PISA dataset indicates that school principals lack a qualified teacher workforce, which may impede the performance of Western Balkans students. When comparing PISA scores in 2018 and 2022 after accounting for determining factors, there is no correlation between the unexplained gap and what principals declare about the lack of teachers hindering quality. There is nevertheless a correlation between this unexplained gap and the principals reporting about (i) a lack of qualified teachers, (ii) truancy, (iii) disrespectful students, and (iv) bullying. None of the recorded information from principals about the lack of devices or software to implement remote learning had a correlation with the unexplained gap. The only factor that is correlated is teachers’ lack of experience in providing remote instruction. 48. However, inefficient teaching practices contributed to the math performance gap in Kosovo, North Macedonia, and Montenegro. Students were asked a wealth of questions about teaching practices in the PISA 2022 survey. One way to go about this is to use the indexes embedded in PISA that combine answers to estimate students’ latent traits. There are five of these indexes that cover (i) teacher support, (ii) teacher-student relationships, (iii) fostering reasoning in mathematics, (iv) encouraging mathematic thinking, and (v) students’ familiarity with mathematical concepts. The latter index could be used as a proxy for the coverage of subject curriculum by teachers and the extent to which current mathematics teachers are covering familiar topics of the national curriculum (as PISA is not based on curricula of participant countries). By adding these indexes – and their school averages – in a linear regression of math scores, along with social factors and country-fixed effects, it is possible to gauge to what extent 46 teaching practices contribute to student performance. The linear combination of all these factors derived from the regression gives an idea of teaching practices in each country. These indices can be compared with the OECD average to get a sense of how much teaching practices contribute to the gap. 49. WeBa countries differ among themselves in terms of the extent to which teaching practices contribute to the math performance of their students. Different components of teaching practices contribute in different degrees to the performance gap in WeBa countries. This decomposition is summarized on Table 3.1. The total contribution of teaching practices to the math performance gap in Kosovo is about 14 points. This corresponds with the difference in average math scores after accounting for social and structural factors. Most of this gap is due to students not being very familiar with mathematical topics, which could signal that the coverage of the content domain of mathematics 22 might not be as comprehensive as in OECD countries and/or that teachers are not introducing mathematical concepts comprehensively enough. Given that most WeBa countries have shorter school year, teachers may not have enough time to cover all topics. The PISA data show that students in Serbia and Albania are more familiar with various math topics than those from the average OECD country, so this broader coverage of math concepts is, in fact, reducing the performance gap. In all of the WeBa countries except Albania, teachers are less efficient at fostering mathematical reasoning by pushing students to explain their reasoning and to plan ahead for what method to use to solve a problem. In Serbia and Montenegro, teaching practices on that front are more efficient, whereas in North Macedonia and Kosovo, they deprive students of a few points. The least efficient country in the math domain is Albania, where this specific issue costs students about 8 points. 50. The metrics for teacher support and the quality of student-teacher relationships are low in Montenegro and Serbia but not in the other WeBa countries. In Albania, both metrics are above the OECD average and make a positive contribution to closing the gap with developed countries. 22 There is no judgement regarding the national curricula of WeBa countries. 47 Table 3.1: Contribution of Different Kinds of Teaching Practices to the Gap between WeBa Scores and the Average PISA Score in Math Fostering Encouraging Topic Teacher Student- All reasoning math coverage support teacher practices thinking relationship Kosovo -1.9 -3.1 -9 0.5 -0.1 -13.6 Macedonia -0.8 -3.4 -3.5 0.6 -0.3 -7.4 Montenegro -1.6 -0.7 -1.2 -1.3 -1.2 -6 Serbia -1.5 0.4 3 -1.5 -0.9 -0.5 Albania 1.7 -8.5 3.6 1.6 1.4 -0.2 Source: Author’s calculations from PISA 2022 microdata. Note: Math scores were regressed over sex, social factors, grade, preschool age, and the embedded indexes for teaching practices (COGACRCO COGACMCO FAMCON TEACHSUP and RELATST), accounting for country-fixed effects on the whole PISA 2022 dataset as well as random fixed effects for school. The indexes are introduced at the individual level and with their school average, except for RELATST where the school averages were not statistically different from zero. The individual level and school level factors with their respective multipliers are linearly combined to produce new indexes that are standardized over the set of OECD countries (with a zero mean). Using the regression levels, it is possible to project the contribution of each index to WeBa’s gap with OECD countries. 51. Socially disadvantaged students are those most likely to be exposed to less efficient teaching practices. In all WeBa countries, students in the most affluent schools are benefiting from better teaching practices than in the poorest ones. The gap is the largest in Albania and Kosovo, where these unequal conditions are widening the performance gap within the country by more than half a year. If teaching practices were of higher quality in all schools at the level they are in the most affluent ones, average math scores would be bumped up by between 5 points in Serbia and 12 points in Albania. North Macedonia is the only country in the region where teaching practices are not statistically better in the wealthiest neighborhood than in the rest of the country (see Figure 3.1). 48 Figure 3.1: Teaching Practices in the WeBa Countries by Quintiles of Average Socioeconomic Factors at the School Level Expressed in Equivalent Math PISA Points 15 Teaching practices in equivalent PISA 10 points (OECD varege is zero) 5 0 -5 -10 -15 -20 Kosovo North Montenegro Serbia Albania Macedonia Q1 Q2 Q3 Q4 Q5 Source: Author’s calculations from PISA 2022 microdata. Note: In Albania, teaching practices in the top quintile (the most advantaged) of schools are better than in the OECD. The contribution to this positive gap in practices adds about 12 points to the PISA math scores of the students of these schools compared to OECD students, after holding all other things equal. 52. The learning climate within schools does not affect student outcomes in WeBa countries. Staff behaviors and pedagogic skills can influence the school climate by reducing bullying, fostering a sense of belonging among students, and establishing a disciplinary atmosphere in classes. In WeBa, bullying is slightly more prevalent than in OECD countries, which does not significantly contribute to the performance gap. WeBa students tend to be less integrated into their school environment compared to students in OECD countries; however, this sense of belonging is not a major factor in the performance gap. The disciplinary climate in WeBa schools is rated slightly lower by students than the climate in schools in an average OECD country. 53. School climate does not vary much within each country. The disciplinary climate, the sense of belonging, and bullying are not much better in more advantaged schools than in less advantaged schools. In all of the Western Balkans countries, differences in the disciplinary climate in the country’s schools are responsible for a performance gap of no more than about 3 points between those in the bottom and the top quintiles on the socioeconomic factors (see Figure 3.2). 49 Figure 3.2: Disciplinary Climate at the School Level in the WeBa Countries by Quintiles of Socioeconomic Factors Expressed in Equivalent Math PISA Points Disciplinary climate in equivalent PISA 1 points (OECD varege is zero) 0 -1 -2 -3 -4 -5 -6 Kosovo North Montenegro Serbia Albania Macedonia Q1 Q2 Q3 Q4 Q5 Source: Author’s calculations from PISA 2022 microdata. Note: In Albania, the disciplinary climate in the top quintile of the most advantaged schools is one point higher than in the OECD and only 3 points higher than in the most disadvantaged schools. 54. Teachers are critical actors in improving students’ learning outcomes. Teaching practices are deemed to be among the most impactful inputs that drive student learning. The role of teachers is well described in the literature. 23/24 Better teachers produce better outcomes, greater well-being for their students, higher future incomes, and also better future life choices. 25 No other school inputs, be they better learning environments or more digital devices, can be effective without a trained and efficient teacher. 55. The World Bank’s recent Making Teacher Policy Work report emphasized the significant role played by teachers and laid out meaningful policies to help them excel. 26 In particular, the report focuses on reforming teacher pre-service and in-service training and on which areas teachers should focus. The report recommends that countries should consider basing their 23 Teachers, Schools, and Academic Achievement Steven G. Rivkin, Eric A. Hanushek, John F. Kain Econometrica, Vol. 73, No. 2 (Mar., 2005), pp. 417-458 24 Hattie, J.A.C. (2009). Visible learning: A synthesis of 800+ meta-analyses on achievement. Oxford, UK: Routledge 25 Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood Raj Chetty, John N. Friedman, Jonah E. Rockoff The American Economic Review, Vol. 104, No. 9 (SEPTEMBER 2014), pp. 2633- 2679 26 World Bank. 2023. Making Teacher Policy Work. © Washington, DC: World Bank. http://hdl.handle.net/10986/40579 License: CC BY 3.0 IGO. 50 teacher qualification policies on the Clear, Doable, and Rewarding (CDR) approach, which can lead to meaningful improvements in teachers’ qualifications. The PISA 2022 data for WeBa countries clearly shows areas where more teacher development is needed. VI. The roles of School Closures and the Digital Transition in Plummeting Learning Outcomes in the Western Balkans A. The global decline in education performance between 2018 and 2022 and its impacts on WeBa countries. 56. Average PISA scores declined significantly in Albania, Kosovo, North Macedonia, and Montenegro between 2018 and 2022, but not in Serbia (see Figure 22). However, after taking into account the fact that socioeconomic factors improved in Serbia, the country did, in fact, suffer a significant drop in performance, amounting to a third of a year of schooling. Socioeconomic factors improved slightly in Kosovo, North Macedonia, and Montenegro, meaning that the actual performance of education systems declined a bit more than the raw scores would suggest. As mentioned above, the drop in Albania’s performance was very large, equivalent to more than one and a half years of schooling. None of these shifts can be explained by changes in social factors. 57. The downward trend in performance was not homogeneous among all students or by gender. The performance of very low achievers remained consistent overall (see Figure 4.1). 27 The scores of high-achieving boys were robust in Montenegro and even increased in Serbia. However, the performance of female students declined throughout the population of the region, except in the lower part of the student population in Serbia. In North Macedonia and Serbia, the performance drop was larger for female students than for their male peers at all points of the distribution. In Kosovo, the performance of the highest-achieving girls fell by slightly less than average. In Montenegro, the performance of boys in the middle of the distribution fell by much more than that of other boys. Performance varied significantly within the student population in all of the WeBa countries. 27 Except in Serbia perhaps but this could be due to the difficulty involved in precisely sampling the ends of the distribution. 51 Figure 4.1: Change in Average PISA Scores between 2018 and 2022 and the Part of the Change Not Explained by the Evolution of Socioeconomic Factors Albania North Macedonia Montenegro Kosovo Serbia 0 Average PISA score 2022 minus average 0 -9.9 -11 -8.7 -10 -17.4 -19.7 -24.4 -25.3 PISA score 2018 -20 -52.3 -54.6 -30 -40 -50 -60 Raw Net of social factors Source: Authors’ computations from PISA microdata. Note: The contribution of social factors is computed from a linear model featuring an index computed at the student level and averaged at the school level. The index is built from contextual questionnaire items on household items, language at home, and urbanization levels. The index is estimated by a hierarchical linear model with random-fixed effects at the school level, see Table 16 in the annex for details. 58. Learning loss curves in WeBa differ from those in OECD countries. Learning losses have been larger for girls than boys in OECD countries, especially in the top half of the proficiency distribution (see Figure 4.2). This is also the case in North Macedonia and Serbia but not in Kosovo, Montenegro, or Albania. In the OECD, learning losses were limited to those at the very bottom of the proficiency distribution but were overall smaller as the initial level of students increased, especially for boys. However, in WeBa, this was only seen in Montenegro. In Serbia, learning losses for the top achievers were indeed lower, with boys at the top of the distribution getting better scores in 2022 than in 2018. This is the only group in the WeBa countries whose scores may have improved during the period. The learning loss curves in Kosovo and North Macedonia are somewhat close to what could be observed in other emerging countries (non- OECD PISA participants), which was that learning losses increased with the initial proficiency of the students and were larger for girls, especially at the higher end of the distribution. In Albania, learning losses were exceptionally large for all students. 52 Figure 4.2: Evolution of Average PISA Scores between 2018 and 2022 by Percentiles of Scores and Gender 20 40 0 0 50 100 10 20 Kosovo North Macedonia -10 Montenegro 0 0 0 50 100 0 50 100 -10 -20 -20 -20 -40 -30 -60 -30 Boys Girls Boys Girls Boys Girls 10 0 5 5 0 50 100 OECD -20 0 0 Serbia Albania 0 50 100 0 50 100 -40 -5 -5 -60 -10 -10 -15 -80 -15 Boys Girls Boys Girls Boys Girls Source: Authors’ calculations from PISA 2018 and 2022 microdata. The Y axis represents the change in the average PISA score between 2022 and 2018 by gender and by percentile of score. For instance in Montenegro, boys in the first percentile of scores in 2022 scored about 10 points higher than boys in the first percentiles of scores in 2018. 59. Learning losses in the WeBa region varied by socioeconomic groups but with no common pattern. In OECD countries, learning losses were close to zero for students in the top socioeconomic quintile and between 6 and 8 points for the rest of the distribution (see Figure 4.3). In non-OECD countries, learning losses also had a parabolic shape, but they were quite uniform across the social levels. The distribution of learning losses by socioeconomic quintiles in North Macedonia is quite close to the distribution in other emerging countries, although the performance gap is two and a half times as large. In Serbia, the losses were concentrated at the bottom of the social distribution while those in the top two quintiles made gains. In Montenegro, the learning losses were large but more limited for students in the top social quintile. In Kosovo, losses were the largest for students in the middle of the distribution and more limited for those in both the lowest and highest quintiles. In Albania, learning losses were enormous for all quintiles. 53 Figure 4.3: Evolution of Average PISA Scores between 2018 and 2022 by Quintiles of the Socioeconomic Index 10 2022 by quintiles of socioeconomic index Change in average PISA scores 2018- 0 -10 -20 -30 -40 -50 -60 Q1 Q2 Q3 Q4 Q5 Source: Authors’ calculations from PISA 2018 and 2022 microdata. B. School closures had an adverse impact on scores. 60. Schools in the WeBa countries stayed closed for extended periods during the pandemic. According to UNESCO data, schools were closed for about a year in North Macedonia and Montenegro and about 18 to 21 weeks in Albania and Serbia, which is close to the EU and OECD average. The nominal impact of one year of school closure is about 25 points, so, at least on paper, the decline in scores in North Macedonia and Montenegro is commensurate with what might have been expected. The shorter length of the closures in Serbia is also consistent with lower learning losses. However, school closures alone cannot account for the massive drop in scores in Albania, which is more than twice as large as might have been expected, especially given that its school closures were shorter than the EU and OECD average. Table 4.1: Pandemic Related School Closures in the Western Balkans (number of weeks) Albania 18 Kosovo* 12 North Macedonia 34 Montenegro 36 Serbia 21 EU average 19 OECD average 20 Source: UNESCO, OECD PISA 2022 54 Note: The number of weeks for Kosovo is not reported by UNESCO and is taken from the PISA 2022 dataset 61. Disparities in school support during closure may have played a limited role. The level of support provided by schools to students in WeBa during the school closures was close to or better than in OECD countries. According to PISA, in all WeBa countries except Serbia, the most support during closures was provided to students enrolled in the most affluent schools. There was also more support provided to students in general upper secondary school than to those at other levels. However, the empirical correlation between school support and PISA scores in the PISA dataset is very limited. Inequities in the way in which different schools managed to support learning during the closures may have contributed to social disparities in learning losses but only in a very limited way. School support was the most homogeneous in Serbia, which is the country that had the largest inequities in learning losses on PISA 2022. In Albania, learning losses were larger for wealthier children, even though these students benefited disproportionally from school support. C. Overuse of digital devices had a negative effect on students in all WeBa countries. 62. There is mounting evidence of the impact of longer hours spent by teenagers on screens on mental health, anxiety, and depression levels. A study in South Korea showed that longer smartphone use causes increased depressive symptoms and higher suicidal ideation for girls, not for boys, and addictive smartphone behavior is frequent among adolescents and more prevalent among girls than boys. 28 In Switzerland, the research showed that the time young men spent using a smartphone was linearly associated with higher rates of social anxiety, depression, attention deficit hyperactivity disorder, and lower levels of life satisfaction. 29 These studies may forecast new research on the impact of smartphone overuse on educational outcomes. PISA 2022 data hints at the potential impacts too. 28 Robert Rudolf; Najung Kim, Smartphone use, gender, and adolescent mental health: Longitudinal evidence from South Korea, SSM - Population Health, Volume 28, 2024 [link] 29 Joseph Studer, Simon Marmet, Matthias Wicki, Yasser Khazaal, Gerhard Gmel, Associations between smartphone use and mental health and well-being among young Swiss men, Journal of Psychiatric Research, Volume 156, 022, ages 602-610 [link] 55 63. Teenagers are experiencing higher levels of anxiety and addiction to digital devices in the Western Balkans. New items in the PISA contextual questionnaire make it possible to gauge the extent to which teens are hooked on their smartphones. This metric was significantly higher in WeBa countries than in the average OECD economy. In Kosovo and North Macedonia, the indicator was about 25 percent of a standard deviation larger than the OECD average. The OECD has recently released the PISA in Focus analysis showing overall trends related to students’ use of screens both in and outside of school (see Figure 4.4). 64. The overall exposure of teens to screens in Albania is similar to the level in the OECD but could be doing more harm. It is possible to model the exposure of 15-year-olds to screens by adding up hours of use as reported by students in the PISA questionnaire and producing a simple average or weighing each type of use by its apparent marginal impact on math 30 scores. The overall amount of time spent before screens is similar in all WeBa countries and in the OECD. However, the amount of time-weighted by use is higher in all WeBa countries except Albania, signaling that the students’ exposure to screens is unlikely to be a key contributor to WeBa’s performance gap with the OECD. Figure 4.4: Students Performance in Mathematics as Related to the Time Spent on Screens During School Hours for Learning and Leisure Learning Leisure 500 Mean score in mathematics 490 480 470 460 450 440 430 420 None Up to 1 hour More than More than More than More than More than 7 1 hour and 2 hours and 3 hours and 5 hours and hours up to 2 hours up to 3 hours up to 5 hours up to 7 hours Time spent on digital devices at school per day Source: OECD, PISA 2022. 30 We find similar a pattern with both reading and science scores, see in appendix. 56 65. Teenagers in Albania are spending a lot of time on their smartphones, which has likely had a huge adverse impact on their scores. PISA 2022 included an optional questionnaire dedicated to eliciting more precise reports of students’ use of and time spent on screens. Albania was the only country in the Western Balkans to administer this questionnaire. It is possible to reiterate the process outlined above and build a variable adding the reported screen times using their marginal impact on scores in a hierarchical linear model. This variable can then be introduced into a more comprehensive model factoring in all of the other determinants considered, such as schools’ disciplinary climate and teaching practices, whether the student had attended preschool, how much support schools provided to students during COVID-19, students’ self-reported absenteeism, and even their anxiety related to their use of digital devices. The marginal impact on students’ math scores of the amount of screen time that they report is large. The more harmful uses are related to the creation, editing, and sharing of digital content and videos, where one additional hour a day spent on such activities decreases math scores by one point. The impact on reading is about 20 percent greater than for math. This overuse of smartphones by 15-year-olds is much more prevalent in Albania than in OECD countries. This indicator alone explains Albania’s 20-point gap in average scores with OECD countries. In fact, Albania ranks last on this indicator among the PISA participants for which this kind of data was collected (see Figure 4.5). Figure 4.5: Impact of the Overuse of Digital Devices by Students on their PISA Scores, WeBa versus OECD Average 15 10 Impact of screen overuse on PISA scores vis-a-vis OECD average 5 0 -5 -10 -15 -20 -25 BGR SVK ROU URY MLT HUN PAN CRI GBR IRL TAP JPN ALB SAU KAZ DOM BRA SVN HRV EST LTU CHL LVA CHE ESP BRN ITA GEO THA POL TUR AUT ISR SWE DNK USA AUS SGP JOR ARG MYS CZE ISL FIN HKG MAR BEL DEU MAC KOR QUR GRC 57 Source: Author’s computations from PISA 2022 microdata. 66. It is unlikely that simply banning smartphones in schools will solve this problem. In PISA 2022, in those schools that had such a ban in place, the gap was cut by only one point, meaning just 5 percent of the standard deviation. A deeper look at the data also hints at the low efficiency of bans, as banning doesn’t change the usage of phones by students in schools. On average in PISA 2022, the phone ban in school reduces the overuse of phones in school by less than 10%. Partly, the detrimental impacts of smartphone use on students’ scores come from their use outside the classroom. However, it also becomes a challenge for schools to adopt policies that move students from the leisure to the learning domain to keep their exposure to a reasonable level. 67. The way in which the main PISA questionnaire was worded may have affected the accuracy of conclusions about their impact on learning. The items on the main questionnaire were not very informative in highlighting the effects of students’ screen exposure on their performance. The imprecise measurement of screen time in broad categories does not make it possible to pinpoint the specific uses that are most harmful to teenagers. As revealed in the detailed questionnaires, interacting on social networks is much more damaging to students’ cognitive development and learning than simply browsing the web or even checking social networks. It is clear from the PISA data that one hour spent on a smartphone can have very different impacts depending on the types of content and interactions. Therefore, simple public messaging warning about time spent on screens is unlikely to be sufficient to protect teenagers from overuse and misuse. 68. More research is needed on the use of digital devices and their effects on students’ outcomes. This research will need to encompass students in lower grades as children are getting access to devices earlier in life. Several existing analyses indicate the complexity of the issue. In Norway, one experimental study confirmed that school bans could have positive impacts on student’s well-being and learning outcomes, especially for girls from disadvantaged backgrounds, as well as reducing bullying 31. Also, a recent global education monitoring report by UNESCO 31 https://openaccess.nhh.no/nhh-xmlui/bitstream/handle/11250/3119200/DP%2001.pdf 58 explored school smartphone bans around the world and discovered some positive effects of focusing children on learning with requiring, however, pedagogical and teacher efforts. 32 More and deeper work will be needed to identify the best policies for schools to adopt to help students overcome their dependency on devices and reduce their detrimental impact on learning. 33 32 https://unesdoc.unesco.org/ark:/48223/pf0000385723/PDF/385723eng.pdf.multi 33 https://www.nytimes.com/2024/03/29/books/jonathan-haidt-gen-z-smartphones-interview.html 59 VII. Conclusion and Policy Recommendations Based on the analysis of this paper, the authors propose several conclusions and recommendations to unlock the potential of the school system in the Western Balkans to improve the academic performance of students across this region. • The mixed outcomes of the pre-primary level should be further investigated. The returns to preschool education are not being sustained throughout the school cycle in WeBa. According to PIRLS, students in WeBa come to school well prepared to read and those kids who received preschool education perform much better than their peers who didn’t attend preschool. However, the results are not sustained up until the age of 15, except for Serbia and Montenegro. PISA demonstrates that students who attended preschool a decade ago in Albania, Kosovo, and North Macedonia show lower results than their peers who did not. The findings indicate a need for detailed monitoring of skills and developmental milestones among children of the region throughout the school cycle. Investing in access and quality of early childhood education and care remains critical for school preparedness and success in later stages of schooling. • The lower-than-expected quality of education in WeBa can be mitigated by increasing class hours. The structural performance gap between WeBa countries and other participating countries in both PIRLS and PISA can be explained by the fact that children and youths in WeBa have much lighter academic schedules than those in OECD and EU countries. Research has shown that increasing class hours is associated with robust learning gains, which are even larger in the early grades. More advanced curricula and more focused school hours would also be beneficial for students in all WeBa countries. This will require investment in education and teacher time. • More efforts are needed to tackle school skipping, which is prevalent in all WeBa school systems. Each country’s policymakers will need to review and revise their current school- 60 skipping policies to increase discipline. The current short school days combined with school skipping appear to be having a severe negative effect on learning outcomes as students are not learning. • Teaching practices could be enhanced. Apart from a lighter curriculum in WeBa, several clues from PISA and PIRLS point to inefficient teaching practices in WeBa at the secondary level for instance, the effects of preschool are fading in many WeBa countries between 4th and 8th grades. Improving teaching practices at a systemic level is a complicated undertaking. It will require improvements in initial teacher training, coaching, and capacity building, as well as fostering a culture of observation and monitoring of learning at the school level. Therefore, deeper studies are needed on what is happening in the classrooms in WeBa countries to identify bottlenecks and increase school efficiency. • The detrimental impact of exposure to digital devices, especially among school students, may be large. This topic is quite new, and there is not enough causal evidence. However, the evidence against the overuse of smartphones and especially social networks is starting to pile up. While there is no compelling evidence of a causal adverse link between the length of smartphone use and mental health the PISA 2022 dataset shows large negative correlations between the use of digital devices and test scores, especially among girls. Given the need to conduct research prior to deciding on policies, this illustrates the need for education systems to monitor smartphone use and associated academic performance in schools to produce data and inform policy reforms. • The unexplained variance in student achievement in WeBa countries compared to other PISA-participating countries needs further investigation. The extensive availability of data in the Western Balkans and the new releases of the TIMSS 2023 database and the PISA 2022 Creative Thinking Assessment taken in combination with PIRLS 2021 data may deepen researchers’ understanding of the reasons behind the gap. This quantitative data 61 needs further support by qualitative methods and classroom observations to form a full picture in each WeBa country. 62 References Almeida Rita, Avitabile Ciro, and Shmis Tigran (2024). Beyond the learning drop: Why countries in Eastern Europe and Central Asia should act now to avoid a teacher crisis. Link Braghieri Luca, Ro'ee Levy, and Alexey Makarin (2022). "Social Media and Mental Health." American Economic Review, 112 (11): 3660-93. Center on PBIS (2024) PBIS: An Evidence-Based Framework for Making Schools Safe, Positive, Predictable, and Equitable. Center on PBIS, University of Oregon. www.pbis.org Chetty, Raj, John N. Friedman, and Jonah E. Rockoff (2014). “Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood,” The American Economic Review, Vol. 104, No. 9 (September), pp. 2633-2679. Hanushek, Eric & Piopiunik, Marc & Wiederhold, Simon. (2018). The Value of Smarter Teachers: International Evidence on Teacher Cognitive Skills and Student Performance. Journal of Human Resources. Forthcoming. 10.3368/jhr.55.1.0317.8619R1. Hattie, J.A.C. (2009). Visible Learning: A Synthesis of 800+ Meta-analyses on Achievement. Oxford, UK: Routledge. IMF (2021). “Inclusivity in the Labor Market”. Link Joseph Studer, Simon Marmet, Matthias Wicki, Yasser Khazaal, Gerhard Gmel, Associations between smartphone use and mental health and well-being among young Swiss men, Journal of Psychiatric Research, Volume 156, 022, ages 602-610. Link Mullis, I.V.S., M. von Davier, P. Foy, B. Fishbein, K.A. Reynolds, and E. Wry (2023). PIRLS 2021 International Results in Reading. Boston College, TIMSS & PIRLS International Study Center. Link OECD (2023). PISA 2022 Results (Volume I): The State of Learning and Equity in Education, PISA, OECD Publishing, Paris. Link OECD (2023), Economic Convergence Scoreboard for the Western Balkans. Link Olefir A., T. Stohr, T. Coupe, and A. Gremalschi (2015). “Student performance and attendance in Moldova from a socio-economic perspective,” The World Bank, Washington, D.C. Orben, A., A.K. Przybylski, S.J. Blakemore, et al. (2022). “Windows of developmental sensitivity to social media.” Nature Communications, 13, 1649. Link Patrinos Harry, Montenegro Claudio (2014). Comparable Estimates of Returns to Schooling Around the World. Link Patrinos,Harry Anthony; Angrist,Noam. Global Dataset on Education Quality: A Review and Update (2000-2017) (English). Policy Research working paper,no. WPS 8592 Washington, D.C. : World Bank Group. Link Rivkin, Steven G., Eric A. Hanushek, and John F. Kain (2005). “Teachers, Schools, and Academic Achievement,” Econometrica, Vol. 73, No. 2 (March), pp. 417-458. 63 Robert Rudolf; Najung Kim, Smartphone use, gender, and adolescent mental health: Longitudinal evidence from South Korea, SSM - Population Health, Volume 28, 2024. Link Santiago-Rosario, M.R., K. McIntosh, S. Izzard, D. Cohen-Lissman, and T.E. Calhoun (2023). “Is Positive Behavioral Interventions and Supports (PBIS) an Evidence-Based Practice?” Center on PBIS, University of Oregon. www.pbis.org. Schleicher (2022). PISA 2022 Insights and Interpretations, OECD, Geneva. Twenge, J.M. (2022). Generations, The Real Differences Between Gen Z, Millennials, Gen X, Boomers, and Silents—and What They Mean for America's Future. Simon and Schuster, New York. World Bank (2023). Making Teacher Policy Work. Washington, DC: World Bank. Link World Bank (2022). Wealth Accounting. Link 64 Annex 1: Technical Appendix Social Determinants of Performance 69. Students’ performance is influenced by social, economic, and cultural factors. In PISA 2022, there is an extensive list of variables useful to characterize the background of pupils, see Table 10. The variables are not necessarily all less favorable in the Western Balkans than in the OECD. For instance, parental education levels are higher in North Macedonia, Montenegro, and Serbia and there are less students who are from abroad or do not speak the language of instruction at home. Students also have at least the same number of books at home in these three countries. Moreover, the PISA economic, cultural, and social index (escs) is similar in Serbia than in the average OECD country. Occupational status (hisei) is lower on average, which is consistent with lower living standards. Annex Table 1: Descriptive Statistics of Socioeconomic Variables Kosovo North Montenegro Serbia Non- OECD Macedonia OECD PISA economic, cultural, -0.34 -0.28 -0.21 -0.20 -1.16 -0.19 escs and social index immig Immigration status 1.02 1.03 1.09 1.12 1.03 1.20 fisced Father's education level 6.64 6.51 6.67 6.29 5.23 6.11 misced Mother's education level 6.06 6.55 6.64 6.52 5.26 6.26 Highest parental 50.6 50.5 48.0 49.3 39.5 51.3 hisei occupational score Student does not speak 0.02 0.07 0.04 0.03 0.33 0.12 folang language of instruction at home st250 Home items (education) 0.94 0.92 0.92 0.88 1.10 0.95 st251 Home items (others) 1.92 1.91 1.95 1.86 1.66 1.95 Number of digital devices 2.82 3.12 3.19 3.32 2.48 3.38 st253 at home st254 Digital devices at home 1.78 1.80 1.82 1.79 1.53 1.85 Number of books at 1.61 1.59 1.93 1.90 1.41 1.86 st255 home st256 Books at home 1.99 2.15 2.32 2.41 1.82 2.15 Source: PISA 2022 microdata. Accounting for Social Factors through Linear Models 70. Interpreting the performance gap between WeBa countries and other participating countries cannot be simply done by feeding a linear regression with the above individual variables. Differences between schools also need to be accounted for. For that purpose, the use of hierarchical linear model is not necessarily ideal because by bundling all together school-level differences in one indicator, one may underestimate the impact of some characteristics or policies that play an additional role through peer effects. We know for instance that children who have fewer books at home are bound to perform less well. However, because the performance of one child interacts with 65 the performance of her peers sharing the same class or school, it will increase further with the average number of books per child at the school level. 71. In fact, the correlation of some factors with scores can be even larger when they are measured at the school level rather than at the individual level. This is the case for instance for the disciplinary climate, as an unorderly classroom hampers the learning of all students while combining the opinion of all students on the unrest in the classroom provides a more accurate measure of disciplinary climate. The same would be true for most measures of teaching practices for instance. 72. The ideal framework would be the following, where the score of student in school , , in country , is a linear function of the vector of individual covariates and capturing potential peer or collective effects through the average of such variable computed among all the students of the school and denoted � , . Country-specific effects are controlled for adding country fixed effects . One would also like to account for unobservable differences between schools thanks to a random fixed , effect, , . The resulting individual error terms are denoted in what follows: , , � , ′ + + , + , (Eq .1) = + 73. Because of correlations within variables of the vector and lower variance of the variables in the vector � , the coefficients of this latter vector, in ′ may not be well identified and easily interpreted. To simplify the approach, one can run rather a two- step model. In a first step, the vector is estimated using a linear hierarchical model, where , are capturing all school/country fixed, including those that stem from the peer effects: , , , = + , + (Eq. 2) 74. Then in a second step, the combined variable ̂ is directly introduced as an � summarizing explaining variable, where ̂ is the vector of multipliers estimated at the first step. Then, one only estimates the relative weight of school-level variables respective to individual variables through two scalars 1 and 2 . Proceeding like this facilitates the interpretation of the results, as several types of controls can be checked sequentially, starting with the most robust ones. , , ̂ ̂ � + , + , (Eq. 3) � , = 1 � � + 2 � 75. Pooling the PISA 2022 data for Moldova and OECD countries, we estimated (Eq. 2) using a mixed model. The regression results for this model are displayed in Annex Table 2 below. Annex Table 2: Regression Results of the Hierarchical Linear Model of Scores with Only Student- level Factors and Random Effects at the School Level Coefficient Standard T stat P-value error girl -3.2 0.3 -10.6 0.000 immig = 2 5.8 0.7 7.8 0.000 immig = 3 11.0 1.1 10.1 0.000 fisced = 2 6.1 1.0 5.9 0.000 fisced = 3 7.1 1.0 7.4 0.000 66 fisced = 4 8.4 1.0 8.3 0.000 fisced = 5 12.0 0.9 12.7 0.000 fisced = 6 8.7 1.0 8.9 0.000 fisced = 7 8.9 1.0 9.1 0.000 fisced = 8 16.5 1.0 17.3 0.000 fisced = 9 13.9 1.0 14.3 0.000 fisced = 10 0.4 1.0 0.4 0.673 misced = 2 3.4 1.0 3.3 0.001 misced = 3 1.4 1.0 1.4 0.156 misced = 4 1.5 1.1 1.4 0.176 misced = 5 4.8 1.0 4.7 0.000 misced = 6 1.4 1.1 1.4 0.174 misced = 7 3.0 1.0 2.9 0.004 misced = 8 10.2 1.0 10.0 0.000 misced = 9 4.6 1.0 4.4 0.000 misced = 10 -13.5 1.1 -12.5 0.000 st250q01ja 3.8 0.3 11.5 0.000 st250q02ja -10.4 0.5 -22.9 0.000 st250q03ja -5.0 0.3 -15.6 0.000 st250q04ja -7.0 0.7 -10.2 0.000 st250q05ja -5.4 0.5 -10.2 0.000 st251q01ja 0.3 0.2 2.1 0.034 st251q02ja -8.6 0.2 -50.3 0.000 st251q03ja -2.8 0.2 -11.6 0.000 st251q04ja 7.2 0.2 30.3 0.000 st251q06ja 6.0 0.1 46.6 0.000 st251q07ja 2.5 0.1 22.0 0.000 st253q01ja 12.1 0.1 99.2 0.000 st254q01ja -12.3 0.2 -50.6 0.000 st254q02ja -6.9 0.2 -36.9 0.000 st254q03ja 3.2 0.2 15.9 0.000 st254q04ja -6.2 0.2 -33.7 0.000 st254q05ja -6.7 0.2 -43.0 0.000 st254q06ja -5.0 0.2 -21.9 0.000 st255q01ja 8.4 0.1 68.8 0.000 st256q01ja -5.2 0.1 -38.4 0.000 st256q02ja 0.0 0.1 0.3 0.760 st256q03ja 7.5 0.1 60.1 0.000 st256q06ja 4.3 0.1 30.9 0.000 st256q07ja -3.8 0.1 -29.4 0.000 st256q08ja 1.7 0.1 14.8 0.000 st256q09ja -4.7 0.2 -30.9 0.000 st256q10ja 0.2 0.1 1.3 0.186 67 hisei 0.4 0.0 65.5 0.000 folang -13.8 0.6 -22.3 0.000 # obs. 369,793 # clusters (schools) 8,518 Source: Author’s calculations from PISA 2022 microdata. Standard errors are computed after clustering the errors at the school level. 76. The main problem of using such a linear approach to create indexes to reflect the student’s learning conditions is that it considerably reduces the sample. There were indeed more than 600,000 observations with test scores in our dataset but using this linear modeling, one can only retain 60 percent of this sample. The basic issue is that all the variables need to be filled out for the index to be computed. Any missing item in the long list would imply dropping the observation. Given that the list of covariates we have selected is quite long, the number of students for which the social index can be computed as the linear combination is reduced. Consequently, 40 percent of the observations are lost. , 77. To solve this problem, we decompose for each student the vector of social item in , two parts the vector of items that are known as they have been filled out by , student and the vector of unknown items. The social index can then be decomposed in two additive components, with their related multipliers. These multiplying vectors are different from one student to another as the list of answered items may differ. , , ̂ , ̂ , ̂ , , = 3 = , + , = + ( . 4) , 78. The strategy for the projection is to use the known part to project the unknown part. For reasons of simplicity, we have assumed without loss of generality , , that all the items are centered, that is [ ] = 0. It follows that [ ] = , [ ] = 0. We simply assume that the relation is linear, and we therefore seek a scalar multiplier , in (eq. 9) such that: , , , [ | ] = ( . 5) 79. Because of the long list of items, the number of possible configurations, defined by the set of missing items and the values of the non-missing ones, is enormous and likely exceeds the number of observations. It is therefore not an option to estimate (Eq. 9) through OLS in each of the possible configurations. It would not only be too long, but it would in many cases give back unreliable results. 80. The identification strategy here is rather to guess the value of the multiplier directly from the covariance matrix. We know from the OLS equation that the expectancy of is as follows, when the expectancies are computed on the whole sample: , , , [ ′ ] [ | ] = , , (. 6) [ ′ ] 81. Using the linearity of the expectancy, one can simplify the numerator and the denominator, where is the covariance matrix of the known items: 68 , , ̂, (. 7) ̂, ′ [ ′ ] = 82. The numerator can be likewise computed from the multipliers and the covariance matrix , between the known and unknown items: , , ̂, ( ( . 8) ̂, ′, [ ′ ] = 83. Such a method allows to compute very quickly for all students with missing items the scalar by which to multiply the known component to get the unknown component. It follows from this algorithm that when there are no missing values, one has , = 0 and = 0 and according to the following equation, the projected social index perfectly matches its actual value: , , , [ | ] = (1 + ) (. 9) 84. The social index of the students with incomplete contextual socioeconomic data tends to be lower than the social index of those who answered all the contextual items. Students who have more difficulties to fill out the contextual questionnaire tend to live in less favorable conditions, see Annex Figure 1. The difference is statistically significant when 6 items or more are missing. Annex Figure 1: Density of the Social Index for Students with Complete and Incomplete Contextual Socioeconomic Data Density -8 -6 -4 -2 0 2 4 Social index Linear index Projected index Source: Author’s calculations from PISA 2022 microdata. 85. The fit of the projected indicator is quite good. The adjusted R2 of the regression of scores on gender and the initial social indicator (when all items are completed) is about 48 percent when the data is weighted and 41 percent when the data is unweighted. Over the sample of students who did not answer all of the socioeconomic items used for the computation of the social index, the adjusted R2 of the regression of scores over gender and the projected social index is still 44 percent for weighted data and 35 percent for weighted data. This is a lot better than when using the embedded ESCS index, which has almost the same number of observations. In that case, the adjusted R2 is only about 26 percent with weighted data and about 22 percent with unweighted data. Therefore, building an optimized index allows to 69 explain twice as much of the variance. Moreover, when adding the school-level of the social index and the country fixed effects, the model explains about 54 percent of the variance, which is remarkably high. 86. Using this social index at both the student level and its weighted average at the school level to account for peer effects, one can correct scores after accounting for socioeconomic factors. To do so, one uses an OLS regression of the scores in each domain over the sex, the social indices at both the student and school level and , country fixed effects, following equation 10 below, where is the pupil’s score , (which can be reading, math, science or the average of the three), is the pupil’s social index, ̅, is the pupil’s school average social index and are country fixed effects. The results of these four regressions are displayed in Annex Table 3. , , , = + α′̅, + + ( . 10) Annex Table 3: OLS Regression with Country-fixed Effects of Scores by Domains by Gender and Social Indices (at Both the Student and School Levels) to Compute the Scores Net of Socioeconomic Factors Average scores means s.e. T-stat P-value girl 0.8 0.3 2.7 0.007 social index (pupil) 51.4 0.3 169.2 0.000 social index (school) 46.6 0.9 54.7 0.000 intercept 442.7 2.1 211.4 0.000 Adj. R2 0.543 Reading girl 18.3 0.3 60.4 0.000 social index (pupil) 53.6 0.3 170.6 0.000 social index (school) 48.5 0.9 55.1 0.000 intercept 428.1 2.2 197.6 0.000 Adj. R2 0.519 Math girl -11.9 0.3 -39.8 0.000 social index (pupil) 49 0.3 157.1 0.000 social index (school) 45.6 0.9 50.5 0.000 intercept 446.6 2.2 201.5 0.000 Adj. R2 0.528 Science girl -4 0.3 -13.3 0.000 social index (pupil) 51.8 0.3 165.7 0.000 social index (school) 45.7 0.9 53 0.000 intercept 453.4 2.2 210.4 0.000 Adj. R2 0.51 #obs 601,192 #cluster 9,269 Source: Authors’ calculations from PISA 2022 microdata. 70 Note: The social index whose methodology of construction is presented in the previous section. The number of observations and clusters is identical for all domains. 87. A similar procedure can be used to compute a social index from the variables in both the 2018 and 2022 waves. Such an index is instrumental in determining the impact of socioeconomic outcomes in the evolution of scores between 2018 and 2022. We also use a mixed model, a hierarchical linear model at the school level. The results are presented in Annex Table 8. Additional Tables Annex Table 4: Standard Deviation of PISA 2022 Scores and the Social Index Reading Math Science Social Index Kosovo 61.9 57.9 61 0.821 North 69.7 78 75.8 0.865 Macedonia Montenegro 83.4 77.5 78.3 0.843 Serbia 84.8 85.6 85.3 0.783 Albania 71.1 79 74.9 0.853 Non-OECD 90.1 81.1 86.7 0.807 OECD 90.5 81.6 87.1 0.763 Source: Authors’ calculations from PISA 2022 microdata. Annex Table 5: Index of Exposure to Mathematical Content (expofa) by Students on the Upper Secondary Track in WeBa Countries General upper sec. Vocational upper sec. Gap means s.e. means s.e. Means s.e. Kosovo 0.239 0.022 0.218 0.025 -0.02 0.03 North 0.229 0.023 0.253 0.02 0.02 0.03 Macedonia Montenegro 0.205 0.025 0.139 0.02 -0.07 0.03 Serbia 0.213 0.026 0.076 0.017 -0.14 0.03 Albania 0.559 0.022 0.376 0.059 -0.18 0.06 Source: Authors’ calculations from PISA 2022 microdata. Annex Table 6: Gap between General and Vocational Upper Secondary Education in Various Measures of Quality in the Western Balkans, PISA 2022 gap se lower upper Expofa -0.186 0.015 -0.215 -0.157 Cogacro -0.282 0.015 -0.313 -0.252 Disclim -0.207 0.014 -0.234 -0.179 Cogamco -0.175 0.015 -0.205 -0.145 teachsup -0.12 0.017 -0.153 -0.087 relatst -0.191 0.013 -0.218 -0.165 71 bullied -0.05 0.013 -0.077 -0.024 anxmat 0.133 0.016 0.101 0.164 Source: Authors’ calculations from PISA 2022 microdata. Note: Data from all WeBa countries has been pooled. Annex Table 7: Relationship between PISA 2022 and PILRS 2021 average reading scores Coef. Std. err. lower upper PIRLS 0.96 0.1 0.76 1.15 INTERCEPT -38 50 -139 63 Number of observations: 45 Adjusted R2: 0.69 Source: Authors’ calculations from PISA 2022 and PIRLS 2021 datasets. Note: OLS regression of the PISA 2022 average reading score by country over the average PIRLS 2021 reading score by country Figure 0.1: Average PISA score by domain and by number of hours a day spent on digital devices at school for learning or leisure. 500 Mean PISA scores in OECE and Balkans 490 480 470 460 450 440 430 420 410 400 none up to 1h betw. betw. betw. betw. betw. betw. more 1&2 2&3 3&4 4&5 5&6 6&7 than 7h Reading: learning Reading: leisure Math: learning Math: leisure Science: reading Science: leisure . Source: Authors’ calculations from PISA 2022 microdata over OECD and WeBa countries. 72 Annex Table 8: Hierarchical Linear Model of Average Scores Over the Whole PISA Samples for 2018 and 2022 Pooled means s.e. T-stat P-value Girl -2.0 0.2 -11.8 0.000 ST250Q01JA 3.2 0.2 16.3 0.000 ST250Q02JA -8.8 0.2 -36.9 0.000 ST250Q03JA -9.5 0.2 -44.6 0.000 ST250Q05JA -0.6 0.2 -2.9 0.004 ST251Q01JA -0.4 0.1 -4.0 0.000 ST251Q03JA 0.3 0.1 2.8 0.006 ST251Q06JA 5.4 0.1 66.6 0.000 ST251Q07JA 1.5 0.1 17.7 0.000 ST254Q01JA -5.9 0.1 -47.7 0.000 ST254Q04JA -4.4 0.1 -45.6 0.000 ST254Q05JA -6.5 0.1 -59.3 0.000 ST254Q06JA 7.1 0.1 56.3 0.000 ST255Q01JA 10 to 25 10.8 0.2 48.3 0.000 26-100 26.4 0.2 110.3 0.000 101-200 36.1 0.3 117.4 0.000 201-500 46.1 0.4 128.3 0.000 More than 500 38.5 0.5 80.1 0.000 ST256Q02JA 1 to 5 -3.2 0.2 -16.7 0.000 6 to 10 2.3 0.4 6.2 0.000 More than 10 11.6 0.4 27.1 0.000 I don't know -0.6 0.4 -1.4 0.161 ST256Q07JA 1 to 5 0.8 0.2 4.6 0.000 6 to 10 -3.0 0.4 -7.8 0.000 More than 10 -4.6 0.5 -9.7 0.000 I don't know 2.6 0.5 5.7 0.000 ST256Q08JA 1 to 5 -0.1 0.2 -0.7 0.482 6 to 10 6.3 0.4 16.1 0.000 More than 10 9.8 0.5 20.7 0.000 I don't know 10.1 0.4 24.5 0.000 ST256Q09JA 1 to 5 -0.2 0.3 -0.6 0.559 6 to 10 -6.1 0.4 -14.6 0.000 More than 10 -18.4 0.6 -31.0 0.000 I don't know -20.2 0.6 -33.6 0.000 ST256Q10JA 73 1 to 5 -3.5 0.2 -15.6 0.000 6 to 10 -1.1 0.3 -3.3 0.001 More than 10 -2.0 0.4 -5.5 0.000 I don't know -2.8 0.5 -5.3 0.000 folang 16.7 0.3 53.0 0.000 urbanization A small town (3 000 to about 15 000 people) 20.3 1.0 20.3 0.000 A town (15 000 to about 100 000 people) 30.8 0.9 32.4 0.000 A city (100 000 to about 1 000 000 people) 39.1 1.0 40.1 0.000 A large city (1 000 000 to about 10 000 000 51.6 1.1 45.6 0.000 people) FISCED ISCED level 1 6.2 0.6 11.1 0.000 ISCED level 2 8.1 0.5 15.6 0.000 ISCED level 3.3 13.3 0.6 23.3 0.000 ISCED level 3.4 14.8 0.5 27.4 0.000 ISCED level 4 13.1 0.6 23.6 0.000 ISCED level 5 11.6 0.5 21.2 0.000 ISCED level 6 17.8 0.5 33.3 0.000 ISCED level 8 3.1 0.6 5.2 0.000 MISCED ISCED level 1 2.4 0.6 4.4 0.000 ISCED level 2 1.3 0.5 2.6 0.011 ISCED level 3.3 5.5 0.6 9.4 0.000 ISCED level 3.4 7.9 0.5 14.5 0.000 ISCED level 4 5.8 0.6 10.4 0.000 ISCED level 5 7.3 0.6 13.2 0.000 ISCED level 6 13.1 0.5 24.4 0.000 ISCED level 8 -9.1 0.6 -15.4 0.000 Intercept 392.1 1.2 314.6 0.000 Source: Author’s calculations from PISA 2018 and 2022 microdata. 74 ABSTRACT Western Balkan countries are consistent participants in international learning assessments like the Program for International Student Assessment (PISA) and Progress in International Reading Literacy (PIRLS). This publication provides an analysis of the recently published data of the PISA 2022 and PIRLS 2021 studies. This study identified four key areas where the performance of Western Balkan education systems could be improved. They include (a) duration of learning in schools, (b) teaching practices, (c) school preparedness, and (d) use of smartphones in schools induced by COVID-related school closures that serve as a major distractor from learning. Based on the analysis, the publication provides policy options and recommendations on advancing the education systems of Western Balkan countries to improve learning outcomes that are critically important for the future well-being of students and the economic development of these countries.