Systems Approach for Better Education Results Is there a Learning Crisis in Punjab? Initial Data Release from the SABER SD Survey August 2019 Acknowledgements This research project has been led by Koen Geven, who also authored the report. There has been an extensive team of research assistants that has supported this research at various points in time. Excellent leadership was provided by Mahjabeen Raza (round 1) and Quentin Daviot (round 2) who coordinated fieldwork from Washington DC, with support from Eema Masood, Fahmina Rahman, Rohit Chabbra, Kabira Namit, and Rucheta Singh. Shubham Adhikari and Ayesha Tahir also contributed as interns. Quentin Daviot deserves an extra mention, for contributing many figures and tables to the report. Owen Ozier at DEC led the sampling process for both rounds. Abdal Mufti coordinated the trainings in Punjab, together with the RCONS team led by Irfan Ahmad, Sarfraz Bhatti and Riaz Ahmed. The survey has also had substantial input from the SABER SD team led by Halsey Rogers, including Ezequiel Molina, Iva Trako, Adelle Pushparatnam and Angela Demas, as well as the LEAPS team led by Jishnu Das, and with input from Alice Danon and Andres Yi Chang. This report received helpful suggestions from Ali Ansari, Ezequiel Molina, Halsey Rogers, Shinsaku Nomura and Isabel McDonald. We are also grateful to the team at LUMS SOE led by Tahir Andrabi as well as various stakeholders of the government (PMIU, PEC, PEF, QAED) for discussions on the early findings from the survey. We would also like to thank various researchers who have contributed ideas to the survey, including Rory Devine, Drew Gitomer, Alina von Davier and Zahra Mansoor. The survey was only possible thanks to the extensive support of the PMIU, particularly the research department led by Abdal Mufti, and previously by Narmeen Adeel and Ali Inam, with support from Faryal Shahid and Tanya Khan. We would also like to extensively thank the PESP-III TTLs Tazeen Fasih and Ali Ansari who have been extremely patient with the execution of the survey and the delivery of the report. SED’s associated departments QAED, PEF and PEC also provided their suggestions to the survey at various points. We would like to extensively thank our partners at DFID, Saima Anwar and Rizwan Azam, as well as Edward Davis in earlier stages, for substantial input, support and endless patience with the survey. Finally, we would also like to thank Luis Benveniste, Keiko Miwa, Omar Arias and Cristian Aedo for their leadership and support to the survey from the beginning to the end inside the World Bank. 1 Table of Contents Acknowledgements................................................................................................................................ 1 Executive Summary ............................................................................................................................... 3 Introduction ............................................................................................................................................ 4 1. Is schooling in Punjab unequally distributed? ..................................................................................... 6 2. Are students learning while they are in school? ................................................................................. 8 3. Are the Main Actors Ready to Improve Learning? ............................................................................ 14 3.1 Are teachers ready to teach?...................................................................................................... 14 3.2 Are learners adequately prepared for school? ............................................................................ 19 3.3 Are principals ready to lead the school? ..................................................................................... 21 3.4 Are school inputs aligned to learning? ........................................................................................ 23 Conclusion: Is there a learning crisis in Punjab? .................................................................................. 26 Appendix: SABER SD Methodology…………………………………………………………………………...28 2 Executive Summary By 2016-17, there were about 14 million students enrolled in primary schools across Punjab, Pakistan. Close to 87% of children of age 6-10 are now in primary school, compared to about 81% five years ago. These children were enrolled in around 119,000 schools of different types (public, private, and public- private partnerships). But are these children in all these schools learning basic literacy and numeracy and if they are not, what can be done about this? This report presents the findings from an innovative survey that aimed to find out whether these children master the basics of the curriculum, and what the main causes of low learning levels are in terms of teaching quality, learner preparedness, school management and school inputs and technology. Teachers: Learners: • 18 % of teachers are absent from the • 32% of children could write Urdu letters classroom before entering school • 56% of teachers have minimum • 25% of children plan to leave school knowledge of the curriculum after finishing Grade 5 • 80% show good practice in classroom • 13 % of children feel too sick to culture concentrate • <10% show good instructional practice • 63 % of children claim that the teacher • <1% show good practice in socio- uses corporal punishment for being emotional support naughty Children in Grade 4 understand 2.0 years of the curriculum 91% correctly answers 3+4 35% correctly answers 384 / 6 School Management: School Inputs and Technology: • 9% of principals are absent from the • 82% of Grade 4 classrooms have school minimum learning environment • 38.2 % has received no training • 56% of schools have minimum • 39% do not know whether school has sanitation facilities handwashing facilities • 24% of schools have multi-grade • 81% do not know the learning outcome classrooms levels of the students in Grade 4 • 25% of Grade 4 classrooms have over • 54% cannot accurately estimate the 40 students content knowledge of teachers in Urdu • 43% of schools has a computer with 3 internet and Math. Introduction In 2018, the World Bank released World Development Report which showed that the world is facing a ‘Learning Crisis’. Following up from that report, the Bank has launched the Human Capital Project to mobilize more resources, including for the improvement of learning outcomes. Subsequently, global leaders have been increasingly focused on solving this learning crisis and have renewed attention for Sustainable Development Goal 4: Quality Education. Pakistan is an early adopter of this Human Capital Project, and the new government is taking the lead to address the root causes of the learning crisis. This report aims to help the government with that agenda, by quantifying the extent of the learning crisis, and to highlight the main causes of the crisis. According the World Development Report, the ‘Learning Crisis’ consists of three main elements. The first is that access to schooling is still unequally distributed. While there have been huge efforts to expand schooling, there are still countries (including Pakistan) with millions of children out of school. Children living in regions with violence, children from poorer families and children with physical or mental disabilities are still often excluded. The second element is that even those who are in school are often not learning anything at all. In Malawi and Zambia, for instance, 89% of students could not read a single word by the end of Grade 2. In India, that figure is 85%. These numbers are important, as children who do not master basic literacy will probably never catch up with the curriculum. In other words, schooling is not necessarily the same as learning. The third element of the learning crisis is that the proximate causes of the learning crisis – low quality teaching, student school readiness, school leadership and school inputs - are not systematically addressed by actors in the system. One of the reasons behind this is that there is no systematic data collection on these factors. To quantify the learning crisis, the World Bank has carried out an extensive research project throughout the last two years. While Punjab is only a single province, it matters profoundly for our global understanding of the learning crisis. Punjab is the largest province in Pakistan, home to about 110 million people, with a variety of enormous challenges in terms of the learning agenda. Moreover, Punjab has been blessed with an exceptional set of policy-makers and has been home to what the Economist calls the ‘most frenetic education reforms in the world’. These reforms have been supported by large donor programs, including from DFID and the World Bank. Despite these ambitious programs, no one has been able to provide an overview of whether these extensive reforms have worked. Third, and this is not a minor issue, there is already a lot of data to work with, meaning it is possible to research the learning crisis in Punjab in great detail. For all these reasons, we believe that what has happened in Punjab will be interesting to a broader audience. Building on a number of other survey instruments (SDI, PETS, SABER), the World Bank’s Education Global Practice has designed the ‘SABER Service Delivery’ survey to quantify the level of learning, while also diagnosing the main causes of the learning crisis in the service delivery chain (teachers, parental support, school management, and school infrastructure). The research team carried out four pilots in the field, programmed the survey into tablets and provided extensive training and certification to a team of enumerators. Throughout 2018, the SABER SD survey was implemented in a stratified random sample of 812 public school, private schools and public-private partnership (PEF) schools, which together provide an overview of education in the province. Findings from the survey were then triangulated with other data sources, including administrative data sources, 4 the LEAPS survey, MICS, HIES and PSLM surveys, as well as data from ASER, EGRA and NEAS. This report is a reflection of that effort and provides the key data to summarize the findings. But note that the report does not analyze the political economy of education reform. This is currently being done by various other researchers, including in the RISE program. 5 1. Is schooling in Punjab unequally distributed? While it was not the aim of the SABER SD survey to collect more data on the out-of-school population, we can use a variety of existing data sources to provide an answer to this question. Collating data from the 2016 Government School Census, the 2016 Private School Census and PEF register data, shows that in 2016-17, about 14 million children were enrolled in about 119,000 primary schools. The growth of the private sector is often credited for the huge increase in schooling, and this dataset confirm that it has remained important. Private schools now enroll around half of all students in the province. Around one fifth of all enrolments (half of private school enrolments) are in public-private partnership programs (managed by the Punjab Education Foundation), which are funded by the government. We should note that there is still some uncertainty about the size of the private sector. in the latest HIES (2016) survey, the estimated size of private school enrolment was over half of all enrolments, while in the 2018 ASER surveys, the estimated size of the private sector is about one-third of all enrolments in this age group. We know from the most recent MICS data (2017/18) that around 13% of primary school aged children (age 6-10) in Punjab are still out of school. This figure is usually referred to as a ‘net enrolment rate’: it is restricted to the children of primary school age (ages 6-10) and is reported by households themselves. An important point is that school access is still highly unequally distributed. Girls are still more likely to be out of school than boys: 11% of boys, and 15% of girls of primary school age (age 6-10) are not in school. Among children from the poorest quintile, the out of school population is at 30%, while in the richest quintile, that is 3%. In rural areas, 16% of children are out of school, while that figure is 8% in urban areas. We can combine the official enrolment data and the survey data to give an estimate of the out of school population at primary age in Punjab. If there are 14 million children enrolled in primary schools, that would mean that about 2 million children remain at home. Out of these 2 million, 1.2 million would be girls, and 0.8 million would be boys. Around 1.8 million of these children have never been enrolled. It should be noted that this is an estimate, which should be updated once the data from the 2017 population census have been released. In the meantime, we can triangulate this with other data sources to show that this estimate seems solid. In the latest round of HIES (2016), the estimated size of the out of school aged population was also estimated at 13%. In the latest round of ASER (2018) of households in rural Punjab, the number is even lower, at about 6%. From HIES (2016), we can also see that Punjab had a much lower out of school population than the other provinces. In KP, the out-of-school population is 20%, in Sindh it is 33% and in Balochistan, it is 36%. From HIES (2016), we can also learn that 91% of the out of school population has never been enrolled in school, while about 10% has dropped out by the time they are surveyed. This is line with earlier surveys (such as the LEAPS survey) which found that households tend not to enroll their children the further they are situated from the school in absolute distance. The good news is that the share of the out of school population seems to have fallen dramatically. In 2013, for instance, the estimated share of children out of school was 26% (PSLM 2013/14). In 2003, that number was 46% (PSLM 2003). The bad news is that Punjab (and Pakistan more broadly) remains an outlier with its large out of school population. Most countries in the region (except 6 Afghanistan) have managed to bring down the out of school population to the single digits. In Bangladesh, for instance, an estimated 6% of children are now out of school, while in India, an estimated 5% of children are out of school. In other words, the out of school population is still a substantial problem, and one that is probably straightforward to solve through tested instruments: public and/or private sector school expansion (including construction), cash transfers, inclusive education policies and school mobilization campaigns. 7 2. Are students learning while they are in school? The answer to this question is perhaps at the core of the debate on the learning crisis. The SABER SD survey finds that virtually all children who are enrolled are falling behind on the curriculum, and some children will never catch up with what they are supposed to learn. Perhaps one of the most intuitive way to understand how students are learning is by showing the percentage that students answer correctly on a subset of test items. We do this in Table 2.1 below, for both our results and some earlier results. Since our sample overlapped with a set of rural schools that were surveyed between 2004 and 2006 (LEAPS), we can compare what has happened over time. In the right column, we display the old numbers (LEAPS), in the middle column, we show the new numbers for a subset of LEAPS schools and in the right column we show the population data for the entire new sample. The table shows that a large population of students correctly answers to some basic items, with over 90% of students are able to add 3 and 4, to write basic letters of the English alphabet and to correctly match pictures to words in Urdu. A substantially smaller share of students (48%) can do a triple-digit subtraction problem with carry-over (238-129); again a smaller share (35%) can do a triple-digit division problem (384/6). The table also shows that, on average, the percentage correct on these items has increased quite substantially. For instance, while the original LEAPS survey found that only 32% of pupils correctly answered 238 – 129, we found that nearly half of the pupils answer this correctly. Similarly, while the LEAPS team found that 70% of pupils could write the letter ‘E’ correctly in between ‘D’ and ‘F’, we found that 92% of student could do so. Table 2.1: Comparing the SABER SD data to the LEAPS data in terms of percentage correct Test item Domain, Level SABER SD SABER SD LEAPS (2018) (2018) (2004- 2006) All schools LEAPS Schools Only 3+4 Math. Grade 1 91 90 92 238-129 Math, Grade 2 48 48 32 384 / 6 Math, Grade 4 35 30 19 Write the letter “D” English, Grade 1 98 96 86 Fill in the right letter D _ F English, Grade 1 92 92 70 Fill in the missing letter BA__ English, Grade 1 79 74 45 Tick the correct answer to Urdu, Grade 2 95 94 52 match the picture (picture of house) Tick the correct answer to Urdu, Grade 2 97 97 73 match the picture (picture of book) Note: Weights applied to SABER SD data. The LEAPS data are taken from the main LEAPS report (Andrabi et al, 2008a). 8 While ideally, we would want students to do all of How to Summarize the Learning Crisis? A this correctly, there is still some good news here for new measure called ‘CAYS’ policy-makers. First, this means that most students know some basic things. In fact, nearly all students Many, if not most, reports on learning use a pick up something from the curriculum, even if measure of ‘percentage correct’ to show how these are lower level items. Secondly, these results much of the curriculum is being absorbed by indicate that students seem to be learning more students. The difficulty is that it is usually not today than they did roughly fifteen years ago. Of straightforward to understand what that means course, we should caveat these findings somewhat, in terms of the overall learning pattern of since these are only a limited sub-sample of items children. We use a different measure here, that do not cover the entire curriculum. which we call the Curriculum Adjusted Years of Nevertheless, these are strong signs that learning Schooling (CAYS) to quantify how much of the levels in the province are improving as the schools curriculum is mastered by Grade 4 students1. in the LEAPS study are typically the most This measure is locally anchored (in the official disadvantaged ones. curriculum as materialized in the teacher guides and student textbooks). Another way to make these results more meaningful is by converting the test results to a CAYS are constructed in the following way. In metric called the ‘Curriculum Adjusted Years of Pakistan, standardized assessments are Schooling’ (CAYS), which we explain in the Box. considered as passed if a student answers 33 Using this conversion, we can say that an percent of the questions correct. We can then average student in a Grade 4 classroom has consider that anyone above this threshold only mastered the Grade 2 curriculum (Figure would pass a grade level. For example, a 3.2). The averages for boys and girls are slightly student who is in Grade 4, but has not even different. Boys do a little worse, mastering at least mastered 33% of the Grade 1 curriculum, 1.9 years of the curriculum, as compared to girls would then have 0 years of CAYS (even if they who acquire 2.1 years. Note again that by have had over 3 years of schooling). If another ‘mastering at least 1.9 years or 2.1 years’, we mean student has understood at least 33% of the that students pass at least 33 percent of the test Grade 1 curriculum (but not the Grade 2) items at that particular grade level. In other words, curriculum, that student will have 1 year of one could also say that the average student in CAYS. Yet another student may have Grade 4 has mastered over 33 percent of what is mastered above 33% of the Grade 1 being taught in Grade 2 but less than 33 percent of curriculum, and 33% of the Grade 2 curriculum, what is being taught in Grade 3. and will have mastered 2 years of CAYS, and so on. On average, across subject fields, children in Grade 4 have only mastered 2 years of the curriculum (see Figure 2.1). Girls are about 2 months ahead in the curriculum as compared to boys (see Figure 2.1). Underlying this figure are some substantial problems. About 40 % of boys, and 29% of girls are stuck at Grade 1 level or below, meaning that they will probably never catch up with the curriculum. This is a problem because students need basic literacy and numeracy skills acquired in the early grades to proceed with learning across their schooling career. 9 Figure 2.1: Children, on average have mastered 2 years of the curriculum, while having been in school for 4 years. Boys are about two months behind girls in their understanding of the curriculum. Note: This figure uses 18,055 students’ observations (9,363 boys and 8,692 girls) and estimations are weighted using sample weights. The assessment was carried out at the beginning of the academic year. Figure 2.2: Distribution of the Curriculum-adjusted years of schooling (33 percent threshold) Note: This figure uses 18,055 students’ observations (9,363 boys and 8,692 girls) and estimations are weighted using sample weights. If we look a little deeper in the distribution, there is still a substantial group of children who do not understand the very basics of what they were taught in Grade 1 and 2. Figure 2.2 below shows that 3 % of boys and 1 % of girls in Grade 4 do not understand the Grade 1 curriculum (i.e. they are at ‘Grade Level 0’), while 37 % of boys and 28 % of girls just understand the basics of the Grade 1 curriculum 10 (Figure 3.5). These students are unlikely to ever catch up with the curriculum, as they are not even mastering very basic literacy and numeracy concepts after four years of schooling. By contrast, only 27 % of boys and 31 % of girls in Grade 4 have mastered the basics of the grade 3 curriculum. On average, only 3 % of boys and 4 % of girls understand the Grade 4 curriculum at which they are being taught. Students learn too slowly across content domains (Figure 2.3). In Urdu, students answer correctly 60% of the test items in which they were asked to associate a word to a picture, and 57% of items in which they construct letters or words in the Urdu alphabet. Students struggle with more difficult domains in Urdu, such as grammar correction (14% correct), and on reading comprehension items (18% correct). In English, students answer correctly to 57% of items in which they associate a word with a picture, while they get 52% of items in which they construct letters and words. Students correctly construct an English sentence 31% of the time, answer 33% of a reading comprehension item correctly, while they can correct grammar only 16% of the time. In mathematics, students answer 67% of addition and subtraction items correctly, but they get only 25% correct of multiplication and division items. This means that most students master the basics of number sense and topics like carry-over, but most students cannot do mathematical operations that are at grade level. Figure 2.3: Average percentage correct for the main curriculum categories by subject Note: This figure has been constructed with sample weights out of the 18,055 students that took the assessment in English, Urdu and Mathematics. Each question has been labeled according to the subject and the domain (Grammar, Sentence, etc.) following the textbook classification. We then graph the average of the average correct on the collection of items within a particular domain. Student in private schools are about 2 months ahead in the curriculum, as compared to students in public schools. Nevertheless, we should probably not conclude that private schools are much better than public 11 schools. The difference between public and private schools is most visible for English (see Figure 2.4), which is typically the language of instruction in private schools, while there is virtually no difference for Urdu or for Math. More importantly, the difference between public and private schools is smaller than the difference between urban and rural schools, when we control for other factors. That being said, students in public-private-partnership schools (PEF schools) do tend to learn more than their peers in both public and private schools on all contain domains (except English, where private schools are still ahead) if we control for background factors. This is a surprising finding for the PEF schools, since the poorest children are typically enrolled in those schools (about 34 % of all children enrolled in PEF schools are from the poorest socio-economic quintile, versus 25% of children in public schools, and about 9% of children in private schools). Figure 2.4 – Density plots of average test scores of public, private and PEF schools Note: Test scores estimated using IRT (1PL) models, then standardized with mean of 500 and standard deviation of 100, the distribution is of mean test scores at school level. Taken together, these results show that we need to be specific when we discuss the ‘learning crisis’ in Punjab. It is not that students don’t know anything at all. In fact, most students in school are learning. The average student understands the Grade 2 curriculum, with girls pulling slightly ahead of boys. The survey found that nearly all children (91 %) in Grade 4 can answer a simple item like 3+4, and 98% of children can write the letter ‘D’. A further 97% of children can tick the correct answer 9 (in Urdu) to match a picture of a book. Even in rural schools, 48% of children can do triple digit subtraction. These findings can be triangulated with other data sources. The 2018 ASER data show that by grade 3, 28% of children in rural Punjab can read a story in Urdu, while that share is 50% by Grade 4 and 69% by Grade 5 (without significant drop-out rates between these grades). Similarly, by Grade 3, 7% of children can read an English sentence, which climbs to 33% by Grade 4 and 64% by Grade 5. Am assessment 12 of reading in Punjab in 2013 showed that the median sampled student in Grade 3 can read 25 words per minute. By Grade 5, the median student could read 84 words per minute. Assessments done by the national assessment center NEAS show that Punjab typically scores much higher than other parts of the countries in multiple subjects. In other words, the majority of children in Punjab are learning something, and most children are advancing in the curriculum. Importantly, these findings are not unique to this survey: other data sources, including from ASER, EGRA and NEAS confirm that students are learning, even if slowly. On the other hand, there are enormous problems that require a complex web of solutions. There are an estimated 2 million children of primary school age in the province who remain out of school. Among those who are in school, 40% of boys and 29% are stuck in a low learning equilibrium from which they are unlikely to recover unless they receive basic numeracy and literacy instruction. 13 3. Are the Main Actors Ready to Improve Learning? 3.1 Are teachers ready to teach? An important prerequisite for learning is that teachers show up in the classroom (Chaudhury et al., 2006). To measure whether this happens, enumerator teams visited a sample of schools on an unannounced day to check whether teachers on the roster were in school and in class teaching. We found that on a regular day, 18 % of the teachers are absent from their classroom. In other words, on any given day, just over 1 in 6 classrooms in Punjab is an ‘orphan classroom’. More worrying perhaps is that in 14% of schools, the teacher is not even present in the school. Internationally, teacher absence is a common problem (see Figure 2.3). In some countries, particularly in Sub Saharan Africa, the problem is so big that nearly half (44%) of teachers are absent from the classroom on any given day. In this context, it is perhaps noteworthy that Punjab performs better than many other regions and countries of the world, including India, Bangladesh, Laos and Sub-Saharan Africa (SSA). At the same time, Punjab performs worse on school absence than Latin America or in Afghanistan. Figure 3.1.1: School & Classroom absenteeism in Punjab, compared to other countries Note: International data from various SDI surveys. School absence data is observed via unannounced visits in a sample of 84 schools. We have government school absenteeism data from PMIU on 44 schools and we can compare the observed school absenteeism to the declared governmental absenteeism. Observations are not weighted. Learning always suffers from being in the classroom without a teacher. Yet, that doesn’t mean that teacher absence is necessarily the fault of the teacher. Teachers, like all humans, get sick and a replacement may not be immediately available. Teachers also have many different demands, such as vaccinations, attending trainings, or helping local administrators. In many countries, improved monitoring by the government allows for a better understanding of when teachers are absent and why this is the case. 14 The survey assessed teachers’ mastery of the curriculum through a basic content knowledge assessment. Based on this assessment, only 56% of teachers across the province have a basic mastery of the curriculum (they correctly answer more than 80% of items in the assessment). There are important differences by school types. In public schools, 68% of teachers have a basic mastery of the curriculum, while in private schools that number is 44% and in PEF schools, the number is 34%. This means that particularly in the private sector, there is a large share of teachers who do not understand what they are supposed to teach. Figure 3.1.2 Teacher grade achievement (80% requirement) Note: Mastery of content knowledge defined as scoring more than 80% on the content knowledge assessment. Based on assessment of 3,373 teachers in 812 schools; 2,861 teachers took the English assessment, 2,911, the Math assessment and 3,049 the Urdu assessment. Assessment domain depending on the subject(s) taught by each teacher. Estimates use sampling weights. Interestingly, Table 3.1.1 indicates that public school teachers understand the curriculum better at every grade level. On this overall score, nearly 70% of teachers master the primary education curriculum while for teachers in private schools that figure is 44% and for PEF schools, the number is 34%. The results for Grade 4 students is also similar. On average 65 % of public-school teachers answer correctly 80 % of the Grade 4 curriculum questions. This is not true for PEF and private school teachers though as only 35 % and 44 % of them respectively, are able to master Grade 4 curriculum. 15 Table 3.1.1: Share of teachers mastering of the curriculum by school type (80% minimal requirement) Public Private PEF Masters the full test (>80%) 68 44 34 Grade 1 (>80%) 87 86 84 Grade 2 (>80%) 75 58 51 Grade 3 (>80%) 67 52 51 Grade 4 (>80%) 65 44 35 Grade 5 (>80%) 63 48 46 Grade 7 (>80%) 61 45 47 Note: Mastery of content knowledge defined as scoring more than 80% on the content knowledge assessment. Based on assessment of 3,373 teachers in 812 schools; 2,861 teachers took the English assessment, 2,911, the Math assessment and 3,049 the Urdu assessment. Assessment domain depending on the subject(s) taught by each teacher. Estimates use sampling weights. Teacher behavior in classrooms was observed by certified enumerators who used a detailed rubric to score whether teachers display good teaching techniques. We found that overall, 80% of teachers show good practice in creating a positive classroom culture. This means that teachers promote positive behavioral expectations (e.g. for both genders) and that they do not waste much time on non- teaching activities. Nevertheless, less than 10% of teachers show good practice in instructional techniques. This means that most teachers do not explain the goals of the lesson clearly, they do not check whether their students understand the lesson, and they provide poor feedback to students. Moreover, less than 1% of teachers show good practice in socio-emotional support. This means that students do not have autonomy in the classroom, they are not provided thinking tasks and students do not get to play a role or ask questions about why they are learning certain things. 16 Figure 3.1.3: Distribution of the teaching practices evaluated by the Teach scale (light yellow and green indicates following good practice) Note: The figure has been constructed from 809 Teach observations in 809 schools. Estimations use sampling weights. Each of the 10 sub-domain have a 1-5 score. Each classroom observation is an average of two datapoints from certified enumerators that used the Teach rubric. Scores are collapsed into four categories: Low is defined as less than 2.5, medium low 2.5-3.5, medium high 3.5-4.5, and high 4.5-5. Table 3.1.2 below shows the detailed scores per component, broken down by school type. Teachers perform poorly when it comes to instructional quality; only 5 % of teachers follow good practice overall. Specifically, 79% of teachers have difficulties with lesson facilitation1, which means that they do not promote lesson comprehension and do not explicitly articulate the objectives of a lesson. 91% of teachers do not adequately check for understanding (and when they do, they do this poorly), which is a basic practice needed in every lesson. Teachers typically do not use strategies like ‘cold-calling’, when a student at random is asked to answer a question related to the lesson. More worryingly is that teachers provide poor feedback: 88% of teachers show bad practice, meaning that students either do not get any feedback on their work, or that comments are of a very general nature. Only 12% provide timely and useful feedback from which students can learn. The worst practice teachers have is that they do not support critical thinking (only 2% of teachers follow good practice). That means that teachers do not ask thinking or open questions, they do not provide thinking tasks (for example, to consider the pros and cons of a particular character in a story), and students do not ask for thinking tasks (e.g. to ask why they are reading a particular story). In terms of socio-emotional support to students, teachers fare even worse: only 1 % of the observed teachers follow good practice overall. That means first that teachers do not develop students’ autonomy 1 Note this is negatively framed, while the Table is positively framed. In other words, the text reports 100% minus the value observed in Table 4.7 17 (only 8% of teachers follow good practice in this area); pupils do not have any choices during a lesson, they do not take on different roles in the classroom (e.g. explaining a problem they understood to their peers) and students are typically not expected to volunteer for tasks. Secondly, teachers do not promote perseverance (only 2% of teachers follow good practice in this area); they do not reward effort (i.e. rewarding trying hard, without necessarily having the right answer), they do not have a positive attitude towards student challenges, and they do not encourage goal setting. Thirdly, teachers do not promote socio-emotional skills (only 3% of teachers followed good practice). This means that teachers do not let students collaborate with one another through peer interaction, and do not promote students to develop interpersonal skills like empathizing, emotional regulation, or perspective taking. Table 3.1.2: Share of teachers meeting good practice under each component by school type Total Public Private PEF Classroom Culture 79 82 75 76 Supportive Learning Environment 66 70 62 59 Setting Positive Behavioral Expectations 33 36 27 30 Providing Equal Opportunities to Learn 89 89 90 88 Instruction 5 7 4 3 Lesson Facilitation 21 28 10 12 Checking for Understanding 9 11 6 5 Providing Feedback 12 14 12 10 Critical Thinking 2 3 3 1 Socio-emotional skills 1 1 0 2 Giving Students Autonomy 8 11 5 3 Stimulating Perseverance 2 3 2 1 Nurturing Social-Emotional Skills 3 2 0 7 Note: The table has been constructed based on 809 TEACH observations in 809 schools. Estimations use sampling weights. Each of the 10 sub-domains has a 1-5 score. Each classroom observation is done by two enumerators and we average the findings. For each domain and sub-domain, we have a 1-4 score that is: Low is defined as less than 2.5, medium low 2.5-3.5, medium high 3.5-4.5, and high 4.5-5. There is also a substantial share of teachers who are not well prepared to teach, although this share varies by school type. In public schools, most teachers have a certificate to teach, although 10% of teachers do not have any of the main certificates (PTC, CT, B.Ed or M.Ed). In private school, by contrast 63% has no teaching certificate, and 77% of teachers has less than a bachelor’s degree. Similarly, in PEF schools, 77% of teacher does not have a teaching certificate, and 87% has less than a bachelor’s degree. This is reflected in teacher salaries, as a median public-school teacher earns about PKR 29,000 per month, while a median private school teacher earns PKR 5,000 per month and a median PEF teacher earns PKR 6,000 per month. 18 3.2 Are learners adequately prepared for school? Many children learn to spell, to write basic words and to develop a substantial vocabulary in early childhood and typically years before entering school. Most of these foundational skills are developed in the family, through interaction between parents, children and siblings. For families whose home language is not the same as the schools’, these kinds of interactions do not prepare children for what they will experience in school (even if those interactions do prepare them for life in many other ways). In Punjab, that is a key difficulty in classrooms, as only 13% of children speak the language of instruction at home (Urdu) and less than 1% speaks English at home, which is often the language of instruction in private schools. The mastery of basic concepts that school builds upon is an important element of children’s ‘s chool readiness’. We measured such ‘school readiness’ through a retrospective2 parent interview. Table 3.2.1 shows that more than half children cannot write any Urdu or English letters before starting school. About two thirds of children cannot write any words in Urdu or English. Perhaps the most surprising thing about these findings is that, on average, Urdu and English are equally unfamiliar for children, even if more households speak Urdu at home. About a third of children cannot even count up to ten by the time they enter school. Further analyses (not shown here) also indicated that there are no major differences between school types, although children in private schools are slightly better prepared than those from the other school types. Table 3.2.1: Children’s preparedness before entering school Total Low SES Average High SES SES Write Urdu letters (%) 32 16 33 43 Write Urdu words (%) 21 10 22 28 Write English letters 31 15 32 42 (%) Write English words 20 10 21 28 (%) Count to 10 (%) 67 53 68 76 Note: Source is a parental interview of a sub-sample on 3,727 children. Socio-economic status (SES) based on a principal component analysis of a list of household items and other background variables. Sitting in school day-after-day requires a significant amount of energy by itself, but learning requires motivation and ambition. If students if they do not plan to finish their grades, if they do not enjoy going to school, or if they find the content too difficult, they are unlikely to absorb much of what is happening in the classroom. We therefore asked a subsample of pupils about their school experience. In general, Table 3.2.2 shows that students plan to leave school at typical exit points (Grade 5 is the end of Primary School, Grade 8 is upper secondary school, Grade 10 and 12 are exit points in higher secondary schooling). The data show that on average, students lower socio-economic strata are mostly 2Retrospective indicating that the interviewer asks parents to reflect on the time when the child was just entering school. Since we do not observe those children histories directly, this is the only way we can access this information. 19 aiming to finish school by Grade 10, while students from higher socio-economic strata are aiming for Grade 12. That being said, still about 1 in 5 or 1 in 4 students from all strata expect to leave school by Grade 5. Table 3.2.2: Students’ Expectations of School Completion Low Average High Total SES SES SES Up to Grade 5 (%) 25 24 27 21 Up to Grade 8 (%) 16 18 17 14 Up to grade 10 (%) 32 36 33 30 Up to Grade 12 (%) 26 22 23 36 Total 100 100 100 100 Note: Source is a parental interview of a sub-sample on 3,727 children. Socio-economic status (SES) based on a principal component analysis of a list of household items and other background variables. Parents in Punjab are not very engaged with their children’s education, nor do they know how well their children perform. About 42% of parents of the worst performing children tend to think that their child performs better than most other children in their classroom. More than a third of parents never speak to the teachers in the school, about a quarter of parents never ask the child what they learn in school, and about three quarters of parents never see their child reading a book. That being said, 38% of parents do hire a private tutor for their child at an average cost of 500 PKR (or about 4 USD) per month, on top of other school costs which are already about 1000 PKR per month on average. 20 3.3 Are principals ready to lead the school? School leadership is increasingly important to make sure that the school improves over time, as school leaders can respond directly to the difficulties faced in the classrooms. In Punjab, most school principals in the province are also teachers, and while they are highly educated, many have never received specific training to prepare them for their jobs as principals. On a given day, about 9% of principals are absent from the school, meaning that the leader is not always present to intervene. The SABER SD survey assessed principals’ knowledge on three main domains: (i) School Infrastructure, (ii) Students’ Academic Knowledge in Language and Mathematics and (iii) Teachers’ Ability in Content Knowledge, Pedagogical Knowledge and Teaching Practices. The central idea behind this assessment was to ask the principal for their opinion or impression of problems in the school, which we could easily verify through observations in the survey in other modules. This approach is perhaps relatively straightforward for school inputs: we ask the principals questions about the presence of electricity or the share of textbooks and then ask the enumerator to make direct observations in the classrooms. This gives us a simple metric of the share of principals who do not know the basic infrastructure of their school. In Table 6.5, we present the principal’s knowledge of the basic school environment. This is mostly good news. Principals tend to correctly estimate whether there is water available in the school (97% correct), and whether children have textbooks (93% correct within a 20% bound). They tend to overrate the handwashing facilities at the school, however (61% correct). The latter is slightly worrying, since in general principals tend to think that the toilets have soap and water, while this is often not the case (as will be discussed in the last chapter). Table 3.3.1: Principal knowledge of school environment Share of principals who are right Drinking water availability 97 % Handwashing (soap + water) 61 % Textbooks 93 % Note: Sample is 811 school principals. We provided the different assessment instruments to the school principals so that they could study what we were trying to understand. We then asked principals to estimate the share of students in the selected Grade 4 stream that would correctly answer to a set of specific test items from the student assessments in Urdu, English and Mathematics. We also asked principals to tell us how they expected teachers to perform on the content knowledge exams, the pedagogic content knowledge exam and the classroom practices. We then compare the principals’ answers to the responses on the assessments and the observations, within categories of (0-20%, 20-40%, 40-60%, 60-80% and 80-100%). The result is a measure of principal knowledge about student and teacher performance. We place the results in Table 6.6. 21 In general, we observe that most principals do not know how well their students and teachers are performing. On the student assessment, only 19% of principals estimate the correct level of their students (within a 20% bound). On average, the principals overestimate students’ performance by 37 percentage points, meaning that they are substantially off from how much students actually learn. On the teacher content knowledge, between one third (32%) and almost half (46%) of principals estimate how their teachers are doing correctly. Again, on average they are about 25-30% off from the actual values that we observe in the test. For teacher quality in the classroom (classroom observation), only 9% of principals provide the right answer. Table 3.3.1: Principals’ knowledge about teachers’ and students’ performance Share of Among those who get it wrong. How principals who much do they underestimate the are right problem? Student content knowledge 19 % 37 % Teacher Urdu content 46 % 26 % knowledge Teacher Math content 46 % 28 % knowledge Teacher English content 32 % 30 % knowledge Teacher pedagogic content 9% 38 % knowledge Teacher classroom practices 13 % 32 % Note: Sample is 811 school principals. This means that principals currently do not feel that their job is to lead the school in an instructional sense. Principals are not aware of how much (or how little) children learn, and whether teachers are adequately responding to the learning needs of the children. In fact, most principals think that the main constraint to learning are the lack of parental involvement and the difficult family background of students. Only a handful of principals believe that teachers’ absenteeism, knowledge or motivation is a key constraint to learning. 22 3.4 Are school inputs aligned to learning? Improvements in infrastructure tend to increase enrolments, but they do not necessarily improve learning itself. This is because infrastructure improvements do not typically affect what is going on in the classroom. The survey has found that most schools (82%) have a minimum learning infrastructure in place (see Figure 3.4.1). A minimum infrastructure means that the classroom has a blackboard, enough visibility, chairs and desks, pencils, and all children have textbooks and exercise books. With regards to sanitation, there are still some major concerns (Figure 3.4.2). Handwashing facilities are lacking in about 41% of schools however, and about 15% of schools do not have minimally functional toilets. Figure 3.4.1: Classroom learning-environment infrastructure Note: Sample of 812 schools A recent high-level report concluded that about half the world’s schools lack clean drinking water, toilets and handwashing facilities (UNICEF & WHO, 2018). Lack of basic hygiene or global infrastructure put children at risk of health issues, affect learning and school completion. Punjab is about average, as half the schools lack either toilets, handwashing facilities or clean drinking water (Figure 3.4.2). More than 80 % of the schools have toilets that are clean, private and accessible to the students and almost all the schools have a clean drinking water source. While 96 % of the schools have a clean drinking water source. However, only 59 % of the schools have soap and water for the students to wash their hands. 23 Figure 3.4.2: Sanitation and handwashing facilities in the schools Note: Sample of 812 schools The SABER SD survey also analyzed whether classrooms are overcrowded (Table 3.4.1). One of the key indicators of inadequate infrastructure is that grades are combined, or students are put together in very large classrooms. In recent years, the government has hired several batches of teachers to make up for these shortfalls but has not necessarily constructed new schools. Still around a quarter of schools have a multi-grade classroom, and the number is about equal across public and private schools (although PEF schools tend to have fewer multi-grade classrooms). In the public sector, it is also very common for classrooms to have more than 40 students. Today, still close to half of classrooms in public primary schools across the province have more than 40 students, while in private and PEF schools that is quite rare. Table 3.4.1: Multi-grade and large classrooms Total Public Private PEF Multi-grade classroom 24 26 26 17 Large classrooms (Share of classrooms with >40 students) Katchi 23 27 15 19 Grade 1 30 48 7 8 Grade 2 29 45 6 9 Grade 3 24 37 5 9 Grade 4 25 40 5 4 Grade 5 23 36 5 7 Note: Sample is 812 schools, weight applied. 24 Security of schools has been a concern for several years across the province. At present, 90 % of the surveyed schools have boundary walls, 64 % are also equipped with razor wires over their walls and 28 % has a concrete barrier around the school. About one third, or 30 %, of schools have an armed guard. There is heterogeneity across school types. Public schools have most security features, with almost all having a boundary wall, and roughly three quarters having razor wire over the wall. In private schools, in contrast, these security features are less present, with only 75 % of schools having a boundary wall, and only about one-third having razor wire. Despite major investments in recent years, many schools still lack enough classrooms and teachers. The survey also found that about a quarter of schools still has multi-grade classroom, and nearly half of classrooms in public schools are overcrowded, having over 40 students. Security of the schools has increased substantially, and about 90 % of schools now have a boundary wall and about two thirds of schools have a razor wire as well. About one third of schools have an armed guard. 25 Conclusion: Is there a learning crisis in Punjab? This initial data release has aimed to provide a summary of the learning crisis and its proximate causes in Punjab, Pakistan. This survey was conducted in a representative sample of schools across the province, and can be broken down into public schools, private schools and public private partnership schools (under the Punjab Education Foundation). The survey built on a wealth of research and data collection efforts and aims to add value by focusing on a representative sample of schools. The report aimed to ask three major questions, to which we provide the answers below, while also discussing the implications for policy-makers. 1. Is Access to School Unequally Distributed? The Out-Of-School population remains a key element of the learning crisis in Punjab, despite impressive reductions. Around 13% of children in the age-group 6-10, or an estimated 2 million children in total remain out of school. Nearly all (91%) of these children have never been enrolled. While this number has come down substantially (from 26% in 2013), this remains a crisis. Most of the world (including most countries in South Asia) has been able to reduce this number to single digits. The policy instruments are well known: expanding public and/or private schooling, using cash transfers, inclusive education policies and school mobilization strategies. 2. Are Children Learning while in School? The second element of the global learning crisis is that in many countries around the world, children are not learning anything at all while in school. The SABER SD survey shows that there is evidence of this problem in Punjab, even though children are learning the very basics of the curriculum. The survey has found that the average child enrolled in Grade 4 understands about 2 years of the curriculum, using a rather low benchmark of curriculum mastery. Importantly, around 40% of boys, and 29% of girls are stuck in a low-level learning equilibrium from which they are unlikely to recover. That being said, children are learning. Most children were able to fill out a 1.5 hour written assessment independently, and most got the very basics right. Out of all 4th grade pupils, 91% know the answer to 3+4, and even in rural schools, 48% of children can do three-digit subtraction with carry-over. This situation has improved remarkably in the last fifteen years or so. In 2004, 52% of children in rural schools could match a picture of a house to the correct Urdu word. In 2018, that share is 94%. 3. Are the main Actors Ready to Improve Learning? The SABER SD Survey has found that the main actors are not well prepared to address the learning crisis. First, teachers are still often absent (18% of classrooms do not have a teacher in front of them), and teachers are not well prepared to teach in a way that students understand. Second, children are not well prepared for schooling by their parents. They do not know the alphabet, counting or basic words before they get to school, and parents do not track learning of their children while in school. Third, school principals do not track what teachers know and do, and they do not know whether and how much children are learning. Finally, while school infrastructure has improved a lot, there are still many overcrowded classrooms, particularly in the public sector. The government has recently launched an ambitious agenda for the education sector in the province called the ‘New Deal’ and is currently operationalizing an Education Sector Plan. The goal of the strategy is that ‘[e]very student in Punjab should be able to demonstrate knowledge and understanding across a 26 set of agreed student learning outcomes (SLOs) in basic competencies by the end of primary school and develop increased higher-order thinking in secondary school’. With this survey, the World Bank has aimed to help the government identify key priority areas for that strategy, and we hope to be of continued assistance to help the government make that vision a reality. 27 Appendix: SABER SD Methodology The SABER Service Delivery (SD) tool was developed in 2016/17 by the Education Global Practice at the World Bank. The main goal of the survey is to identify and uncover the barriers that obstruct student learning outcomes in low and middle- income countries. It does so by building an evidence base on learning outcomes and metrics that may affect learning in schools. The theory of change behind this initiative is to help policy-makers make decisions based on the latest evidence, from a representative sample of schools. The SABER SD survey builds upon and contributes to two current World Bank Group initiatives that produce comparative data and knowledge on education systems: SABER (Systems Approach for Better Education Results) and SDI (Service Delivery Indicators). It also builds on earlier facility surveys, such as the QSDS (Quality of Service Delivery Surveys) and PETS (Public Expenditure Tracking Surveys). Conceptually, SABER SD is an extension of the SABER framework, which documents and assesses education policies to reveal how inputs are transformed into outcomes. It builds on the evidence base from the core SABER domains and captures implementation dimensions of those policies. Combined, the data provide a diagnostic that assesses the functionality and state of a given education system (Vegas et al 2011; Hanushek, 2003; McEwan 2015; Ganimian & Murnane, 2016). This includes issues like classroom practices (Kane el al., 2013; Kane & Cantrell 2013), school leadership and management practices (Bloom et al., 2015; Lemos & Scur 2016), teacher pedagogical content knowledge (Phelps 2015), and the role of the home environment and parental involvement (Davis-Keane 2005). While the SABER SD survey maintains comparability with SDI, its focus on the education sector allows it to go deeper and innovate in key areas. Table 1 below provides an overview of different survey modules included in the SABER SD survey. Table A.1: Overview of the SABER SD survey structure Module Description 1 Rosters Provides a list of all teachers in the school, all students in Grade 4 classrooms, and basic demographic data (e.g. sex, age). 2 School Collects information on school infrastructure, state of facilities, number Information of students, number of shifts, number of grades taught, and number of teachers. 3 School Collects information on school governance, management, and finance. Governance, Also collects professional history and background of school principals. Management, The instrument also assesses management quality in two ways: by and Finance using the World Management Survey and direct assessment of principals’ knowledge of the school environment. 4 Classroom Measures teacher practices and classroom behaviors with the use of an Observation open source tool called Teach, which isdesigned to measure the quantity (i.e. time on task) and quality (i.e. classroom culture, instruction, socio-emotional support) of teaching. This is supplemented with an optional student report on their perceptions of teacher practices. 28 5 Pupil Measures learning outcomes in mathematics and reading in the Assessment language of instruction. Aligned to the LEAPS assessment, which has been implemented in Punjab since 2003. The module also includes innovative measures of student non-cognitive or socio-emotional skills (such as “Grit” and “Growth Mindset). 6 Teacher Collects data on the professional history and working conditions of Interview teachers. Also assesses teacher knowledge of mathematics and the language of instruction; includes a written test that measures both content knowledge and pedagogic content knowledge. 7 Parent Captures information on the home environment, parental involvement, Interview and socio-economic background of students. A.1 Sample structure The goal of the survey is to provide a representative picture of all primary schools in the province of Punjab, Pakistan. While this a straightforward task in theory, this is extremely difficult in practice. The first step was to create a sampling frame from the extremely dynamic schooling system in Punjab. With very efficient support from the PMIU and the PESP-III project, as well as DFID support, several censuses of the school system have been carried out, which include the 2016 Public School Census, the 2016 Private School Census and a census of program schools supported by the Punjab Education Foundation3 (for an overview, see Table 1). The latter category includes schools under the Foundation Assisted School (FAS) program, the Education Voucher Scheme (EVS), the Public School Support Program (PSSP)4 which was still under PEF at the time of planning the survey, and the New School Program (NSP). The research team put together these recent censuses to create a full overview of all primary schools in Punjab. This sampling frame includes 118,691 schools offering primary education to at least 1 student. in all 36 districts of Punjab. The census data makes clear that sector is subdivided into three main sectors, namely a public sector that caters to roughly 6.72 million students (about 48% of the total), a private, mostly for-profit, sector which caters to 4.26 million students (or 30% of the total) and a public-private partnership model (the Punjab Education Foundation) that caters to about 3 million students (or about 21 % of the total). 3 The Punjab Education Foundation was established under the Punjab Education Foundation Act of 1991 as an autonomous statutory body to encourage and promote education on non-commercial/ non-profit basis though Public Private Partnership (PPP). 4 As mentioned in the introduction, the PSSP schools have recently been spun off to another authority. 29 Table A.2 – An overview of the primary school system in Punjab in 2016 Number of schools offering Total number of Relative Median primary grades to at least 1 students in sector shares school size student millions (by number of (primary (primary grades students) students only) only) Public schools 51,210 6.72 48 % 89 Private schools 53,958 4.26 30 % 52 PEF schools 13,523 3.00 21 % 170 (public-private partnership schools) Totals 118,691 13.99 100%5 75 Note: PEF data from PEF programs, Public School Census 2016, Private School Census 2016. Including all schools on which we have primary enrolment data and in which enrolment in primary grades is at least 1 student. One thing that stands out from this data is that the median primary school in Punjab is still very small, enrolling around 75 children. This is partly because of the geographical and demographic nature of the province, which has a dispersed population and large rural areas. The government tries to cater to the out of school population by placing a public school in every village. The private sector is also extremely active, and places schools in villages that typically also have a public school. The PEF, on the other hand, may tend to place schools in remote locations where there are fewer competitors, and hence these schools may be somewhat bigger. In order to be able to present estimates at the sectoral level, the sample was stratified in several ways. Strata included the three sub-sectors, i.e. public schools, private schools and public private partnership schools and urban and rural areas. A separate stratum was created for LEAPS schools (including both public and private schools), in order to create an overlap with this historical sample. This would allow us to provide an understanding of whether learning outcomes and the schooling environment are improving in rural schools over time. Third, due to operational constraints, it was impossible to draw a random sample of all schools at province level. The survey team drew a convenience sample of 6 districts that is representative of North, Central and South Punjab, which includes both richer and poorer districts. A convenience sample was appropriate due to security and operational constraints of working in Punjab, particularly in an election year (elections were held two months after survey completion). The selected districts were Attock, Faisalabad, Lahore, Muzaffargarh, Rahimyar Khan and Sargodha. In consultation with the government, it was decided that schools in military areas (including the cantonment area in Lahore) were excluded. In order to deal with potential refusals and closed schools, a set of replacement schools was also drawn. Within the final strata, schools were sampled proportional to size (number of total enrolled children in grades 1-5, i.e. primary school). 5 Numbers do not add up to 100 in the column due to rounding. 30 During implementation, a substantial number of private schools in Lahore refused to participate, despite repeated attempts by the survey firm, a letter from the government to enter and calls from governmental authorities to the schools. The survey team finally decided to exclude the stratum from the sample, rather than replace these schools. This is because a large number of refusals would likely bias the data in this stratum (chain schools, which are typically attended by the children of the elite, had particularly high refusal rates). The final sample is presented in Table A.3 below. The sample includes public schools, private schools (outside Lahore), as well as public private partnership schools under the Punjab Education Foundation (PEF). The sample includes data on 812 schools, 18,055 children, 3,373 teachers, 812 principals, and 3,532 parents. Note that the distributions are not the same as in the overall census (Table 2). We therefore use survey weights in any of the calculations presented throughout the report. This final sample should be considered representative of all schools in Punjab, excluding military areas and private schools in Lahore (in substantive terms, this probably means that we miss information on the schools attended by the elite of the province). Within schools, the survey team selected a random Grade 4 stream from which all students were assessed in listening, mathematics, English and Urdu (see section A.2 on the student assessment design). In each Grade 4 classroom, 5 students were sampled who would be interviewed in more detail. The parents of these students were also interviewed using a phone protocol (Module 7 of the survey). In each school, we sampled (maximum) 6 teachers, teaching grades 2-5 (i.e. primary school teachers). These teachers were interviewed and assessed for the teacher module. 31 Table A.3: Overview of Sample of first round data collection (the main sample) Percentage Min Max Observations or Mean with data Schools 812 % Public 50% 0 1 812 % Private 20% 0 1 812 % PEF 30%6 0 1 812 % Urban 23% 0 1 7327 % Co-Ed 78% 0 1 812 Principal 812 Age 43 20 79 811 % Female 44.2 0 1 812 Teachers 6695 Teachers tested 3373 Age 32.4 15 72 5441 % Female 73 0 1 6692 Students 21,537 Students tested 18,055 Age 10 6 17 21,494 % Girl 47.3 0 1 21,537 Parents 3,532 % Direct parent 93.1 0 1 3,532 % Mother 18 0 1 3,532 Age of 39.2 16 78 3,175 respondent Note: No weights included in this descriptive overview. For urban school’s information is available only for public and private schools. Co-Ed means at least 1 boy and 1 girl are enrolled in the primary grades of the school (but they may be separated within the school). For Principals, out of 812 observations, around 90 % of the respondents have an official principal/head teacher position (the other 10 % vice-principals, acting principals, owners, etc.). Percentage of Direct parent: around 7 % of the respondents are the legal guardians of the children (aunt, uncle, grand-parents) A sample of 200 schools was drawn for a second round of data collection. The goal of the second round of data collection was to understand how students learn throughout the school year. The second round 6 Note that the share of PEF differs somewhat from the full distribution given in Table 2. This is because the final sample in Table 3 shows the relative share of the number of schools, while above we present the distribution of the relative size by number of students. 7 Note that the share of urban is low due to sample construction, which includes 200 schools from rural villages (LEAPS) as well as 200 PEF schools which are mostly in villages. 32 also provided more data on classroom practices, student health and a broader understanding of socio- emotional issues like corporal punishment. For this round of data collection, LEAPS schools were excluded. Table A.4 provides a descriptive picture of the schools included in the second round of data collection. Table A.4: An overview of the second round of data collection Percentage Min Max Observations (or Mean) with data Schools 200 Public 48.5% - - 200 Private 14% - - 200 PEF 37.5% - - 200 Urban 29% - - 200 Co-Ed 82% - - 200 Teachers 378 Teachers - - 222 observed Age 31 16 69 375 Female 65.5 % - - 378 Students 6,152 Student - - - 4,899 assessment Age 10.4 6 17 1,444 Female 46 % 0 1 4,438 Note: No weights included in this descriptive overview. A.2 Assessment Design The student assessment aims to assess to what extent children in Punjab schools master the curriculum that they are being taught in primary school. The research team decided to use a previously validated assessment, which has been used in the LEAPS survey since 2003 (Andrabi et al, 2008a). This assessment includes 141 test items related to the primary school curriculum, up to Grade 5. In order to validate the assessment, the research team mapped each test item to its sequencing in the curriculum (see Table A.5). The source of this mapping were the textbooks distributed to public schools, and teacher guides, rather than the official curriculum (i.e. the grade distribution refers to the realized curriculum, rather than the ‘official curriculum’). Table A.5: Distribution of questions by grade level and subject Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Total Listening 5 2 - - - 7 Urdu 3 24 18 3 - 48 33 English 14 17 10 3 - 44 Math 10 13 8 10 1 42 Total 32 56 36 16 1 Note: mapping of test items to the curriculum was done by World Bank research team, based on a combination of textbooks and teacher guides. The LEAPS assessment is a written assessment that consists of four parts. The assessment starts with a ‘listening section’, during which students are supposed to write letters and words that are spoken out loud by the enumerator. The three remaining sections evaluate students’ knowledge in Urdu, English and Mathematics. In order to avoid close students copying off of each other, the enumerator randomly distributed three different booklets in the classroom (each booklet has a different order for Urdu, English and Mathematics). The enumerator stays in the classroom during the assessment and all the questions of a specific section (whatever its order in the booklet) are answered within a time constraint (30 minutes for each subject). At the end of a specific timeline, the enumerator informs the students to start the next section, even if they have not finished the previous one. When the assessment is finished, the enumerator collects all the booklets and these are sent to the data entry team for digitization. The ‘listening and writing’ section is composed of seven questions; for three of them, students have to write a letter that is spoken out loud in English by the enumerator and for the four remaining questions, students have to write a word. The Urdu, English and Mathematics sections are respectively composed of 48, 44 and 42 questions. The questions span a broad range of topics covered by the curriculum: in Urdu and English, the questions refer to alphabet and word construction, word recognition, grammar, sentence construction and reading comprehension. In the Mathematics section, questions are about counting and numbers, addition and subtraction, multiplication and division, fractions, percentages and problem solving. These assessment items included both open-ended and multiple-choice items. After administration, the data entry team was asked to transcribe the actual answers of the students, except for a small sub-set of items that were more complicated to transcribe. This sub-set was corrected by trained enumerators using a pre-defined template. After data entry, this resulted in 18,055 valid assessments, which were then marked using a pre-defined do-file in Stata. A sub-section of student assessment was also used to assess the teachers’ content knowledge in Mathematics, English and Urdu. During piloting, the research team learned that teachers are more comfortable correcting a student exam, rather than fill out their own exam. The team therefore filled in a sample student exam with both correct and incorrect answers. Teachers were given a separate booklet, in which they would mark each item as incorrect or correct. In case the teacher marked an item as ‘incorrect’, they were then asked to provide the correct answer to these items. Trained enumerators then scored the teacher answers using a pre-defined template. The completed and revised booklets were then sent to the data entry team, and later corrected by the research team. A.3 Survey Implementation Implementing any complex survey requires a careful implementation plan and a well-trained team. The survey was coordinated by a small team based in Washington DC, while a research firm (RCons) was 34 contracted to implement the survey. The research team developed the field manual, the tablet software and the enumerator training and certification. The survey firm coordinated the survey implementation plan and hired the enumerators that would visit the schools. The coordinators of the survey firm had previously been involved in the LEAPS survey and were broadly familiar with the details of the survey, and highly familiar with the schooling environment in Punjab. The survey firm also hired and trained separate teams of enumerators to do back-checks, to do telephone interviews, and to do data entry. The team developed a detailed survey manual and a training plan for enumerators. The coordinators of the survey participated in all four field pilots; they were then trained on the final modules during a 5 day- training led by the research team. The World Bank research team then led the training of the enumerators on the high inference modules (classroom observation, management module), while the research firm led training on the low inference modules. This was an interactive training during which enumerators extensively practiced the modules and included a school visit to practice the survey modules. During the training, enumerators undertook various certification tests of the classroom observation module (96% pass rate) and the world management survey (100% pass rate). While the survey was initially scheduled to be fielded in November 2017 (with a second round one year later), the survey was delayed because the necessary approvals could not be obtained. With guidance from the PMIU and the SED, the team therefore decided to launch the survey in the new school year, that started on 1 March 2018, and to finish data collection before the summer break that would start in May 2018. The second round of data collection was moved to November 2018 (end of academic year), in order to comply with the funding and operational regulations for the survey. Actual field dates for round 1 were between 12 March and 16 May 2018, while round two data collection happened between 17 November to 14 December 2018. Data collection was done using tablets with SurveyCTO software. This allowed the research team to live- monitor the quality of implementation. During fieldwork, the team regularly provided feedback to the survey firm on data quality issues (mostly on a daily basis). Errors in the software were mostly ironed out during the field pilots, although some (mostly minor) errors still affected data collection. These errors and updates to the software were carefully recorded to be integrated into data processing. 35 Is there a Learning Crisis in Punjab? This report provides an initial data release of the first comprehensive analysis of the learning crisis and its proximate causes in Punjab. The report provides an initial answer to three key questions about the learning crisis: (1) Is schooling unequally distributed, (2) Are children learning anything at all while in school? And (3) Are the main factors (teachers, families, school principals and school inputs) aligned to address the learning crisis? The data for this report were collected through the SDI survey, which was designed by the World Bank to quantify the level of learning. The survey also diagnoses the main causes of the learning crisis in the service delivery chain (teachers, parental support, school management, and school infrastructure). The SDI survey was implemented in a stratified random sample of 812 public school, private schools and public-private partnership (PEF) schools throughout 2018. Findings from the survey were triangulated with other data sources, including administrative data sources, the LEAPS survey, MICS, HIES and PSLMN surveys, as well as data from ASER, EGRA and NEAS. This public data release is the first public release from that effort and provides an opportunity to jointly reflect on these findings. 36