Policy Research Working Paper 10764 Early Learning in South Punjab, Pakistan Investigating Child Development and Classroom Quality Jonathan Seiden Amer Hasan Diego Luna Bazaldua Education Global Practice A verified reproducibility package for this paper is May 2024 available at http://reproducibility.worldbank.org, click here for direct access. Policy Research Working Paper 10764 Abstract In Pakistan, learning poverty among primary school aged relationships between teaching practices and early child- children is estimated to be as high as 77 percent, but little hood development outcomes, the analysis finds a strong data exist on early learning experiences. This paper describes positive relationship across the areas of process quality and the state of classroom quality in 1,395 classrooms and the domains of childhood development. Children studying in early childhood development status of 8,249 children in a high-quality classroom have outcomes that are equivalent a representative sample of 894 public schools in South to having been in school nine months longer than children Punjab, using two measurement tools: The Teach ECE of similar ages in an average quality classroom, suggesting classroom observation tool, which describes the struc- that a sharper focus on teaching quality may improve early tural and process quality features of classrooms, and the childhood development outcomes and school readiness. Anchor Items for the Measurement of Early Childhood The findings also show that after accounting for teaching Development Direct Assessment which reports on early quality, degrees and certification are not associated with learning and developmental outcomes of children aged 4 to early childhood development outcomes, but that classes 6 years. The paper finds key gaps in the foundational skills taught by female teachers have better early childhood devel- of young children and areas for improvement in both the opment outcomes. physical classroom and teaching practices. In examining the This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at ahasan1@worldbank.org. A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click here for direct access. RESEA CY LI R CH PO TRANSPARENT ANALYSIS S W R R E O KI P NG PA The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Early Learning in South Punjab, Pakistan: Investigating Child Development and Classroom Quality1 Jonathan Seiden 2 Amer Hasan 3 Diego Luna Bazaldua2 Keywords: Early Childhood Development, Classroom quality, Pakistan, pre-primary, primary JEL Codes: I20, I21 1 The authors would like to thank the Programme Monitoring and Implementation Unit of the School Education Department of the Government of the Punjab for sharing the microdata used in this paper. Ghulam Dastgeer provided helpful clarifications on survey protocols. The authors would also like to thank Elizabeth Hentschel for the analysis reported in Table 1. All errors are our own. This work is a product of the staff of the World Bank with external contributions. The World Bank does not guarantee the accuracy of the data included in this work and does not assume responsibility for any errors, omissions, or discrepancies in the information, or liability with respect to the use of or failure to use the information, methods, processes, or conclusions set forth. The boundaries, colors, denominations, links/footnotes, and other information shown in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. The citation of works authored by others does not mean the World Bank endorses the views expressed by those authors or the content of their works. 2 Harvard Graduate School of Education and World Bank, Education Global Practice 3 World Bank, Education Global Practice Introduction Pakistan is the fifth largest country in the world by population and home to over 53 million children under the age of eight (Pakistan Bureau of Statistics, 2017). Twenty million children are out of school (Ersado et al., 2023) and 77% of 10-year-olds are estimated to be unable to read and understand a simple, age-appropriate paragraph (World Bank, processed). Yet, little is known about young children’s school readiness or about the quality of their earliest educational experiences. This study aims to fill this knowledge gap and documents the skills pre-primary-aged children can demonstrate and the quality of the classrooms they attend in South Punjab, Pakistan. The Punjab province is home to over half of the country’s population but there is widespread variation across the province in human development outcomes. As shown in Table 1, data from the most recent Multiple Indicator Cluster Surveys (MICS) study show that enrollment in early childhood education is far lower in South Punjab (27% of children enrolled) than in the rest of the province (36% of children enrolled) (Bureau of Statistics Punjab, Planning & Development Board, Government of the Punjab & UNICEF, 2018). Not surprisingly, the share of children developmentally on track is also markedly different – 52% of 3 and 4 year olds are reported by their parents as developmentally on track in South Punjab compared to 63% of children in the rest of the province (Bureau of Statistics Punjab, Planning & Development Board, Government of the Punjab & UNICEF, 2018). Table 1. Share of 3 to 4-year-old Children Enrolled in ECE and Share Reported to be Developmentally on Track % Enrolled in ECE % Developmentally on Track Mean 95% Confidence Interval Mean 95% Confidence Interval South Punjab 27.32 (25.73, 28.92) 52.42 (50.67, 54.17) Male 27.85 (25.91, 29.78) 50.21 (47.91, 52.50) Female 26.79 (24.57, 29.00) 54.72 (52.49, 56.96) Central & North Punjab 35.71 (34.69, 36.72) 62.55 (61.54, 63.55) Male 35.53 (34.18, 36.87) 60.67 (59.29, 62.05) Female 35.90 (34.48, 37.31) 64.54 (63.16, 65.91) Punjab Overall 33.34 (32.48, 34.20) 59.69 (58.81, 60.57) Male 33.37 (32.36, 34.49) 57.73 (56.54, 58.92) Female 33.30 (32.10, 34.50) 61.75 (60.57, 62.93) Source: MICS, Punjab 2017-2018 (Bureau of Statistics Punjab, Planning & Development Board, Government of the Punjab & UNICEF, 2018) The quality of pre-primary education plays an important role not just in early childhood development but potentially also in primary education (Akyeampong et al., 2023). Research shows that children who have high-quality early childhood education experiences have better cognitive and socio-emotional outcomes and that early school readiness skills are predictive of later schooling success (Duncan et al., 2007; Engle et al., 2011) – though whether this success is short- 2 term or long-term is far from a settled point in studies from low- and middle-income countries (See Akyeampong et al., 2023). There is growing evidence of the benefits of quality pre-primary education as a cost- effective catalyst for improvements in learning outcomes and long-term economic benefits later in life. Compared to other educational policies to improve foundational learning, global experts propose that investing in quality pre-primary education can help address learning inequities and reduce deficits by the time children enter primary school (Global Education Evidence Advisory Panel, 2020). This is confirmed by results from international large-scale assessment studies, which have shown that enrollment in pre-primary education results in higher average learning outcomes in reading, mathematics and science (Mullis et al., 2020; Mullis et al., 2023). Yet, even when investing in improving access to and quality of early childhood education can be seen as smart policy, most of the studies that have investigated this linkage come from high-income countries (Holla et al., 2021). Fortunately, more and more evidence on this linkage is being generated in low- and middle-income contexts to support the investment in quality early childhood education (Augsburg et al., 2022; Hasan et al., 2019; Nakajima et al., 2021). Hence, this study provides a window into one of the less explored and more important policy aspects of young children’s education: what goes on in the early childhood education classroom in a lower-middle-income setting such as Pakistan – specifically in South Punjab – and how early childhood settings are associated with children’s cognitive and socioemotional development. It attempts to fill a knowledge gap by describing what relevant skills pre-primary aged children can do and the quality of classrooms in which they study. We also report on how children’s skills and classroom quality vary by student, teacher, and school characteristics and assess the correlations between these variables to understand the role that quality may play in supporting young children’s acquisition of school readiness skills. We use a large, representative dataset of public schools in South Punjab to answer the following core research questions: 1: What do pre-primary classrooms in South Punjab look like? 2: What is the structural and process quality of these classrooms? 2.1: How does quality differ by classroom and teacher factors? 3: What can children in pre-primary classrooms do? 4: How are children’s early learning and development related to individual, teacher, and class factors? Data The data presented in this paper draw on the Baseline Survey on Early Childhood Care and Education (ECCE) carried out under the Education Component of the Punjab Human Capital Investment Project (PHCIP). Child-level data on the Anchor Items for the Measurement of Early Childhood Development: Direct Assessment (AIM-ECD DA) was collected from January to April 2022. AIM-ECD DA data was collected by 32 certified enumerators. Observations of classroom 3 quality using Teach ECE were collected by 30 enumerators who had similarly completed a Teach ECE training and passed a certification quiz. 4 Sample At the time of data collection, PHCIP operated in 11 districts of South Punjab (Bahawalnagar, Bahawalpur, Bhakkar, Dera Ghazi Khan, Khushab, Layyah, Lodhran, Mianwali, Muzaffargarh, Rahim Yar Khan and Rajanpur). Education activities covered 3,400 schools across these districts. Figure 1 shows the geographical coverage of the study in South Punjab with the 11 districts in turquoise. These 11 districts were intentionally selected for the PHCIP project on the basis of higher multidimensional poverty index (MPI) scores compared to the rest of the province. Approximately 2.9 million primary school aged children live in these districts. Figure 1. Eleven South Punjab Districts Included in Sample Source: Authors’ calculation The 3,400 schools selected for the project were chosen out of a universe of 16,439 schools in the 11 districts located in South Punjab. At the time of project initiation, eligible schools were those with a minimum of three or more functional classrooms, three or more sanctioned teachers, and those that did not have a dedicated ECCE classroom. The study took a representative sample 4 Enumerators attended a 3-day AIM-ECD DA training, and were certified upon successfully: • Attending all training sessions and participating actively. This included role-playing and demonstrating administration of the tool. • Participating in two field-tests. During the first field test, they had to work with a partner to practice the assessment and scoring. On the second field test, they worked one-on-one with children. • Passing a written quiz that tested their knowledge of the DA and proper administration and had their administration during the field visits observed by a trainer to ensure quality. 4 of 900 project schools. Within each school, all classes containing pre-primary-aged children 4-6 years were included for observation with Teach ECE. In most schools, a single Katchi and a single Grade 1 class were observed. Katchi classes are an informal grade before primary grade 1. A random sample of children aged 4-6 in attendance on the date of data collection was drawn by selecting five children from the Katchi class and five children from the Grade 1 class when both were available. In cases where attendance was fewer than 10, all age-eligible children were asked to participate. Out of 900 sampled schools, the survey contained 1,482 Teach ECE observations from classrooms in 899 schools (one school did not have an adult present) and attempted to administer AIM-ECD DA to 8,613 children. The survey excluded data from children who either did not assent to participate in AIM-ECD DA (n=45) or did not meet the quality check requirements (n=5) and reported a final working sample of 8,563 children. Despite attempting to interview 10 children per school, in 137 schools, fewer than 10 age-appropriate children were available and in four schools the number of children interviewed was 11. Additional cleaning and analysis of the data revealed 27 classrooms with repeated observations. In these cases, the first observation of the classroom was kept and the later observation was discarded. Six classrooms had no corresponding child data and two classrooms with child data had no corresponding quality observation. To allow for correlational analyses, we focus on classrooms with both child and classroom-level data. In addition to excluding observations due to data issues, we also focus on a substantive subset of the original data consisting exclusively of Katchi and Grade 1 classes. This led to the exclusion of data from 3 higher-grade classrooms (which had pre-primary aged children) as well as 51 ECE classrooms, a new type of classroom with a dedicated curriculum in Punjab province. The subset of data remaining allows us to better reflect on the baseline status of pre-primary education in South Punjab before the widespread implementation of ECE. After removing these classrooms and corresponding children from our sample, we are left with a final analytical sample of 8,249 children (4,806 boys and 3,443 girls) studying in 1,395 classrooms (taught by nearly equal numbers of male and female teachers) in 894 schools. The total number of schools and primary-aged enrollment in South Punjab as a whole, in the project schools, and in the analytical sample is shown below in Table 2. This approach creates a sample that is statistically representative of the 3,400 PHCIP schools with nearly 530,000 primary-aged children. While not statistically representative of the 11 districts as a whole, the general composition of the sample closely reflects the composition of districts, with Rahimyar Khan being under-represented in both the project and sample, and Rajanpur being slightly over- represented. 5 Table 2. Sample and project area by district compared to South Punjab District Number of schools Primary enrollment South Analytical South Analytical PHCIP PHCIP Punjab Sample Punjab Sample 2,145 473 126 269,425 71,282 17,888 Bahawalnagar (13%) (13.9%) (14.1%) (12.5%) (13.5%) (13.1%) 1,652 400 107 196,742 52,606 14,436 Bahawalpur (10%) (11.8%) (12%) (9.2%) (10%) (10.6%) 1,242 270 72 152,944 39,264 10,023 Bhakkar (7.6%) (7.9%) (8.1%) (7.1%) (7.4%) (7.4%) 1,567 194 45 203,814 31,322 6,967 D.G. Khan (9.5%) (5.7%) (5%) (9.5%) (5.9%) (5.1%) 929 235 63 101,302 33,400 8,677 Khushab (5.7%) (6.9%) (7%) (4.7%) (6.3%) (6.4%) 1,506 454 117 187,575 65,181 15,778 Layyah (9.2%) (13.4%) (13.1%) (8.7%) (12.4%) (11.6%) 750 241 63 110,317 38,601 10,146 Lodhran (4.6%) (7.1%) (7%) (5.1%) (7.3%) (7.5%) 1,201 282 80 144,850 44,758 12,465 Mianwali (7.3%) (8.3%) (8.9%) (6.7%) (8.5%) (9.2%) 1,731 372 96 266,126 67,497 17,512 Muzaffargarh (10.5%) (10.9%) (10.7%) (12.4%) (12.8%) (12.9%) 2,773 165 45 386,011 22,974 5,848 Rahim Yar Khan (16.9%) (4.9%) (5%) (18%) (4.4%) (4.3%) 942 314 80 129,564 60,853 16,441 Rajanpur (5.7%) (9.2%) (8.9%) (6%) (11.5%) (12.1%) Total 16,438 3,400 894 2,148,670 527,738 136,181 Source: Author’s calculations using the Punjab Annual School Census, 2021. Tools The AIM-ECD DA (or AIM-ECD for short) was used to measure early childhood developmental outcomes for children aged 4-6 years old (Pushparatnam et al., 2021; World Bank, 2021). The Teach Early Childhood Education (Teach ECE) tool was used to conduct classroom observations to capture the quality of teaching practices and interactions between teachers and children (Ding et al., 2024; Pushparatnam & Ding, 2021). Each tool is described below. The AIM-ECD DA is a tool that measures early childhood development through a set of activities that a trained enumerator engages in with a child. The AIM-ECD DA contains items in four critical domains of early development: Early Numeracy, Early Literacy, Executive Function, and Social-Emotional development and shares items from the Measure of Learning and Quality Outcomes Module on Early Learning and Development (MELQO MODEL) as well as the International Development and Early Learning Assessment (IDELA) (Pisani et al., 2018; Raikes et al., 2019). 6 The standard global version of the assessment comprises 93 items grouped in 14 subtasks (activities) chosen after analyzing a large cross-country dataset consisting of 24,533 children in 23 countries and identifying items with robust psychometric properties across contexts (Pushparatnam et al., 2021). In addition to the standard AIM-ECD subtasks items, during the contextualization process for Pakistan, a single subtask testing English letters was included so that children could be assessed on both their Urdu and English letter knowledge. Figure 2 shows the AIM-ECD DA subtasks and the developmental domains to which each subtask belongs (World Bank, 2022). While the AIM-ECD was not designed for specific use in Pakistan, the constructs it assesses are highly relevant according to both provincial and national sources. Appendix A: Crosswalk between Student Learning Objectives and AIM-ECD Subtasks reviews the alignment of the assessment with the Single National Curriculum and finds good coverage (National Curriculum Council, 2020). Appendix B: Crosswalk between Punjab Early Learning and Development Standards and AIM-ECD reviews the contents of Punjab-specific standards and the AIM-ECD assessment, again finding good matches for many constructs deemed important by the provincial education authorities (Quaid-e-Azam Academy for Educational Development & UNICEF, n.d.). Figure 2. AIM-ECD DA Subtasks and Domain Assignments AIM-ECD Overall English Letter (Early Learning and Identification Development) Early Early Executive Social English Letter Numeracy Numeracy Functioning Emotional Identification Head, Toes, Listening Receptive Perspective Knees, Comprehensio Spatial Taking/Empat Shoulders n Story Vocabulary hy Task Urdu Letter Producing a Forward Digit Identification Set Span Simple Initial Sound Backwards Addition and Discrimination Digit Span Subtraction Number Name Writing Identification Number Comparison Shape Identification The Teach ECE tool was developed to measure both structural and process aspects of classroom quality (World Bank, 2021). Structural aspects of quality refer to static aspects of quality that exist in the classroom, for example the quality and availability of learning materials, adequacy of the classroom environment, teacher-to-student ratio, and qualifications of the ECE 7 teacher. In contrast, process aspects of quality refer to the quality of the interactions that children experience in the classroom, e.g., the teacher’s promotion of critical thinking skills during learning activities, sensitivity to children’s needs, and the promotion of a positive classroom culture to enable development and early learning. In general, process quality elements, and the “quantity and quality of interactions between a young child and his or her primary caregivers, including ECE providers, are the most important factors in early development” (Burchinal, 2018, p. 3). Teach ECE captures information regarding the process quality of classrooms during two 15-minute segments of class time. Observers collect information on whether the teacher is engaging the class in a learning activity and the proportion of the class that is actively engaged three times during each segment to measure the Time on Task of the classroom. Observers then code 28 behaviors indicative of quality during the lesson which fall under nine elements. These nine elements fall under three primary areas of process quality—Classroom Culture, Guided Learning, and Socioemotional Skills. Figure 3 shows the structure of the Teach ECE measures of process quality (Ding et al., 2023; World Bank, 2021). In addition to the process quality of classrooms, observers also collect information regarding the structural aspects of quality, including the teacher’s level of education, teacher-child ratio, attendance and composition of enrolled children, language-match in the classroom, availability of learning materials, and safety and hygiene issues. Consistent with the literature emphasizing the importance of process quality in ECE, the majority of this paper focuses on these process quality scores from the Teach ECE classroom observation tool. Figure 3. Teach ECE Framework of Areas and Elements Quality of Teaching Practices Time on Task Socioemotional Classroom Culture Guided Learning Time on Learning Skills Supportive Facilitation of Learning Autonomy Learning Environment Positive Behavioral Feedback Perserverance Expectations Social & Critical Thinking Collaborative Skills Checks for Understanding 8 Training and Field Work Comprehensive knowledge and practice training and examples are provided to AIM-ECD and Teach ECE enumerators before any field data collection activity. The training is a quality assurance activity to ensure that enumerators correctly interpret each item correctly and score the assessments in a reliable and consistent manner to ensure the fidelity of scores. Enumerators participated in a four-day training course on Teach ECE and a three-day training on AIM-ECD consistent with the tool recommendations (World Bank, 2021, 2022). On the first day, the Teach ECE trainer explained the tool in detail, its intended measurement purposes and permitted uses, its teaching behaviors and elements. On the second and third days, enumerators practiced Teach ECE coding using practice videos of local teachers previously recorded in each location to maximize local adaptation, which were 20 minutes long to match the time they would be coding in the field. On the last day, enumerators went to schools to practice coding live and receive feedback from trainers. A similar structure is followed in the case of AIM-ECD, but with enumerators practicing the direct assessment administration with a child. Before enumerators are allowed to conduct classroom observations or child assessments in the field, they must pass a certification exam. In the case of Teach ECE, the exam involves scoring three videos within 1 point of the master codes (i.e., codes developed by Teach ECE experts after observing the same video) for at least 8 out of 9 elements for each of the three videos (80% of the time). In the case of AIM-ECD, the certification involves a reliability quiz and observation from the master trainer. This certification increases the quality assurance in the use of the tool and reduces potential scoring bias; at the same time, the exam is an excellent opportunity to clarify any questions regarding the scoring process. Methods Calculation of Scores Both the AIM-ECD and Teach ECE contain many items that must be consolidated into aggregate scores for hypothesis testing. Below we describe the procedures used to calculate these aggregate scores. Overall, both instruments appeared to function well. AIM-ECD DA Score Calculation We followed standardized guidance from the tool creators in order to calculate subtask, domain, and overall AIM-ECD scores (World Bank, 2022). Table 3 describes the sub-tasks in each domain as well as the number of items in each sub-task. In general, AIM-ECD scores are calculated on a 0-100% scale roughly equivalent to the percentage of items children scored correctly. For each subtask, the score is calculated by scoring each question (usually as correct/incorrect) and then dividing the total score by the maximum possible score. For each domain, the average is taken of all subtask scores of the four domains. Finally, the AIM-ECD Overall score is calculated by taking the average of the four domain scores. To ensure consistency of scoring and comparability with other datasets, the English Letters subtask was analyzed separately and not included in the calculation of the AIM-ECD DA Overall Score. 9 Table 3. AIM-ECD DA domains, subtasks, and items Domain Subtask Number of items Early Listening Comprehension Story 5 Literacy Letter Identification 20 Initial Sound Discrimination 3 Name Writing 1 Early Receptive Spatial Vocabulary 4 Numeracy Producing a Set 6 Simple Addition 3 Number Identification 20 Shape Identification 5 Executive Head, Toes, Knees, Shoulders Task 15 Function Forward Digit Span 4 Backwards Digit Span 4 Social- Perspective Taking/Empathy 3 Emotional English English Letter Identification 20 Letters For the more formal hypothesis tests used later in this report, we rely on the AIM-ECD Overall and domain scores. The advantage of this approach is that we are able to analyze and summarize a spectrum of ECD that captures many different skills in a single score. To further add meaning to our results and to concretize interpretations about what children can and cannot do, we also analyze a set of intentionally selected individual exemplar AIM-ECD items as described in Table 4. Table 4. Exemplar AIM-ECD Direct Assessment items Domain Subtask Exemplar Description Identified Urdu letter Alif presented out of sequence: ‫ا‬ Letter Identification Identified Urdu letter small ye presented out of Early sequence: ‫ی‬ Literacy Listening Comprehension Identified color of hat from story Name Writing Wrote at least some letters when asked to write name Sound Discrimination Identified word starting with /s/ sound Identified numeral 10 presented out of sequence Number Identification Identified numeral 1 presented out of sequence Early Counted 6 objects correctly Producing a Set Numeracy Counted 14 objects correctly Identified a circle among shapes Shape Identification Identified a triangle among shapes 10 Simple Addition Added 2+1=3 Correctly identified the word describing a picture of a Spatial Vocabulary ball under a box Backward Digit Span Could repeat 2 digits in reverse order Executive Forward Digit Span Could repeat 3 digits in same order Functioning Head Toes Knees Correctly followed instructions of a confusing task Shoulder Task Social Empathy Perspective Could identify emotion crying girl was feeling Emotional Taking English English Letter Identified English letter A Letters Identification Identified English letter Z Teach ECE Score Calculation As with the AIM-ECD DA, we followed the standardized guidance for calculating Teach ECE behavior, element, and area scores (World Bank, 2021). Behaviors are the individual items that comprise the Teach ECE assessment. When observing a class, the observer scores each of the 28 behaviors during two segments of an observation. Behaviors are coded in an ordinal scale as “Low,” “Medium,” and “High” (1, 3, or 5). To facilitate reporting and interpretation, more emphasis is placed on the coding of the ten element scores. Based on the behavior scores within each element, the observer scores each element on a scale of 1-5, with 1 indicating low quality and 5 indicating high quality. It is important to note that, unlike AIM-ECD DA, element scores are not simply the average of behavior scores. While element scores are based on behavior scores, observers have flexibility of assigning slightly higher or lower element scores based on their judgment. Finally, we calculate the area scores and an overall Teach ECE score for each segment. The area score is calculated by taking the average of the elements that comprise that area and the Teach ECE Overall score is the average of the Classroom Culture, Guided Learning, and Socioemotional Support scores. Because of the structure of the assessment, there are essentially two sets of scores for each classroom observed—one from the first segment and one from the second segment. Segments are therefore the basic unit of analysis when analyzing Teach ECE scores. As such, when reporting percentages in relation to benchmarks, the actual interpretation is “percentage of segments” rather than “percentage of classes.” This paper primarily focuses on Area and Element scores as well as the overall Teach ECE score. In addition, it presents the results of selected exemplar items (i.e., scored behaviors) as shown in Table 5. Table 5. Exemplar behaviors from Teach ECE Area Element Exemplar Behavior 0.2 The teacher leads a learning activity and Time on Task Time on Task - Student most children are on task Classroom Supportive Learning 1.1 The teacher treats all children respectfully Culture Environment 11 2.1 The teacher sets clear behavioral Positive Behavioral expectations for classroom activities and/or Expectations routines 3.3 The teacher makes connections during the Facilitation of Learning day that relate to other concepts or children’s daily lives 4.3 The teacher adjusts teaching to the level of Guided Checks for Understanding the children Learning 5.1 The teacher provides specific comments or Feedback prompts that help clarify children’s misunderstandings 6.3 The children ask open-ended questions or Critical Thinking perform thinking tasks 7.3 Children volunteer to participate in the Autonomy classroom Socioemotional 8.2 The teacher responds positively to children's Perseverance Skills challenges Social & Collaborative 9.2 Children collaborate with one another Skills through peer interaction Reliability of Tools The Cronbach’s alpha of the domains and overall scores for the AIM-ECD and Teach ECE are shown below in Table 6. Overall, the AIM-ECD demonstrates superb reliability with an overall alpha of 0.978 and excellent reliability for each domain. Overall Teach ECE segment scores were very good with a Cronbach’s alpha of 0.884, but individual domains were lower—while Guided learning demonstrated very good reliability (0.820), Socio-Emotional Skills was acceptable (0.771) and Classroom Culture was questionable/fair (0.651). These figures are similar, and slightly better, than reliabilities from a review of international Teach ECE data (Ding et al., 2023, 2024). Table 6. Cronbach’s Alpha of AIM-ECD Domain and Teach ECE Areas Tool Domain/Area Cronbach’s Alpha AIM-ECD DA Early Literacy 0.938 Early Numeracy 0.964 Executive Functioning 0.943 Social Emotional 0.951 12 Overall 0.978 Teach ECE Classroom Culture 0.651 Guided Learning 0.820 Socio-Emotional Skills 0.771 Overall 0.889 Analysis Given the sampling design, results do not require weighting to be representative of the PCHIP project area. In addition, on reporting the results of composite items calculated as described above, we also present results based on the proportion of segments demonstrating a certain degree of quality and the proportion of children with different skills. Below, we outline the process used to set benchmarks needed to dichotomize these results on AIM-ECD and Teach ECE. In addition to the simple descriptive statistics and disaggregation, we also build a statistical model for inferential purposes. The details of this model-building process are outlined below. Quality Benchmarks, Exemplar Items, and Standardized Scores As described above, both the AIM-ECD DA and Teach ECE generate continuous scores to describe child developmental outcomes and quality of teaching practices, respectively. AIM- ECD scores range from 0-100% while Teach ECE element and area scores range from 1-5. These continuous scores are limited in their comprehensibility. To improve the communicability of our results, we also utilize benchmarks on Teach ECE to report on how much of the time (i.e., what proportion of segments) classrooms are demonstrating minimal quality during lessons. When reporting on the proportion of segments demonstrating minimum quality, we use the proportion of segments that received at least a “Medium” rating, i.e., an average rating of at least 3. Three represents the score achieved by a class with no extremely serious quality issues and thus is a reasonably interpretable benchmark. Teach ECE quality benchmarks are useful for creating concrete descriptives about the status of ECD in South Punjab but should not be interpreted as the result of best practice standards recommended for external use. 5 We do not set similar benchmarks for AIM-ECD that allow us to talk about the proportion of children demonstrating minimal developmental levels. Instead, we report the results from a set of exemplar items on the assessment to describe the proportion of children able to accomplish certain skills. In addition to using quality benchmarks for Teach ECE and exemplar items AIM-ECD, we also transform AIM-ECD and Teach ECE continuous scores to improve interpretability of our multiple regression analysis. We transform both scores to a standardized variable with mean 0 and standard deviation 1 (i.e., Z-scores). This allows us to interpret coefficients from our models as differences in outcome standard deviations (for AIM-ECD scores) and in teaching quality (for 5 Standard setting is a complex normative process that requires buy-in from relevant stakeholders and experts, careful consideration of what children “should” be able to do at a given age or stage of education, and an examination of the consequences of these decisions (American Educational Research Association et al., 2014; Cizek & Bunch, 2007). A formal standard setting exercise to provide additional meaning to AIM-ECD DA and Teach ECE scores is currently under consideration and will be part of the public set of resources available for users of these tools. 13 Teach ECE scores). Transforming scores in this way deviates from standardized AIM-ECD and Teach ECE guidance, but doing so does not alter any statistical approaches (World Bank, 2021, 2022). Multiple Regression Analysis In addition to the simple descriptive statistics and disaggregation by covariates that help answer research questions 1-3, we build a model to investigate the joint relationships between classroom quality and child outcomes. To accomplish this, we use the overall and domain-specific AIM-ECD scores discussed in Calculation of scores (above) to analyze the associations between AIM-ECD by various child and classroom factors. As described above in Quality Benchmarks, Exemplar Items, and Standardized Scores, we standardize both the AIM-ECD and Teach ECE scores prior to fitting these models. Data management and statistical analysis was conducted using R 4.2.1 with the lme4 and tidyverse packages and in Stata 17.0 (Bates et al., 2015; R Core Team, 2021; StataCorp, 2019; Wickham et al., 2019). Due to the sampling and data collection design, children are nested within classrooms as well as enumerators. This leads to a complex error structure that should not be ignored in analysis. Children studying in the same classroom may have non-independent errors given within-site homogeneity and exposure to the same teacher’s lessons. Similarly, enumerators may be systematically more or less lenient, leading to correlations within enumerators. To account for this data structure, we use a mixed effects (random effects) model with children’s AIM-ECD scores nested in classrooms and also add a random effect to account for differences in enumerators with the covariance of enumerator and class intercepts constrained to zero (i.e., assuming that enumerator leniency across classrooms was not correlated with overall classroom averages). For each AIM-ECD domain or overall score, we fit the following model to predict the overall or domain standardized Z-score for child in classroom assessed by enumerator . Random effects at the enumerator and classroom allow us to examine associations both within and between classrooms and to partially control for differences by enumerator leniency. = 0, + 1, + 2 + 3 + 0, = 00 + 01 + 02 + 03 1 + 04 × 1 + 05 ℎ + 06 ℎ + 07 ℎ + 08 + 1, = 20 + where ~(0, ) ~(0, Ψ) � , � = 0 Given the above model, we are primarily interested in the coefficients given by 01—i.e., the estimate of the association of a given aspect of quality and child development conditional on the age and sex of a child and structural aspects of the classroom. 14 Results Q1. What Do the Study Pre-Primary Classrooms in South Punjab Look Like? We begin our results with purely descriptive analysis of the composition of classrooms focusing on the characteristics of their teachers and the children enrolled in them. As shown in Figure 4, Katchi classes are designed for children aged 4-5 years old and Grade 1 is designed for children about 6 years old and above. While these distinctions exist in theory, in practice age requirements are not followed. In addition, a new type of class, ECE class, has a two-year curriculum according to the Punjab ECCE policy. As described above in Sample, we do not include ECE classes in this analysis. Figure 4. Intended Age Range and Curriculum by Class Type Age 4 5 6+ Katchi No curriculum ECE 2 year curriculum Grade 1 Grade 1 curriculum Table 7 below describes the composition of classes in the sample. Determining the grade/level of a classroom is not always straightforward given that many classes operate in a multi-graded fashion–i.e., multiple levels are taught in the same room by the same teacher. For analytical purposes, we classify grades according to the grade/level that was being taught at the time of observation and additionally record whether a classroom was multi-graded or not. Nearly four in ten classrooms were taught in a multi-graded fashion. In research question 4, we will address multi- graded classes, but for the subsequent sections, we focus only on disaggregating by Katchi and Grade 1, regardless of whether or not they were multi-graded. Table 7. Number of Classrooms by Grade and Multi-graded Status Grade Single graded Multi-graded Total Katchi 394 295 689 Grade 1 451 255 706 Total 845 550 1,395 Enrollment and Attendance Enrollment size varies substantially, with as few as 5 and as many as 136 children enrolled in a single class. The average classroom has about 29 children enrolled. As shown in Figure 5, Katchi classrooms were much larger than Grade 1 classes, with the average Katchi class enrolling 15 33.4 children compared to 24.4 in Grade 1. Figure 6 shows the distribution of attendance in sampled classes. On average, the 82% of children were in attendance on the date of observation. However, 25% of classes had a quarter or more students absent on the day of observation as shown by the orange shading. Figure 5. Average and Median Enrollment Size by Class Type Figure 6. Distribution of Attendance on Day of Observation There are sex differences in both enrollment and attendance as shown in Figure 7. Across classrooms, the average number of boys enrolled was about 17 compared to 12 for girls. As shown 16 in the figure, the discrepancy between sexes is particularly large for older children, with the average classroom having about 7 boys for every 5 girls enrolled over the age of six. Figure 7. Average Number of Children Enrolled and Present in Classrooms by Age and Sex Class Composition Figure 8 displays the observed composition of the average Katchi and Grade 1 classrooms. As we would expect, the average age of enrolled children in Grade 1 classes is higher at 6.34 years compared to Katchi at 5.49. Beyond these average differences, there is large age heterogeneity in classrooms, above and beyond what is dictated by policy. Not only are there underage students in Katchi, but nearly half of students in Katchi are six years old or older. This aligns with anecdotal evidence which suggests that Katchi is being used to place students that are deemed unready for Grade 1. In contrast, we also see that about one in five students enrolled in Grade 1 classes is less than six years old. 17 Figure 8. Actual Age Composition of Classes Teacher and Staff Characteristics Nearly all (99%) classrooms had a single teacher present in the classroom, and only 3.4% of classrooms had any other adult serving as an assistant to the teacher. Across the sample, teachers were equally likely to be male or female—however we note that there was a difference by class type, with 53% of Katchi teachers being female, compared to 46% of Grade 1 teachers. There are some differences by gender when it comes to experience and certification as shown in Figure 9. Female teachers, on average, have more experience teaching pre-primary classes than male teachers—63% of male teachers reported a year or less experience teaching preschool aged children, compared to 57% of female teachers. In contrast, the educational status of male teachers is higher on average with 93% of male teachers reporting either having completed a bachelor’s or master’s degree in education or possessing a teaching certificate. While most (83%) female teachers also possessed a degree or a certificate, 16% reported having no certification. 18 Figure 9. Teacher Experience and Certification Level by Sex Figure 10 shows the proportion of teachers achieving different levels of certification by class type. Grade 1 classes were usually taught by a teacher with a B.Ed. and just 8% of classes were taught by teachers 19 without a certification. Most Katchi classes were also taught by a certified teacher, but 15% of Katchi classes were taught by teachers with neither a degree nor a certificate. Figure 10. Teacher Certification by Class Type 20 Q2. What Is the Quality of These Classrooms? Having described who is teaching and learning in sampled preschool-aged classrooms in South Punjab, we now turn our attention to the quality demonstrated in these classrooms as measured by the Teach ECE. Structural Aspects of Quality Figure 11 displays the Water, Sanitation, and Health (WASH) resources available in Katchi and Grade 1 classrooms. Overall, most classrooms have adequate access to most WASH resources—with over 90% of classrooms having access to clean drinking water, child-friendly handwashing, and toilet facilities. However, nearly 20% of schools did not have clean toilet facilities, and just 40% had gender-segregated toilets for girls. Figure 11. Water, Sanitation, and Health Resources of Classrooms Figure 12 displays the availability of learning materials in these classrooms. We record both whether the class has any of these resources available at all in the classroom, and the proportion of children who have access to these resources. For example, while 59% of Katchi classes have some writing materials (e.g., pens or pencils) available for students, only approximately 26% of children can access them. There are many aspects of the material 21 environment of classes which could be improved. Fewer than one in four Grade 1 classes have access to any art supplies orstorybooks. Even when available, less than 10% of children can use them at any one time. Figure 12. Availability of Learning Materials in Classrooms The Teach ECE instrument also assesses the physical infrastructure of the classroom to inform about possible safety concerns. Overall, there were few major issues; however, 20% of Katchi classrooms had broken windows or doors. 13% had broken chairs, and 18% did not have adequate fencing on school grounds to protect children from road traffic. In Grade 1 classes, the safety situation appears broadly similar. Process Aspects of Quality Teach ECE tracks the Time on Task of teachers and children, and also reports on the quality levels of nine elements of process quality nested in three areas (Classroom Culture, Guided Learning, Socioemotional Skills). Elements are scored from 1-5 based on the quality ratings of multiple behaviors. To efficiently communicate the level of quality in classrooms for these descriptive purposes, we report the proportion of segments meeting minimum levels of quality. For the Time on Task indicator, this is the proportion of segments with the teacher leading a 22 learning activity with at least half of children engaged. For the other elements, this is the proportion of segments with at least a score of three. 6 Figure 13. Proportion of Segments with Minimum Levels of Quality in Teach ECE Elements Figure 13 shows the proportion of segments that classes were observed with minimum levels of quality in each element as defined above. While the teacher was creating an adequately supportive learning environment in 90% of observed segments and engaging the classes in learning activities with at least half of the class on-task in about three fourths of segments, most other aspects of quality were much less frequently observed. In the Guided Learning area, teachers were only providing feedback and guidance during one third of segments and rarely provided tasks that engaged children’s critical thinking. The Socioemotional area shows the weakest aspects of quality, with teachers only providing children with some degree of autonomy in their learning during one third of segments, and very few (7%) segments fostering social and collaborative skills through small group work or cooperative play. 6 See Selection of Benchmarks for Descriptive Analysis and the Teach ECE tool for additional clarification regarding these decisions and Appendix C: Proportion of Classrooms Meeting Minimum Quality Levels on Teach ECE Behaviors for results for each Teach ECE behavior. 23 To further concretize these findings, Figure 14 shows the proportion of segments that classes were observed meeting at least a “Medium” quality rating on 10 exemplar behaviors (items). Full results of all Teach ECE behaviors are presented in Appendix C: Proportion of Segments Meeting Minimum Quality Levels on Teach ECE Behaviors. These findings give depth to the elements in Figure 13. In terms of Classroom Culture, we can see that in 95% of segments, teachers were treating all children respectfully during the observation, but that behavioral expectations were not clear for children in about half of observed segments. In terms of support for Guided Learning, while teachers are providing specific comments to clarify misunderstandings in about half of the observed segments, in just 10% of segments were teachers observed using open-ended questions to prompt children’s critical thinking. Under Socioemotional Skills, we see that children are often volunteering (58% of segments) and that teachers respond positively to individual children’s challenges in 57% of observed segments, but that in just 13% of segments did children have opportunities to meaningfully collaborate with one another. Figure 14. Proportion of Segments Meeting Minimum Quality Levels: Exemplar Teach ECE Behaviors Q2.1. How Does Quality Differ by Classroom and Teacher Characteristics? We now explore how these elements of quality differ by classroom and teacher characteristics. The first characteristic we examine is how the Teach ECE elements differ by class type as shown in Figure 15. Overall, Katchi and Grade 1 classes appear to have similar levels of quality in most elements. The largest differences are in the likelihood of being on task and Facilitation of Learning, in which Grade 1 classes have higher quality on average, and Autonomy, in which Katchi classes more consistently provide opportunities for choice. 24 Figure 15. Teach ECE Elements by Class Type Note: Error bars represent the 95% confidence interval The next disaggregate we examine is how Teach ECE quality varies by teacher certification. Figure 16 displays the proportion of segments that demonstrate minimum levels of quality in the Teach ECE elements according to whether the teacher has a higher education, a certificate, or no certification. Here we see a small but clear trend of higher quality in classes taught by more highly educated teachers. The elements that this is most apparent in are the Time on Task indicator and Guided Learning area. Across segments, 76% of the time classes taught by a teacher with higher education were on task compared to 65% of those with no certification. The Facilitation of Learning was also an area with a dramatic difference—45% of segments in classrooms taught by a teacher with a higher education had adequate supports for learning, compared to 27% in a classroom led by a teacher without any certification. Other elements showed smaller differences, but the trend remained clear across all aspects of the Teach ECE. Despite this overall positive trend, even highly educated teachers have substantial room to improve the quality of the learning in their classrooms. Recall that the bar here has been arbitrarily set as a rating of 3- Medium level quality. 25 Figure 16. Teach ECE Elements by Teacher Certification We also examine how these aspects of quality vary by teacher gender as shown in Figure 17. We did not expect to see major differences by this factor based on findings using these tools in other country settings. However, in the case of the sampled classrooms in South Punjab, across all elements of Teach ECE, female teachers have a small, consistent advantage in quality. While this relationship was true across the entirety of Teach ECE elements, the largest differences were found in the Guided Learning and Socioemotional Skills areas. Female teachers were observed supporting children’s facilitation of learning in 44% of segments, compared to 36% for male teachers. Female teachers also more consistently provided adequate checks for understanding in 55% of segments compared to 45% for male teachers. Finally, while neither male nor female teachers supported children’s perseverance in more than half of segments, female teachers were more likely (32% compared to 21%) to be observed demonstrating this aspect of quality in their classrooms. Overall, there is a clear pattern of higher quality in female-led classrooms. 26 Figure 17. Teach ECE Elements by Teacher Gender The final disaggregate that we examine is the language of instruction. Unlike class type or teacher certification or gender, there is not a consistent pattern between language and teaching quality as shown in Figure 18. Teachers using Saraiki and Other languages were more likely to be on task than those using Urdu (82% and 81% compared to 71%) and to set positive behavioral expectations (58% and 46% compared to 39%) but teachers in Urdu classes were more likely to provide helpful feedback (38% compared to 11% in Saraiki and 27% in Other) and support children’s Perseverance (29% compared to 11% and 14%). The relatively small number of teachers using Saraiki and other languages in the classroom complicates this analysis but there are not consistent differences in quality according to language across Teach ECE elements. 27 Figure 18. Teach ECE Elements by Language of Instruction Q3. What Can Children in Pre-Primary Classrooms Do? Having described the levels of quality present in classrooms for preschool-aged children – those between the ages of 4 and 6 – we now describe the skills that children are able to demonstrate and how these skills differ by age and gender. Later we will aggregate these measures into the domains and overall estimates of children’s skills described in AIM-ECD DA Score Calculation, but for the purposes of answering our third research question, we will focus on the proportion of children that demonstrated the abilities to successfully complete exemplar items on the AIM-ECD as mentioned previously in Table 1. Age is commonly the most important predictor of a child’s early learning and developmental level. As such, we begin our description by examining how the proportion of correct responses on these exemplar items increases with age. Figure 19 displays the proportion of children getting exemplar items correct by age. Across all items, we see the expected pattern of older children scoring higher. However, the age gradient varies substantially by item and many items have extremely low levels of correct responses even at high ages. 28 Among the items in Early Literacy, the steepest age gradients are found in identifying Urdu letter choti ye (66% of six-year-olds vs. 45% of four-year-olds) and writing letters of their name (60% of six-year-olds vs. 33% of four-year-olds). In the Early Numeracy domain, the steepest differences were found in identifying the numeral 10 (73% of six-year-olds vs. 47% of four-year- olds), and counting 6 and 14 objects, which had a difference of over 20 percentage points. Identifying English Letters also exhibited a large age difference with 58% of six-year-olds identifying letter Z compared to just 35% of four-year-olds. In contrast, other items show smaller differences between ages. On a listening comprehension item recalling details about the story, six- year-olds were only 10 percentage points more likely to respond correctly than four-year-olds (61% compared to 51%). Similarly, the difference between scores by age on shape identification by age was small—with just 20% of six-year-olds and 14% of four-year-olds able to identify a triangle. While all items exhibited an age gradient, it was much less pronounced in the Social Emotional and Executive Functioning domains. Focusing exclusively on the six-year-old children, who are either in Grade 1 or about to be, reveals significant gaps in the knowledge that these children possess. In Early Literacy, 6% could not identify the first letter of the Urdu alphabet, 34% could not identify the last letter of the Urdu alphabet, and 40% of six-year-olds could not write any letters of their name. In Early Numeracy, nearly one in four children could not identify the numeral 1 presented out of order, half of children could not count 14 objects correctly, 41% could not identify a circle, and 80% could not identity a triangle. In the Executive Functioning and Social Emotional domain, 11% of children could not remember and repeat a three-digit-sequence of numbers and just 17% of children could name an emotion that a picture of a girl crying was feeling. The skills tested on the AIM-ECD Direct Assessment serve as the foundational skills upon which later learning is built. As such, these findings strongly suggest room for improvement in the outcomes of preschool-aged children in at least some districts in South Punjab. 29 Figure 19. AIM-ECD Exemplar Item Correct Percentage by Age In addition to examining performance by age, we also disaggregate results by child sex as shown in Figure 20. In general, most items (11 out of 19) show little difference in correct response rates by sex, but when there are differences (in 8 of the 19 items) girls are outperforming boys. These differences are most evident in the Early Literacy domain, where girls were 6 p.p. more likely to identify the last letter of the Urdu alphabet, 5 p.p. more likely to answer a listening comprehension item correctly, and 7 p.p. more likely to be able to write at least a few letters of their name. Other items showed smaller differences in favor of girls or were similar. 30 Figure 20. Exemplar Items Correct Percentage by Child Sex Q4. How Are Children’s Early Learning and Development Related to Individual, Teacher, and Class Factors? This final section focuses on unpacking the results from a multiple mixed effects regression model. This represents the first time we are using formal statistical inference to make claims about the associations between quality and outcomes. In contrast to the above results focusing on exemplar items, we use domain-level and overall AIM-ECD and Teach ECE Z-scores. We report on two sets of models: those predicting the AIM-ECD Overall and Domain scores by Teach ECE Overall Quality and those predicting AIM-ECD Overall scores by the various areas of Teach ECE. Full model results are presented in Appendix D. Regression Tables. 31 The model building process used to arrive at our final model is described below in Table 8. We use Model 6 as our primary analytical model, which includes variables controlling for child’s age and sex, Grade 1 classroom, multi-graded classroom, the interaction between multi-graded and Grade 1 classroom, teacher’s sex, education, and experience, and classroom language. In addition to interactions between class type and multi-graded status, we tested a number of additional models including more complex interactions but found little of substance. As such, we do not report these models. Table 8. Model Building Process for AIM-ECD Overall Z-scores by Teach ECE Overall Z-scores (1) (2) (3) (4) (5) (6) Teach ECE Overall Z-score 0.0887*** 0.0892*** 0.0860*** 0.0777*** 0.0759*** 0.0755*** (0.0162) (0.0155) (0.0142) (0.0138) (0.0140) (0.0140) Child’s age (years) 0.232*** 0.203*** 0.198*** 0.198*** 0.197*** (0.0131) (0.0130) (0.0129) (0.0129) (0.0129) Child is female 0.0295 0.0205 0.0178 0.00631 0.00511 (0.0203) (0.0198) (0.0197) (0.0202) (0.0202) Multi-graded class -0.168*** 0.0935* 0.0976** 0.0932* (vs. Single-graded) (0.0284) (0.0376) (0.0376) (0.0376) Grade 1 class 0.566*** 0.816*** 0.820*** 0.814*** (vs. Katchi) (0.0276) (0.0364) (0.0374) (0.0375) Multi-graded x Grade 1 Class -0.554*** -0.553*** -0.546*** (0.0541) (0.0543) (0.0543) Teacher has teaching -0.0238 -0.0257 certificate or ed. degree (0.0428) (0.0428) Teacher has 3+ years pre- -0.0126 -0.0140 primary experience (0.0331) (0.0330) Teacher is female 0.0692* 0.0594* (0.0281) (0.0291) Saraiki language class -0.0271 (vs. Urdu language class) (0.0479) Other language class -0.156* (vs. Urdu language class) (0.0691) Constant 0.0663 -1.189*** -1.239*** -1.330*** -1.337*** -1.314*** (0.0605) (0.0929) (0.0922) (0.0925) (0.102) (0.103) N 8249 8249 8249 8249 8249 8249 Var u_k (Enum) 0.111 0.112 0.110 0.112 0.113 0.114 Var u_j (Class) 0.308 0.274 0.200 0.181 0.180 0.179 Var e_ijk (Residual) 0.589 0.576 0.571 0.571 0.571 0.571 Standard errors in parentheses ~ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001 32 Including the factors discussed in Q3 jointly makes several inferences abundantly clear. As expected, older children score higher on the AIM-ECD than younger children. An additional year of age is associated with a 0.197 standard deviation ( = 0.197, < .001) increase in AIM-ECD Z- score. However, we do not see any significant differences in overall development by child sex. In addition to age, the grade level of the class explains a huge amount of variation in AIM-ECD scores. Children in Grade 1 classes performed 0.814 standard deviations ( = 0.814, < .001) higher than those in Katchi classes. However, this difference is greatly attenuated in multi-graded classes. While multi-graded Katchi classes have slightly higher ( = 0.093, < .05) AIM-ECD Z- scores than children in single-graded Katchi classes, children in multi-graded Grade 1 classes score -.546 SD worse than those in single-graded Grade 1 classes ( = −0.546, < .001). Classes taught in Urdu and Saraiki appear to have similar levels of achievement on average, but those taught in another language had -0.156 SD lower scores on average than Urdu classes ( = −0.156, < .05). Our model finds strong evidence about the importance of ECE teaching quality as shown in Table 8 and Figure 21. A one-standard deviation difference in Overall Teaching Quality is associated with a 0.076 standard deviation difference in AIM-ECD Z- scores ( = 0.076, < .001), even after controlling for other factors (Model 6). After controlling for classroom quality, other teacher-level factors that might be hypothesized to be related to outcomes are not significantly associated with AIM-ECD. In our final model, teachers with three or more years of experience and those with higher levels of certification did not have classes with higher levels of AIM-ECD scores. One factor appears to remain salient—teacher gender. Classrooms led by female teachers have 0.060 SD higher ( = 0.060 < 0.05) AIM-ECD Z-scores on average, even after controlling for other factors of quality. Using association between age and AIM-ECD scores allows us to convert differences in Z-scores to differences by age. Given that a year of age is associated with a 0.197 SD difference in AIM-ECD scores, and that a 1 SD difference in Teach ECE scores is associated with a .076 SD difference in AIM-ECD scores, we can conclude that a child in a high-quality classroom (+2SD) is predicted to have an AIM-ECD score that is similar to a child who is 0.77 years (~9 months) older in an average quality classroom. 33 Figure 21. Predicted AIM-ECD Z-Scores by Child Age and Teach ECE Overall Quality Score Note: Predicted Z-scores from Model 6 holding all other variables at their mean. AIM-ECD Domain Models The models shown in Panel A of Appendix D. Regression Tables report on the associations between Teach ECE Overall quality and the domains of AIM-ECD. Overall, most models tell a similar story as the Overall AIM-ECD but there are some differences to note. The age gradient differs by domain—the slope is over twice as steep for the Early Literacy and Early Numeracy domains as it is for the Social-Emotional domain. We also find a large and significant difference by gender when it comes to the Early Literacy domain—girls are significantly outperforming boys ( = 0.073, < 0.001). In other domains, the differences between girls and boys are insignificant. The domain-level models also reaffirm the consistent and significant relationship between quality and outcomes, but show that this relationship differs by domain. Figure 22 illustrates the predicted AIM-ECD domain Z-scores for children in low (-2 SD), average, and high-quality (+2 SD) classrooms according to the Teach ECE Overall Z-score. The strongest relationships between quality and outcomes are found in the Early Numeracy ( = 0.077, < .001), and Executive Functioning ( = 0.070, < .001) domains. In contrast, while still significant, the relationship was about half as strong in the Social Emotional domain ( = 0.028, < .05). As before, we can 34 use the age gradient to convert these differences to age. Compared to an average quality classrooms, a child studying in a high-quality (+2 SD) classroom is predicted to have similar Early Literacy scores to a child 8 months older, similar Early Numeracy scores to a child 9 months older, similar Executive Functioning scores to a child one year older, and similar Social Emotional scores to a child 7 months older. Other factors also varied in the strength of the relationship. Children studying in Grade 1 strongly outperformed those in Katchi and ECE. This association was strongest for the pre- academic domains of Early Literacy ( = 0.851, < .01) and Early Numeracy ( = 0.829, < .01) domains. In contrast, the gender of the teacher was an insignificant predictor for the Social Emotional and Early Numeracy domains, but relatively strong for the Executive Functioning ( = 0.071, < .05) and Early Literacy ( = 0.065, < .05) domains. Figure 22. AIM-ECD Domain Z-scores Predicted by Teach ECE Overall Quality Note: Holding all other variables at their mean, the plot shows the estimated AIM-ECD Z-scores for children studying in low (-2), average (0) and high (+2) quality classrooms. Teach ECE Domain Models Above we examined how the various domains of AIM-ECD were differently associated with overall quality as represented by the Teach ECE Overall Z-score. Now we shift to whether 35 the Overall AIM-ECD Z-score is differently associated with the different areas of Teach ECE. Panel B of Appendix D. Regression Tables shows the associations between each Teach ECE area and AIM-ECD Overall. All Teach ECE areas are strongly related to AIM-ECD Overall scores. Somewhat surprisingly, the strength of associations between the various areas of quality on Teach ECE and AIM-ECD Overall Z-scores does not appear to vary much. The strong associations are seen for the Time on Task areas ( = 0.087, < .001), but this was not much larger than the association with Social Emotional Skills ( = 0.065, < .001). Figure 23 shows this similarity, with the predicted AIM-ECD Overall Z-scores in high (+2 SD) and low (-2 SD) quality classrooms by each area of Teach ECE nearly equal to each other. Figure 23. Predicted AIM-ECD Overall Score in High- and Low-Quality Classrooms by Teach ECE Area for a 5-year-old Child Discussion The data presented in this paper provide a unique opportunity to present answers to an array of descriptive and inferential research questions focused on pre-primary classrooms in Pakistan. The subset of relationships highlighted in this paper are reflective of the key policy priorities that routinely emerge in popular discourse and in debates among decision-makers and practitioners on 36 the quality of pre-primary schooling and children’s development outcomes in the early years. A brief recap of what the data indicate: Pre-primary classrooms are often large and composed of children of diverse ages. Many children are studying in classrooms with age-levels that are not consistent with the assigned age range of the class. Large age ranges in a classroom can make it difficult for a single teacher to effectively engage the entire class in activities. Children’s absenteeism is a challenge. About a quarter of classes had more than one in four children absent on the day of an announced visit. Ensuring children and teachers are present in schools is, of course, a prerequisite for learning. As such, better understanding the causes for absenteeism of pre-primary children is an important area for future research. Many children lack critical skills. Many children lack the pre-academic and school- readiness skills necessary to succeed later on in life as evidenced by their performance on basic exemplar items such as number identification and two digit addition. We find the results for six- year-old children particularly illustrative in this regard, with a third of 6-year-olds unable to identify the last letter of the Urdu alphabet, 41% unable to identify a circle on a piece of paper, and 23% unable to identify the numeral 1 out of sequence. Girls are not attending in equal numbers. Even at the earliest ages and in the youngest classes, there are more boys than girls. The gap is larger for older children and in Grade 1. It appears that the enrollment gap in primary school starts extremely early, and there is no age at which there are equal numbers of boys and girls. Policy makers interested in girls’ education cannot wait to address the issue of non-enrollment of girls when children are older—it is something that must be addressed when children are as young as four years old. Girls in school can thrive. Despite being underrepresented in pre-primary classes, they appear to be doing at least as well as boys in many early learning and development skills. Indeed, in the Early Literacy domain of AIM-ECD, we find that girls significantly outperform boys. An examination of structural quality reveals both areas of strength and of weakness. Structural quality features of classrooms reveal many positive findings, such as a relative absence of safety hazards in classrooms and adequate water and sanitation facilities in most schools. However, we also find substantial material deficiencies. It is unusual for pre-primary children to be in classrooms with adequate levels of learning materials. Learning resources are rare in classrooms—with just 25% of Katchi classrooms having storybooks, and 27% with any sort of educational toys or blocks. Even when classrooms have such materials in the classroom, many times children are not able to access or use them. Multi-graded classrooms are a challenge for older children. When teachers have multiple grades in the same classroom, it presents a challenge to target instruction to the right level, manage the classroom effectively, and ensure that all children are learning. According to our results, these factors appear to be especially challenging for Grade 1 classrooms, where multi- grading was strongly associated with negative outcomes. We do not find the same challenge in Katchi classrooms. 37 Process quality indicators show substantial room for improvements in how teachers engage young students. The findings from Teach ECE provide ample evidence of areas in both structural and process quality that would benefit from further investment. In more than one-fourth of segments observed, the teacher was not engaging children in a learning activity. Classroom Culture was largely positive, but in just one third of segments were teachers observed providing children with constructive feedback, in just 12% were children engaged in any kind of critical thinking activities, and in just one third of segments did children have any sort of autonomy over their participation in class activities. Teaching quality matters – more than teacher education and experience. Despite low levels of teaching quality in many classrooms, we find strong evidence of its association with outcomes. In earlier analysis, we disaggregated levels of children’s learning and development by classroom and teacher factors, we found that children’s skills varied by teacher level of certification, with children in classes taught by teachers with higher education exhibiting stronger outcomes. We also found that classes with more highly educated teachers exhibited higher levels of quality. However, once we controlled for multiple factors simultaneously, we find consistent and positive associations between teaching quality (i.e., practices employed by teachers in the classroom) and outcomes, but that other teacher- and classroom-level factors matter less. Our results cannot be taken as causal evidence, as we discuss more in Limitations, but these results suggest that that differences in outcomes by teacher certification may be due to the higher quality of classrooms taught by more certified teachers. Given the low levels of quality observed in many classrooms, these results are encouraging as a potential path forward for further investment in both the material resources preprimary classes have access to and use as well as encouragement for teachers to engage in more evidence-based classroom practices. Female teachers perform. As mentioned above, most teacher and classroom factors were not associated with outcomes after controlling for quality. One exception was teacher gender. We find a small but consistent advantage in AIM-ECD performance from classrooms led by a female teacher that are not explained by differences in quality or background. Limitations This work is, to the best of our knowledge, the first to systematically examine the early learning and developmental status of children enrolled in ECE settings and the quality of such settings in selected districts of South Punjab, Pakistan. Its strengths lie in its ability to present rich data on both teaching quality and child outcomes using a large, representative sample of schools from a particularly disadvantaged setting in a low-income country. We must also acknowledge the limitations of this work. These limitations fall under three main categories: sampling, design, and interpretation. Sample The sample of children included in this study is a representative sample of pre-primary aged children studying in the 3,400 PHCIP public schools and can be reasonably argued to be representative of the full population of public pre-primary aged children in the 11 districts in South 38 Punjab. However, as noted earlier, many pre-primary aged children in Pakistan are either not enrolled in school or are studying in private schools that were not included in this study. Further, given the sampling design, children who are chronically absent are less likely to be included. As such, the generalizability of the sample may be somewhat limited to this albeit large and important subset of public pre-primary aged children in South Punjab. Design We have reported outcomes disaggregated by numerous factors and statistically tested associations between classroom quality and early learning and developmental outcomes. These findings represent the results of a cross-sectional analysis at a single point of time and do not utilize any quasi-experimental sources of exogenous variation. As such, the inherently correlational nature of our analysis prohibits us from making any causal claims. We attempt to address some sources of confounding in the model building process used to address Q4, but it is possible that there are additional unobserved factors that may be influencing results. This is particularly important given the literature on longer-term impacts of classroom quality on learning outcomes in primary education or better income in life. Further research utilizing stronger causal designs must be implemented in order to better understand these types of conclusions. Interpretation Our exemplar items on AIM-ECD and Teach ECE are a useful framework to understand in concrete terms what the children in the study in South Punjab are and are not able to do and the levels of quality present in classrooms. However, these comparisons are inherently limited without a more robust normative interpretation. Our descriptive analysis is an empirical examination of what children are able to do and what quality levels teachers demonstrate, but we do not report on what level they should be able to achieve. Further work is required to define minimum levels of acceptable proficiency on the AIM-ECD and stronger definitions of minimum benchmarks on Teach ECE in order to better interpret the gaps that exist. Conclusion This study is the first large-scale, in-depth study of pre-primary aged children’s outcomes and classroom quality in South Punjab, Pakistan. We find that while some classrooms demonstrate adequate levels of quality, there is substantial room for improvement. We find variable levels of achievement on a range of important school readiness and early learning and developmental skills and identify several factors that may be important to consider for policy. The data also present some findings that run contrary to popular belief. These include the findings on teacher certification, the schooling outcomes of girls, and the strengths and weaknesses of being in a multigrade classroom – to name a few. 39 References Akyeampong, K., Andrabi, T., Banerjee, A., Banerji, R., Dynarski, S., Glennerster, R., Grantham- McGregor, S., Muralidharan, K., Piper, B., Ruto, S., Saavedra, J., Schmelkes, S., & Yoshikawa, H. (2023). Cost-effective approaches to improve global learning. What does recent evidence tell us are “smart buys” for improving learning in low- and middle-income countries. FCDO; World Bank; UNICEF; USAID. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing (Joint Committee on Standards for Educational and Psychological Testing (U.S.), Ed.). American Educational Research Association. Augsburg, B., Attanasio, O. P., Dreibelbis, R., Nketiah-Amponsah, E., Phimister, A., Wolf, S., & Krutikova, S. (2022). Lively Minds: Improving health and development through play–a randomised controlled trial evaluation of a comprehensive ECCE programme at scale in Ghana. BMJ Open, 12(10), e061571. Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1). Burchinal, M. (2018). Measuring Early Care and Education Quality. Child Development Perspectives, 12(1), 3–9. Bureau of Statistics Punjab, Planning & Development Board, Government of the Punjab & UNICEF. (2018). Multiple Indicator Cluster Survey Punjab, 2017-18, Survey Findings Report. Bureau of Statistics Punjab, Planning & Development Board, Government of the Punjab. Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Sage Publications. Ding, E., Pushparatnam, A., Seiden, J., Avedaño, E., Molina, E., Cloutier, M.-H., Luna Bazaldua, D. A., Hasan, A., & Gregory, L. (2023, July). Validity evidence for an ECE classroom observation tool 40 [Poster presentation]. 2023 International Meeting of the Psychometric Society, College Park, MD, USA. Ding, E., Pushparatnam, A., Seiden, J., Avedaño, E., Molina, E., Cloutier, M.-H., Luna Bazaldua, D., Hasan, A., & Gregory, L. (2024). Validity evidence for an ECE classroom observation tool. In H. Wu (Ed.), Quantitative Psychology: The 88th Annual Meeting of the Psychometric Society, College Park, MD, 2023. Springer. Duncan, G. J., Dowsett, C. J., Claessens, A., Magnuson, K., Huston, A. C., Klebanov, P., Pagani, L. S., Feinstein, L., Engel, M., Brooks-Gunn, J., Sexton, H., Duckworth, K., & Japel, C. (2007). School readiness and later achievement. Developmental Psychology, 43(6), 1428–1446. Engle, P. L., Fernald, L. C., Alderman, H., Behrman, J., O’Gara, C., Yousafzai, A., de Mello, M. C., Hidrobo, M., Ulkuer, N., Ertem, I., & Iltus, S. (2011). Strategies for reducing inequalities and improving developmental outcomes for young children in low-income and middle-income countries. The Lancet, 378(9799), 1339–1353. Ersado, L., Hasan, A., Geven, K., Martijn, K., Ashi, K., Baron, J., Bend, M., & Ahmed, S. A. (2023). Pakistan Human Capital Review: Building Capabilities throughout Life. World Bank. Global Education Evidence Advisory Panel. (2020). Cost Effective Approaches to Improve Global Learning What Does Recent Evidence Tell Us Are Smart Buys for Improving Learning in Low- and-Middle-Income Countries. World Bank; FCDO; Building Evidence in Education. Hasan, A., Jung, H., Kinnell, A., Maika, A., Nakajima, N., & Pradhan, M. (2019). Contrasting Experiences: Understanding the Longer-Term Impact of Improving Access to Preschool Education in Rural Indonesia. The World Bank. Holla, A., Bendini, M., Dinarte, L., & Trako, I. (2021). Is Investment in Preprimary Education Too Low? Lessons from (Quasi) Experimental Evidence across Countries. The World Bank. Nakajima, N., Hasan, A., Jung, H., Kinnell, A., Maika, A., & Pradhan, M. (2021). Built to Last: Sustainability of Early Childhood Education Services in Rural Indonesia. The Journal of Development Studies, 57(10), 1593–1612. 41 National Curriculum Council. (2020). Single National Curriculum Early Childhood Care and Education Curriculum Grade Pre I. Ministry of Federal Education and Professional Training, Government of Pakistan. Pakistan Bureau of Statistics. (2017). Population Census 2017: Population by single year age, sex and rural/urban. Pakistan Bureau of Statistics. Pisani, L., Borisova, I., & Dowd, A. J. (2018). Developing and validating the International Development and Early Learning Assessment (IDELA). International Journal of Educational Research, 91, 1– 15. Pushparatnam, A., & Ding, E. (2021, June 11). Scaling up quality early childhood education: What gets measured gets done [World Bank Blogs]. Education for Global Development. Pushparatnam, A., Luna Bazaldua, D. A., Holla, A., Azevedo, J. P., Clarke, M., & Devercelli, A. (2021). Measuring Early Childhood Development Among 4–6 Year Olds: The Identification of Psychometrically Robust Items Across Diverse Contexts. Frontiers in Public Health, 9, 569448. Quaid-e-Azam Academy for Educational Development & UNICEF. (n.d.). Punjab Early Learning and Development Standards Assessment Tool (Children Age 3, 4, and 5 Year). Quaid-e-Azam Academy for Educational Development. R Core Team. (2021). R: A language and environment for statistical computing (4.2.1 (Bird Hippie)) [Computer software]. R Foundation for Statistical Computing. Raikes, A., Koziol, N., Janus, M., Platas, L., Weatherholt, T., Smeby, A., & Sayre, R. (2019). Examination of school readiness constructs in Tanzania: Psychometric evaluation of the MELQO scales. Journal of Applied Developmental Psychology, 62, 122–134. Saavedra, J., & Sherburne-Benz, L. (2022, September 28). Pakistan’s Floods are Deepening its Learning Crisis. End Poverty in South Asia. StataCorp. (2019). Stata Statistical Software (Release 16) [Computer software]. StataCorp LLC. Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T., Miller, E., Bache, S., Müller, K., Ooms, J., 42 Robinson, D., Seidel, D., Spinu, V., … Yutani, H. (2019). Welcome to the Tidyverse. Journal of Open Source Software, 4(43), 1686. World Bank. (2021, March 18). Teach ECE: Helping Countries Track and Improve Teaching Quality in Early Childhood Education. World Bank. (2022, September 1). A Set of Resources to Help Countries Measure and Improve Early Childhood Development [Text/HTML]. World Bank. 43 Appendix A: Crosswalk between Student Learning Objectives and AIM-ECD Subtasks Below we review the contents of the Single National Curriculum Early Childhood Care and Education Curriculum Grade Pre I to review the linkages between the contents of Chapter 2 (“Key Learning Areas, Competencies and Expected Learning Outcomes for ECCE”) and the constructs measured by AIM-ECD (National Curriculum Council, 2020). Domain Competency Sub-competency Related AIM-ECD Subtask PERSONAL, SOCIAL 1: Children will develop an understanding of their b. Identify different Empathy/Perspective AND EMOTIONAL likes, dislikes, strengths, emotions and self-grooming, occasions when they Taking DEVELOPMENT decision making and problem solving skills. Children feel happy, sad, scared, will further enhance their positive sense of self- loved, angry, excited identity and see themselves as capable learners. and bored. PERSONAL, SOCIAL 2: Children will be willing to share and work in a. Show an Empathy/Perspective AND EMOTIONAL collaboration with their peers, teachers, family understanding and Taking DEVELOPMENT members and neighbours, regardless of any respect for the feelings differences, such as, in gender, ability, culture, of their peers and others. language and ethnicity. LANGUAGE AND 1: Children will engage confidently with others using h. Recognition of letters Initial Sound LITERACY language in a variety of ways for a variety of with their initial sounds. Discrimination purposes and contexts. LANGUAGE AND 1: Children will engage confidently with others using i. Recognition of Initial Sound LITERACY language in a variety of ways for a variety of phonemes (phonemic Discrimination purposes and contexts. awareness) in spoken words. LANGUAGE AND 2: Children will describe objects, events and their b. Describe and talk Listening Comprehension LITERACY plans for the day. about pictures, drama, Story animated video etc 44 LANGUAGE AND 6: Children will recognise letters and familiar words b. Identify and name the Listening Comprehension LITERACY in simple texts. characters in a story. Story LANGUAGE AND 6: Children will recognise letters and familiar words c. Recognise their Name Writing LITERACY in simple texts. names in print (Urdu & English). LANGUAGE AND 6: Children will recognise letters and familiar words d. Begin to recognise Urdu Letter Identification LITERACY in simple texts. letters of the Alphabet. / English Letter Identification LANGUAGE AND 6: Children will recognise letters and familiar words f. Identify letter sounds Initial Sound LITERACY in simple texts. through words that have Discrimination personal meaning for them. LANGUAGE AND 6: Children will recognise letters and familiar words g. Associate initial letter Initial Sound LITERACY in simple texts. sounds with names of Discrimination objects in their classroom environment. LANGUAGE AND 6: Children will recognise letters and familiar words h. Think of a variety of Initial Sound LITERACY in simple texts. objects beginning with a Discrimination single letter of the alphabet. LANGUAGE AND 7: Children will use pictures, symbols and familiar a. Make marks and Name Writing LITERACY letters and words to communicate meaning, show scribble to communicate awareness of some (symbols, letters and words), for meaning, representing the different purposes of writing. some sounds correctly and in sequence. 45 LANGUAGE AND 7: Children will use pictures, symbols and familiar b. Use some clearly Name Writing LITERACY letters and words to communicate meaning, show identifiable letters in awareness of some (symbols, letters and words), for their writing to the different purposes of writing. communicate meaning LANGUAGE AND 7: Children will use pictures, symbols and familiar h. Trace, copy and write Name Writing LITERACY letters and words to communicate meaning, show the letter of Urdu awareness of some (symbols, letters and words), for alphabet. the different purposes of writing. LANGUAGE AND 7: Children will use pictures, symbols and familiar i. Trace, copy and write Name Writing LITERACY letters and words to communicate meaning, show the letters of the English awareness of some (symbols, letters and words), for alphabet. the different purposes of writing. LANGUAGE AND 7: Children will use pictures, symbols and familiar m. Write their own Name Writing LITERACY letters and words to communicate meaning, show names in English & awareness of some (symbols, letters and words), for Urdu and their native the different purposes of writing. language with appropriate use of upper and lower case letters. BASIC 1: Children will develop basic logical, critical, i. Group objects together Shape Identification MATHEMATICAL creative and problem solving skills by demonstrating according to their shapes CONCEPTS an understanding of the different attributes of objects and colours. (such as colour, size, weight and texture) and match, sequence, sort and classify objects based on one/two attributes. BASIC 1: Children will develop basic logical, critical, m. Differentiate between Number Comparison MATHEMATICAL creative and problem solving skills by demonstrating 'more', 'less' and 'equal. CONCEPTS an understanding of the different attributes of objects (such as colour, size, weight and texture) and match, 46 sequence, sort and classify objects based on one/two attributes. BASIC 2: Children will develop a basic understanding of b. Understand one to Producing a Set MATHEMATICAL quantity, counting up to 50 and simple number one correspondence. CONCEPTS operations of 0-9. BASIC 2: Children will develop a basic understanding of e. Compare quantities of Number Comparison MATHEMATICAL quantity, counting up to 50 and simple number objects in different sets CONCEPTS operations of 0-9. and describe which sets are equal, which have more objects, and which have lesser objects than another. BASIC 2: Children will develop a basic understanding of g. Identify and write Number Identification MATHEMATICAL quantity, counting up to 50 and simple number correct numerals to CONCEPTS operations of 0-9. represent numbers from 0-50. BASIC 2: Children will develop a basic understanding of i. Identify which Number Comparison MATHEMATICAL quantity, counting up to 50 and simple number numeral represents a CONCEPTS operations of 0-9. bigger quantity or lesser quantity. BASIC 2: Children will develop a basic understanding of l. Use concrete objects Simple Addition MATHEMATICAL quantity, counting up to 50 and simple number to develop the concept CONCEPTS operations of 0-9. of addition and subtraction BASIC 2: Children will develop a basic understanding of m. Substitute numerals Simple Addition MATHEMATICAL quantity, counting up to 50 and simple number for concrete objects CONCEPTS operations of 0-9. 47 during the process of addition. BASIC 2: Children will develop a basic understanding of n. Use the concept of Simple Addition MATHEMATICAL quantity, counting up to 50 and simple number addition in their daily CONCEPTS operations of 0-9. lives with oral examples. BASIC 2: Children will develop a basic understanding of q. Use the concept of Simple Addition MATHEMATICAL quantity, counting up to 50 and simple number subtraction in their daily CONCEPTS operations of 0-9. lives with oral examples BASIC 3: Children will recognise basic geometrical shapes a. Recognise, name and Shape Identification MATHEMATICAL and the position of objects in relation to each other draw two dimensional CONCEPTS and surroundings shapes, such as circle, oval, square, rectangle or triangle, using features such as number of sides, curved BASIC 3: Children will recognise basic geometrical shapes c. Identify the shapes in Shape Identification MATHEMATICAL and the position of objects in relation to each other their environment. CONCEPTS and surroundings BASIC 3: Children will recognise basic geometrical shapes e. Develop Receptive Spatial MATHEMATICAL and the position of objects in relation to each other understanding and Vocabulary CONCEPTS and surroundings describe the position and order of objects using position words such as, in front of, behind, up, down, under, inside, outside, between and next to. 48 Appendix B: Crosswalk between Punjab Early Learning and Development Standards and AIM-ECD The Punjab Early Learning and Development Standards are a series of age-specific achievements that are used to monitor early childhood development in South Punjab (Quaid-e- Azam Academy for Educational Development & UNICEF, n.d.). While the AIM-ECD assessment was not designed with the ELDS in mind, below we illustrate the overlapping constructs from the AIM-ECD subtasks and the ELDS items which attempt to capture similar skills. Most AIM-ECD subtasks have very close matches to one or more ELDS. ELDS with conceptual similarities, but less strong connections, are marked with a star (*). AIM-ECD RELEVANT AGE 4 ELDS RELEVANT AGE 5 ELDS SUBTASK LETTER 4.2.4 Recognizes the capital letters and small 4.1.3. Knows the sounds of vowels a, e, IDENTIFICATION letters and also recognizes sounds of letters. I, o, and u and consonants. 4.3.2 Knows that text carries meaning and is written from right to left (Urdu) LISTENING 3.1.2 Remembers parts of a story/sequence of 3.1.2. Enjoys stories with riddles, COMPREHENSION the story. guessing and suspense. * 3.1.10 Identifies and can name colours such as 3.2.1. Identifies and can name colours. * red, green, blue, yellow, organ, and purple. * 3.2.4. Demonstrates awareness and 4.2.2 Identifies names and characters in a story. appreciation for family and cultural stories. * 4.1.1. Demonstrates an understanding of conversations by responding to questions. 4.2.3 Predicts what comes next in the story. 4.2.5 Understands sentences with two or more phrases or ideas. 5.6 Understands questions and answers actively. NAME WRITING 1.2.3 Holds a crayon or marker using a tripod 4.3.1 Writes two and three letter words. grasp. 4.3.3 Tries to write letters (Urdu/English) SOUND 4.1.2 Recognizes phonics in spoken language. 4.1.3. Knows the sounds of vowels a, e, DISCRIMINATION 4.2.4 Recognizes the capital letters and small I, o, and u and consonants. letters and also recognizes sounds of letters. 4.1.4 Uses rhymes and relates it to spelling patterns (words building). * 4.2.2 Uses sounds and rhymes to relate this to spelling patterns. * 4.2.6 Identifies rhyming words* NUMBER 3.1.6 Recognizes, counts, and writes numbers 0- 3.1.3. Counts, recognizes, and writes 0-- IDENTIFICATION 20 and can separate objects/things etc. 50 numerals. PRODUCING A 3.1.3 Understands idea of counting 3.1.1 Counts 10 or more things and SET objects/people. compares the groups formed. 3.1.6 Recognizes, counts, and writes numbers 0- 20 and can separate objects/things etc. 3.1.7 Counts reliably ten everyday objects. SHAPE 1.2.2 Forms shapes and objects out of clay. * 3.1.7 Recognizes and names 3-D shapes IDENTIFICATION 3.1.6 Recognizes shapes circle, square, triangle, such as sphere, cube, cylinder and cone. rectangle, and oval. 49 SIMPLE 3.1.11 Understands the concept of addition 3.1.1 Counts 10 or more things and ADDITION through objects. compares the groups formed. 3.1.4. Adds and subtracts with sets of objects smaller than three. SPATIAL 3.2.2 Enquires about the world around them. * 3.1.7 Describes position and order of VOCABULARY objects using position words such as in front of, behind, up, down, under, inside, outside, between and next 3.2.5 Understand location of objects, areas at school and home. * BACKWARD 4.1.4 Sings a song or a poem from memory. * 5.6 Understands questions and answers DIGIT SPAN actively.* FORWARD DIGIT 4.1.4 Sings a song or a poem from memory.* 5.6 Understands questions and answers SPAN actively.* HEAD TOES 2.3 Enjoys doing new things* 2.2 Agrees with rules and follows clear KNEES 4.1.6 Understands and follows instructions instructions SHOULDER TASK 5.3 Initiates games and activities with others 2.5 Uses play to explore, practices and (children and adults) understands social roles. * 4.1.6 Demonstrates an understanding of non-verbal cues. * 5.5 Plays show and tell games* EMPATHY 2.1 Behaves in an outgoing and friendly; 2.4 Expresses needs and feelings PERSPECTIVE overenthusiastic way at times. * appropriately and is sensitive to the TAKING 2.6 Cooperates with other children. needs and feelings of others. 2.6 Appreciates love, care, friendship. Tolerance, kindness, and respect for others. ENGLISH LETTER 4.3.1 Knows that text carries meaning and is 4.1.3. Knows the sounds of vowels a, e, IDENTIFICATION written from left to right (English) i, o, and u and consonants. 50 Appendix C: Proportion of Segments Meeting Minimum Quality Levels on Teach ECE Behaviors 51 Appendix D. Regression Tables Panel A: AIM-ECD Domain Z-scores Predicted by Teach ECE Overall Z-scores (1) (2) (3) (4) AIM-ECD AIM-ECD AIM-ECD AIM-ECD Early Literacy Early Numeracy Executive Social- Z-score Z-score Functioning Z- Emotional Z- score score Teach ECE Overall Z-score 0.0656*** 0.0766*** 0.0704*** 0.0278* (0.0139) (0.0141) (0.0137) (0.0132) Child's age 0.193*** 0.205*** 0.144*** 0.0923*** (0.0125) (0.0130) (0.0132) (0.0153) Child is female 0.0729*** -0.0251 -0.0137 -0.0219 (0.0197) (0.0204) (0.0206) (0.0235) Multi-graded class 0.0810* 0.0996** 0.0793* 0.0368 (0.0375) (0.0380) (0.0365) (0.0345) Grade 1 class 0.851*** 0.829*** 0.644*** 0.286*** (0.0374) (0.0379) (0.0367) (0.0359) Multi-graded X Grade 1 -0.547*** -0.552*** -0.465*** -0.200*** (0.0542) (0.0549) (0.0529) (0.0505) Teacher has teaching -0.0297 -0.0563 0.00395 0.000347 certificate or ed. degree (0.0427) (0.0433) (0.0417) (0.0396) Teacher has 3+ years pre- -0.0112 0.00568 -0.00588 -0.0245 primary experience (0.0330) (0.0334) (0.0321) (0.0304) Teacher is female 0.0653* 0.0402 0.0705* 0.0197 (0.0290) (0.0295) (0.0285) (0.0275) Saraiki language class 0.0112 -0.0317 0.00148 -0.0590 (0.0479) (0.0483) (0.0469) (0.0441) Other language class -0.115~ -0.128~ -0.141* -0.103 (0.0689) (0.0698) (0.0674) (0.0641) Constant -1.355*** -1.318*** -0.981*** -0.551*** (0.105) (0.0999) (0.112) (0.101) N 8249 8249 8249 8249 Var u_k (Enum) 0.136 0.0883 0.181 0.0493 Var u_j (Class) 0.187 0.185 0.155 0.0644 Var e_ijk (Residual) 0.531 0.577 0.603 0.892 Standard errors in parentheses ~ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001 52 Panel B: AIM-ECD Overall Z-scores Predicted by Teach ECE Area Z-scores (1) (2) (3) (4) AIM-ECD AIM-ECD AIM-ECD AIM-ECD Overall Overall Overall Overall Teach ECE Time on Task Z- 0.0867*** score (0.0151) Teach ECE Classroom 0.0709*** Culture Z-score (0.0148) Teach ECE Guided Learning 0.0659*** Z-score (0.0141) Teach ECE Socioemotional 0.0652*** Skills Z-score (0.0140) Child's age 0.196*** 0.197*** 0.197*** 0.197*** (0.0129) (0.0129) (0.0129) (0.0129) Child is female 0.00363 0.00455 0.00513 0.00511 (0.0202) (0.0203) (0.0203) (0.0203) Multi-graded class 0.0839* 0.0953* 0.0937* 0.0971** (0.0376) (0.0376) (0.0376) (0.0376) Grade 1 class 0.794*** 0.816*** 0.813*** 0.820*** (0.0378) (0.0376) (0.0376) (0.0376) Multi-graded X Grade 1 -0.531*** -0.548*** -0.548*** -0.552*** (0.0545) (0.0544) (0.0544) (0.0544) Teacher has teaching -0.0263 -0.0225 -0.0142 -0.0175 certificate or ed. degree (0.0427) (0.0429) (0.0427) (0.0428) Teacher has 3+ years pre- 0.00421 -0.0105 -0.0115 -0.0159 primary experience (0.0331) (0.0331) (0.0331) (0.0331) Teacher is female 0.0654* 0.0629* 0.0613* 0.0663* (0.0290) (0.0292) (0.0292) (0.0291) Saraiki language class -0.0384 -0.0293 -0.0255 -0.0211 (0.0480) (0.0480) (0.0480) (0.0480) Other language class -0.175* -0.158* -0.161* -0.153* (0.0691) (0.0692) (0.0692) (0.0692) Constant -1.294*** -1.317*** -1.324*** -1.332*** (0.102) (0.103) (0.103) (0.102) N 8249 8249 8249 8249 Var u_k (Enum) 0.106 0.112 0.115 0.110 Var u_j (Class) 0.179 0.180 0.180 0.180 Var e_ijk (Residual) 0.571 0.571 0.571 0.571 Standard errors in parentheses ~ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001 53