EDUCATION WORKING PAPER No. 10 | February 2024 Validating Teaching Observation Tools: A Content-Based Approach for PLAY and Teach Primary Diego Luna Bazaldua and Emma Carter © 2025 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Email: AskEd@worldbank.org Internet: www.worldbank.org/en/topic/education This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy, completeness, or currency of the data included in this work and does not assume responsibility for any errors, omissions, or discrepancies in the information, or liability with respect to the use of or failure to use the information, methods, processes, or conclusions set forth. The boundaries, colors, denominations, links/footnotes and other information shown in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. The citation of works authored by others does not mean the World Bank endorses the views expressed by those authors or the content of their works. Nothing herein shall constitute or be construed or considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Cover design: Marianne Siblini Validating Teaching Observation Tools: A Content-Based Approach for PLAY and Teach Primary Diego Luna Bazaldua and Emma Carter Table of Contents Abstract ......................................................................................................................................... 3 Key words ...................................................................................................................................... 3 Acknowledgements ....................................................................................................................... 4 Executive Summary ....................................................................................................................... 5 1. Study objective ...................................................................................................................... 6 2. Description of observation tools ........................................................................................... 6 3. Method ............................................................................................................................... 10 4. Results ................................................................................................................................. 12 a. Content validity results: Content matching of Teach Primary behaviors and PLAY items ............. 12 b. Concurrent validity results: Relationship among Teach Primary and PLAY scores......................... 19 c. Construct validity results for PLAY: Internal structure for the PLAY items ..................................... 19 5. Conclusion ........................................................................................................................... 22 References .................................................................................................................................. 23 Annexes ...................................................................................................................................... 25 Annex A: PLAY Constructs of support for children’s engagement in learning ........................................ 25 Annex B: Table comparing the main features of the Teach Primary classroom observation tool and PLAY toolkit ............................................................................................................................................. 30 Annex C: Overview of experts ................................................................................................................ 31 Annex D: Full mapping results for Phase 1 ............................................................................................. 32 Annex E: Mapping results for Phase 2 .................................................................................................... 47 Annex F: Correlation of Teach Primary and PLAY scores ........................................................................ 52 Annex G: Correlation of PLAY items, item-total correlations and visual representations of internal structure ................................................................................................................................................. 53 Annex H: Exploratory Factor Analysis solution ....................................................................................... 58 Abstract This study compares two tools used to observe and assess how teachers support student learning: the LEGO Foundation’s Engage tool, formerly known as PLAY, and the World Bank’s Teach Primary tool. The goal was to evaluate their similarities and overlap in their measurement of teaching practices. A group of nine experts reviewed both tools in detail, carefully matching items from PLAY to those in Teach Primary. Results indicate that about one-third of the items closely matched, showing that PLAY is built on a solid framework to capture engaging teaching practices. Another third of the items did not have a clear match, meaning that each tool separately captures unique aspects of teaching that the other does not. The study also looked at how well the two tools’ scores align using real classroom data from Ethiopia, Peru, and Sierra Leone. The results showed a small positive relationship between the two tools, meaning they measure related but different aspects of teaching. The study also found that the way PLAY is structured may need some adjustments to better reflect the skills it is designed to measure. By combining expert opinions with data analysis, this study provides strong evidence that both tools are valuable but serve different purposes. The findings highlight the importance of using multiple approaches to assess teaching quality. The study also offers recommendations for improving tool improvement and suggests next steps on how classroom observation tools can help support improved teaching and learning worldwide. JEL: I25 - Education and Economic Development Key words Classroom observations, teaching quality, engaging teaching, socio-emotional learning, teaching practices, content validity, construct validity. Acknowledgements This document was developed by Diego Luna Bazaldua (Senior Education Specialist; HEDGE), Emma Carter (Consultant; HEDGE). The team worked under the overall guidance of Halil Dundar (Practice Manager, HEDGE). Laura Gregory (Senior Education Specialist; HEDGE), Sergio Venegas (Economist, HEDGE) and Maryam Akmal (Economist, HEDGE) provided overall support in the different stages of this study. Peer reviewers include Ali Hasan Ansari (Senior Economist, HAWE3), Huma Kidwai (Senior Education Specialist, HAEE1), Ezequiel Molina (Senior Economist, HLCED), and Lauri Pynnonen (Senior Education Specialist, HEAED). Additional valuable inputs were received from Amer Hasan and Harshil Kumar Sahai, and other members of the Education Global Practice who participated in discussions on the paper. The next list of international experts contributed to content reviews of the tools: Abbie Raikes, Carolina Melo Hurtado, Carolina Moreira Vasquez, Dawit Tiruneh, Elizabeth Hentschel, Jonathan Seiden, Lenka Janik Blaskova, Rabea Malik, and Tadesse Teferi. This work is sponsored by the Foundational Learning Compact (FLC) Trust Fund. Executive Summary Engaging teaching has emerged as a key approach to fostering children’s socio-emotional and cognitive skills. Because of the importance of embedded engaging activities in the teaching and learning process, LEGO Foundation and the World Bank have collaborated to advance a more evidence- based approach to enhancing holistic teaching and learning. These efforts emphasize practices that focus on engagement-based and socio-emotional learning. This report presents findings from a significant component of the partnership, focusing on comparing the LEGO Foundation's Engage, formerly known Playful Learning Across the Years (PLAY) 1, classroom observation tool with the World Bank’s Teach Primary classroom observation instrument. The goal was to assess the extent of overlap between these tools, both qualitatively and quantitatively, providing content and concurrent validity evidence for the newly designed PLAY instrument. This study also aimed to enhance understanding of the technical properties of both tools, emphasizing synergies and distinctions. Methodologically, the qualitative aspect of the study followed the Delphi method, involving nine experts from diverse international backgrounds. They conducted two phases of content review. Phase 1 included a detailed content comparison and item-to-item matching, with rationales provided for matching decisions made. Phase 2 focused on reviewing ambiguous cases—items that did not clearly match on Phase 1. In addition, quantitative analysis assessed the correlation between Teach Primary and PLAY aggregate scores to evaluate concurrent validity, using data from Ethiopia, Peru, and Sierra Leone. Key qualitative results showed that over a third of the items in the PLAY tool strongly corresponded with behavioral items from the Teach Primary instrument, providing evidence of the empirical basis of the former. Additionally, the study identified a similar proportion of items with minimal or no correspondence to Teach Primary behaviors, highlighting the PLAY tool's unique ability to capture other aspects of teaching conducive to quality engagement-based instruction and learning. Quantitative findings included a total score correlation of 0.15 between the Teach Primary and PLAY instruments, supporting concurrent validity and reinforcing the tools’ measurement of separate but related constructs of quality teacher practice and engagement-based teaching and learning, respectively. Taken together, these findings revealed that combining qualitative learnings from the Delphi method with robust quantitative validation results creates a comprehensive validation framework, strengthening confidence in these classroom observation tools to measure quality engagement-based and socio-emotional learning. In addition to the analytical findings, valuable lessons were learned from the application of the Delphi technique, which could improve its effectiveness for consensus-building and decision-making in future research. Among these were the provision of sufficient time for tasks to facilitate data accuracy and the importance of consistent communication with experts. Finally, clear incentives along with streamlined processes for labor-intensive exercises were recommended as helpful strategies for enhancing participants’ engagement. 1 The name PLAY is used throughout this paper as it corresponds to the name used in publicly available materials at the time of writing. The authors acknowledge that the tool is now called Engage. 1. Study objective The LEGO Foundation and the World Bank have collaborated to support education systems in adopting a more evidence-informed approach to improve holistic teaching and learning, particularly through engagement-based and socio-emotional teaching practices. A crucial aspect of this objective is enhancing the measurement of these practices, especially within low- and middle-income countries (LMICs). One activity included in this collaboration involved a detailed content comparison of the LEGO Foundation's Playful Learning Across the Years (PLAY) and the World Bank's Teach Primary classroom observation tools. This comparison aimed to understand the extent of content and score overlap, as well as their connection with students' learning experiences and outcomes. Additionally, the exercise focused on identifying important content distinctions between the instruments and how they may provide complementary information to users. This comparison has also served as a form of content validity evidence and facilitates subsequent quantitative analysis aimed at understanding and comparing the technical properties of the tools. For instance, this content matching analysis highlights the degree of concordance between items, subscales, and underlying latent constructs measured by PLAY and Teach Primary. It also enhances our understanding of the association between constructs indicating the quality of engagement-based teaching practices and those reflecting general teaching practices and teacher-student interactions. This can be achieved by implementing a similar methodology to that used by the World Bank’s Teach team in relation to inclusive teaching classroom practices (see Molina et al., 2024), examining the relationship between the underlying construct—engaging teaching practices—and Teach Primary behaviors. This approach enables mutual learning and increases awareness of the synergies between the PLAY and Teach Primary measurement approaches, facilitating measurement scale-up and promoting engagement-based and socio-emotional learning in the classroom. This paper is structured as follows: Section 1 begins with a description of the LEGO Foundation’s PLAY and the World Bank’s Teach Primary classroom observation tools. Section 2 details the study’s methodological approach, including the study’s design, profiles of experts involved, and analytical methods. Section 3 provides an overview of the results from the two key phases of this study. Finally, Section 4 presents key conclusions and discusses the limitations related to the method used. 2. Description of observation tools PLAY Classroom Observation Tool The LEGO Foundation has collaborated with RTI International and New York University (NYU) Global TIES for Children to develop a culturally responsive toolkit that includes classroom observation and survey measurement tools focused on capturing the quality of engaging teaching and learning in classrooms. The PLAY toolkit aims to measure how adults support children's engagement in learning, which leads to a broad range of learning outcomes. PLAY includes tools for multiple age groups across different settings. For students aged 6 to 12, the measurement tools are designed specifically for classroom use. These tools include a classroom inventory, teacher-child observation, a teacher survey, and a child survey. The teacher-child classroom observation tool, which is the focus of the comparison task described in the next section of this document, encompasses four key dimensions essential for supporting children's engagement in learning. These dimensions, also referred to as constructs, are summarized below and detailed further in Annex A. • Support for exploration: Adult support for children's interaction with things and ideas to expand thought. • Support for agency: Adult support for children's ability to influence how and what they learn. • Support for personal and social connection: Adult support for (1) relating learning to children's personal experiences, (2) children learning through social interaction, and/or (3) children feeling a sense of belonging and being socially connected. • Support for emotional climate: Adults foster an environment where interactions between adult(s), child(ren), and peers are warm, respectful, and positive. These four constructs are captured in 25 items within the tool. Each item is scored on a 3-point scale, measuring the frequency and quality of teacher and student behaviors, as well as participation in the classroom. The World Bank's Teach Primary (2nd Edition) Classroom Observation Tool The Teach Primary tool captures teaching practices that support quality learning and nurture children's cognitive and socio-emotional skills. Initially launched in 2019 after a rigorous development and validation process, Teach Primary underwent a revision in 2020-2021 to enhance its measurement of inclusive teaching practices that are responsive and facilitate whole-child development. Similar to PLAY, the Teach Primary tool is designed for classrooms with students aged 6 to 12. The Teach Primary observation tool assesses two main aspects: (i) the time teachers spend on learning and the extent to which students remain on task, and (ii) the quality of teaching practices that develop students' socio-emotional and cognitive skills. The Time on Task component uses three "snapshots" of 1– 10 seconds to record the teacher's actions and the number of students who are on task. Observers note whether the teacher provides a learning activity for most students. If so, they scan the classroom to determine if students are on task. Off-task behavior is rated as follows: 0-1 students being off-task (High); 2-5 students being off-task (Medium); 6 or more students being off-task (Low). The Quality of Teaching Practices component is organized into three primary areas: 1. Classroom Culture: The teacher creates a culture conducive to learning. This focuses on the extent to which the teacher: (i) creates a supportive learning environment by treating all students respectfully, consistently using positive language, responding to students' needs, and both challenging stereotypes and not exhibiting bias in the classroom. (ii) establishes positive behavioral expectations by setting clear behavioral expectations, acknowledging positive student behavior, and effectively redirecting misbehavior. 2. Instruction: The teacher facilitates lessons that deepen understanding and encourage critical thinking and analysis. This focuses on the extent to which the teacher: (i) facilitates the lesson by explicitly articulating lesson objectives that are aligned to the learning activity, explaining content using multiple forms of representation, and connecting the learning activity to other content knowledge or students' daily lives, and by modeling the learning activity through enacting or thinking aloud. (ii) does not simply move from one topic to the next but checks for understanding by using questions, prompts, or other strategies to determine students' level of understanding, by monitoring students during group and independent work, and by adjusting his/her teaching to the level of students. (iii) gives feedback by providing specific comments or prompts to help clarify students' misunderstandings or identify their successes. (iv) encourages students to think critically by asking open-ended questions and providing them with thinking tasks that require them to analyze content actively. Students exhibit critical thinking ability by asking open-ended questions or performing thinking tasks. 3. Socioemotional Skills: The teacher fosters socio-emotional skills to help students succeed inside and outside the classroom. This focuses on the extent to which the teacher: (i) instills autonomy by providing students with opportunities to make choices and take on meaningful roles in the classroom. Students exhibit their autonomy by volunteering to participate in classroom activities. (ii) promotes perseverance by acknowledging students' efforts rather than focusing solely on their intelligence or natural abilities, having a positive attitude toward students' challenges, framing failure and frustrations as part of the learning process, and encouraging students to set short- and long-term goals. (iii) fosters social and collaborative skills by encouraging collaboration through peer interaction and by promoting interpersonal skills, such as perspective-taking, empathizing, emotion regulation, and social problem-solving. Students exhibit social and collaborative skills by collaborating with one another through peer interaction. These three areas encompass nine elements, summarized above, captured in 28 items linked to specific teaching behaviors. Based on the quality of observed teaching practices captured in observed behaviors, these items are rated on a rubric as low, medium, or high. The scores are then translated into a 5-point scale derived from two 15-minute lesson observations of the same teacher. Further details about each behavior and element can be found in the Teach Primary manual (Molina et al., 2021). Tool adaptation Teach Primary Observation Tool The Teach Primary tool was originally developed in English but has been adapted and implemented across various linguistic and cultural contexts to ensure its applicability. For Spanish-speaking countries, the tool has been translated into Spanish and used extensively, requiring minimal content adaptation due to its broad alignment with teaching practices in these regions. In African countries, where multiple official languages of instruction coexist, Teach has been translated into several local languages to ensure accessibility for enumerators and relevance for diverse classroom settings, but overall the implementation relied on English in Sierra Leone and Ethiopia. These adaptations of Teach Primary have focused primarily on linguistic translation while maintaining fidelity to the original constructs, ensuring the tool's validity across different contexts and is understood by enumerators. PLAY Classroom Observation Tool The PLAY tool, developed in English, required translation into Spanish for its implementation in Peru. This translation facilitated the scoring of teachers in Spanish-speaking classrooms while maintaining the theoretical constructs and scoring framework of the original tool. While no additional adaptations were made for other countries included in the study, the flexibility of the PLAY tool allows for further customization should future contexts require it. Enumerator training Training enumerators in classroom observation tools is essential for ensuring that data collected is accurate, consistent, and reliable. It helps enumerators understand how to use the tool correctly, observe and record behaviors objectively, and avoid personal biases. Proper training also ensures ethical observations, protecting the rights of students and teachers, and helps enumerators adapt to different classroom settings. By aligning all enumerators on how to apply the tool, the quality of the data improves, making it more useful for decision-making and creating better insights into classroom practices. Teach Primary Observation Tool Enumerator training for the Teach Primary tool follows a structured and rigorous process facilitated by certified master trainers. Publicly available resources, including implementation guidelines and training materials, provide a comprehensive framework for preparing enumerators in the Teach Primary tool administration and scoring. Training typically spans several days and involves multiple components: • Conceptual Understanding: Trainees are guided through the theoretical underpinnings of the Teach tool, including its constructs and scoring framework. • Practical Exposure: Enumerators engage with context-relevant videos of teachers in classrooms, gaining practical experience in scoring classroom behaviors. • Skill Practice: Trainees practice administering and scoring Teach Primary to build confidence and accuracy. • Certification: Enumerators must pass a certification exam to demonstrate mastery of the tool. This includes achieving a high level of agreement in their ratings of standardized videos, ensuring reliability and readiness for field data collection. This rigorous approach ensures consistency and reliability in data collection across diverse implementation contexts. PLAY Classroom Observation Tool Training for the PLAY tool was designed to build capacity among enumerators through a cascading training model. A certified master trainer conducted a comprehensive training session for a select group of trainers, who then replicated the training with enumerators. Key components of the PLAY training included: • Orientation: Enumerators were introduced to the tool’s constructs and the methods for observing and scoring engaging teaching and learning. • Practice Opportunities: Sessions included opportunities to practice scoring based on observation scenarios and examples. Unlike Teach, the PLAY training did not require enumerators to reach a predetermined level of inter-rater agreement for certification. However, the training ensured that enumerators were familiar with the tool and confident in their ability to apply it during data collection. 3. Method Study description This study compared the item content of the PLAY and Teach Primary classroom observation tools. Specifically, it involved mapping each of the 25 items from PLAY to one or more of the 30 items from Teach Primary to determine the extent of overlap and difference between the instruments. A table outlining the main features of these tools is provided as Annex B. Participants in this study comprised a group of experts with backgrounds in the development and/or use of classroom observation measures, particularly within low- and middle-income country contexts and across multiple world regions. Initially, 15 experts were identified for potential involvement. After being contacted via email and provided with a brief overview of the study, nine consented to participate. Following this confirmation, participants received a more detailed outline of the task and Terms of Reference concerning their consultancy role. Participants engaged in content review and comparison of the two tools, item-to-item matching tasks, and participated in any follow-up discussions or tasks. An overview of the experts is provided as Annex C. • The Delphi method for content validity This study strongly drew upon the principles of the Delphi method. The Delphi method, utilized in multiple fields, including education, health, and psychology, allows experts to provide judgments on a topic, which can then be modified or refined based on reactions to the collective views of the group (Green, 2014; Hyatt et al., 2024; Manizade and Mason, 2010). Unlike face-to-face techniques, the Delphi method promotes independence of thought and allows researchers to leverage the expertise of numerous experts from different locations while fostering an environment that encourages group acceptance and shared responsibility for the outcome. The Delphi method typically involves several key characteristics for obtaining group consensus on a topic, including (1) participant anonymity, (2) anonymity of responses, (3) multiple and structured iterations of consultations, (4) group consensus, (5) controlled feedback of responses to participants, and (6) statistical analysis of data (Granta and Kinney, 1992, pp. 13-14). Studies utilizing the Delphi method have been conducted with as few as seven and as many as 1,000 participants, though samples between 10 and 50 are typically advised due to the extensive data generated and the analyses required for each individual response (Iqbal and Pipon-Young, 2009). The Delphi method has emerged as a popular approach for determining content validity (Green, 2014; Manizade and Mason, 2010). Content validity denotes the extent to which the items of a measurement instrument are relevant to and representative of the targeted construct it is intended to measure. It considers the importance, relevance, and clarity of instrument items, domains, and definitions, as well as the appropriateness and adequacy of item response scales (Roebianto et al., 2023). Content validity is particularly relevant as it increases the accuracy of measuring constructs. For example, in classroom observation tools, content validity ensures that all developed items cover aspects of classroom practice that learning theories define as relevant for improving student learning. Additionally, content validity helps reduce measurement bias by carefully selecting and reviewing items to cover all relevant aspects of a construct. There are many ways to establish content validity for classroom observation measures. The current study is particularly relevant to provide evidence on whether the instrument has an empirical foundation reflecting key practices and processes of other validated observational methods designed to capture teaching quality within similar contexts (Molina et al., 2020; Seidman et al., 2018). Equally important is the ability to capture a teaching quality dimension distinct from other measurement tools. Considering these aims underpinned by the Delphi method principles, this study involved two key phases, as outlined in detail below. • Phase 1 (Mapping of PLAY items to Teach Primary behaviors) Phase 1 involved participants carefully reviewing both the PLAY and Teach Primary tools. They then mapped each PLAY item to the Teach Primary behavior with the closest level of agreement on a scale of 0-4, with 0 indicating no perceived agreement and 4 indicating near identical agreement. Participants could select multiple options if a single PLAY item corresponded to more than one Teach Primary behavior. Participants were also encouraged to provide a rationale for their placement decisions and explain any ambiguities they encountered. After this step, participants repeated the process in reverse by mapping Teach Primary behaviors to PLAY items. They were advised to complete this task within one day and submit their results within two weeks of receiving the materials. • Phase 2 (Review of cases of ambiguity) Following a careful review of Phase 1 results, clear matches and non-matches of items and behaviors were determined, along with examples where ambiguity was present. The next section describes in detail how matches, non-matches, and ambiguity cases are defined. For cases of ambiguity in Phase 2, participants were contacted again and asked to review an anonymized overview of the mapping analysis, specifically focusing on items requiring follow-up revisions. Participants received individual feedback to reconsider their initial mapping decisions in light of the overall results. They were allowed to adjust their previous decisions or keep them unchanged and were allotted one week to complete this task. • Analytical approach For Phase 1, results and rationales were compiled into an Excel sheet for review. Only items reflecting participants’ highest level of matching were considered for consistency. Some participants were contacted to clarify inconsistencies (for instance, where written rationales did not reflect rank placements) and to provide responses for missing data (for instance, where rankings were missed for certain items). Results for each item, including the number of raters indicating agreement and their ranking levels for each mapped example, were summarized using Excel. Averages were calculated for the number of participants in agreement and the entire sample (n=9). After a joint review and discussion with the Teach team, conditions for matches, non-matches, and items requiring follow-up were summarized as follows: • Matches: Items with strong agreement among experts in both directions, indicated by an aggregated score over 1.5 on average across the sample and over 50 percent of the sample in agreement. • Non-matches: Items with strong variation in expert responses or predominantly low-level rankings, reflecting weak agreement or non-matches. • Items requiring follow-up: Items with inconsistencies in expert agreement depending on the direction of mapping or results close to the thresholds for matches. For Phase 2, summarized results for items requiring follow-up were adjusted based on participants' indicated changes. As in Phase 1, participants were contacted to resolve inconsistencies between mapping decisions and rationales. The same criteria used to determine matches and non-matches in Phase 1 were applied to the revised results. 4. Results a. Content validity results: Content matching of Teach Primary behaviors and PLAY items • Phase 1 results Phase 1 revealed nine clear matches between the items of the PLAY and Teach Primary classroom observation tools using the above criteria (see Annex D for the full results and Table 1 for a summary). For instance, all experts considered that the PLAY item COX1: "Teacher connects concepts in the lesson to students’ backgrounds or life outside the classroom" and Teach Primary behavior 3.3: "The teacher makes connections in the lesson that relate to other content knowledge or students’ daily lives" had a near identical or strong match. This pattern was also reflected in the reverse mapping process, where eight participants indicated a comparable level of concordance (for detailed information, see Example A). Conversely, we found six clear non-matches for items from the PLAY instrument and 12 from the Teach Primary tool in the review of the summarized data (see Annex C for the full results and Table 2 for a summary). For instance, for PLAY item COE3: "Teacher includes students who did not volunteer to answer," five participants indicated no matches from Teach Primary. While four participants noted matches for four different items from the Teach Primary tool, all levels of agreement were marked as either ‘some agreement’ or ‘weak agreement.’ Similarly, for the Teach Primary item 0.1: "Teacher provides learning activity to most students," eight experts indicated no matches from the PLAY tool, and three participants noted matches for three different items, albeit at low levels of the scale (for detailed information, see Example B). Lastly, we identified nine PLAY items and nine Teach Primary items that necessitated follow-up from participants (see Annex C for the full results and Table 3 for a summary). This included PLAY item COA4: "Students choose who plays each role" and Teach Primary item 7.2: "The teacher provides students with opportunities to take on roles in the classroom." While these items were well matched when mapping Teach Primary to PLAY, as demonstrated by seven participants indicating a match with an overall average across the full sample of 1.56, the reverse mapping was not as strong. Only four participants noted a match, with an overall average of 1.3, a difference that warranted follow-up (for detailed information, see Example C). Table 1: Matched Items (After Phase 1 review) PLAY Teach Primary COX1: Teacher connects concepts in the lesson to ↔ 3.3 The teacher makes connections in the lesson that students' backgrounds, or life outside the classroom. relate to other content knowledge or students’ daily lives. COX3: Teacher promotes students thinking by ↔ 6.2 The teacher provides thinking tasks. themselves by asking a comparison, categorization, or prediction question or through a task they give them. COX4: Teacher gives hints or suggestions to encourage ↔ 5.1 The teacher provides specific comments or prompts students to continue to get to the answer or explore that help clarify students’ misunderstandings. the concept. COX6: Teacher uses multiple methods to help students ↔ 3.2 The teacher explains content using multiple forms of learn about a concept. representation. COA5: Students decide the WHAT or the HOW to do an ↔ 7.1 The teacher provides students with choices. academic task. COC1: Teacher has students work together on an ↔ 9.1 The teacher promotes students’ collaboration instructional activity towards a common instructional through peer interaction. goal or understanding. COE2: Teacher responds to students' emotional needs. ↔ 1.3 The teacher responds to students' needs. COE6: Teacher is inclusive of children with diverse ↔ 1.4 The teacher does not exhibit bias and challenges backgrounds and learning needs through meaningful stereotypes in the classroom. interactions, such as individualized attention. COE7: The teacher explains a student's actions, ↔ 9.2 The teacher promotes students' interpersonal skills. intentions, and/or feelings to other students. Total matched PLAY items = 9 Total matched Teach Primary items = 9 Table 2: Non-matched items (after Round 1 Review) PLAY Teach Primary COX2: Teacher gives student(s) a chance to try or 0.1 Teacher provides learning activity to most students. explore something first before being shown how to use/answer it. COA1: Students try different solutions (iteration). 0.2 Students are on task. COA6: Teacher uses student ideas or examples in 1.1 The teacher treats all students respectfully. instruction. COC5: Teacher sets up a task that involves lively social 1.2 The teacher uses positive language with students. interaction between students COE1: Teacher uses a mode of instruction that is 3.1 The teacher explicitly articulates the objectives of the explicitly upbeat. lesson and relates classroom activities to the objectives. COE3: Teacher includes students who did not volunteer 3.4 The teacher models by enacting or thinking aloud. to answer. Total non-matched PLAY items = 6 4.2 The teacher monitors most students during independent/group work. 4.3 The teacher adjusts teaching to the level of the students. 6.3 The students ask open-questions or perform thinking tasks. 7.3 The students volunteer to participate in the classroom. 8.3 The teacher encourages goal-setting. 9.3 Students collaborate with one another through peer interaction. Total non-matched Teach Primary items =9 Example A: Matched items. PLAY COX1 to Teach Primary 3.3 from Phase 1 analysis In this example, there is a very strong level of agreement in both directions of mapping, with thresholds for matches exceeded in both cases. Of the nine experts, three indicated a near identical match between PLAY item COX1 and Teach Primary behavior 3.3 and provided the highest matching score of 4 points. Six experts considered the match to be strong but not identical and provided the second-highest matching score of 3 points. This resulted in an average level of agreement of 3.33 among all raters in the sample. This strong agreement was confirmed when experts matched Teach Primary behavior 3.3 with PLAY item COX1. In this case, four participants indicated a near-identical level of agreement with a matching score of 4 points, and four noted a strong level of agreement with a matching score of 3 points, resulting in an average agreement of 3.11 among all experts. PLAY item Mapped Teach Primary No. L.4 L.3 L.2 L.1 Av. agreement Av. agreement Status items Responses raters (1-4) all (1-4) (max 9) SUPPORT FOR EXPLORATION COX1: Teacher connects 3.3 The teacher makes 9 3 6 0 0 3.33 3.33 Match concepts in the lesson to connections in the lesson students' backgrounds, that relate to other content or life outside the knowledge or students’ classroom. daily lives Teach item Mapped PLAY items No. L.4 L.3 L.2 L.1 Av. agreement Av. agreement Status Responses raters (1-4) all (1-4) (max 9) AREA B: INSTRUCTION 3. Lesson facilitation 3.3 The teacher makes COX1 Teacher connects 8 4 4 0 0 3.5 3.11 Match connections in the concepts in the lesson to lesson that relate to students' backgrounds, or other content life outside the classroom knowledge or students’ COC3 Teacher invites 1 1 0 0 0 4 0.44 daily lives students to share their thoughts and/or personal experience about themselves, their families, or communities connected to the learning goal COC4 Teacher connects 2 0 2 0 0 3 0.67 student personal interests to learning activities to help them engage Example B: Non-matched items. PLAY COE3 and Teach Primary 0.1 In both cases, over half of the sample indicated "none," denoting no items from the other tool reflected similar content. All other items matched by one respondent only received low-level matching rankings with matching scores of 1 or 2 points. For example, the PLAY item COE3 was considered to have no match to any Teach Primary behavior by five of the experts. Three experts noted some agreement with Teach Primary behaviors 4.1, 7.2, and 7.3, while one expert found a weak level of agreement with Teach Primary behavior 1.1. Similarly, for the mapping of Teach Primary item 0.1, eight experts indicated no match to any PLAY item. One expert noted some agreement with PLAY item COC5, and two experts found a weak level of agreement with PLAY items COX6 and COX2. PLAY item Mapped Teach No. L.4 L.3 L.2 L.1 Av. Av. Status Primary items Responses agreement agreement (max 9) raters (1-4) all (1-4) SUPPORT FOR PERSONAL AND SOCIAL CONNECTION COE3: Teacher 4.1 The teacher uses 1 0 0 1 0 2 0.22 Non-match includes questions, prompts or students who other strategies to did not determine students’ volunteer to level of answer. understanding 7.2 The teacher 1 0 0 1 0 2 0.22 provides students with opportunities to take on roles in the classroom 7.3 The students 1 0 0 1 0 2 0.22 volunteer to participate in the classroom 1.1 The teacher treats 1 0 0 0 1 1 0.11 all students respectfully None 5 NA NA NA NA NA NA Teach Primary Mapped PLAY items No. L.4 L.3 L.2 L.1 Av. Av. Status item Responses agreement agreement (max 9) raters (1-4) all (1-4) 0. Time on Learning 0.1 Teacher COC5 Teacher sets 1 0 0 1 0 2 0.22 Non-match provides up a task that learning activity involves lively social to most students interaction between students COX6 Teacher uses 1 0 0 0 1 1 0.11 multiple methods to help students learn about a concept COX2 Teacher gives 1 0 0 0 1 1 0.11 student(s) a chance to try or explore something first before being shown how to use / answer it None 8 NA NA NA NA NA NA Example C: Items requiring follow-up. PLAY COA4 and Teach Primary 7.2 In this example, while the thresholds for matches are met when mapping from Teach Primary to PLAY, they are not met when mapping from PLAY to Teach Primary. Specifically, four experts indicated that PLAY item COA4 mapped to Teach Primary behavior 7.2, with one noting a nearly identical matching level, two noting a strong matching level, and one noting some level of agreement. However, the strength of agreement was higher for the mapping of Teach Primary behavior 7.2 to PLAY item COA4. Seven experts mapped the items, with one noting a near-identical matching level, four noting a strong matching level, and two noting some level of agreement. Due to this inconsistency, the team sent these examples to the experts for a second round of review, providing feedback on their scores and the overall results from Phase 1. PLAY item Mapped Teach No. L.4 L.3 L.2 L.1 Av. Av. Status Primary items Responses agreement agreement (max 9) raters (1-4) all (1-4) SUPPORT FOR AGENCY COA4: Students 7.2 The teacher 4 1 2 1 0 3 1.33 Follow-up choose WHO provides students with plays each role in opportunities to take an activity. on roles in the classroom 7.1 The teacher 3 0 1 2 0 2.33 0.78 provides students with choices 7.3 The students 1 0 0 1 0 2 0.22 volunteer to participate in the classroom 9.1 The teacher 1 0 0 0 1 1 0.11 promotes students’ collaboration through peer interaction Teach Primary Mapped PLAY items No. L.4 L.3 L.2 L.1 Av. Av. Status item Responses agreement agreement (max 9) raters (1-4) all (1-4) AREA C: SOCIOEMOTIONAL SKILLS 7: Autonomy 7.2 The teacher COA4 Students 7 1 1 2 3 2 1.56 Follow- provides students choose WHO plays up with opportunities each role in an to take on roles in activity the classroom COC2 Teacher 1 0 1 0 0 3 0.33 discusses or otherwise creates a sense of student/class togetherness None 1 NA NA NA NA NA NA • Phase 2 results For Phase 2, all experts were contacted once again to review ten PLAY and nine Teach Primary items that did not match those from the other tool in Phase 1. As seen in Annex E, only marginal changes in results were noted following the experts’ second review of the summarized mapping and results for the first matching round. Additionally, for some items, reviewers indicated no changes in their ratings from Phase 1 to Phase 2. These trends are illustrated in Example D, which displays the revised mapping for PLAY item COA4 and Teach Primary behavior 7.2. Overall, these results revealed no additional matches. Instead, ten non-matched items from PLAY and nine from Teach Primary were confirmed. Please refer to Table 4 for details these 19 items that found no match on the other tool. Example D: Results from Phase 2 followed-up items [PLAY COA4 and Teach Primary 7.2] PLAY item Mapped Teach No. L.4 L.3 L.2 L.1 Av. Av. Status Primaryitems Responses agreement agreement (max 9) raters (1-4) all (1-4) SUPPORT FOR AGENCY COA4: Students 7.2 The teacher 5 0 3 1 1 2.4 1.33 Non- choose WHO provides students match plays each role with opportunities to in an activity. take on roles in the classroom 7.1 The teacher 3 0 1 2 0 2.33 0.77 provides students with choices 9.1 The teacher 1 0 0 0 1 1 0.11 promotes students’ collaboration through peer interaction Teach Primary Mapped PLAY items No. L.4 L.3 L.2 L.1 Av. Av. Status item Responses agreement agreement (max 9) raters (1-4) all (1-4) AREA C: SOCIOEMOTIONAL SKILLS 7: Autonomy 7.2 The teacher COA4 Students 7 1 1 3 2 2.14 1.66 Non- provides students choose WHO plays match with each role in an opportunities to activity take on roles in COC2 Teacher 1 0 0 1 0 2 0.22 the classroom discusses or otherwise creates a sense of student/class togetherness None 1 NA NA NA NA NA NA Table 4: Non-matched items (After Phase 2 Review) PLAY Teach Primary COX5: Teacher asks questions to generate 2.1 The teacher sets clear behavioral expectations for explanations/reasons (e.g., for phenomena/ classroom activities. experiments/ results). COA2: Students practice a skill introduced by the 2.2 The teacher acknowledges positive student behavior. teacher. COA3 Students create something connected to the 2.3 The teacher redirects misbehavior and focuses on the current lesson. expected behavior, rather than the undesired behavior. COA4: Students choose WHO plays each role in an 4.1 The teacher uses questions, prompts or other activity. strategies to determine students’ level of understanding. COA7: Teacher asks open-ended questions or prompts 5.2 The teacher provides specific comments or prompts for students to share their opinions or preferences. that help identify students’ successes. COC2: Teacher discusses or otherwise creates a sense 6.1 The teacher asks open-ended questions. of student/class togetherness. COC3: Teacher invites students to share their thoughts 7.2 The teacher provides students with opportunities to and/or personal experience about themselves, their take on roles in the classroom. families, or communities connected to the learning goal. COC4: Teacher connects student personal interests to 8.1 The teacher acknowledges students' effort. learning activities to help them engage. COE4: Teacher makes students know that it is okay to 8.2 The teacher has a positive attitude towards students' participate even if their answer or contribution is wrong challenges. or different. They will not get punished or teased. COE5: Teacher reinforces rules or uses positive Total non-matched Teach Primary items = 9 discipline and rewards to support a positive emotional climate. Total non-matched PLAY items = 10 b. Concurrent validity results: Relationship among Teach Primary and PLAY scores The qualitative results presented in section 4.1 are complemented with quantitative evidence using the scored Teach Primary and PLAY data from Ethiopia, Peru and Sierra Leone. For more information on the data collection on these three countries and additional quantitative results, refer to the report produced by the World Bank for LEGO foundation. Concurrent validity refers to the degree to which the results of a measurement tool correlate with those of a related measure. It assesses whether aggregated scores from two different measurement tools administered at the same time yield similar results, indicating that they are measuring the same or similar constructs. High correlations provide evidence of good concurrent validity and can help validate new tools when an existing one is already recognized as a valid criterion for the same construct. In this case, the concurrent validity focuses on the correlation between Teach Primary and PLAY aggregated scores and can inform whether these two tools capture similar constructs linked with quality teaching practices. This section summarizes key correlational results, while Annex F includes the Pearson correlation matrix between Teach Primary and PLAY dimension-level scores and total scores. c. Construct validity results for PLAY: Internal structure for the PLAY items In addition to the evidence tied to the concurrent validity, it was also relevant to explore the construct validity of the PLAY classroom observation tool. In this context, construct validity refers to the degree to which individual items within a measurement tool fit the underlying constructs with the use of statistical and psychometric techniques. For instance, in the case of the PLAY classroom observation tool, it could be argued that the theory proposes four underlying constructs – Support for Exploration, Support for Agency, Support for Personal and Social Connection, and Support for Emotional Climate – so item scores measuring each should be show a stronger correlation among them, while items measuring separate constructs should exhibit smaller correlations or not be correlated at all (Beckman et al., 2005). A series of statistical evidence can provide information on construct validity by determining the dimensional structure of a measurement tool and determining its reliability, including statistics of internal consistence (for instance, item-total score correlations and Cronbach’s alpha coefficient), visual representations of item clusters (for instance, multidimensional scaling techniques), and multivariate analysis of latent variables (for instance, exploratory and confirmatory factor analysis models). Psychometric analyses linked to construct validity evidence are summarized in Annexes G and H. In terms of reliability, when the 25 PLAY items are included to calculate the Cronbach’s alpha coefficient, the reliability coefficient for this tool is equal to 0.79, which indicates a good level of internal consistency. In general, this result would indicate that nearly 80% of the total PLAY score variance is due to the accurate measurement of engaging practices and close to 20% of the score variance may be due to measurement errors. Annex G presents the correlations between PLAY items and the item-total score correlations. The average inter-item correlation was 0.25, indicating that the PLAY items seem to have moderate positive correlations and are internally consistent. It is important to note that while most item-level correlations were positive, item COE6 “Teacher is inclusive of children with diverse backgrounds and learning needs through meaningful interactions, such as individualized attention” showed a negative correlation with the total score and with many PLAY items; this inconsistency for item COE6 should be examined by subject-matter experts to understand why it is presenting suboptimal psychometric properties that negatively impact the reliability of the tool. If this item were to be dropped, the overall reliability of this classroom observation tool would go to 0.80. To a lesser extent, item COE5 “Teacher reinforces rules or uses positive discipline and rewards to support a positive emotional climate” also presented some level of inconsistency with a few other items part of PLAY, but it still contributes to the overall test reliability. Additional diagnostics on the polychoric correlation matrix were conducted to determine its suitability for factor analysis modelling. The Kaiser-Meyer-Olkin (KMO) statistic yielded an overall value of 0.103, indicating that the matrix is not suitable for factor analysis, which did not improve after excluding items COE6 and COE5. On the other side, the Bartlett sphericity test (χ2 = 28,546.24, df = 300, p-value < 0.001) was statistically significant, flagging that the correlation matrix was suitable for factor analysis. The contradictory results between KMO and Bartlett test statistics led to a visual exploration of the data using non-metric multidimensional scaling techniques (MDS) using the polychoric correlation matrix as the input to calculate a distance matrix. While the use of MDS as an internal structure analysis technique has decreased over time in the field of psychometrics, it is still useful to use item clustering techniques to explore the optimal number of latent variables underlying the data and identify items that are dissimilar with respect to the others. Figures 1 and 2 in Annex G include two different visual representations of similarity among PLAY items using hierarchical clustering and non-metric MDS techniques. Both figures show that there may be item clusters, one composed mainly of Support for Exploration items with some Support for Emotional Climate items, and the other with a mix of Support for Agency and Support for Personal and Social Connection items. Moreover, consistent with the Cronbach’s alpha results, these visual representations also indicate items COE5 and COE6 tend to cluster with each other, but not with the rest of the PLAY items. For the exploratory factor analysis, the first step was to determine visually the optimal number of latent factors using a scree plot (see Figure 3 in Annex G). The criteria to determine the number of factors were (1) that the eigen values were above 1 and (2) identify the number of factors before the point the curve changed drastically. These criteria led to the identification of a three-factor solution as the optimal one. Together, the three factors explained 47% of the total variance (F1 = 23% of variance, F2 = 14%, and F3 = 10%). All factors presented positive latent correlations (F1-F2 = 0.33; F1-F3 = 0.14, and F2-F3 = 0.29). Results for an exploratory factor analysis model with correlated latent factors using oblimin rotation is presented in Annex H. In addition, Annex H presents results for one-, two-, and four-factor solutions following the same analytical approach for the three-factor model. Using the standardized factor loadings as the criterion to guide the interpretation of the three-factor model, the first factor is composed mainly by Support for Exploration and Support for Emotional Climate items. The second factor comprises all Support for Personal and Social Connection items and most of the Support for Agency items. The third factor includes a residual mix of Agency, Exploration and Emotional Climate items. Items COA2 and COE6 did not show a high factor loading with any of the three factors, probably indicating something unique about the qualitative nature of these two items that makes them distinct from the rest. When looking at the results for the one-, two- and four-factor solutions, there is no clear item-by-factor clustering that is consistent with the framework behind PLAY, but it is worth noting that the one-factor solution indicates all items being linked to a single latent factor, except for COE5 and COE6. Overall, the exploratory factor analysis results are consistent with the MDS clustering representation of the PLAY items. Key results • The average observed correlation among the four PLAY dimensions is 0.46. The overall Cronbach’s alpha is 0.79, indicating a good level of reliability among the PLAY items. Item COE6 presents some inconsistencies with the rest of the PLAY items. • The average correlation among the three Teach Primary areas is 0.66. The overall Cronbach’s alpha is 0.97, indicating an excellent level of reliability among the Teach Primary items. No Teach Primary item shows internal inconsistency with the others or the total score. • The average correlation of Teach Primary and PLAY scores is 0.11, which is positive but not particularly strong in terms of effect size. • The maximum correlation between PLAY and Teach Primary was 0.16, corresponding to the PLAY total score and the Teach Primary Socioemotional dimension. This result is consistent with those from the content validity analysis given that PLAY emphasizes the measurement of engaging teaching practices that foster students’ socioemotional skills development. • The minimum correlation between PLAY and Teach Primary was -0.01, corresponding to PLAY’s agency support construct and Teach Primary’s Classroom Culture area. All other correlations between Teach Primary and PLAY scores are positive. This result is consistent with the content validity results since Support for Agency focuses mainly on students’ actions, while Teach Primary mainly focuses on teachers’ teaching practices. • The total score correlation for the two tools is 0.15, indicating evidence of convergent validity. The two classroom observation tools measure separate but related constructs of overall quality teaching practice and engagement-based teaching and learning, respectively. • Given that the MDS and exploratory factor analysis results produce outputs different from the four-factor theoretical framework suggested for the PLAY classroom observation tool, it is important to discuss and enrich these results with feedback from subject matter experts. Experts can help to determine whether revisions to the PLAY framework or the classroom tool content are needed to improve the description and measurement of the engaging teaching and learning dimensions the tool intends to measure. 5. Conclusion In conclusion, this study has highlighted the potential of the Delphi method in establishing the content validity of classroom observation measures. It revealed that over a third of the items in the LEGO Foundation’s PLAY tool showed a strong level of agreement with items from the World Bank’s Teach Primary tool. This finding offers evidence of the empirical underpinnings of the PLAY tool, given Teach Primary’s status as a rigorously tested and validated classroom observation measure applicable to LMICs (Molina et al., 2020). Additionally, the study identified a similar proportion of items with minimal to no correspondence to Teach Primary behaviors, highlighting the PLAY tool's unique ability to capture other aspects of teaching conducive to quality engaging instruction and learning. Moreover, the research identified items falling between these extremes, suggesting areas of both tools that could benefit from further refinement and clarification. Integrating quantitative studies can significantly enhance the validation of findings obtained through the Delphi method or other qualitative techniques. One approach could involve statistical analyses of data collected from classroom observations using the PLAY tool and established measurement tools like Teach Primary, such as the concurrent validity analysis here presented. This quantitative approach enabled a rigorous comparison of the constructs measured by each tool, providing additional evidence of their validity and reliability consistent with the qualitative results. Combining qualitative insights from the Delphi method with robust quantitative validation studies establishes a comprehensive validation framework, enhancing the reliability and utility of classroom observation tools in promoting quality instruction and learning. In addition to the challenges mentioned, several valuable lessons were learned from navigating the limitations of the Delphi technique in this study. Firstly, ensuring sufficient time for task completion emerged as crucial. Given the complex nature of the task and the need for careful consideration by experts, allocating ample time for reviewing and mapping items is essential to prevent rushed judgments that may compromise data accuracy. Furthermore, maintaining ongoing communication with experts proved instrumental in addressing issues related to delayed responses and ensuring continued engagement throughout the study. Regular updates, reminders, and follow-up communications can help keep experts informed and motivated, reducing the risk of sample attrition and ensuring a more comprehensive data set. Clear incentives were also identified as essential for maintaining experts' motivation and data quality. Incentives can take various forms, such as monetary compensation, acknowledgment of expertise, or opportunities for professional development. Providing tangible rewards or recognition can incentivize experts to invest time and effort in the task, enhancing the quality and reliability of the data collected. Moreover, streamlining the process and minimizing labor-intensive tasks can help alleviate participant burden and improve overall engagement during the matching tasks. Simplifying instructions, reducing unnecessary iterations, and optimizing data collection and analysis procedures can enhance efficiency without compromising the rigor of the study. Overall, addressing these lessons learned can contribute to the successful implementation of the Delphi method in future research endeavours, enhancing its effectiveness as an approach for consensus-building and decision-making in diverse contexts. Finally, embedding engaging teaching practices into education systems is a transformative step toward achieving holistic learning goals. Policymakers are urged to take bold, evidence-based actions to scale these practices, leveraging tools like PLAY and Teach Primary to gain information for continuous improvement and monitoring. By investing in engaging teaching, education systems can enhance classrooms to be more inclusive, engaging, and capable of nurturing every child's potential. The ripple effects of such investments go far beyond the classroom. They help build societies where individuals are equipped with the skills and confidence to contribute meaningfully to their communities. By fostering creativity, collaboration, and critical thinking from an early age, engaging teaching cultivates a foundation for lifelong learning that adapts to the demands of an ever-changing world. Furthermore, these practices help bridge gaps in educational equity, creating opportunities for all learners to thrive regardless of their socio- economic background. Policymakers have a unique opportunity to lead this change by prioritizing resources and strategies that place engaging teaching at the heart of education reform. Together, policymakers and development partners can shape a future where learning is not just a process but a joyful and empowering journey for every child. References Beckman, T. J., Cook, D. A., & Mandrekar, J. N. (2005). What is the validity evidence for assessments of clinical teaching?. Journal of general internal medicine, 20, 1159-1164. https://link.springer.com/article/10.1111/j.1525-1497.2005.0258.x Grant, J. S., & Kinney, M. R. (1992). Using the Delphi technique to examine the content validity of nursing diagnoses. International Journal of Nursing Terminologies and Classifications, 3(1), 12– 22. https://doi.org/10.1111/j.1744-618X.1992.tb00193.x Green, R. A. (2014). The Delphi technique in educational research. SAGE Open, 4(2), 215824401452977. https://doi.org/10.1177/2158244014529773 Hyatt, A., Gough, K., Chung, H., Wood, W., Aston, R., Cockwill, J., Galetakis, S., & Krishnasamy, M. (2024). Development of consensus quality indicators for cancer supportive care: A Delphi study and pilot testing. BMC Health Services Research, 24(1), 377. https://doi.org/10.1186/s12913-024-10876-6 Iqbal, S., & Pipon-Young., L. (2009) The Delphi method with a step-by-step guide. The Psychologist. Methods. British Psychological Society. Vol 22. no. 7. Manizade, A. G., & Mason, M. M. (2011). Using Delphi methodology to design assessments of teachers’ pedagogical content knowledge. Educational Studies in Mathematics, 76(2), 183–207. https://doi.org/10.1007/s10649-010-9276-z Molina, E., Carter, E., Luna Bazaldua, D., Pushparatnam, A., & Singal, N. (2024). Teaching for all? Measuring the quality of inclusive practices across eight countries. Compare: A Journal of Comparative and International Education, 1–19. https://doi.org/10.1080/03057925.2024.2304293 Molina, E., Fatima, S. F., Ho, A. D., Melo, C., Wilichowski, T. M., & Pushparatnam, A. (2020). Measuring the quality of teaching practices in primary schools: Assessing the validity of the Teach observation tool in Punjab, Pakistan. Teaching and Teacher Education, 96, 103171. https://doi.org/10.1016/j.tate.2020.103171 Molina, E., Pushparatnam, A., Melo Hurtado, C., Wilichowski, T., Del Toro Mijares, A., Ding, E., Aloys, J.B., Carter, E., Singal, N. (2021). Teach Primary: Observer Manual (English). Teach Primary Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/872291641201520569/Teach-Primary-Observer- Manual Roebianto, S., Aulia, S., & Mubarokah, L. (2023). Content validity: Definition and procedure of content validation in psychological research. Testing, Psychometrics, Methodology in Applied Psychology, 30(1), 5–18. https://doi.org/10.4473/TPM30.1.1 Seidman, E., Kim, S., Raza, M., Ishihara, M., & Halpin, P. F. (2018). Assessment of pedagogical practices and processes in low and middle income countries: Findings from secondary school classrooms in Uganda. Teaching and Teacher Education, 71, 283–296. https://doi.org/10.1016/j.tate.2017.12.017 Annexes Annex A: PLAY Constructs of support for children’s engagement in learning [From: Developing the “Playful Learning Across the Years” (PLAY) Toolkit (RTI International and NYU, 2023, pp.3-11). • Support for Exploration What is support for exploration? Exploring the physical and social environment is a basic developmental task. As children grow, the complexity of this exploration also grows—for example, an infant can explore a new material by manipulating it, mouthing it, or throwing it, while an older child may try to combine it with something else to represent an animal or an object. Why is it important and how is it linked to learning through play? A child who is exploring their environment is more likely to be voluntarily and intrinsically motivated, as they are learning something new about their surroundings. Exploration can take different forms depending on the characteristics of the environment the child has access to, how the child’s exploration of that environment is facilitated, and the developmental stage of the child. The environment in early childhood includes, for example, the range of objects and social interactions that are part of everyday life (both inside the home and outside the home), whether the child is with immediate family or exposed to a wide range of people and surroundings, and as an infant whether the child is on the floor with room to move or being held by caregivers. Each of these environments has a different set of objects to explore and, often, people to explore them with (whether adults or peers). In addition to providing children with access to a variety of interesting learning environments, the way in which their exploration is facilitated is also critical. As a caregiver, support for exploration could include encouraging children to explore an object in multiple ways. In childcare or preschool environments, support for exploration builds on similar principles. For example, children’s exploration of materials and phenomena in nature—such as sand, water, or plants—can be encouraged in ways that bring engagement (e.g., by asking questions about the properties or characteristics of a material like sand when wet versus dry). As children explore their environment with increasingly advanced cognitive skills over the course of childhood, they generate, test, and revise “hypotheses” iteratively about the natural and social worlds (Hirsh-Pasek et al., 2020). Physical environments can be designed to encourage exploration (e.g., engaging learning landscapes that have been integrated into public community space and foster different ways of using motor, language, and other skills to interact with features like shapes, inclines, steps, textures of walls or seats, and images embedded in them; see Schlesinger and Hirsh-Pasek, 2019). Support for exploration in the primary schooling context can acknowledge the integration of developmentally specific competencies of middle childhood and early adolescence (Del Giudice, 2014). Teachers may encourage children to extend their understanding of a concept by approaching it using different methods or perspectives. For example, teachers may encourage children to compare similar concepts, connect a concept to one they have learned previously, or explore a topic by expressing their ideas about it. They may also encourage students’ independent exploration of a concept or learning material, either solo or in peer groups, in ways that can build engagement. How might support for exploration vary across cultures? In communities where learning occurs by observing and pitching in (Rogoff, 2014), learning through exploration varies depending on the everyday community activity. For example, in a community where fishing is the primary livelihood, learning through exploration will occur in very different ways than in communities where land-based agriculture or small shops and businesses are the mai livelihood. In infancy, support for exploration may be experienced very differently in cultures where infants are more physically free compared to those where they are more restricted in their physical motion (Karasik et al., 2018). • Support for Agency What is support for agency? Agency refers to the extent to which children are able to “exert their thinking and actions in a social context in which others hold the same rights” (Zosh et al., 2017). Adair and colleagues define it as being able to influence and make decisions about what and how something is learned in order to expand capabilities (Adair, 2014; Adair and Sanchéz-Suzuki Colegrove, 2021). Having the choice of an activity or material or being able to choose peers with whom to interact in an activity are all examples of child agency in center or school settings. Similarly, in interacting with a caregiver, having choice of a material to play with or how to play with it can vary depending on the particular caregiver and child. In early caregiving, in addition to child choice in material or movement, support for agency might be reflected in following the child’s lead, rather than the adult leading an interaction. Why is it important and how is it linked to learning through play? Children with lower agency have fewer opportunities for creativity and collaboration in learning. For example, children’s level of agency is high at one end of the spectrum—free play—and may be lower in direct instruction, where children often have limited agency (for example, to initiate an interaction with a peer to share something they are learning, or to get up, move around, or even gesture freely). In settings that are more restricted, such as large pre- primary or primary classrooms, agency may be reflected in the degree of freedom of movement, even in a fairly constrained task such as lining up to write on the blackboard. Some freedom of how to do an activity like writing on the blackboard may result in more individualized learning and solicit more individual feedback from an adult or a peer. How might support for agency vary across cultures? Support for agency may be restricted in ways that reflect equity or inequity in classrooms and therefore vary across programs Adair and Colegrove (2021) note that in the United States, black and brown children have substantially lower levels of agency in early childhood settings than white children, observing that learning activities with greater child agency, like free play, are more likely to be “rewards” for sitting still and being silent in other learning activities. • Support for Personal and Social Connection Connection to Experience What is support for connection to experience? Support for connection to experience means bringing experiences that are not physically present or that occurred in the past into a current learning interaction. For example, in early education, linking exploration with children’s home or community experiences may foster deeper learning, including the application of ideas across contexts. Why is it important and how is it linked to learning through play? Linking new learning to children’s past experiences may make them more engaged and interested in the new content or activity. Indeed, in a national study of preschools in Colombia, when new learning was connected to children’s prior knowledge and experience, children’s executive function and emergent language skills were higher (Maldonado et al., 2021). In a study of Ghanaian preschools, when teachers connected learning to children’s past experiences, this pedagogical behavior predicted growth in early literacy and social-emotional skills (Wolf et al., 2018). In primary schooling, connection to experience may mean connection to children’s individual interests. These interests could, for example, incorporate elaborated play sequences with peers in school or community settings. Providing community-based mentorship for youth to pursue their own arts-based interests and activities in schools in fact boosted students’ performance on traditional achievement measures and on school attendance in urban El Salvador (Dinarte Diaz and Egana-del Sol, 2019). And when secondary school teachers linked learning to the everyday experiences of students, this led to improved science achievement in secondary schools in Uganda (Seidman et al., 2018). How might support for connection to experience vary across cultures? Linking learning in the home, with family, or in centers or schools with indigenous cultural traditions of socialization is a kind of programming that respects and deepens cultural variation in learning. For example, project-based pedagogy that incorporates learning as it occurs in nature and non-human systems, as reflected in indigenous cultural ways of knowing, has been used to build systems-based and critical thinking skills in both Native American and non-Native children (Bang, 2020). Social Connectedness What is support for social connectedness? Social connectedness is the relational trust, friendship, and camaraderie that occur among children or between children and adults, and it is a building block of functioning communities and societies. The increasingly complex social interactions that occur across infancy, early childhood, and later childhood are the context for social connectedness. At the beginning of life, dyadic interactions with caregivers, peers, siblings, and relatives provide the trust and attachment that allow for exploration and learning about the social and physical world. As group-based interactions become more complex after children start to walk, play begins to occur in peer groups, and not just in dyadic contexts. The well-known transition from parallel play (with children engaged in solo activity with only occasional interaction) to peer interactive play (with children collaborating in pairs or small groups) occurs during the early childhood and pre- primary years, with teachers and caregivers playing a critical role. Social connectedness, or the extent of relational trust, friendship, and camaraderie, can vary in peer groups and in classrooms and centers as a result. Parents or caregivers may support social connectedness by encouraging peer play in early childhood in community settings. Teachers may support social connectedness in early childhood by first encouraging children’s basic interactive skills—their ability to form friendships by initiating conversation; inviting a peer into an activity; and encouraging children in the classroom to listen to one another. An emphasis on prosocial behaviors and social norms in the classroom or center can facilitate such emerging friendships and social connectedness. In primary education, teachers may continue to support such prosocial behaviors, but there may be increasing focus on group work that is sustained across time. Collaborative group work can be encouraged by teachers and break up predominant patterns of didactic, teacher-led, or lecture-based instruction. Working toward a common goal can be made explicit by the teacher. There may be a resulting overall sense of camaraderie and connectedness of the classroom as a whole. Connectedness may also constitute an important component of children’s intrinsic motivation. In collectivist cultures, children’s engagement in activities may be motivated by a desire to strengthen relationships with caregivers or teachers. For example, parents and teachers in Tanzania said that one of the qualities children needed to succeed at school was to “love” their teacher (Jukes et al., 2018). In this study, both children and teachers reported that children’s motivation for most of their actions—from being polite to studying hard to being curious—was for the teacher to love them. The importance of connectedness was mentioned by both children (“I was happy to meet my teacher, I love her deeply from my heart”) and teachers (“[the children who learn well are those who] love to be close to teachers, they trust their teachers in a way that nobody else can teach them”). Why is it important and how is it linked to learning through play? Without the trust in others that is fostered through social connectedness, children will be less engaged—that is, more withdrawn in social interactions, less likely to take creative risks, and less likely to engage in peer o group play activities. A class without sustained and close friendships among students will likely be one in which children are less likely to disclose to classmates what they are thinking or trying in a learning activity. This may result in lower levels of peer-to-peer learning, with a larger proportion of the learning coming directly from a teacher’s didactic lesson content and less of the learning in the form of new ideas coming from the students themselves. How does support for social connectedness vary across cultures? In many cultures, social connectedness and related areas of social responsibility, including behaviors such as respect, obedience, acknowledging the wisdom of elders, and care for others, are valued alongside individual agency and initiative, as has been observed among the Ewe in Togo and A-Chewa speakers in Zambia, or among Latinx immigrant communities in North America (Ng et al., 2012; Noyau and Gbeto, 2004; Serpell, 1993). In contrast, in many WEIRD (Western, educated, industrialized, rich, and democratic) societies, children often do not contribute to community or household activities without being asked (Rogoff, 2014). Consideration of the overall direction of the group in a collaborative play task was higher in Mexican-origin children in the United States compared with European American middle-class children (Alcalá, Rogoff, and Fraire, 2018). Specific activities may support social responsibility in addition to social connectedness in ways that are in accordance with culture. For example, creative storytelling in early education and creative writing in primary and secondary education can encourage drawing moral and social lessons related to responsibility, as was observed in a Chinese preschool in the well-known Preschool in Three Cultures study (Tobin, Hsueh, and Karasawa, 2009). In a primary school intervention in Bangladesh, traditions of indigenous, oral storytelling in the Chittagong Hills Tracts were incorporated with the interpretation of moral lessons into a curriculum, with positive impacts on writing, comprehension, and vocabulary (Nyeu, 2020). • Support for Emotional Climate What is emotional climate? In studies of the observed quality of preschool or primary school classrooms, emotional climate has often been prominent as a predictor of both social- emotional and academic learning outcomes (Wolf et al., 2018). In such studies, this construct has been defined as individualized attention by the teacher, behaviors such as praise and positive reinforcement, and the observed presence of specific positive emotions such as enthusiasm or smiling. In addition, behavioral aspects of pedagogy such as behavior management (the response to difficult or problem behaviors) or encouragement of prosocial behavior has been part of emotional climate in these studies. Why is it important and how is it linked to learning through play? Without a positive emotional climate, learning may “shut down.” Consider the extreme of a parent-child relationship characterized by frequent conflict, or a classroom with a high proportion of children who are aggressive. The frequent tussles that occur—whether physical or verbal—may disrupt attention and distract from learning activities. A context with relatively little positive reinforcement of children’s actions may result in less motivation to engage with the materials or ideas in a lesson. Indeed, a more positive classroom emotional climate has been linked to higher learning and social-emotional outcomes in many studies, although primarily in rich countries (Von Suchodoletz et al., 2022). In some studies in low- and middle-income countries, however, emotional climate does predict child learning and engagement (e.g., in secondary schools in Uganda, as studied by Seidman et al., 2018, or student engagement in preschools in Ghana, as studied by Wolf et al., 2018). How does emotional climate vary across cultures? Considered from a cross-cultural perspective, learning through play might not occur in contexts where the affective climate is in conflict with support for intrinsic motivation and flow aspects of engagement. Positive emotions can be powerful in motivating learning, whether such emotions are displayed by the teacher or encouraged among students. However, the type and valence of emotion may differ by cultural context. In the WEIRD literature on classroom quality and child learning, for example, emotional climate characteristics such as warmth, praise, and smiling have been predictive of academic and social outcomes in children. However, in other cultures, learning through play might not be as strongly associated with warmth and praise. An emotional climate that fosters sustained attention may include individualized attention in centers, classrooms, or homes, but without strong affective display. Annex B: Table comparing the main features of the Teach Primary classroom observation tool and PLAY toolkit Aspect Teach Primary Observation Tool PLAY (Playful Learning Across the Years) Toolkit Focus Age Primarily for students in the 6-to-12-year- Primarily for students in the 6-to-12-year- Group old age range old age range Setting Classroom Classroom Main Time on Learning Support for Exploration Components Classroom Culture Support for Agency Instruction Support for Personal and Social Socioemotional skills Connection Support for Emotional Climate Measurement Teacher-child observation (Scored using Teacher-child observation Method "Snapshots" of 1–10 seconds and quality (Scored using quality ratings) ratings) Classroom inventory Checklist capturing other aspects of Teacher survey education quality Child survey Method of Teacher-child observation Teacher-child observation Focus for Current Task Key Teacher's behaviors Teacher’s behaviors Observational Student’s behaviors Student’s behaviors Elements Student’s on-task/off-task behaviors Ratings/Scori One item focused on the teacher’s All items are rated on 3-point scale ng provision of learning activity (from the Time measuring frequency, effectiveness, or on Learning component) is measured using participation a dichotomous (Yes/No) scale. All other items are rated on a 3-point scale (low, medium or high) measuring frequency, effectiveness, or participation. Objectives Captures time spent on learning and extent Measures the quality of adults’ support for to which students are on task. learning through play. Designed to focus on Captures quality of teaching practices that the school setting and how teachers help develop students’ cognitive and support children’s engagement in learning. socio-emotional skills as well as other aspects of the classroom environment which can influence learning. Annex C: Overview of experts ID Role Organisation Country of Residence Nationality 1 Senior Research University of Cambridge United Kingdom Ethiopia Associate 2 Lecturer University of Exeter United Kingdom Slovakia /United Kingdom 3 Research Fellow Institute of Development and Economic Pakistan Pakistan Alternatives 4 Assistant Universidad de los Andes Chile Chile Professor 5 Teach Master University of Virginia United States Chile trainer 6 Associate University of Nebraska United States United States Professor 7 PhD researcher Harvard University United States United States 8 PhD Researcher Yale University United States United States 9 Teach Frontieri Consultanting Group Ethiopia Ethiopia enumerator Annex D: Full mapping results for Phase 1 Analysis of Mapping PLAY to Teach Primary Items PLAY item Mapped Teach Primary items No. L.4 L.3 L.2 L.1 Av. agreement Av. Status Responses raters (1-4) agreement (max 9) all (1-4) SUPPORT FOR EXPLORATION COX1: Teacher connects concepts in the 3.3 The teacher makes connections in the 9 3 6 0 0 3.33 3.33 Match lesson to students' backgrounds, or life lesson that relate to other content outside the classroom. knowledge or students’ daily lives COX2: Teacher gives student(s) a chance 6.2 The teacher provides thinking tasks 3 0 0 2 1 1.66 0.56 Non-match to try or explore something first before being shown how to use / answer it 7.1 The teacher provides students with 3 0 0 1 2 1.33 0.44 choices 3.2 The teacher explains content using 1 0 0 1 0 2 0.22 multiple forms of representation 6.3 The students ask open-questions or 1 0 0 1 0 2 0.22 perform thinking tasks 0.1 Teacher provides learning activity to 1 0 0 1 0 2 0.22 most students None 1 NA NA NA NA NA NA COX3: Teacher promotes students 6.2 The teacher provides thinking tasks 9 2 4 3 0 2.89 2.89 Match thinking by themselves by asking a 6.1 The teacher asks open-ended questions 3 0 3 0 0 3 1 comparison, categorization or prediction question or through a task they give them. COX4: Teacher gives hints or suggestions 5.1 The teacher provides specific comments 7 1 3 1 2 2.43 1.88 Match to encourage students to continue to get or prompts that help clarify students’ to the answer or explore the concept. misunderstandings 8.2 The teacher has a positive attitude 1 0 1 0 0 3 0.33 towards students' challenges 8.1 The teacher acknowledges students' 1 0 1 0 0 3 0.33 effort 5.2 The teacher provides specific comments 1 0 1 0 0 3 0.33 or prompts that help identify students’ successes COX5: Teacher asks questions to generate 6.1 The teacher asks open-ended questions 7 2 4 1 0 3.14 2.44 Follow-up explanations/reasons (e.g., for 4.1 The teacher uses questions, prompts or 2 1 1 0 0 3.5 0.78 phenomena/ experiments/results). other strategies to determine students’ level of understanding COX6: Teacher uses multiple methods to 3.2 The teacher explains content using 9 4 2 1 2 2.89 2.89 Match help students learn about a concept. multiple forms of representation SUPPORT FOR AGENCY COA1: Students try different solutions 8.1 The teacher acknowledges students' 1 0 1 0 0 3 0.33 Non-match (iteration). effort 4.2 The teacher monitors most students 1 0 1 0 0 3 0.33 during independent/group work 6.3 The students ask open-questions or 1 0 0 0 1 1 0.11 perform thinking tasks 7.2 The teacher provides students with 1 0 0 0 1 1 0.11 opportunities to take on roles in the classroom 7.1 The teacher provides students with 1 0 0 0 1 1 0.11 choices 9.1 The teacher promotes students’ 1 1 0 0 0 4 0.44 collaboration through peer interaction 9.3 Students collaborate with one another 1 1 0 0 0 4 0.44 through peer interaction None 3 NA NA NA NA NA NA COA2: Students practice a skill 4.2 The teacher monitors most students 4 1 2 0 1 2.75 1.22 Follow-up introduced by the teacher. during independent/group work 6.3 The students ask open-questions or 2 0 2 0 0 3 0.66 perform thinking tasks 9.1 The teacher promotes students’ 1 0 0 1 0 2 0.22 collaboration through peer interaction None 2 NA NA NA NA NA NA COA3 Students create something 6.3 The students ask open-questions or 4 0 1 1 2 1.75 0.78 Follow-up connected to the current lesson. perform thinking tasks 6.2 The teacher provides thinking tasks 1 0 0 0 1 1 0.11 7.1 The teacher provides students with 1 0 1 0 0 3 0.33 choices 9.1 The teacher promotes students’ 1 0 1 0 0 3 0.33 collaboration through peer interaction 9.3 Students collaborate with one another 1 0 1 0 0 3 0.33 through peer interaction None 2 NA NA NA NA NA NA COA4: Students choose WHO plays each 7.2 The teacher provides students with 4 1 2 1 0 3 1.33 Follow-up role in an activity. opportunities to take on roles in the classroom 7.1 The teacher provides students with 3 0 1 2 0 2.33 0.78 choices 7.3 The students volunteer to participate in 1 0 0 1 0 2 0.22 the classroom 9.1 The teacher promotes students’ 1 0 0 0 1 1 0.11 collaboration through peer interaction COA5: Students decide the WHAT or the 7.1 The teacher provides students with 8 4 3 1 0 3.38 3.00 Match HOW to do an academic task. choices 9.3 Students collaborate with one another 1 0 0 0 1 1 0.11 through peer interaction COA6: Teacher uses student ideas or 8.2 The teacher has a positive attitude 1 0 1 0 0 3 0.33 Non-match examples in instruction. towards students' challenges 6.3 The students ask open-questions or 1 0 0 1 0 2 0.22 perform thinking tasks 7.2 The teacher provides students with 2 0 0 2 0 2 0.44 opportunities to take on roles in the classroom 3.3 The teacher makes connections in the 1 0 0 0 1 1 0.11 lesson that relate to other content knowledge or students’ daily lives None 4 NA NA NA NA NA NA COA7: Teacher asks open-ended 6.1 The teacher asks open-ended questions 7 3 3 1 0 3.29 2.55 Follow-up questions or prompts for students to share their opinions or preferences. 7.2 The teacher provides students with 1 0 0 1 0 2 0.22 opportunities to take on roles in the classroom 6.2 The teacher provides thinking tasks 1 0 0 0 1 1 0.11 None 0 NA NA NA NA NA NA SUPPORT FOR PERSONAL AND SOCIAL CONNECTION COC1: Teacher has students work 9.1 The teacher promotes students’ 8 4 4 0 0 3.5 3.11 Match together on an instructional activity collaboration through peer interaction towards a common instructional goal or 8.3 The teacher encourages goal-setting 1 0 1 0 0 3 0.33 understanding. COC2: Teacher discusses or otherwise 9.2 The teacher promotes students' 4 1 2 0 1 2.75 1.22 Follow-up creates a sense of student/class interpersonal skills togetherness. 9.1 The teacher promotes students’ 2 2 0 0 0 4 0.89 collaboration through peer interaction 9.3 Students collaborate with one another 2 1 0 0 1 2.5 0.56 through peer interaction 7.2 The teacher provides students with 1 0 1 0 0 3 0.33 opportunities to take on roles in the classroom 2.1 The teacher sets clear behavioral 1 0 0 0 1 1 0.11 expectations for classroom activities None 1 NA NA NA NA COC3: Teacher invites students to share 3.3 The teacher makes connections in the 5 3 0 0 2 2.8 1.67 Follow-up their thoughts and/or personal lesson that relate to other content experience about themselves, their knowledge or students’ daily lives families, or communities connected to 6.1 The teacher asks open-ended questions 1 0 0 1 0 2 0.22 the learning goal. None 3 NA NA NA NA NA NA COC4: Teacher connects student personal 3.3 The teacher makes connections in the 5 2 1 1 1 2.8 1.56 Follow-up interests to learning activities to help lesson that relate to other content them engage. knowledge or students’ daily lives 1.3 The teacher responds to students' 1 0 1 0 0 3 0.33 needs (TT); 8.3 The teacher encourages goal-setting 1 0 0 0 1 1 0.11 None 2 NA NA NA NA NA NA COC5: Teacher sets up a task that 9.3 Students collaborate with one another 2 1 1 0 0 3.5 0.77 Non-match involves lively social interaction between through peer interaction students 9.1 The teacher promotes students’ 3 1 2 0 0 3.33 1.11 collaboration through peer interaction 9.2 The teacher promotes students' 1 0 0 1 0 2 0.22 interpersonal skills 6.2 The teacher provides thinking tasks 1 0 0 1 0 2 0.22 None 3 NA NA NA NA NA NA SUPPORT FOR EMOTIONAL CLIMATE COE1: Teacher uses a mode of instruction 1.1 The teacher treats all students 1 0 1 0 0 3 0.33 Non-match that is explicitly upbeat. respectfully 1.2 The teacher uses positive language with 3 0 1 2 0 2.33 0.77 students None 5 NA NA NA NA NA NA COE2: Teacher responds to students' 1.3 The teacher responds to students' 5 2 1 2 0 3 1.67 Match emotional needs. needs 8.2 The teacher has a positive attitude 2 0 1 1 0 2.5 0.56 towards students' challenges 9.2 The teacher promotes students' 1 0 0 1 0 2 0.22 interpersonal skills 1.2 The teacher uses positive language with 1 0 0 0 1 1 0.11 students COE3: Teacher includes students who did 4.1 The teacher uses questions, prompts or 1 0 0 1 0 2 0.22 Non-match not volunteer to answer. other strategies to determine students’ level of understanding 7.2 The teacher provides students with 1 0 0 1 0 2 0.22 opportunities to take on roles in the classroom 7.3 The students volunteer to participate in 1 0 0 1 0 2 0.22 the classroom 1.1 The teacher treats all students 1 0 0 0 1 1 0.11 respectfully None 5 NA NA NA NA NA NA COE4: Teacher makes students know that 8.2 The teacher has a positive attitude 4 1 1 2 0 2.75 1.22 Follow-up it is okay to participate even if their towards students' challenges answer or contribution is wrong or 1.2 The teacher uses positive language with 1 1 0 0 0 4 0.44 different. They will not get punished or students teased. 8.1 The teacher acknowledges students' 1 0 1 0 0 3 0.33 effort 2.1 The teacher sets clear behavioral 2 0 0 2 0 2 0.44 expectations for classroom activities 1.1 The teacher treats all students 1 0 0 0 1 1 0.11 respectfully None 1 NA NA NA NA NA NA COE5: Teacher reinforces rules or uses 2.1 The teacher sets clear behavioral 5 1 2 2 0 2.8 1.56 Follow-up positive discipline and rewards to expectations for classroom activities support a positive emotional climate. 2.3 The teacher redirects misbehavior and 3 0 2 1 0 2.67 0.89 focuses on the expected behavior, rather than the undesired behavior 9.2 The teacher promotes students' 1 0 1 0 0 3 0.33 interpersonal skills COE6: Teacher is inclusive of children 1.4 The teacher does not exhibit bias and 8 4 4 0 0 3.5 3.11 Match with diverse backgrounds and learning challenges stereotypes in the classroom needs through meaningful interactions, 4.2 The teacher monitors most students 1 0 0 0 1 1 0.11 such as individualized attention. during independent/group work COE7: The teacher explains a student's 9.2 The teacher promotes students' 5 1 3 0 1 2.8 1.56 Match actions, intentions, and/or feelings to interpersonal skills other students. 2.1 The teacher sets clear behavioral 1 0 0 1 0 2 0.22 expectations for classroom activities 2.2 The teacher acknowledges positive 1 0 0 0 1 1 0.11 student behavior 5.2 The teacher provides specific comments 1 0 0 0 1 1 0.11 or prompts that help identify students’ successes None 1 NA NA NA NA NA NA Analysis of Mapping Teach Primary to PLAY Items Teach Primary item Mapped PLAY items No. L.4 L.3 L.2 L.1 Av. Av. Status Responses agreement agreement (max 9) raters (1-4) all (1-4) 0.TIME ON LEARNING 0.1 Teacher provides learning activity to COC5 Teacher sets up a task that involves lively social 1 0 0 1 0 2 0.22 Non-match most students interaction between students COX6 Teacher uses multiple methods to help students 1 0 0 0 1 1 0.11 learn about a concept COX2 Teacher gives student(s) a chance to try or explore 1 0 0 0 1 1 0.11 something first before being shown how to use / answer it None 6 NA NA NA NA NA NA 0.2 Students are on task COA4 Students choose WHO plays each role in an activity 1 0 0 0 1 1 0.11 Non-match None 8 NA NA NA NA NA NA AREA A: CLASSROOM CULTURE 1. Supportive learning environment 1.1 The teacher treats all students COE2 Teacher responds to students' emotional needs 2 1 0 0 1 2.5 0.56 Non-match respectfully COE1 Teacher uses a mode of instruction that is explicitly 2 0 1 1 0 2.5 0.56 upbeat COE6 Teacher is inclusive of children with diverse 2 0 1 0 1 2 0.44 backgrounds and learning needs through meaningful interactions, such as individualized attention COE5 Teacher reinforces rules or uses positive discipline 1 0 1 0 0 3 0.33 and rewards to support a positive emotional climate None 1 NA NA NA NA NA NA 1.2 The teacher uses positive language COE1 Teacher uses a mode of instruction that is explicitly 5 0 1 1 3 1.6 0.89 Non-match with students upbeat COE5 Teacher reinforces rules or uses positive discipline 1 0 0 1 0 2 0.22 and rewards to support a positive emotional climate COE4 Teacher makes students know that it is okay to 1 1 0 0 0 4 0.44 participate even if their answer or contribution is wrong or different. They will not get punished or teased. None 2 NA NA NA NA NA NA 1.3 The teacher responds to students' COE2 Teacher responds to students' emotional needs 8 3 3 1 1 3 2.67 Match needs None 1 NA NA NA NA NA NA 1.4 The teacher does not exhibit bias COE6 Teacher is inclusive of children with diverse 9 6 3 0 0 3.67 3.67 Match and challenges stereotypes in the backgrounds and learning needs through meaningful classroom interactions, such as individualized attention 1.4a The teacher does not exhibit COE6 Teacher is inclusive of children with diverse 7 3 4 0 0 3.43 2.67 gender bias and challenges gender backgrounds and learning needs through meaningful stereotypes in the classroom interactions, such as individualized attention None 2 NA NA NA NA NA NA 1.4b The teacher does not COE6 Teacher is inclusive of children with diverse 7 3 4 0 0 3.43 2.67 exhibit disability bias and challenges backgrounds and learning needs through meaningful disability stereotypes in the classroom interactions, such as individualized attention None 2 NA NA NA NA NA NA 2. Positive behavioral expectations 2.1 The teacher sets clear behavioral COE5 Teacher reinforces rules or uses positive discipline 6 1 2 2 1 2.5 1.66 Follow-up expectations for classroom activities and rewards to support a positive emotional climate COC2 Teacher discusses or otherwise creates a sense of 2 1 0 0 1 2.5 0.56 student/class togetherness COE7 The teacher explains a student's actions, intentions, 1 0 0 1 0 2 0.22 and/or feelings to other students 2.2 The teacher acknowledges positive COE5 Teacher reinforces rules or uses positive discipline 7 2 3 1 1 2.85 2.22 Follow-up student behavior and rewards to support a positive emotional climate COE7 The teacher explains a student's actions, intentions, 1 1 0 0 0 4 0.44 and/or feelings to other students None 1 NA NA NA NA NA NA 2.3 The teacher redirects misbehavior COE5 Teacher reinforces rules or uses positive discipline 6 2 1 1 2 2.5 1.67 Follow-up and focuses on the expected behavior, and rewards to support a positive emotional climate rather than the undesired behavior COE7 The teacher explains a student's actions, intentions, 1 1 0 0 0 4 0.44 and/or feelings to other students None 2 NA NA NA NA NA NA AREA B: INSTRUCTION 3. Lesson facilitation 3.1 The teacher explicitly articulates the None 9 NA NA NA NA NA NA Non-match objectives of the lesson and relates classroom activities to the objectives 3.2 The teacher explains content using COX6 Teacher uses multiple methods to help students 9 4 2 3 0 3.11 3.11 Match multiple forms of representation learn about a concept 3.3 The teacher makes connections in COX1 Teacher connects concepts in the lesson to 8 4 4 0 0 3.5 3.11 Match the lesson that relate to other content students' backgrounds, or life outside the classroom knowledge or students’ daily lives COC3 Teacher invites students to share their thoughts 1 1 0 0 0 4 0.44 and/or personal experience about themselves, their families, or communities connected to the learning goal COC4 Teacher connects student personal interests to 2 0 2 0 0 3 0.67 learning activities to help them engage 3.4 The teacher models by enacting or COX6 Teacher uses multiple methods to help students 1 0 0 1 0 2 0.22 Non-match thinking aloud. learn about a concept (EH); COX5 Teacher asks questions to generate explanations/reasons (e.g., for phenomena/ experiments/results) COX5 Teacher asks questions to generate 1 0 0 1 0 2 0.22 explanations/reasons (e.g., for phenomena/ experiments/results) COX4 Teacher gives hints or suggestions to encourage 1 0 0 0 1 1 0.11 students to continue to get to the answer or explore the concept None 6 NA NA NA NA NA NA 4. Checks for understanding 4.1 The teacher uses questions, prompts COX5 Teacher asks questions to generate 4 2 2 0 0 3.5 1.56 Follow-up or other strategies to determine explanations/reasons (e.g., for phenomena/ students’ level of understanding experiments/results) COA7 Teacher asks open-ended questions or prompts for 1 0 1 0 0 3 0.33 students to share their opinions or preferences COA2 Students practice a skill introduced by the teacher 1 0 0 1 0 2 0.22 COX4 Teacher gives hints or suggestions to encourage 1 0 0 0 1 1 0.11 students to continue to get to the answer or explore the concept None 2 NA NA NA NA NA NA 4.2 The teacher monitors most students COA2 Students practice a skill introduced by the teacher 3 0 1 1 1 2 0.67 Non-match during independent/group work COA1 Students try different solutions (iteration) 2 0 0 2 0 4 0.44 COC1 Teacher has students work together on an 1 0 0 0 1 1 0.11 instructional activity towards a common instructional goal or understanding None 2 NA NA NA NA NA NA 4.3 The teacher adjusts teaching to the COA7 Teacher asks open-ended questions or prompts for 1 1 0 0 0 4 0.44 Non-match level of the students students to share their opinions or preferences COX4 Teacher gives hints or suggestions to encourage 2 0 1 0 1 2 0.44 students to continue to get to the answer or explore the concept COE6 Teacher is inclusive of children with diverse 2 0 1 1 2.5 0.56 backgrounds and learning needs through meaningful interactions, such as individualized attention COA6 Teacher uses student ideas or examples in 1 0 0 0 1 1 0.11 instruction COE2 Teacher responds to students' emotional needs 1 0 0 0 1 1 0.11 None 2 NA NA NA NA NA NA 5. Feedback 5.1 The teacher provides specific COX4 Teacher gives hints or suggestions to encourage 5 1 3 0 1 2.8 1.56 Match comments or prompts that help clarify students to continue to get to the answer or explore the students’ misunderstandings concept COE4 Teacher makes students know that it is okay to 1 0 0 1 0 2 0.22 participate even if their answer or contribution is wrong or different. They will not get punished or teased None 1 NA NA NA NA NA NA 5.2 The teacher provides specific COX4 Teacher gives hints or suggestions to encourage 2 0 1 0 1 2 0.44 Follow-up comments or prompts that help identify students to continue to get to the answer or explore the students’ successes concept COE7 The teacher explains a student's actions, intentions, 1 0 0 1 0 2 0.22 and/or feelings to other students COE5 Teacher reinforces rules or uses positive discipline 1 0 0 0 1 1 0.11 and rewards to support a positive emotional climate COA6 Teacher uses student ideas or examples in 1 0 0 0 1 1 0.11 instruction None 3 NA NA NA NA NA NA 6. Critical thinking 6.1 The teacher asks open-ended COX5 Teacher asks questions to generate 5 1 3 0 0 3.2 1.78 Follow-up questions explanations/reasons (e.g., for phenomena/ experiments/results) COA7 Teacher asks open-ended questions or prompts for 3 3 0 0 0 4 1.33 students to share their opinions or preferences COX3 Teacher promotes students thinking by themselves 2 0 2 0 0 3 0.67 by asking a comparison, categorization or prediction question or through a task they give them 6.2 The teacher provides thinking tasks COX3 Teacher promotes students thinking by themselves 6 3 2 1 0 1.67 2.22 Match by asking a comparison, categorization or prediction question or through a task they give them COX5 Teacher asks questions to generate 2 0 1 1 0 2.5 0.56 explanations/reasons (e.g., for phenomena/ experiments/results) (DT) COX2 Teacher gives student(s) a chance to try or explore 1 0 0 0 1 1 0.11 something first before being shown how to use / answer it 6.3 The students ask open-questions or COA2 Students practice a skill introduced by the teacher 1 1 0 0 0 4 0.44 Non-match perform thinking tasks COX3 Teacher promotes students thinking by themselves 2 1 1 0 0 3.5 0.78 by asking a comparison, categorization or prediction question or through a task they give them COA1 Students try different solutions (iteration) 5 0 1 1 3 1.6 0.89 None 1 NA NA NA NA NA NA AREA C: SOCIOEMOTIONAL SKILLS 7: Autonomy 7.1 The teacher provides students with COA5 Students decide the WHAT or the HOW to do an 6 3 2 1 0 3.33 2.22 Match choices academic task COA3 Students create something connected to the 1 1 0 0 0 4 0.44 current lesson COA4 Students choose WHO plays each role in an activity 2 1 1 0 0 3.5 0.78 COX3 Teacher promotes students thinking by themselves 1 0 0 1 0 2 0.22 by asking a comparison, categorization or prediction question or through a task they give them 7.2 The teacher provides students with COA4 Students choose WHO plays each role in an activity 7 1 1 2 3 2 1.56 Follow-up opportunities to take on roles in the COC2 Teacher discusses or otherwise creates a sense of 1 0 1 0 0 3 0.33 classroom student/class togetherness None 1 NA NA NA NA NA NA 7.3 The students volunteer to COC2 Teacher discusses or otherwise creates a sense of 1 0 1 0 0 3 0.33 Non-match participate in the classroom student/class togetherness COA6 Teacher uses student ideas or examples in 1 0 1 0 0 3 0.33 instruction COE3 Teacher includes students who did not volunteer to 3 0 0 1 2 1.33 0.44 answer COA4 Students choose WHO plays each role in an activity 1 0 0 1 0 2 0.22 COA7 Teacher asks open-ended questions or prompts for 2 0 0 1 1 1.5 0.33 students to share their opinions or preferences None 1 NA NA NA NA NA NA 8. Perseverance 8.1 The teacher acknowledges students' COE4 Teacher makes students know that it is okay to 6 0 1 1 4 1.5 1 Follow-up effort participate even if their answer or contribution is wrong or different. They will not get punished or teased COE5 Teacher reinforces rules or uses positive discipline 1 0 0 0 1 1 0.11 and rewards to support a positive emotional climate COA1 Students try different solutions (iteration) 1 0 0 0 1 1 0.11 None 1 NA NA NA NA NA NA 8.2 The teacher has a positive attitude COE4 Teacher makes students know that it is okay to 6 0 2 2 2 2 1.33 Follow-up towards students' challenges participate even if their answer or contribution is wrong or different. They will not get punished or teased COA6 Teacher uses student ideas or examples in 1 0 1 0 0 3 0.33 instruction None 2 NA NA NA NA NA NA 8.3 The teacher encourages goal- COC1 Teacher has students work together on an 2 0 1 0 1 2 0.44 Non-match setting instructional activity towards a common instructional goal or understanding COC4 Teacher connects student personal interests to 1 0 0 0 1 1 0.11 learning activities to help them engage None 6 NA NA NA NA NA NA 9. Social and Collaborative Skills 9.1 The teacher promotes students’ COC1 Teacher has students work together on an 6 3 3 0 0 3.5 2.33 Match collaboration through peer interaction instructional activity towards a common instructional goal or understanding COC5 Teacher sets up a task that involves lively social 3 2 1 0 0 3.67 1.22 interaction between students 9.2 The teacher promotes students' COE7 The teacher explains a student's actions, intentions, 5 1 3 1 0 3 1.66 Match interpersonal skills and/or feelings to other students COE5 Teacher reinforces rules or uses positive discipline 1 1 0 0 0 4 0.44 and rewards to support a positive emotional climate COC3 Teacher invites students to share their thoughts 1 1 0 0 0 4 0.44 and/or personal experience about themselves, their families, or communities connected to the learning goal COC2 Teacher discusses or otherwise creates a sense of 1 0 1 0 0 3 0.33 student/class togetherness COC5 Teacher sets up a task that involves lively social 1 0 0 1 0 2 0.22 interaction between student 9.3 Students collaborate with one COC1 Teacher has students work together on an 3 1 1 0 1 2.67 0.88 Non-match another through peer interaction instructional activity towards a common instructional goal or understanding COA3 Students create something connected to the 1 1 0 0 0 4 0.44 current lesson COC5 Teacher sets up a task that involves lively social 1 0 1 0 0 3 0.33 interaction between students COA4 Students choose WHO plays each role in an 1 0 0 1 0 2 0.22 activity COC2 Teacher discusses or otherwise creates a sense of 1 0 0 0 1 1 0.11 student/class togetherness COA2 Students practice a skill introduced by the teacher 1 0 0 0 1 1 0.11 Annex E: Mapping results for Phase 2 Analysis of Mapping PLAY to Teach Primary Items (Revised results) PLAY item Mapped Teach Primary items No. L. L.3 L.2 L.1 Av. Av. Status Respon 4 agree agreeme ses(max ment nt all (1- 9) raters 4) (1-4) SUPPORT FOR EXPLORATION COX5: Teacher asks questions to 6.1 The teacher asks open-ended questions 8 1 7 0 0 3.13 2.77 Non- generate explanations/reasons match 4.1 The teacher uses questions, prompts or other 1 1 0 0 0 4 0.33 (e.g., for phenomena/ strategies to determine students’ level of experiments/results). understanding SUPPORT FOR AGENCY COA2: Students practice a skill 4.2 The teacher monitors most students during 4 0 3 0 1 2.5 1.11 Non- introduced by the teacher. independent/group work match 6.3 The students ask open-questions or perform 1 0 1 0 0 3 0.33 thinking tasks None 4 N NA NA NA NA NA A COA3 Students create something 6.3 The students ask open-questions or perform 4 0 1 1 3 2 0.89 Non- connected to the current lesson. thinking tasks match 7.1 The teacher provides students with choices 1 0 0 1 0 2 0.22 9.1 The teacher promotes students’ collaboration 1 0 1 0 0 3 0.33 through peer interaction 9.3 Students collaborate with one another through 1 0 1 0 0 3 NA peer interaction None 2 N NA NA NA NA NA A COA4: Students choose WHO 7.2 The teacher provides students with 5 0 3 1 1 2.4 1.33 plays each role in an activity. opportunities to take on roles in the classroom 7.1 The teacher provides students with choices 3 0 1 2 0 2.33 0.77 Non- 9.1 The teacher promotes students’ collaboration 1 0 0 0 1 1 0.11 match through peer interaction SUPPORT FOR PERSONAL AND SOCIAL CONNECTION COC2: Teacher discusses or 9.2 The teacher promotes students' interpersonal 6 1 2 1 2 2.33 1.55 Non- otherwise creates a sense of skills match student/class togetherness. 9.1 The teacher promotes students’ collaboration 1 0 1 0 0 3 0.33 through peer interaction 9.3 Students collaborate with one another through 2 1 0 0 1 2.5 0.56 peer interaction 7.2 The teacher provides students with 1 0 0 0 1 1 0.11 opportunities to take on roles in the classroom 2.1 The teacher sets clear behavioral expectations 1 0 0 0 1 1 0.11 for classroom activities COC3: Teacher invites students to 3.3 The teacher makes connections in the lesson 6 3 3 1 0 3.83 2.56 Non- share their thoughts and/or that relate to other content knowledge or match personal experience about students’ daily lives themselves, their families, or 6.1 The teacher asks open-ended questions 1 0 0 1 0 2 0.22 communities connected to the None 3 N NA NA NA NA NA learning goal. A COC4: Teacher connects student 3.3 The teacher makes connections in the lesson 8 2 1 4 1 2.5 2.22 Non- personal interests to learning that relate to other content knowledge or match activities to help them engage. students’ daily lives 1.3 The teacher responds to students' needs 1 0 0 0 1 1 0.11 SUPPORT FOR PERSONAL AND SOCIAL CONNECTION COE4: Teacher makes students 8.2 The teacher has a positive attitude towards 5 1 1 3 0 2.6 1.44 Non- know that it is okay to participate students' challenges match even if their answer or 1.2 The teacher uses positive language with 1 0 0 0 1 1 0.11 contribution is wrong or different. students They will not get punished or 8.1 The teacher acknowledges students' effort 1 0 1 0 0 3 0.33 teased. 2.1 The teacher sets clear behavioral expectations 1 0 0 1 0 2 0.22 for classroom activities 1.1 The teacher treats all students respectfully 1 0 0 0 1 1 0.11 COE5: Teacher reinforces rules or 2.1 The teacher sets clear behavioral expectations 6 0 4 2 0 2.33 1.56 uses positive discipline and for classroom activities rewards to support a positive 2.3 The teacher redirects misbehavior and focuses 3 0 1 2 0 2.33 0.77 emotional climate. on the expected behavior, rather than the Non- undesired behavior match 9.2 The teacher promotes students' interpersonal 1 1 0 0 0 4 0.44 skills Analysis of Mapping Teach Primary to PLAY Items (Revised results) Teach Primary item Mapped PLAY items No. L.4 L.3 L.2 L.1 Av. Av. Status Responses(ma agreeme agreeme x 9) nt raters nt all (1- (1-4) 4) AREA A: CLASSROOM CULTURE 2. Positive behavioral expectations 2.1 The teacher sets clear COE5 Teacher reinforces rules or uses 7 0 3 3 1 2.42 1.88 Non- behavioral expectations for positive discipline and rewards to match classroom activities support a positive emotional climate COC2 Teacher discusses or otherwise 1 0 1 0 0 3 0.33 creates a sense of student/class togetherness COE7 The teacher explains a student's 1 0 0 1 0 2 0.22 actions, intentions, and/or feelings to other students 2.2 The teacher acknowledges COE5 Teacher reinforces rules or uses 7 2 4 0 1 3 2.33 Non- positive student behavior positive discipline and rewards to match support a positive emotional climate COE7 The teacher explains a student's 1 0 0 1 0 2 0.22 actions, intentions, and/or feelings to other students None 1 N N NA NA NA NA A A 2.3 The teacher redirects COE5 Teacher reinforces rules or uses 8 0 2 3 3 1.88 1.67 Non- misbehavior and focuses on the positive discipline and rewards to match expected behavior, rather than support a positive emotional climate the undesired behavior COE7 The teacher explains a student's 1 1 0 0 0 4 0.44 actions, intentions, and/or feelings to other students None 0 N N NA NA NA NA A A AREA B: INSTRUCTION 4. Checks for understanding 4.1 The teacher uses questions, COX5 Teacher asks questions to generate 5 0 5 0 0 3 1.66 Non- prompts or other strategies to explanations/reasons (e.g., for match determine students’ level of phenomena/ experiments/results) understanding COA7 Teacher asks open-ended 1 0 1 0 0 3 0.33 questions or prompts for students to share their opinions or preferences COX4 Teacher gives hints or suggestions 1 0 0 0 1 1 0.11 to encourage students to continue to get to the answer or explore the concept None 2 N N NA NA NA NA A A 5. Feedback 5.2 The teacher provides specific COX4 Teacher gives hints or suggestions 4 0 1 0 3 1.4 0.67 Non- comments or prompts that help to encourage students to continue to get match identify students’ successes to the answer or explore the concept None 5 N N NA NA NA NA A A 6. Critical thinking 6.1 The teacher asks open- COX5 Teacher asks questions to generate 4 1 3 0 0 3.3 1.44 Non- ended questions explanations/reasons (e.g., for match phenomena/ experiments/results) COA7 Teacher asks open-ended 3 2 0 1 0 2 0.67 questions or prompts for students to share their opinions or preferences COX3 Teacher promotes students 2 0 2 0 0 3 0.67 thinking by themselves by asking a comparison, categorization or prediction question or through a task they give them AREA C: SOCIOEMOTIONAL SKILLS 7: Autonomy 7.2 The teacher provides COA4 Students choose WHO plays each 7 1 1 3 2 2.14 1.66 Non- students with opportunities to role in an activity match take on roles in the classroom COC2 Teacher discusses or otherwise 1 0 0 1 0 2 0.22 creates a sense of student/class togetherness None 1 N N NA NA NA NA A A 8. Perseverance 8.1 The teacher acknowledges COE4 Teacher makes students know that 8 0 0 2 6 1.23 1.11 Non- students' effort it is okay to participate even if their match answer or contribution is wrong or different. They will not get punished or teased None 1 N N NA NA NA NA A A 8.2 The teacher has a positive COE4 Teacher makes students know that 8 0 2 2 4 1.75 1.56 Non- attitude towards students' it is okay to participate even if their match challenges answer or contribution is wrong or different. They will not get punished or teased None 1 N N NA NA NA NA A A Annex F: Correlation of Teach Primary and PLAY scores The matrix below presents the correlations between Teach Primary and PLAY scores using data from Ethiopia, Peru, and Sierra Leone. The diagonal represents the unity standardized covariance. The row titles identify the corresponding domain or total score of each tool and the column number follows corresponds to the same variable. 1 2 3 4 5 6 7 8 9 1 Classroom Culture 1 2 Instruction 0.465 1 3 Socioemotional skills 0.419 0.637 1 4 Teach Primary Total Score 0.786 0.853 0.818 1 5 Support for Exploration 0.098 0.17 0.105 0.152 1 6 Support for Agency -0.012 0.07 0.124 0.07 0.515 1 7 Support for Personal and Social Connection 0.092 0.077 0.156 0.13 0.506 0.596 1 8 Support for Emotional Climate 0.061 0.087 0.115 0.105 0.489 0.356 0.33 1 9 PLAY Total Score 0.079 0.139 0.157 0.15 0.842 0.773 0.752 0.718 1 Annex G: Correlation of PLAY items, item-total correlations and visual representations of internal structure The matrix below presents the polychoric correlations between PLAY item scores using data from Ethiopia, Peru, and Sierra Leone. The diagonal represents the unity standardized covariance. The row and column titles identify the item name code. cox1 cox2 cox3 cox4 cox5 cox6 coa1 coa2 coa3 coa4 coa5 coa6 coa7 coc1 coc2 coc3 coc4 coc5 coe1 coe2 coe3 coe4 coe5 coe6 coe7 cox1 1 cox2 0.018 1 cox3 0.327 0.356 1 cox4 0.294 0.158 0.397 1 cox5 0.302 0.236 0.371 0.292 1 cox6 0.082 0.280 0.294 -0.020 0.268 1 coa1 0.071 0.297 0.380 0.288 0.435 0.279 1 coa2 0.059 0.235 0.323 0.361 -0.071 0.026 0.212 1 coa3 0.232 0.041 0.304 0.366 0.364 0.172 0.494 0.045 1 coa4 0.272 0.235 0.254 0.381 0.269 0.055 0.562 0.228 0.511 1 coa5 0.251 0.102 0.244 0.315 0.321 0.135 0.591 0.096 0.656 0.712 1 coa6 0.206 0.418 0.472 0.219 0.209 0.416 0.441 0.248 0.387 0.530 0.359 1 coa7 0.372 0.329 0.598 0.191 0.362 0.497 0.208 0.205 0.340 0.030 0.225 0.515 1 coc1 0.377 0.131 0.361 0.258 0.397 0.178 0.156 0.264 0.538 0.499 0.551 0.269 0.364 1 coc2 0.075 0.310 0.329 0.252 0.499 0.265 0.489 0.093 0.514 0.485 0.348 0.469 0.313 0.395 1 coc3 0.560 0.141 0.300 0.407 0.397 0.173 0.428 0.129 0.460 0.695 0.571 0.507 0.227 0.395 0.417 1 coc4 0.299 0.261 0.523 0.332 0.326 0.361 0.583 0.363 0.509 0.630 0.581 0.597 0.366 0.452 0.549 0.638 1 coc5 0.335 0.395 0.511 0.424 0.263 0.423 0.416 0.308 0.607 0.580 0.363 0.630 0.420 0.512 0.525 0.549 0.709 1 coe1 0.138 0.265 0.378 0.189 0.271 0.352 0.139 0.201 0.126 0.170 0.225 0.191 0.362 0.198 0.119 0.128 0.296 0.376 1 coe2 0.280 0.325 0.286 0.275 0.108 0.118 0.451 0.237 0.216 0.467 0.379 0.294 0.304 0.199 0.369 0.546 0.423 0.478 0.274 1 coe3 0.016 0.451 0.288 0.013 0.135 0.421 0.412 0.095 -0.045 0.106 0.053 0.335 0.322 0.010 0.266 0.113 0.369 0.363 0.319 0.282 1 coe4 -0.109 0.352 0.329 0.217 0.396 0.344 0.368 -0.006 0.275 -0.117 0.098 0.211 0.476 0.126 0.564 -0.069 0.273 0.323 0.281 0.153 0.389 1 coe5 0.098 0.093 0.223 0.202 0.113 0.243 -0.042 0.113 0.018 -0.254 -0.143 0.069 0.252 0.184 0.140 -0.168 0.008 0.078 0.226 -0.016 0.227 0.356 1 coe6 0.121 -0.085 0.057 -0.001 -0.142 -0.303 -0.274 0.105 -0.128 -0.195 -0.088 -0.136 0.069 0.011 -0.068 -0.170 -0.115 -0.061 -0.064 0.054 -0.156 -0.118 0.071 1 coe7 0.353 0.316 0.279 0.376 0.271 0.165 0.548 0.195 0.442 0.732 0.519 0.487 0.013 0.304 0.406 0.780 0.644 0.632 0.107 0.604 0.287 -0.117 -0.241 -0.199 1 The table below presents the correlations between individual PLAY item scores and the total PLAY score using data from Ethiopia, Peru, and Sierra Leone. Items with a positive correlation with the total score contribute to the other psychometrics properties of the measurement tool, including the discrimination between teachers with high and low scores on engaging teaching practices and the total reliability of the tool. Negative item- total correlations, such as the case of COE6, require an examination from experts to determine why this item is inconsistent with the rest of the measurement tool. PLAY item Item-Total correlation cox1 0.294 cox2 0.378 cox3 0.542 cox4 0.387 cox5 0.402 cox6 0.361 coa1 0.419 coa2 0.257 coa3 0.435 coa4 0.460 coa5 0.368 coa6 0.474 coa7 0.503 coc1 0.402 coc2 0.502 coc3 0.490 coc4 0.557 coc5 0.587 coe1 0.351 coe2 0.346 coe3 0.331 coe4 0.382 coe5 0.186 coe6 -0.063 coe7 0.425 Figure 1. Hierarchical clustering of PLAY items using distance matrix Figure 2. Non-Metric MDS representation of PLAY items using distance matrix Figure 3. Scree plot for Exploratory Factor Analysis Annex H: Exploratory Factor Analysis solution Table H3. Standardized factor loadings, communalities and uniqueness for one-factor solution PLAY item Loading F1 Communality Uniqueness cox1 0.394 0.156 0.844 cox2 0.424 0.180 0.820 cox3 0.602 0.363 0.637 cox4 0.477 0.228 0.772 cox5 0.502 0.252 0.748 cox6 0.407 0.166 0.834 coa1 0.673 0.453 0.547 coa2 0.305 0.093 0.907 coa3 0.640 0.410 0.590 coa4 0.717 0.514 0.486 coa5 0.649 0.421 0.579 coa6 0.685 0.470 0.530 coa7 0.513 0.263 0.737 coc1 0.561 0.315 0.685 coc2 0.652 0.426 0.574 coc3 0.721 0.520 0.480 coc4 0.834 0.696 0.304 coc5 0.823 0.678 0.322 coe1 0.375 0.141 0.859 coe2 0.567 0.322 0.678 coe3 0.379 0.144 0.856 coe4 0.352 0.124 0.876 coe5 0.081 0.007 0.993 coe6 -0.150 0.022 0.978 coe7 0.714 0.509 0.491 Table H2. Standardized factor loadings, communalities and uniqueness for two-factor solution PLAY item Loading F1 Loading F2 Communality Uniqueness cox1 0.391 0.060 0.171 0.829 cox2 0.113 0.470 0.266 0.734 cox3 0.236 0.569 0.463 0.537 cox4 0.397 0.169 0.228 0.772 cox5 0.275 0.366 0.272 0.728 cox6 0.010 0.585 0.346 0.654 coa1 0.533 0.275 0.450 0.550 coa2 0.202 0.174 0.093 0.907 coa3 0.585 0.159 0.426 0.574 coa4 0.953 -0.169 0.837 0.163 coa5 0.734 -0.010 0.534 0.466 coa6 0.452 0.398 0.474 0.526 coa7 0.030 0.724 0.538 0.462 coc1 0.464 0.203 0.314 0.686 coc2 0.402 0.417 0.439 0.561 coc3 0.870 -0.069 0.724 0.276 coc4 0.668 0.330 0.690 0.310 coc5 0.575 0.434 0.674 0.326 coe1 0.062 0.466 0.239 0.761 coe2 0.503 0.163 0.330 0.670 coe3 0.012 0.541 0.297 0.703 coe4 -0.188 0.777 0.549 0.451 coe5 -0.307 0.518 0.265 0.735 coe6 -0.128 -0.049 0.023 0.977 coe7 0.888 -0.101 0.744 0.256 Table H3. Standardized factor loadings, communalities and uniqueness for three-factor solution PLAY item Loading F1 Loading F2 Loading F3 Communality Uniqueness cox1 0.447 0.261 -0.308 0.336 0.664 cox2 0.088 0.163 0.500 0.342 0.658 cox3 0.579 0.141 0.178 0.507 0.493 cox4 0.369 0.307 -0.106 0.283 0.717 cox5 0.385 0.212 0.105 0.287 0.713 cox6 0.231 0.033 0.494 0.375 0.625 coa1 -0.040 0.601 0.405 0.570 0.430 coa2 0.214 0.160 0.030 0.100 0.900 coa3 0.374 0.494 -0.112 0.477 0.523 coa4 -0.057 0.940 -0.079 0.840 0.160 coa5 0.183 0.671 -0.125 0.543 0.457 coa6 0.210 0.449 0.319 0.488 0.512 coa7 0.727 -0.093 0.230 0.639 0.361 coc1 0.589 0.316 -0.256 0.523 0.477 coc2 0.261 0.388 0.286 0.440 0.560 coc3 0.127 0.812 -0.133 0.721 0.279 coc4 0.252 0.639 0.213 0.690 0.310 coc5 0.374 0.524 0.218 0.670 0.330 coe1 0.344 0.027 0.248 0.239 0.761 coe2 0.080 0.503 0.160 0.341 0.659 coe3 -0.029 0.104 0.719 0.536 0.464 coe4 0.397 -0.183 0.545 0.540 0.460 coe5 0.491 -0.385 0.161 0.321 0.679 coe6 0.238 -0.219 -0.283 0.129 0.871 coe7 -0.182 0.945 0.098 0.838 0.162 Table H4. Standardized factor loadings, communalities and uniqueness for four-factor solution PLAY item Loading F1 Loading F2 Loading F3 Loading F4 Communality Uniqueness cox1 0.198 0.540 -0.303 -0.214 0.411 0.589 cox2 0.243 0.223 0.187 0.412 0.383 0.617 cox3 0.123 0.654 0.084 0.073 0.552 0.448 cox4 0.249 0.359 -0.015 -0.159 0.281 0.719 cox5 0.121 0.190 0.427 -0.232 0.380 0.620 cox6 0.069 0.236 0.373 0.252 0.360 0.640 coa1 0.605 -0.140 0.480 0.081 0.637 0.363 coa2 0.188 0.389 -0.207 0.144 0.208 0.792 coa3 0.348 0.110 0.400 -0.515 0.702 0.298 coa4 0.890 -0.052 -0.005 -0.157 0.830 0.170 coa5 0.572 0.020 0.206 -0.365 0.619 0.381 coa6 0.469 0.292 0.160 0.180 0.501 0.499 coa7 -0.124 0.728 0.219 0.039 0.646 0.354 coc1 0.188 0.454 0.076 -0.415 0.528 0.472 coc2 0.330 0.077 0.543 -0.101 0.550 0.450 coc3 0.764 0.180 -0.113 -0.150 0.726 0.274 coc4 0.618 0.280 0.177 0.034 0.688 0.312 coc5 0.511 0.444 0.125 0.073 0.689 0.311 coe1 0.042 0.409 0.116 0.159 0.258 0.742 coe2 0.541 0.228 -0.066 0.180 0.404 0.596 coe3 0.230 0.131 0.296 0.598 0.580 0.420 coe4 -0.244 0.173 0.826 0.071 0.781 0.219 coe5 -0.395 0.452 0.181 0.043 0.314 0.686 coe6 -0.242 0.321 -0.323 -0.097 0.178 0.822 coe7 0.983 -0.032 -0.114 0.124 0.889 0.111 ABSTRACT This study compares two tools used to observe and assess how teachers support student learning: the LEGO Foundation’s Engage tool, formerly known as PLAY, and the World Bank’s Teach Primary tool. The goal was to evaluate their similarities and overlap in their measurement of teaching practices. A group of nine experts reviewed both tools in detail, carefully matching items from PLAY to those in Teach Primary. Results indicate that about one-third of the items closely matched, showing that PLAY is built on a solid framework to capture engaging teaching practices. Another third of the items did not have a clear match, meaning that each tool separately captures unique aspects of teaching that the other does not. The study also looked at how well the two tools’ scores align using real classroom data from Ethiopia, Peru, and Sierra Leone. The results showed a small positive relationship between the two tools, meaning they measure related but different aspects of teaching. The study also found that the way PLAY is structured may need some adjustments to better reflect the skills it is designed to measure. By combining expert opinions with data analysis, this study provides strong evidence that both tools are valuable but serve different purposes. The findings highlight the importance of using multiple approaches to assess teaching quality. The study also offers recommendations for improving tool improvement and suggests next steps on how classroom observation tools can help support improved teaching and learning worldwide.