80069 Sindh Province, Pakistan SABER Province Report STUDENT ASSESSMENT 2012 Key Policy Areas for Student Assessment Status 1. Classroom Assessment The 2006 National Curriculum provides very general, non-comprehensive guidelines on classroom assessment. Although available online, this document is not offered through other channels (such as teacher training courses or libraries) that are easily accessible to teachers and other stakeholders. There are some system-level mechanisms in place (such as pre- and in-service teacher training opportunities) that are meant to ensure that teachers develop appropriate skills and expertise in classroom assessment. However, on-the-ground classroom assessment practices tend to be mainly about students recalling information, provide little feedback to students and parents, and are generally considered weak. 2. Examinations The Secondary School Certificate exam is administered to students in Grades 9 and 10. Examination results are used only for making decisions about the promotion of individual students to the next level of schooling. The Sindh Boards of Intermediate and Secondary Education Ordinance of 1972 and the Sindh Boards of Intermediate and Secondary Education (Amendment) Act of 2008 authorize the exam. The Examination Rules and Procedures document (known as the “Calendar”) for each Sindh Board covers key aspects of the exam. Officially, the examination is meant to measure student attainment of the knowledge and skills outlined in the 2006 National Curriculum. However, in practice, it assesses textbook knowledge. Moreover, the textbooks are based on an earlier (2002) curriculum, not the more recent (2006) one. There is a permanent staff, but it is insufficient to meet the needs of the examination due to staff members’ lack of technical capacity. Additionally, there is no technical documentation available on the examination. 3. National Large-Scale Assessment (NLSA) The Provincial Education Assessment Center (PEACE) Sindh Assessment was administered for the first time in 2008 as an independent, provincial-level assessment exercise. Previously, PEACE was part of the National Education Assessment System (NEAS). Under NEAS, Sindh administered the assessment to a provincially-representative sample of students and published a provincial report separate from the national report. Although the PEACE office is a permanent agency, it is inadequately staffed to effectively carry out PEACE activities. This also was the case when PEACE was part of NEAS. PEACE results and information have not been adequately used. 4. International Large-Scale Assessment (ILSA) Sindh has not participated in an ILSA, and it does not have plans to do so in the near future. THE WORLD BANK SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Introduction SABER-Student Assessment methodology The Sindh province in Pakistan has focused on The SABER-Student Assessment framework is built on increasing student learning outcomes by improving the the available evidence base for what an effective quality of education in the province. An effective assessment system looks like. The framework provides student assessment system is an important component guidance on how countries can build more effective to improving education quality and learning outcomes student assessment systems. The framework is as it provides the necessary information to meet structured around two main dimensions of assessment stakeholders’ decision-making needs. In order to gain a systems: the types/purposes of assessment activities better understanding of the strengths and weaknesses and the quality of those activities. of its existing student assessment system within the context of the province’s Public Sector Education Assessment types and purposes system, the Sindh province decided to benchmark this system using standardized tools developed under The Assessment systems tend to be comprised of three World Bank’s Systems Approach for Better Education main types of assessment activities, each of which Results (SABER) program. SABER is an evidence-based serves a different purpose and addresses different program to help countries systematically examine and information needs. These three main types are: strengthen the performance of different aspects of their classroom assessment, examinations, and large-scale, education systems. system level assessments. What is SABER-Student Assessment? Classroom assessment provides real-time information to support ongoing teaching and learning in individual classrooms. Classroom assessments use a variety of SABER-Student Assessment is a component of the formats, including observation, questioning, and paper- SABER program that focuses specifically on and-pencil tests, to evaluate student learning, generally benchmarking student assessment policies and systems. on a daily basis. The goal of SABER-Student Assessment is to promote stronger assessment systems that contribute to Examinations provide a basis for selecting or certifying improved education quality and learning for all. students as they move from one level of the education system to the next (or into the workforce). All eligible National governments and international agencies are students are tested on an annual basis (or more often if increasingly recognizing the key role that assessment of the system allows for repeat testing). Examinations student learning plays in an effective education system. cover the main subject areas in the curriculum and The importance of assessment is linked to its role in: usually involve essays and multiple-choice questions. (i) providing information on levels of student learning and achievement in the system; Large-scale, system-level assessments provide feedback (ii) monitoring trends in education quality over on the overall performance of the education system at time; particular grades or age levels. These assessments (iii) supporting educators and students with real- typically cover a few subjects on a regular basis (such as time information to improve teaching and every 3 to 5 years), are often sample based, and use learning; and multiple-choice and short-answer formats. They may be (iv) holding stakeholders accountable for results. national or international in scope. Appendix 1 summarizes the key features of these main types of assessment activities. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 2 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Quality drivers of an assessment system Table 1: Framework for building an effective assessment system, with indicator areas The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or The indicators are identified based on a combination of policy framework for assessment activities; institutional criteria, including: and organizational structures for designing, carrying x professional standards for assessment; out, or using results from the assessment; the x empirical research on the characteristics of effective availability of sufficient and stable sources of funding; assessment systems, including analysis of the and the presence of trained assessment staff. characteristics that differentiate between the assessment systems of low- versus high-performing System alignment refers to the extent to which the nations; and assessment is aligned with the rest of the education x theory — that is, general consensus among system. This includes the degree of congruence experts that it contributes to effective between assessment activities and system learning assessment. goals, standards, curriculum, and pre- and in-service teacher training. Levels of development Assessment quality refers to the psychometric quality of The World Bank has developed a set of the instruments, processes, and procedures for the standardized questionnaires and rubrics for collecting assessment activity. It covers such issues as design and and evaluating data on the three assessment types implementation of assessment activities, analysis and and related quality drivers. interpretation of student responses to those activities, and the appropriateness of how assessment results are The questionnaires are used to collect data on the reported and used. characteristics of the assessment system in a particular country or system. The information from the Crossing the quality drivers with the different questionnaires is then applied to the rubrics in order to assessment types/purposes provides the framework judge the development level of the country’s and broad indicator areas shown in Table 1. This assessment system in different areas. framework is a starting point for identifying indicators that can be used to review assessment systems and The basic structure of the rubrics for evaluating plan for their improvement. data collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice in each area. For each indicator, the rubric displays four development SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 3 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 levels—Latent, Emerging, Established, and Advanced. progress in poverty reduction have slowed significantly These levels are artificially constructed categories over the past five years due to a weak macroeconomic chosen to represent key stages on the underlying environment, political and security conditions, and continuum for each indicator. Each level is catastrophic natural shocks. accompanied by a description of what performance on the indicator looks like at that level. Sindh is the second-largest province in Pakistan and has the second-highest income among the four provinces in x Latent is the lowest level of performance; it the country. School participation in Sindh, as in the rest represents absence of, or deviation from, the of the country, has shown very slow improvement over desired attribute. the past several years, especially for girls in rural areas. x Emerging is the next level; it represents partial In 2010/2011, primary net enrollment rate was 62 presence of the attribute. percent and secondary net enrollment rate was 35 x Established represents the acceptable minimum percent in Sindh. The level of student achievement is standard. also low; a 2009 diagnostic assessment of grade 4 students in mathematics and language showed an x Advanced represents the ideal or current best average score of less than 50 percent. practice. The Sindh government recognizes poor sector A summary of the development levels for each governance and accountability, and weak administrative assessment type is presented in Appendix 3. systems as principal problems in the education sector. To address these challenges, in 2006/07, the In reality, assessment systems are likely to be at Government of Sindh initiated a multi-faceted Sindh different levels of development in different areas. For Education Sector Reform Program (SERP) with support example, a system may be Established in the area of from the World Bank. The objectives of this program examinations, but Emerging in the area of large- were to increase school participation and student scale, system-level assessment, and vice versa. While progression, reduce gender and rural-urban disparities, intuition suggests that it is probably better to be and improve the measurement of student learning. further along in as many areas as possible, the evidence is unclear as to whether it is necessary to Detailed information was collected on Sindh province’s be functioning at Advanced levels in all areas. student assessment system using the SABER-Student Therefore, one might view the Established level as a Assessment questionnaires and rubrics in 2012. This desirable minimum outcome to achieve in all areas, but SABER-Student Assessment report reflects the status of only aspire beyond that in those areas that most the student assessment system in the Sindh province as contribute to the national vision or priorities for of 2012. It is important to remember that these tools education. In line with these considerations, the ratings primarily focus on benchmarking a system’s policies and generated by the rubrics are not meant to be additive arrangements for assessment activities at the system or across assessment types (that is, they are not meant to macro level. Additional data would need to be collected be added to create an overall rating for an assessment to determine actual, on-the-ground practices in the system; they are only meant to produce an overall Sindh province, particularly by teachers and students in rating for each assessment type). The methodology for schools. The following sections discuss the findings by assigning development levels is summarized in each assessment type, accompanied by suggested Appendix 4. policy options. The suggested policy options were determined in collaboration with key local stakeholders Education in the Sindh province based on Sindh province’s immediate interests and needs. Detailed, completed rubrics for each assessment Pakistan is a lower-middle-income country in South type in the Sindh province are provided in Appendix 5. Asia. Current GDP per capita (US, 2012) is $1,257 with annual growth of 4 percent. Economic growth and SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 4 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Classroom Assessment tend to provide little feedback on student performance to students and parents. Classroom assessment results are often used as an administrative tool rather than a Level of Development: EMERGING pedagogical resource as classroom assessment information is primarily used internally to promote or In the Sindh province, the 2006 National Curriculum fail students. document provides general, non-comprehensive guidelines on classroom assessment. The document There are currently no mechanisms in place to provides information on assessment purposes, types systematically monitor the quality of classroom and questions, and includes some sample test content assessment practices. and scoring criteria. Although available online, this document is not offered through other more easily Suggested policy options: accessible channels for teachers and other stakeholders (such as teacher training courses or libraries). 1. Make the curriculum document more easily accessible to stakeholders through, for example, The 2006 National Curriculum document has not been libraries, teacher training courses, and schools. provided to all practicing classroom teachers, and because not all teachers have access to the internet, 2. Develop detailed guidelines for teachers on how to many teachers are not aware of the guidelines provided carry out classroom assessment activities to support in the document. In addition to the 2006 National student learning. Curriculum document, there are scarce system-wide resources available for teachers to engage in classroom 3. Ensure more effective uses of classroom assessment activities. Existing resources include assessment information by: (i) communicating to textbooks, which contain questions at the end of the teachers appropriate uses of the data; and (ii) chapters, and guidelines for teachers to use when making it compulsory for classroom assessment conducting classroom assessment activities. However, information to be disseminated to key stakeholders. the majority of these textbooks are based on a previous version of the national curriculum that was developed in 2002, and all textbooks have not been updated based on the 2006 curriculum. There are some system-level mechanisms in place (such as pre- and in-service teacher training opportunities) that are meant to ensure that teachers develop appropriate skills and expertise in classroom assessment. For example, a course on assessment is also offered in both undergraduate and graduate education programs. However, on-the-ground classroom assessment practices are generally considered to be weak. There are no required uses of classroom assessment information to support student learning. Classroom assessment practices tend to focus on students recalling information, and teachers often rely on the questions and the end-of-chapter exercises provided in the textbooks to assess students. Teachers generally do not use explicit criteria for evaluating students’ work, and SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 5 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Examinations examination design, administration, data processing and data reporting. Level of Development: EMERGING There is a permanent staff responsible for running the examination, but it is insufficient to meet the needs of The Secondary School Certificate examination is the examination due to staff members’ lack of technical administered to students in Grades 9 and 10. The Sindh capacity. Issues identified with the performance of the Boards of Intermediate and Secondary Education human resources responsible for the examination Ordinance of 1972 and the Sindh Boards of include item errors, repetition, limited skills tested and Intermediate and Secondary Education (Amendment) inadequate curriculum coverage. Act of 2008 authorize the examination. The Examination Rules and Procedures document (known as the In general, comprehensive material to help prepare for “Calendar”) for each Sindh Board covers key aspects of the examination is accessible to most students. Such the exam including the rules for preparation, the format material includes the textbook (which is considered the of the examination, and the procedures to investigate main resource) and previous examination papers (as inappropriate behavior. there is little repetition of examination questions from year to year). The Secondary School Certificate examination is intended to measure student attainment of the There is no technical documentation on the knowledge and skills outlined in the 2006 National examination and there are no mechanisms in place to Curriculum. In practice, the examination tests textbook ensure the quality of the examination. Inappropriate knowledge. At the same time, textbooks are not well behavior surrounding the examination process is aligned with the curricula due to two main issues: (1) moderate and includes leakage of the content of the majority of the textbooks in circulation are based on a paper prior to the examination, and copying and using previous version of the national curricula developed in unauthorized materials such as prepared answers and 2002 (new textbooks based on the new curricula notes. As a result, the credibility of the examination and developed in 2006 are in various stages of its results are often called into question by many development) and (2) textbooks are not perceived to stakeholders. adequately cover all aspects of the curricula such as higher order thinking skills. Thus, there is no clear link Suggested policy options: between the examination and the student learning outcomes or the skills identified in the curriculum, 1. Build the capacity of individuals responsible for resulting in inadequate coverage and the lack of higher examination activities, especially in the areas of test order skills tested. design, curriculum alignment, and report writing. This could be done by: (i) making training Efforts to improve the examination have been led by opportunities available within the Boards; and (ii) the government and by donors. For example, several allocating funding for attending international efforts to improve aspects of the examinations have courses or workshops. been made by the Boards along with the Education 2. Develop a technical report on the examination and Department, including the introduction of multiple- disseminate it to key stakeholders. choice and short answer questions. The Boards have also tried to ensure that the examiners marking the 3. Identify and agree upon the official purposes and papers are from the same language background as the possible uses of examination results and introduce language of the paper they are marking. mechanisms, such as an oversight committee, to monitor actual uses. Funding for the examination is provided by student 4. Institutionalize preventive and reactive mechanisms fees, and covers all core examination activities including to address inappropriate behavior during the examination process. Ensure that the mechanisms SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 6 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 are clearly defined, and are communicated through Educational Development (AKU-IED) also offers a course various communication methods (for example, on assessment at the M.Ed level. through documents, the radio, etc.) to all key stakeholders in a timely manner. The PEACE assessment measures competencies National Large-Scale Assessment (NLSA) identified in the curriculum. However, since assessments are based on the 2006 curricula and the textbooks available are based on the previous curricula Level of Development: EMERGING (developed in 2002), some skills assessed on PEACE are not available in the textbooks and could cause some The Provincial Education Assessment Center (PEACE) confusion amongst teachers who only use the textbook Sindh Assessment was administered for the first time in to teach students. 2008 as an independent, provincial-level assessment exercise. Previously, PEACE was part of the National Various mechanisms are in place to ensure the quality Education Assessment System (NEAS). Under NEAS, the of the PEACE, including: (i) all administrators are trained Sindh province administered the assessment to a according to a protocol; (ii) there is a standardized provincially representative sample of students and manual for administrators; (iii) a pilot is conducted published a provincial report separate from the national before the main data collection takes place; (iv) all report. booklets are numbered; and (v) external and internal reviewers and observers are employed. Coding and Regular funding for PEACE is allocated by the provincial scoring is a centralized activity, conducted in one place government. Funding covers core PEACE activities, but and overseen by persons involved in data analysis. does not cover research and development activities. Training of scorers, provision of code sheets and rubrics Various donors are also providing technical assistance for open-ended items, and a 10 percent check of for improving the design of the PEACE assessment and scoring by another person also takes place. for the training of PEACE administrators. Computerized data is also compared to the data entered manually to cross-check for errors. Although the PEACE office is a permanent agency working under the Bureau of Curriculum, it is In general, PEACE results are poorly disseminated, and inadequately staffed to effectively carry out assessment their use has been minimal. activities. In general, there is a lack of qualified staff who can ensure that tasks are conducted in an efficient Suggested policy options: and timely manner. Specifically, there is lack of technical staff, pyschometricians, and statisticians for 1. Develop the capacity of PEACE staff by (i) data analysis. There is also lack of clarity on the role introducing workshops and trainings on education that focal persons in the districts are to have in test measurement and evaluation, including, in administration and monitoring. No final review of particular, on data analysis; (ii) providing funding to assessment instruments takes place, which results in PEACE staff to attend international programs on errors when the instruments are printed; there is also a educational measurement and evaluation; and (iii) need for translation verification between Urdu and identifying external human resources for Sindhi. conducting translation verifications between Urdu and Sindhi. Introduce a mandate for capacity The Sindh province offers some opportunities to development of PEACE staff. Identify which prepare individuals for work on the PEACE. Graduate individuals or organizations will be responsible for and non-graduate university courses on educational the workshops and trainings, and provide them with measurement and evaluation are made available. The regular funding for and access to the necessary University of Sindh offers a course on educational training resources and key experts (at the measurement and evaluation in its B.Ed and M.Ed provincial, national, and international levels). programs. The Aga Khan University Institute for SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 7 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 2. Identify, with the participation of relevant persons at the district level, the intended uses of PEACE results; develop and implement a plan to disseminate the results; introduce mechanisms, such as an oversight committee, to monitor actual uses of PEACE results. International Large-Scale Assessment (ILSA) Level of Development: LATENT The Sindh province has not participated in an ILSA, and it does not have plans to do so in the near future. Suggested policy options: 1. Create an opportunity for high-level discussion among key stakeholders on key education policy questions or problems for which ILSA data could be useful. 2. Determine the need for, and possible next steps in relation to, participation in an ILSA exercise. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 8 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations Surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select immediate feedback on feedback on the students as they students for feedback to overall health of comparative move from one further inform the system at performance of level of the educational classroom particular the education education system opportunities instruction grade/age system at to the next (or level(s), and to particular into the monitor trends in grade/age workforce) learning level(s) Frequency Daily For individual For individual Annually and Annually and subjects offered subjects offered more often more often on a regular on a regular where the system where the system basis (such as basis (such as allows for allows for every 3-5 years) every 3-5 years) repeats repeats Who is All students Sample or A sample of All eligible All eligible tested? census of students at a students students students at a particular grade particular grade or age level(s) or age level(s) Format Varies from Usually multiple Usually multiple Usually essay Usually essay observation to choice and short choice and short and multiple and multiple questioning to answer answer choice choice paper-and-pencil tests to student performances Coverage of All subject areas Generally Generally Covers main Covers main curriculum confined to a few confined to one subject areas subject areas subjects or two subjects Additional Yes, as part of Frequently Yes Seldom Seldom information the teaching collected from process students? Scoring Usually informal Varies from Usually involves Varies from Varies from and simple simple to more statistically simple to more simple to more statistically sophisticated statistically statistically sophisticated techniques sophisticated sophisticated techniques techniques techniques SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 9 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development Level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—ENABLING CONTEXT EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—SYSTEM ALIGNMENT SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—ASSESSMENT QUALITY AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 10 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Appendix 3: Summary of the Development Levels for Each Assessment Type Assessment Type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system- institutional capacity to wide institutional system-wide institutional wide institutional support and ensure the capacity to support and capacity to support and capacity to support and quality of classroom ensure the quality of ensure the quality of ensure the quality of assessment practices. classroom assessment classroom assessment classroom assessment practices. practices. practices. CLASSROOM ASSESSMENT There is no standardized There is a partially There is a stable There is a stable examination in place for stable standardized standardized standardized key decisions. examination in place, examination in place. examination in place and and a need to develop There is institutional institutional capacity and institutional capacity to capacity and some strong mechanisms to run the examination. The limited mechanisms to monitor it. The EXAMINATIONS examination typically is monitor it. The examination is of high of poor quality and is examination is of quality and is perceived perceived as unfair or acceptable quality and is as fair and free from corrupt. perceived as fair for corruption. most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA There is a stable NLSA place. NLSA in place and a in place. There is in place and institutional need to develop institutional capacity and capacity and strong institutional capacity to some limited mechanisms to monitor run the NLSA. mechanisms to monitor it. The NLSA is of high NATIONAL (OR SYSTEM- Assessment quality and it. The NLSA is of quality and its LEVEL) LARGE-SCALE impact are weak. moderate quality and its information is ASSESSMENT information is effectively used to disseminated, but not improve education. always used in effective ways. There is no history of Participation in an ILSA There is more or less There is stable participation in an ILSA has been initiated, but stable participation in an participation in an ILSA nor plans to participate there still is need to ILSA. There is and institutional capacity in one. develop institutional institutional capacity to to run the ILSA. The capacity to carry out the carry out the ILSA. The information from the INTERNATIONAL LARGE- ILSA. information from the ILSA is effectively used SCALE ASSESSMENT ILSA is disseminated, to improve education. but not always used in effective ways. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 11 SINDH PROVINCE, PAKISTAN | STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Appendix 4: Methodology for Assigning the score for these key dimensions. These key Development Levels variables include formal policy, regular funding, having a permanent assessment unit, and the quality 1. The country team or consultant collects of assessment practices. information about the assessment system in the country or system. 2. Based on the collected information, a level of development and score is assigned to each dimension in the rubrics: x Latent = 1 score point x Emerging = 2 score points x Established = 3 score points x Advanced = 4 score points 3. The score for each quality driver is computed by aggregating the scores for each of its constituent dimensions. For example: The quality driver, ‘Enabling Context,’ in the case of ILSA, has 3 dimensions on which a hypothetical country or system receives the following scores: Dimension A = 2 points; Dimension B = 2 points; Dimension C = 3 points. The hypothetical country’s or system’s overall score for this quality driver would be: (2+2+3)/3 = 2.33 4. A preliminary level of development is assigned to each quality driver. 5. The preliminary development level is validated using expert judgment in cooperation with the country team and The World Bank Task Team Leader. For scores that allow a margin of discretion (i.e., to choose between two levels of development), a final decision has to be made based on expert judgment. For example, the aforementioned hypothetical country or system has an ‘Enabling Context’ score of 2.33, corresponding to a preliminary level of development of ‘Emerging or Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses ‘Emerging’ as the most appropriate level. 6. Scores for certain key dimensions under ‘Enabling Context’ (in the case of EXAM, NLSA, and ILSA) and under ‘System Alignment’ (in the case of CLASS) were set as ceiling scores, i.e., the overall mean score for the particular assessment type cannot be greater than SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 12 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Appendix 5: SABER-Student Assessment Rubrics Sindh Province, Pakistan This appendix provides the completed SABER-Student Assessment rubrics for each type of assessment activity in the Sindh province, Pakistan. In each row of the rubric, the relevant selection is indicated by a thick border and an asterisk. The selection may include a superscript number that refers to the justification or explanation for the selection (as indicated by a thick border and an asterisk). The explanation or justification text can be located in the “Development-level rating justifications” section at the end of each rubric. If a row includes a superscript, but not a thick border and an asterisk, this means that insufficient information was available to determine the relevant selection in the row. SINDH PROVINCE, PAKISTAN Classroom Assessment SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 13 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. 1 assessment. classroom assessment. assessment. * This option does not apply to this This option does not apply to this The availability of the document is The document is widely available. 2 dimension. dimension. restricted. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide 3 teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment. resources available for teachers for classroom assessment. * There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or students are expected to learn, but the students are expected to learn and to to what level of performance. level of performance required is not what level of performance. 4 clear. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms This option does not apply to this There are some system-level There are a variety of system-level to ensure that teachers develop skills dimension. mechanisms to ensure that teachers mechanisms to ensure that teachers and expertise in classroom assessment. develop skills and expertise in classroom develop skills and expertise in classroom 5 assessment. assessment. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 14 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are 6 from widespread weaknesses or there is known to be weak. known to be of moderate quality. known to be generally of high quality. no information available on classroom assessment practices. There are no mechanisms to monitor the There are ad hoc mechanisms to monitor There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the practices. practices. assessment practices. quality of classroom assessment practices. * ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key 7 stakeholders. stakeholders. stakeholders. * There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support classroom assessment to support classroom assessment to support student student learning. student learning, excluding its use as an learning, including its use as an input for input for external examination results. external examination results. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 15 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Classroom Assessment: Development-level rating justifications 1. The only system-level document that provides some recommendations on assessment practices is the 2006 National Curriculum, which provides general information on assessment purposes, types, and questions, as well as some sample test content and scoring criteria. These are not, however, comprehensive guidelines. Additionally, teachers in schools have not been provided the curriculum document, for the most part, and are therefore not aware of these guidelines. 2. The document is available online. However, the document is not available through other channels, such as libraries and teacher training courses and colleges, which teachers and other stakeholders can easily access. 3. Textbooks or workbooks that provide support for classroom assessment are typically available on a system-wide basis to teachers. Textbooks contain questions at the end of the chapter; there also are guidelines for teachers on conducting class activities for assessment purposes. For example, it suggests that teachers hold speech competitions, form groups and give them various tasks, etc. The 2006 National Curriculum document, although available on the Ministry of Education website, has not been provided to all practicing classroom teachers. Not all teachers have access to the internet and can download large documents. The textbook board has yet to develop new textbooks based on the 2006 curriculum. 4. Currently, this is the 2006 National Curriculum. After the 18th constitutional amendment, the authority to develop curriculum has been transferred to the provinces, and Sindh is in the final stages of adopting a new curriculum. 5. Available system-level mechanisms include pre-service and in-service teacher training. A course on assessment is offered in the B.Ed and M.Ed program. 6. It is very common for classroom assessment activities to be mainly about recalling information. Teachers mostly develop questions directly from the textbook and use the exercises at the end of the chapters. Teachers also do not use explicit or a priori criteria for scoring or grading students' work. Parents are poorly informed about students' grades, which, in turn, are very school-dependent. Sharing annual exam results with parents, however, is very common. Classroom assessment activities also provide little useful feedback to students, as students are told only their overall test results. Classroom assessment activities are also mainly used as an administrative or control tool rather than a pedagogical resource, and for the most part are used internally to promote or fail students. (Textbooks are generally not aligned with the curricula because the textbooks in circulation are based on a previous version of the national curricula that was developed in 2002. There are no new textbooks that are based on the new curricula that went into effect in 2006.) 7. Results are reported to the district education department. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 16 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 SINDH PROVINCE, PAKISTAN Examinations SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 17 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this place. operating on an irregular basis. has been operating regularly. dimension There is no policy document that There is an informal or draft policy There is a formal policy document that * This option does not apply to this 1 authorizes the examination. document that authorizes the authorizes the examination. dimension. examination. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 2 dimension. the public. public. dimension. This option does not apply to this This option does not apply to this * The policy document addresses some The policy document addresses all key 3 dimension. dimension. key aspects of the examination. aspects of the examination. * ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the 4 the examination or are indifferent to it. examination. examination. examination. There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder 5 groups. groups. * Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this not welcomed by the leadership in dimension. generally welcomed by the leadership in dimension. 6 charge of the examination. charge of the examination. (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 18 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this 7 examination. the examination. examination. dimension. * This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. 8 processing or reporting. processing and reporting. * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and 9 dimension. development. dimension. development. * ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or The examination office is newly The examination office is a stable This option does not apply to this 10 is newly established. established. organization. dimension. * The examination office is not This option does not apply to this The examination office is accountable to This option does not apply to this 11 accountable to an external board or dimension. an external board or agency. dimension. agency. * Examination results are not recognized Examination results are recognized by Examination results are recognized by Examination results are recognized by by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection 12 country/system. another country/system. system in another country/system. * The examination office does not have The examination office has some of the The examination office has all of the The examination office has state of the the required facilities to carry out the required facilities to carry out the required facilities to carry out the art facilities to carry out the examination. 13 examination. examination. examination. (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 19 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment 14 examination, issues are pervasive. effectively, with minimal issues. effectively, with no issues. * The country/system does not offer This option does not apply to this The country/system offers some The country/system offers a wide range opportunities that prepare for work on dimension. opportunities that prepare for work on of opportunities that prepare for work 15 the examination. the examination. on the examination. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 20 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what This option does not apply to this 16 measures. dimension. the examination measures. dimension. What the examination measures is This option does not apply to this * What is measured by the examination is This option does not apply to this questioned by some stakeholder groups. dimension. largely accepted by stakeholder groups. dimension. 17 * Material to prepare for the examination There is some material to prepare for the There is comprehensive material to There is comprehensive material to is minimal and it is only accessible to examination that is accessible to some prepare for the examination that is prepare for the examination that is 18 very few students. students. accessible to most students. accessible to all students. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses There are up-to-date compulsory courses examinations available to teachers. workshops on examinations available to or workshops on examinations available or workshops on examinations for teachers. to teachers. teachers. Teachers are excluded from * all Teachers are involved in very few Teachers are involved in some Teachers are involved in most 19 examination-related tasks. examination-related tasks. examination-related tasks. examination-related tasks. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 21 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical There is a comprehensive, high quality documentation. examination, but it is not in a formal report but with restricted circulation. technical report available to the general report format. public. * There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the examination. quality of the examination. * ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the 20 examination process is high. examination process is moderate. examination process is low. examination process is marginal. The examination results lack credibility * The examination results are credible for The examination results are credible for This option does not apply to this 21 for all stakeholder groups. some stakeholder groups. all stakeholder groups. dimension. * The majority of the students (over 50%) A significant proportion of students A small proportion of students (less than All students can take the examination; may not take the examination because of (10%-50%) may not take the examination 10%) may not take the examination there are no language, gender or other language, gender, or other equivalent because of language, gender, or other because of language, gender, or other equivalent barriers. 22 barriers. equivalent barriers. equivalent barriers. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 22 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all 23 proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way. * Student names and results are public. This option does not apply to this Students’ results are confidential. This option does not apply to this dimension. dimension. * ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who There are very limited options for There are some options for students who There is a variety of options for students do not perform well on the examination, students who do not perform well on the do not perform well on the examination. who do not perform well on the 24 or students must leave the education examination. examination. system. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the examination. examination. examination. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 23 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Examinations: Development-level rating justifications 1. Sindh Boards of Intermediate & Secondary Education Ordinance 1972 (amended in 1973, 1974, 1978), and most recently the Sindh Boards of Intermediate & Secondary Education (Amendment) Act 2008 (Sindh Act No. XXV of 2011). 2. The acts and ordinances, as well as the Calendar, are technically public documents, but not necessarily easily accessible. Both types of documents should be available on request at the Boards. However, there does not appear to be a clear procedure for making such a request. When a request for these documents was made, the Board did not supply the documents. 3. The act outlines the governance, distribution of power, and responsibilities among key stakeholders. The Calendar outlines the procedures to investigate and address security breaches, cheating, or other forms of inappropriate behavior, and the procedures for special/disadvantaged students, while specifying who can sit for the examination. It also identifies the rules about the preparation and format of the examination. 4. There is little publicly available evidence on stakeholder support or opposition, and there is very little public debate on this issue. 5. Efforts to improve the examination have been led by the government and most recently, by donors. For example, there have been several efforts to improve different aspects of the examinations by the Boards along with the Education Department. The Inter Board Committee of Chairmen (IBCC) has changed the paper pattern in recent years to include multiple choice and short answer questions. The Boards have tried to ensure that the examiners marking the papers are from the same language background as the language of the paper they are marking. There appear to be efforts by the Aga Khan University Examination Board (AKUEB) and USAID-supported EDLINKS project to improve the quality of the exam paper items and curriculum coverage in Boards across Pakistan. However, this is a fairly new initiative and apart from a workshop with paper setters from the Boards, it’s not clear what other activities have been implemented. Only Karachi Board respondents mentioned they may be working on developing an item bank in the future. 6. The Boards at the higher levels appear to be aware of the various issues in the examination system and as a result have taken steps to address the issues themselves. However, since efforts to improve the examination from outside the government, such as those by the AKU-EB, have either not occurred or not been implemented as yet, it is difficult to say whether they would be receptive to such changes. 7. The funding is from student fees. 8. Data analysis and data reporting are limited to development and reporting of the exam results for individual candidates. 9. The Karachi Board has a research section; however, this section does not necessarily conduct research. Its functions are usually to develop descriptive analysis of exam results (i.e., by gender or academic groups or over the last 10 years) and handle non-research activities such as scholarships, debates and essay competitions. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 24 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 10. The oldest Board, the Karachi Board of Secondary & Intermediate Education, was established in 1951, while the most recent Board, the Mirpur Khas Board of Secondary & Intermediate Education, was established in 2004. 11. Accountable to the Governor of Sindh province. 12. The IBCC / Foreign Ministry attests to the use of the Secondary School Certificates (SSC) abroad. Although the SSC exam would not be used for admission into university per se, it would presumably form part of the secondary school education record for admissions. 13. The consultant only visited Karachi and Hyderabad Boards. Given that the Boards are located in very different areas, it is difficult to determine whether these responses in the questionnaire, particularly for technological facilities, are representative of the other Boards as well. 14. While there is permanent full-time staff for running the examination, it is insufficient to meet the needs of the examination. For example, there is a lack of staff who can perform research and data analysis, as well as a lack of technical staff with skills in examination design to guide the paper development process. Issues that have been identified with the performance of the human resources that are responsible for the examination include item errors, repetition, limited skills tested, and inadequate curriculum coverage. 15. Graduate and non-graduate university courses on educational management and evaluation are offered. The University of Sindh, for example, offers a course on educational measurement and evaluation in its B.Ed and M.Ed programs. 16. Officially, the examination seeks to measure the national curriculum. However, there is no clear link to the student learning outcomes or the skills identified in the curriculum, often resulting in inadequate coverage and the lack of higher order skills tested as noted by research studies. In practice, it is essentially the textbook content that is measured by the exam. (Textbooks cannot be said to be well aligned with the curricula: (1) The majority of the textbooks in circulation are based on a previous version of the national curricula developed in 2002 (new textbooks based on the new curricula, i.e. 2006, are in various stages of development), and (2) they are not perceived to adequately cover all aspects of the curricula such as higher order thinking skills.) 17. Different stakeholders have different concerns. Many stakeholders (such as parents, students, and the general public) are more concerned with issues related to cheating, speed, and accuracy of scoring. However, those who recognize that there are inadequacies in the curriculum coverage and skills tested, such as respondents at the Reform Support Unit, do not accept what is measured. 18. Materials required for preparing for the exam include the textbook (the main resource, as the exam is essentially textbook-based) and past papers (as there is apparently not a lot of repetition in exam questions from year to year). Textbooks in government schools are now provided free of cost. Respondents noted that the textbooks are of good quality and that was all that students need to prepare for the exams. 19. Retired teachers select or create examination questions, while active teachers administer and score the examination, supervise the examination procedures, and act as a judge in, for example, orals. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 25 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 20. Various inappropriate behaviors diminish the credibility of the examinations. Such behaviors include leakage of the content of an examination paper or part of a paper prior to the examination, impersonation, copying from other candidates, using unauthorized materials such as prepared answers and notes, collusion among candidates via mobile phones, passing of paper, or equivalent, and provision of external assistance via the supervisor, mobile phone, etc. To reduce inappropriate behavior, the Boards this year sought to adhere to the scheduled registration date to prevent students from manipulating their center allocation. Also, in addition to the Board’s supervisory staff monitoring, there are governor-appointed Special Vigilance Teams, consisting of retired teachers, who monitor and provide independent reports on conduct. To address the leakage of exam papers, there is a secure room for developing papers and processes for preventing leakage at that stage. In Hyderabad, papers are made available at the Banks and center staff must collect and deposit filled papers on the day of the exam. And in Karachi, the papers are delivered to the exam centers by Board representatives themselves. Finally, answer books are coded so that the identity of the candidate is not known to the examiner. 21. The credibility of the exams and its results are called into question with regards to the need for quality improvement at all levels (exam design, administration, scoring, and determining results) and the perceptions in the general public that cheating is still widespread. The views of stakeholders tend to vary in terms of their area of concern. Media reports tend to focus on the conduct (i.e., occurrence of cheating) and accuracy or delays in results, as do parents and students. This year, there has been a substantial amount of criticism over the computerization of the process. Staff within the Boards, government departments, and teachers appears to be concerned with issues in marking. Finally, from reports and interviews, stakeholders such as those in teacher education institutions, NGOs, and selected government officials and educators point out the poor quality of the exam paper and its questions. Universities do not appear to trust the quality of the exams and take their own entrance exams. 22. Respondents have noted that if a student manages to make it to secondary school (i.e., does not drop out at earlier stages or face access-related barriers), then fees or location should not serve as a barrier to take the examinations, as it is the key certification for proving that they have completed secondary school. Exam centers are usually close to the students' schools, and there are no registration barriers based on ethnicity or gender. 23. Because it is unclear what the official purposes and uses of examination results are, it is difficult to identify "improper" uses. Generally, however, exam results are not used beyond determining pass/fail and admission into the next level of education, which are in line with the intended purposes. 24. Students may retake the examination, may opt for less selective schools/universities/tracks, or they may repeat the grade. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 26 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 SINDH PROVINCE, PAKISTAN National (or System-Level) Large-Scale Assessment (NLSA) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 27 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this 1 irregular basis. been operating regularly. dimension. * There is no policy document pertaining There is an informal or draft policy There is a formal policy document that This option does not apply to this 2 to NLSA. document that authorizes the NLSA. authorizes the NLSA. dimension. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. dimension. * There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the 3 dimension. NLSA will take place. coming years. * ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the NLSA. 4 the NLSA or are indifferent to it. NLSA. NLSA. (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 28 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the There is irregular funding allocated to There is regular funding allocated to the This option does not apply to this 5 NLSA. the NLSA. NLSA. dimension. This option does not apply to this Funding covers some core NLSA * Funding covers all core NLSA activities: This option does not apply to this dimension. activities: design, administration, analysis design, administration, analysis and dimension. and reporting. reporting. This option does not apply to this Funding does not cover research and This option does not apply to this * Funding covers research and dimension. development activities. dimension. development activities. * ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this team. group of people. institution or unit. dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes * Political considerations never hamper dimension. technical considerations. hamper technical considerations. technical considerations. This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a This option does not apply to this * 6 dimension. clearly recognized body. clearly recognized body. dimension. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 29 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed The NLSA office is adequately staffed to The NLSA office is adequately staffed to 7 NLSA. to effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no minimal issues. issues. * The country/system does not offer This option does not apply to this The country/system offers some The country/system offers a wide range opportunities that prepare individuals dimension. opportunities to prepare individuals for of opportunities to prepare individuals 8 for work on NLSA. work on the NLSA. for work on the NLSA. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 30 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this curriculum or learning standards. dimension. curriculum or learning standards. dimension. * What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned What the NLSA measures is largely 9 questioned by stakeholder groups. dimension. by some stakeholder groups. accepted by stakeholder groups. * There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it dimension. 10 measures what it is supposed to intended to measure. is intended to measure. measure. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on There are occasional courses or There are some courses or workshops on There are widely available high quality 11 the NLSA. workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 31 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include Different options are offered to include 12 groups of students in the NLSA. dimension. all groups of students in the NLSA. all groups of students in the NLSA. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in 13 ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. place to ensure the quality of the NLSA. There is no technical report or other There is some documentation about the There is a comprehensive technical There is a comprehensive, high quality * documentation about the NLSA. technical aspects of the NLSA, but it is report but with restricted circulation. technical report available to the general not in a formal report format. public. * ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA 14 NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way. dimension. * NLSA information is not used or is used This option does not apply to this NLSA results are used by some NLSA information is used by all in ways inconsistent with the purposes dimension. stakeholder groups in a way that is stakeholder groups in a way that is or the technical characteristics of the consistent with the purposes and consistent with the purposes and 15 assessment. technical characteristics of the technical characteristics of the assessment. assessment. There are no mechanisms in place to * This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA. place to monitor the consequences of 16 the NLSA. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 32 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 National (or System-Level) Large Scale Assessment (NLSA): Development-level rating justifications 1. The first year that the Provincial Education Assessment Center (PEACE) Sindh assessment was administered was in 2008. PEACE Sindh was previously part of NEAS, established to conduct assessments to a provincially representative sample as part of a national study. In 2008, it began conducting assessments independently. 2. There is the original PEACE PC-1 approved by Sindh Education and Literacy Department for the period 2003-2009 and a re-modified PC-1, which includes a three-year plan for 2009-12. 3. The plan for the next three years, 2012-2015, has been drafted and is awaiting approval by the Secretary of Education. 4. There does not appear to be any publicly available evidence for stakeholder perceptions of the large-scale assessments. 5. There is regular funding allocated by the government, as well as technical assistance (for assessment design and training of assessment administrators) from the EU. 6. The NLSA office is accountable to the Sindh Department of Education and Literacy. 7. There is permanent or full-time staff, but it is insufficient to meet the needs of the assessment. Generally, there is lack of qualified staff who can ensure that the tasks are conducted in an efficient and timely manner. Specifically, according to respondents, there is a lack of technical staff, psychometricians, and statisticians for data analysis. The issues that have been identified include poor training of test administrators or unclear instructions and guidelines for administering the assessment, weaknesses in test design, and frequent errors in the test questions. It is difficult to find relevant/good-quality persons to develop items, particularly the kind that go beyond the textbook and assess higher order thinking skills. There is also no in-house capacity to develop the sample, and the available in-house persons lack the necessary skills to benefit from technical assistance provided. Finally, data management and storage of scripts is also a key issue. Generally, there is a lack of clarity on the part of focal persons in the districts with regards to test administration and monitoring. There is also no final review of tools, which results in some errors in the printed tools and a need for translation verification between Urdu and Sindhi. Finally, there is lack of qualified staff for statistical analysis that caused delays. 8. Graduate and non-graduate university courses on educational measurement and evaluation are made available. The University of Sindh offers a course on educational measurement and evaluation in their B.Ed and M.Ed programs. The Aga Khan University Institute for Educational Development (AKU-IED) also offers a course on assessment at the M.Ed level. In the past, a specialized course was offered by the University of Melbourne in collaboration with the Institute of Educational Research (IER) for NEAS staff. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 33 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 9. The PEACE assessment measures competencies identified in the curriculum, and it has been conducting these assessments for several years, including under NEAS. Therefore, stakeholders are now familiar with it and are likely to accept it. However, since assessments are based on the 2006 curricula and the textbooks available are based on the previous curricula (developed in 2002), PEACE staff note that some skills are not available in the textbooks and could cause some confusion amongst teachers who are used textbook-based tests. 10. There are several reviews at different stages, during the development of the assessment framework, item development and after the pilot. These are internal reviews that include external experts from the universities, teacher professional development colleges, and schools. 11. PEACE has conducted 13 awareness workshops with teachers in which they shared results of the PEACE assessment and provided guidelines on assessment practices in general, such as how to develop them and how to develop reports, etc. 12. The NLSA is offered in the language of instruction for almost all student groups. Additionally, the sampling frame includes both urban and rural areas. It does not appear that any other plans have been made to ensure inclusion or exclusion of students from hard-to-reach areas. 13. Various mechanisms are in place to ensure the quality of the NLSA. Specifically, all proctors or administrators are trained according to a protocol, there is a standardized manual for large-scale assessment administrators, a pilot is conducted before the main data collection takes place, all booklets are numbered, and external and internal reviewers/observers are employed. Coding and scoring is a centralized activity, conducted in one place and overseen by persons involved in analysis. There is training of scorers, provision of code sheets and rubrics for open-ended items, and a 10 percent check of scoring by another person. In terms of processing of data, the computerized data is compared to the manual data to cross-check for errors. The persons involved in monitoring the administration of the assessment consisted of district focal persons and PEACE staff, as well as the Bureau of Curriculum and Provincial Institute of Teacher Education staff. 14. The NLSA results are disseminated in a variety of ways to different stakeholders. Reports containing results include a technical report, a general report for a general audience, a teacher feedback report, and a leaflet or flyer that provides brief information. For dissemination, they have conducted a provincial workshop in which they called the Executive District Education Officer from the districts and people from other provinces. In the future, they would like to hold district-based workshops as well. For NEAS assessment, there were also yearly reports and newsletters, stakeholder conferences, and coverage in the media. Dissemination of both the current PEACE and previous NEAS assessments is noted as one of the weaker aspects of the program by a current and former staff member. For the current PEACE assessment, one district officer noted that the relevant persons from the districts are not involved in dissemination, which has caused a lack of awareness in the assessment and its uses at the district level. 15. There has been little use of results by teachers, districts, or policymakers. Reasons cited include that there are no mechanisms for ensuring that recommendations are implemented or that the assessment results inform teacher practices. There has only recently been some interest by apex body Sindh Teacher Education Development Authority (STEDA) to use the results for teacher training. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 34 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 16. Working groups have been established under PEACE to look at the gaps in curricula in relation to results. While PEACE recognizes the need to conduct research, and there is now two years’ worth of data available, no initiatives have been made to conduct research. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 35 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 SINDH PROVINCE, PAKISTAN International Large-Scale Assessment (ILSA) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 36 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in The country/system has participated in 1 in an ILSA in the last 10 years. dimension. at least one ILSA in the last 10 years. two or more ILSA in the last 10 years. * The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. the next 5 years. the next 5 years. * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. dimension. ILSA. This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. dimension. ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, ILSA. donors. discretion. decree or norm. This option does not apply to this Funding covers some core activities of Funding covers all core activities of the This option does not apply to this dimension. the ILSA. ILSA. dimension. Funding does not cover research and This option does not apply to this This option does not apply to this Funding covers research and development activities. dimension. dimension. development activities. (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 37 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. activities. activities. activities. This option does not apply to this The national/system coordinator or The national/system coordinator is fluent This option does not apply to this dimension. other designated team member may not in the language of the assessment. dimension. be fluent in the language of the assessment. This option does not apply to this The ILSA office is inadequately staffed or The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, effectively. with minimal issues. with no issues. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 38 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this international workshops or meetings. international workshops or meetings. workshops or meetings. dimension. The country/system offers no This option does not apply to this The country/system offers some The country/system offers a wide range opportunities to learn about ILSA. dimension. opportunities to learn about ILSA. of opportunities to learn about ILSA. This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are dimension. dimension. available to the country's/system's ILSA available to a wide audience, in addition team members only. to the country's/system's ILSA team members. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 39 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical This option does not apply to this published. standards to have its data presented standards required to have its data dimension. beneath the main display of the presented in the main displays of the international report or in an annex. international report. The country/system has not contributed This option does not apply to this This option does not apply to this The country/system has contributed new new knowledge on ILSA. dimension. dimension. knowledge on ILSA. ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely the country/system. in the country/system. the country/system. disseminated in the country/system. Products to provide feedback to schools This option does not apply to this Products to provide feedback to schools Products to provide feedback to schools and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are not made available. sometimes made available. systematically made available. There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the There is wide media coverage of the ILSA results. ILSA results. ILSA results. results. If any, country/system-specific results Results from the ILSA are used in a Results from the ILSA are used in some Results from the ILSA are used in a and information from the ILSA are not limited way to inform decision making in ways to inform decision making in the variety of ways to inform decision used to inform decision making in the the country/system. country/system. making in the country/system. country/system. It is not clear that decisions based on This option does not apply to this This option does not apply to this Decisions based on the ILSA results have ILSA results have had a positive impact dimension. dimension. had a positive impact on students' on students' achievement levels. achievement levels. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 40 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 International Large Scale Assessment (ILSA): Development-level rating justifications 1. The Sindh province has not participated in an ILSA, and it does not have plans to do so in the near future. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 41 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 Acknowledgements References This report was prepared by The World Bank SABER- Clarke, M. 2012. “What Matters Most for Student Student Assessment team in collaboration with Assessment Systems: A Framework Paper.” Umbreen Arif, World Bank Senior Education Specialist READ/SABER Working Paper Series. Washington, D.C.: and Task Team Leader for education projects in World Bank. Pakistan. World Bank. 2012. Pakistan Country Indicator Data. Washington, D.C.: World Bank. Data retrieved from http://data.worldbank.org/ on March 10, 2014. World Bank. 2009. Project Appraisal Document on a Proposed Credit in the Amount of SDR 200.6 Million (US $300 Million Equivalent) to the Islamic Republic of Pakistan for a Sindh Education Sector Project (SEP). Report No. 47642-PK. Washington D.C.: The World Bank. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 43 SINDH PROVINCE, PAKISTAN ǀ SABER-STUDENT ASSESSMENT SABER PROVINCE REPORT | 2012 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policymakers and business people—an accessible, objective snapshot showing how well the policies of their country’s education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. THE WORLD BANK SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 44