79944 Zambia SABER Country Report STUDENT ASSESSMENT 2009 Key Policy Areas for Student Assessment Status 1. Classroom Assessment A formal and widely-available, system-level document provides guidelines to teachers for classroom assessment activities. Additionally, there appear to be adequate required uses of classroom assessment information to support student learning, including as an input to external examination results. At the same time, actual classroom assessment practices are considered to be weak and there are few system-wide resources available to teachers to help them engage in better quality classroom assessment practices. 2. Examinations The examinations program has been operating on a regular basis. A formal, publically-available, policy document authorizes the program and provides guidance on its key aspects. Regular funding for core examination activities is provided by the government. The examinations are run by a stable organization that has all of the required facilities to carry out examination activities. 3. National Large-Scale Assessment (NLSA) The NLSA program has been operating on an ongoing basis, with regular funding provided by the government. The funding covers all core NLSA activities as well as research and development. The NLSA office is inadequately staffed to effectively carry out the NLSA, and the country does not offer opportunities that prepare individuals for work on the NLSA. Efforts are made to include all student groups in the NLSA, including providing accommodations or alternative assessments for students with disabilities. NLSA results are poorly disseminated, and few teachers use them to inform their teaching or improve student learning. 4. International Large-Scale Assessment (ILSA) Zambia has participated in three SACMEQ surveys (I, II, and III) in the last ten years, and has taken concrete steps to participate in SACMEQ IV. Although some opportunities to learn about ILSAs are offered to SACMEQ team members, they are nevertheless inadequately trained to carry out this ILSA effectively. SACMEQ results are used in some ways to inform decision making. For example, policy makers and education leaders have used the results to track the impact of reforms on student achievement levels and to inform curriculum improvement, teacher training, and resource allocation. However, it is unclear whether decisions based on the results have had a positive impact on student achievement levels. THE WORLD BANK ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Introduction SABER-Student Assessment methodology In 2009, Zambia joined the Russia Education Aid for The SABER-Student Assessment framework is built on Development (READ) Trust Fund program, the goal of the available evidence base for what an effective which is to help countries improve their capacity to assessment system looks like. The framework provides design, carry out, analyze, and use assessments for guidance on how countries can build more effective improved student learning. As part of the READ Trust student assessment systems. The framework is Fund program, and in order to gain a better structured around two main dimensions of assessment understanding of the strengths and weaknesses of its systems: the types/purposes of assessment activities existing assessment system, Zambia participated in a and the quality of those activities. formal exercise to benchmark this system under The World Bank’s Systems Approach for Better Education Assessment types and purposes Results (SABER) program. SABER is an evidence-based program to help countries systematically examine and Assessment systems tend to be comprised of three strengthen the performance of different aspects of their main types of assessment activities, each of which education systems. serves a different purpose and addresses different information needs. These three main types are: What is SABER-Student Assessment? classroom assessment, examinations, and large-scale, system level assessments. SABER-Student Assessment is a component of the Classroom assessment provides real-time information SABER program that focuses specifically on to support ongoing teaching and learning in individual benchmarking student assessment policies and systems. classrooms. Classroom assessments use a variety of The goal of SABER-Student Assessment is to promote formats, including observation, questioning, and paper- stronger assessment systems that contribute to and-pencil tests, to evaluate student learning, generally improved education quality and learning for all. on a daily basis. National governments and international agencies are Examinations provide a basis for selecting or certifying increasingly recognizing the key role that assessment of students as they move from one level of the education student learning plays in an effective education system. system to the next (or into the workforce). All eligible The importance of assessment is linked to its role in: students are tested on an annual basis (or more often if (i) providing information on levels of student the system allows for repeat testing). Examinations learning and achievement in the system; cover the main subject areas in the curriculum and (ii) monitoring trends in education quality over usually involve essays and multiple-choice questions. time; (iii) supporting educators and students with real- Large-scale, system-level assessments provide feedback time information to improve teaching and on the overall performance of the education system at learning; and particular grades or age levels. These assessments (iv) holding stakeholders accountable for results. typically cover a few subjects on a regular basis (such as every 3 to 5 years), are often sample based, and use multiple-choice and short-answer formats. They may be national or international in scope. Appendix 1 summarizes the key features of these main types of assessment activities. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 2 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Quality drivers of an assessment system Table 1: Framework for building an effective assessment system, with indicator areas The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or policy framework for assessment activities; institutional and organizational structures for designing, carrying The indicators are identified based on a combination of out, or using results from the assessment; the criteria, including: availability of sufficient and stable sources of funding; • professional standards for assessment; and the presence of trained assessment staff. • empirical research on the characteristics of effective assessment systems, including analysis of the System alignment refers to the extent to which the characteristics that differentiate between the assessment is aligned with the rest of the education assessment systems of low- versus high-performing system. This includes the degree of congruence nations; and between assessment activities and system learning • theory—that is, general consensus among goals, standards, curriculum, and pre- and in-service experts that it contributes to effective teacher training. assessment. Assessment quality refers to the psychometric quality of Levels of development the instruments, processes, and procedures for the assessment activity. It covers such issues as design and The World Bank has developed a set of standardized implementation of assessment activities, analysis and questionnaires and rubrics for collecting and evaluating interpretation of student responses to those activities, data on the three assessment types and related quality and the appropriateness of how assessment results are drivers. reported and used. The questionnaires are used to collect data on the Crossing the quality drivers with the different characteristics of the assessment system in a particular assessment types/purposes provides the framework country. The information from the questionnaires is and broad indicator areas shown in Table 1. This then applied to the rubrics in order to judge the framework is a starting point for identifying indicators development level of the country’s assessment system that can be used to review assessment systems and in different areas. plan for their improvement. The basic structure of the rubrics for evaluating data collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice in each area. For each indicator, the rubric displays four development levels—Latent, SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 3 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Emerging, Established, and Advanced. These levels are Zambia spent about 1.35 percent of GDP on education artificially constructed categories chosen to represent in 2008 (latest available data). At the primary level, key stages on the underlying continuum for each gross school enrollment in 2009 was approximately 116 indicator. Each level is accompanied by a description of percent; at the lower secondary level (grades 8 and 9), what performance on the indicator looks like at that it was 76 percent. Gross enrollment rate at the level. secondary level (grades 8-12) is not available. • Latent is the lowest level of performance; it represents absence of, or deviation from, the Since 2002, when the Government of Zambia desired attribute. introduced free basic education up to grade 7, • Emerging is the next level; it represents partial enrolment in basic education has steadily increased. presence of the attribute. However, according to UNICEF, children from poor • Established represents the acceptable minimum households and rural areas, as well as girls, have tended standard. to be the last to enroll in school and the first to drop • Advanced represents the ideal or current best out. They also are significantly underrepresented in the practice. higher grades of basic education as well as at the secondary level. To address these issues, the A summary of the development levels for each Government of Zambia introduced bursaries to help assessment type is presented in Appendix 3. offset indirect school costs for low-income families, as well as a pregnancy readmission policy that allows girls In reality, assessment systems are likely to be at who have become pregnant to re-enter school. different levels of development in different areas. For example, a system may be Established in the area of Additionally, the country has increased the supply of examinations, but Emerging in the area of large-scale, qualified and motivated teachers, is adjusting the school system-level assessment, and vice versa. While intuition curriculum to clearly specify the skills and competencies suggests that it is probably better to be further along in that children should attain in different subjects and at as many areas as possible, the evidence is unclear as different grade levels, and has increased the supply of to whether it is necessary to be functioning at teaching and learning materials. Advanced levels in all areas. Therefore, one might view the Established level as a desirable minimum outcome to Detailed information on Zambia’s student assessment achieve in all areas, but only aspire beyond that in those system was obtained from the country’s self-diagnosis areas that most contribute to the national vision or report that was prepared in 2009 as part of the READ priorities for education. In line with these considerations, Trust Fund program. At the time of the review, SABER- the ratings generated by the rubrics are not meant to be Student Assessment questionnaires had not been additive across assessment types (that is, they are not developed. Self-diagnosis information was analyzed meant to be added to create an overall rating for an using the SABER-Student Assessment rubrics. It is assessment system; they are only meant to produce an important to remember that the SABER-Student overall rating for each assessment type). The methodology Assessment rubrics primarily focus on benchmarking a for assigning development levels is summarized in country’s policies and arrangements for assessment Appendix 4. activities at the system or macro level. Additional data would need to be collected to determine actual, on-the- Education in Zambia ground practices in Zambia, particularly by teachers and students in schools. The following sections discuss the findings for each assessment type, accompanied by Zambia is a low income country in Southern Africa. GDP suggested policy options. The suggested policy options per capita (current US$, 2009) is $1006, with annual were determined in collaboration with key local growth of approximately 3.5 percent. Employment is stakeholders based on Zambia’s immediate interests primarily in the agricultural sector (approximately 72 and needs. Detailed, completed rubrics for each percent), services sector (approximately 20 percent), and assessment type are provided in Appendix 5. in industry (approximately 7 percent). SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 4 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Classroom Assessment pedagogical resource. Classroom assessment activities also are not aligned with the pedagogical or curricular framework. Level of development Ad hoc mechanisms have been introduced in pilot In Zambia, a formal and publically-available, system- schools to monitor the quality of classroom assessment level document provides guidelines for classroom practices. Specifically, these ad hoc mechanisms were assessment activities. Additionally, an official curriculum put in place during the pilot phase of the Continuous or standards document specifies what students are Assessment project. The program is in the process of expected to learn, although the level of performance being expanded to non-pilot schools. required of students is not clear. At the same time, there are adequate required uses of Few system-wide resources are available to teachers for classroom assessment to support student learning, conducting classroom assessment activities. For including diagnosing student learning issues, providing example, teacher guides and student task booklets feedback to students on their learning, informing were developed during the four years of the Continuous parents about their child’s learning, and planning next Assessment project pilot. However, these resources are steps in instruction. In practice, however, not all of the available only in select schools. 1 teachers have the skills to use classroom assessment information to improve student learning. Currently, there are no system-level mechanisms in place to ensure that teachers develop skills and Suggested policy options: expertise in classroom assessment. During the pilot phase of the Continuous Assessment project, certain 1. Build teachers' capacity to engage in classroom system-level mechanisms, such as in-service teacher assessment by making available system-wide training programs, were introduced and used to train resources/materials to teachers in all schools; teachers in classroom assessment practices in the pilot institutionalize system-level mechanisms to schools. ensure that teachers develop skills and expertise in classroom assessment. Classroom assessment practices in Zambia are known to be generally weak. For example, classroom assessment 2. Ensure that classroom assessment activities activities tend to focus only on information recall. improve student learning by introducing varied Additionally, teachers tend not to use explicit or a priori and systematic mechanisms to monitor the criteria for scoring or grading students' work, and quality of classroom assessment practices. uneven application of standards for grading is a serious problem. Although teachers are required to report on student performance to individual students and their parents, parents tend to be poorly informed about their child’s grades. Classroom assessment is mainly used as an administrative or control tool rather than as a 1 The main objective of the Continuous Assessment program pilot is to develop school-based assessment procedures to help teachers appropriately assess student competencies and identify areas of need, carry out remedial work, and maintain student performance records. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 5 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Examinations What is measured by the examinations is largely accepted by stakeholder groups. Comprehensive material on the examinations, including information on Level of development how to prepare for them, and a report on the strengths and weaknesses of previous examinees’ performance, is In Zambia, annual examinations are conducted at the accessible to most students. end of grades 7, 9, and 12 in all subject areas and are used for selection and certification purposes. There are Inappropriate behavior surrounding the examination formal, publically-available policy documents process is quite high, and has included, for example, authorizing the examinations, including the leakage of the content of an examination paper and Examinations Council of Zambia Act and the Education copying from other candidates. At the same time, the Policy, Educating Our Future document. These examination results are basically viewed as credible by documents address all key aspects of the examinations, key stakeholder groups and are used by them in an including their purpose, the distribution of power and appropriate way. For example, examination results are responsibilities among key entities, and the authorized used for certification and selection to higher levels of uses of results. education, and for hiring by employers. The examinations office, the Examinations Council of Some options are available for students who do not Zambia, is a stable, semi-autonomous organization that perform well on the examinations. For example, is covered by an Act of Parliament. The Minister of students may retake the examinations, attend remedial Education appoints the Board of the Examinations or preparatory courses in order to prepare to retake the Council of Zambia, and the Board reports to the examinations, opt for less selective education tracks, Minister. The examinations office has all of the required repeat the grade, or leave the education system. facilities to carry out the examinations, including computers for all technical staff and secure storage Some mechanisms are in place to monitor the facilities. consequences of the examinations. For example, predictive validity studies are periodically commissioned Funding for the examinations is regular and provided by by the Examinations Council of Zambia. Additionally, the government. Funding covers all core examination funding is provided for independent research on the activities (design, administration, data processing, and impact of the examinations, and expert review groups reporting), including a small component on research are organized. and development. Suggested policy options: While the examinations office has a permanent or full- time staff, this has proved insufficient to meet the 1. Improve the capacity of existing permanent needs of the examinations program. Issues with staff of the examinations office by providing carrying out examination activities have included them with a range of opportunities to prepare frequent errors in data processing, poor training of test for work on the examinations, such as making administrators, and unclear instructions and guidelines available university courses and non-university for administering the examination. Currently, no training workshops on educational opportunities are offered in the country to prepare measurement and evaluation. individuals for work on the examinations. Teachers are involved in some examination-related tasks, including 2. Ensure that the examinations are fair by making administration and scoring. Up-to-date, voluntary provisions for students in hard-to-reach areas courses on examination setting, marking, and analyzing to enable them to take the examinations, as results are available only to teachers who meet certain well as by addressing and combating criteria. inappropriate behavior surrounding the examinations. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 6 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 National Large-Scale Assessment collection takes place. Optical Mark Reader forms are used to collect student responses, and then scanned. A (NLSA) comprehensive technical report is also prepared, but its circulation is restricted. Level of development Reports with results are made available for all stakeholder groups. The main reports contain In Zambia, the NLSA program has been operating information on overall achievement levels, results for regularly since 1999. 2 Every few years, the NLSA is subgroups, and information on performance trends administered to grade 5 students to assess their literacy over time. Workshops and presentations for key and numeracy skills. A formal, publically-available policy stakeholders are held to communicate the results. document, Focus on Learning, authorizes the NLSA. While there is no detailed NLSA plan for the coming NLSA results are used by some stakeholder groups in a years, there is a general understanding that the NLSA way that is consistent with the purposes and technical will take place according to schedule. characteristics of the assessment. For example, the Ministry of Education, the donor community, and civil Funding is allocated for NLSA activities by the Ministry society use the results to monitor the overall quality of of Education. Funding covers all core NLSA activities education. Very few teachers, however, use the NLSA (design, administration, analysis, and reporting), as well results to inform their teaching or to improve student as long- or medium-term planning of program learning. milestones and staff training. Funding also covers research and development activities. Currently, no mechanisms are in place to monitor the consequences of the NLSA. The NLSA is run by the Examination Council of Zambia. The Council is a permanent agency that is accountable Suggested policy options: to the Ministry of Education. However, the Council is inadequately staffed to effectively carry out all NLSA 1. Make available a wide range of opportunities to activities. Currently, no courses or workshops on the prepare individuals for work on the NLSA, NLSA are available in Zambia. including funding for attending international programs or courses on educational Efforts are made to include all groups of students in the measurement and evaluation; provide teachers NLSA. For example, accommodations or alternative with opportunities to learn about the NLSA by assessments are provided for students with disabilities, offering high-quality courses or workshops on and special plans are made to ensure that the NLSA is NLSA on a regular basis. administered to students in hard-to-reach areas. The NLSA is also offered in the language of instruction for 2. Introduce a variety of mechanisms to monitor almost all student groups. the consequences of the NLSA, such as providing funding for independent research on Some mechanisms are in place to ensure the quality of the impact of the NLSA, instituting a permanent the NLSA, including that all booklets are numbered, and oversight committee, or organizing themed there is a standardized manual for administrators. conferences to provide a forum to discuss Additionally, a pilot is conducted before the main data research on the consequences of the NLSA. 2 The NLSA program in Zambia is referred to as the National Assessment Programme. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 7 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 International Large-Scale Assessment Country-specific results and information for SACMEQ (ILSA) are regularly disseminated in Zambia. National-level dissemination of the results is conducted in all Level of development provinces and to all key stakeholders. National and international reports, as well as brochures and PowerPoint presentations with Zambia’s results are Zambia has participated in three surveys (I, II, and III) distributed to key stakeholders. However, products to carried out by the Southern and Eastern Africa provide feedback to schools and educators about the Consortium for Monitoring Educational Quality results are not made available, and there is no media (SACMEQ) in the last ten years. Zambia has taken coverage of the results. concrete steps to participate in SACMEQ IV. The results from SACMEQ have been used by Zambian There is a formal, publically-available policy document policy makers and education leaders to try to improve that addresses participation in SACMEQ. It is the education quality by tracking the impact of reforms on SACMEQ general policy document, which is agreed student achievement levels, as well as by informing upon by the SACMEQ Assembly of Ministers. The curriculum improvement, teacher training programs, Assembly of Ministers meets every two years. Zambia resource allocation, and other assessment activities does not have a country-specific policy document that (e.g., classroom assessment, examinations). However, it deals with participation in ILSAs. is not clear that decisions based on SACMEQ results have had a positive impact on students' achievement Funding for ILSA activities (specifically, SACMEQ) in levels. Zambia has come from loans or external donors, and covers participation fees, implementation of the assessment exercise (e.g., printing booklets, travel to Suggested policy options: schools), processing and analyzing the data collected, reporting and disseminating the assessment results in 1. Institutionalize participation in ILSA by having a Zambia, and attending international expert meetings for formal policy document at the country level, the assessment. Funding does not cover research and and by introducing regular government funding development activities. for ILSA activities, including research and development. There is a team and a national coordinator responsible for carrying out the SACMEQ activities. The team has 2. Ensure effective uses of ILSA results by attended all international workshops or meetings for introducing widespread media coverage of the the assessment. While Zambia offers some results; develop products that provide opportunities to learn about ILSAs, such as workshops information about ILSA results and on using international assessment databases as well as systematically make these products available to funding for attending international workshops or key stakeholders, including teachers and training, these opportunities are available only to the educators. country’s SACMEQ team members. Zambia met all technical standards required to have its data presented in the main displays of the international SCAMEQ report. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 8 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations Surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select immediate feedback on feedback on the students as they students for feedback to overall health of comparative move from one further inform the system at performance of level of the educational classroom particular the education education system opportunities instruction grade/age system at to the next (or level(s), and to particular into the monitor trends in grade/age workforce) learning level(s) Frequency Daily For individual For individual Annually and Annually and subjects offered subjects offered more often more often on a regular on a regular where the system where the system basis (such as basis (such as allows for allows for every 3-5 years) every 3-5 years) repeats repeats Who is All students Sample or A sample of All eligible All eligible tested? census of students at a students students students at a particular grade particular grade or age level(s) or age level(s) Format Varies from Usually multiple Usually multiple Usually essay Usually essay observation to choice and short choice and short and multiple and multiple questioning to answer answer choice choice paper-and-pencil tests to student performances Coverage of All subject areas Generally Generally Covers main Covers main curriculum confined to a few confined to one subject areas subject areas subjects or two subjects Additional Yes, as part of Frequently Yes Seldom Seldom information the teaching collected from process students? Scoring Usually informal Varies from Usually involves Varies from Varies from and simple simple to more statistically simple to more simple to more statistically sophisticated statistically statistically sophisticated techniques sophisticated sophisticated techniques techniques techniques SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 9 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development Level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—ENABLING CONTEXT EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—SYSTEM ALIGNMENT SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—ASSESSMENT QUALITY AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 10 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Appendix 3: Summary of the Development Levels for Each Assessment Type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system- institutional capacity to wide institutional system-wide institutional wide institutional support and ensure the capacity to support and capacity to support and capacity to support and quality of classroom ensure the quality of ensure the quality of ensure the quality of assessment practices. classroom assessment classroom assessment classroom assessment practices. practices. practices. CLASSROOM ASSESSMENT There is no standardized There is a partially There is a stable There is a stable examination in place for stable standardized standardized standardized key decisions. examination in place, examination in place. examination in place and and a need to develop There is institutional institutional capacity and institutional capacity to capacity and some strong mechanisms to run the examination. The limited mechanisms to monitor it. The EXAMINATIONS examination typically is monitor it. The examination is of high of poor quality and is examination is of quality and is perceived perceived as unfair or acceptable quality and is as fair and free from corrupt. perceived as fair for corruption. most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA There is a stable NLSA place. NLSA in place and a in place. There is in place and institutional need to develop institutional capacity and capacity and strong institutional capacity to some limited mechanisms to monitor run the NLSA. mechanisms to monitor it. The NLSA is of high NATIONAL (OR SYSTEM- Assessment quality and it. The NLSA is of quality and its LEVEL) LARGE-SCALE impact are weak. moderate quality and its information is ASSESSMENT information is effectively used to disseminated, but not improve education. always used in effective ways. There is no history of Participation in an ILSA There is more or less There is stable participation in an ILSA has been initiated, but stable participation in an participation in an ILSA nor plans to participate there still is need to ILSA. There is and institutional capacity in one. develop institutional institutional capacity to to run the ILSA. The capacity to carry out the carry out the ILSA. The information from the INTERNATIONAL LARGE- ILSA. information from the ILSA is effectively used SCALE ASSESSMENT ILSA is disseminated, to improve education. but not always used in effective ways. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 11 ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Appendix 4: Methodology for Assigning Development Levels 1. The country team or consultant collects information 5. The preliminary development level is validated using about the assessment system in the country. expert judgment in cooperation with the country team and The World Bank Task Team Leader. 2. Based on the collected information, a level of development and score is assigned to each dimension in For scores that allow a margin of discretion (i.e., to the rubrics: choose between two levels of development), a final decision has to be made based on expert judgment. For • Latent = 1 score point example, the aforementioned hypothetical country has • Emerging = 2 score points an ‘Enabling Context’ score of 2.33, corresponding to a • Established = 3 score points preliminary level of development of ‘Emerging or • Advanced = 4 score points Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the 3. The score for each quality driver is computed by country team chooses ‘Emerging’ as the most aggregating the scores for each of its constituent appropriate level. dimensions. For example: 6. Scores for certain key dimensions under ‘Enabling The quality driver, ‘Enabling Context,’ in the case of Context’ (in the case of EXAM, NLSA, and ILSA) and ILSA, has 3 dimensions on which a hypothetical country under ‘System Alignment’ (in the case of CLASS) were receives the following scores: Dimension A = 2 points; set as ceiling scores, i.e., the overall mean score for the Dimension B = 2 points; Dimension C = 3 points. The particular assessment type cannot be greater than the hypothetical country’s overall score for this quality score for these key dimensions. These key variables driver would be: (2+2+3)/3 = 2.33 include formal policy, regular funding, having a permanent assessment unit, and the quality of 4. A preliminary level of development is assigned to assessment practices. reach quality driver. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 12 ZAMBIA ǀ STUDENT ASSESSMENT Appendix 5: SABER-Student Assessment Rubrics for Zambia This appendix provides the completed SABER-Student Assessment rubrics for each type of assessment activity in Zambia. In each row of the rubric, the relevant selection is indicated by a thick border and an asterisk. The selection may include a superscript number that refers to the justification or explanation for the selection (as indicated by a thick border and an asterisk). The explanation or justification text can be located in the “Development level rating justifications� section at the end of each rubric. If a row includes a superscript, but not a thick border and an asterisk, this means that insufficient information was available to determine the relevant selection in the row. 13 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ZAMBIA Classroom Assessment 14 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. assessment. classroom assessment. assessment. * This option does not apply to this This option does not apply to this The availability of the document is The document is widely available dimension. dimension. restricted. . * ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide 1 teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment. resources available for teachers for classroom assessment. * There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or students are expected to learn, but the students are expected to learn and to to what level of performance. level of performance required is not what level of performance. clear. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms This option does not apply to this There are some system-level There are a variety of system-level to ensure that teachers develop skills dimension. mechanisms to ensure that teachers mechanisms to ensure that teachers 2 and expertise in classroom assessment. develop skills and expertise in classroom develop skills and expertise in classroom * assessment. assessment. 15 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are 3 from widespread weaknesses or there is known to be weak. known to be of moderate quality. known to be generally of high quality. no information available on classroom assessment practices. * There are no mechanisms to monitor the There are ad hoc mechanisms to monitor There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the 4 practices. practices. assessment practices. quality of classroom assessment * practices. ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key 5 stakeholders. stakeholders. stakeholders. * There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support classroom assessment to support classroom assessment to support student learning. student learning, excluding its use as an student learning, including its use as an 6 input for external examination results. input for external examination results. * 16 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Classroom Assessment: Development level rating justifications 1. Teacher Guides and Pupil task booklets were developed during the four years of the Continuous Assessment pilot, and are now available in select schools. 2. During the pilot phase of the Continuous Assessment project, some system-level mechanisms were developed and used to train teachers in classroom assessment practices in the pilot schools. Currently, these mechanisms are available only in select schools. Such mechanisms include pre- and in- service teacher training and opportunities for teachers to participate in conferences and workshops. Additionally, all teacher training programs include a required course on classroom assessment, and school inspection/teacher supervision includes a component that is focused on classroom assessment. 3. Additionally, classroom assessment activities are mainly about recalling information. Teachers do not use explicit or a priori criteria for scoring or grading students' work, and uneven application of standards for grading students’ work is a serious problem. Furthermore, parents are poorly informed about students' grades. Classroom assessment is mainly used as an administrative or control tool rather than as a pedagogical resource, and is not aligned with the pedagogical or curricular framework. 4. Ad hoc mechanisms were in place to monitor the quality of assessment during the pilot phase of the Continuous Assessment project. The program is in the process of being scaled up. 5. Teachers are required to report on individual student’s performance to students and parents. 6. Required uses of classroom assessment include diagnosing student learning issues, providing feedback to students on their learning, informing parents about their child’s learning, planning next steps in instruction, grading students for internal classroom uses, and providing input to an external examination program. In practice, however, not all of the teachers and Standards Officers have the skills to use classroom assessment information to improve student learning. 17 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ZAMBIA Examinations 18 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this place. operating on an irregular basis. has been operating regularly. dimension * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this 1 authorizes the examination. document that authorizes the authorizes the examination. dimension. examination. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public public. dimension. * This option does not apply to this This option does not apply to this The policy document addresses some The policy document addresses all key 2 dimension. dimension. key aspects of the examination. aspects of the examination. * ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the 3 the examination or are indifferent to it. examination. examination. examination. * There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder groups. groups. * Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this not welcomed by the leadership in dimension. generally welcomed by the leadership in dimension. charge of the examination charge of the examination. * (CONTINUED) 19 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this 4 examination. the examination. examination. dimension. * This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. processing or reporting. processing and reporting * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and dimension. development. dimension. development. * ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or The examination office is newly The examination office is a stable This option does not apply to this is newly established. established. organization. dimension. * The examination office is not This option does not apply to this The examination office is accountable to This option does not apply to this 5 accountable to an external board or dimension. an external board or agency. dimension. agency. * Examination results are not recognized Examination results are recognized by Examination results are recognized by Examination results are recognized by by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection 6 country. another country. system in another country. * The examination office does not have The examination office has some of the The examination office has all of the The examination office has state of the the required facilities to carry out the required facilities to carry out the required facilities to carry out the art facilities to carry out the examination. 7 examination. examination. examination. * (CONTINUED) 20 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment 8 examination, issues are pervasive. effectively, with minimal issues. effectively, with no issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities The country offers a wide range of that prepare for work on the dimension. that prepare for work on the opportunities that prepare for work on examination. examination. the examination. * 21 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what This option does not apply to this 9 measures. dimension. the examination measures. dimension. What the examination measures is This option does not apply to this What is measured by the examination is * This option does not apply to this questioned by some stakeholder groups. dimension. largely accepted by stakeholder groups. dimension. * Material to prepare for the examination There is some material to prepare for the There is comprehensive material to There is comprehensive material to is minimal and it is only accessible to examination that is accessible to some prepare for the examination that is prepare for the examination that is 10 very few students. students. accessible to most students. accessible to all students. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses There are up-to-date compulsory courses examinations available to teachers. workshops on examinations available to or workshops on examinations available or workshops on examinations for 11 teachers. to teachers. teachers. * Teachers are excluded from all Teachers are involved in very few Teachers are involved in some Teachers are involved in most 12 examination-related tasks. examination-related tasks. examination-related tasks. examination-related tasks. * 22 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical There is a comprehensive, high quality documentation. examination, but it is not in a formal report but with restricted circulation. technical report available to the general report format. public. * There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the 13 examination. quality of the examination. * ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the 14 examination process is high. examination process is moderate. examination process is low. examination process is marginal. * The examination results lack credibility The examination results are credible for The examination results are credible for This option does not apply to this 15 for all stakeholder groups. some stakeholder groups. all stakeholder groups. dimension. * The majority of the students (over 50%) A significant proportion of students A small proportion of students (less than All students can take the examination; may not take the examination because of (10%-50%) may not take the examination 10%) may not take the examination there are no language, gender or other language, gender, or other equivalent because of language, gender, or other because of language, gender, or other equivalent barriers. * 16 barriers. equivalent barriers. equivalent barriers. (CONTINUED) 23 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all 17 proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way. 18 * Student names and results are public. This option does not apply to this Students’ results are confidential. This option does not apply to this dimension. dimension. * ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who There are very limited options for There are some options for students who There is a variety of options for students do not perform well on the examination, students who do not perform well on the do not perform well on the examination. who do not perform well on the 19 or students must leave the education examination. examination. system. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place 20 monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the examination. examination. examination. * 24 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Examinations: Development level rating justifications 1. Examinations Council of Zambia Act and the Education Policy, Educating Our Future policy document. 2. Specifically, the policy document outlines governance, the distribution of power and responsibilities among key entities; it describes the purpose of the examination and the authorized uses of results; identifies the funding sources; outlines procedures to investigate and address security breaches, cheating or other forms of inappropriate behavior; outlines procedures for special/disadvantaged students and specifies who can sit for the examination; identifies the rules about preparation of the examination, explains alignment with curricula and standards and explains the format of the examination questions. 3. All stakeholder groups support the examination, including policymakers, teacher unions, educators, students, parents, the media, think-tanks and NGOs, universities and employers. 4. Specifically, the Examinations Council of Zambia is semi-autonomous and is covered by an Act of Parliament. The Minister of Education appoints the Board of the Examinations Council of Zambia, and the Board reports to the Minister. The Ministry provides policy guidance to the Council and subvention grants. Therefore, funding for examinations is regular and is only from the government. 5. The Examinations Council of Zambia is accountable to the government through its parent, the Ministry of Education. 6. In all British Commonwealth countries and all foreign countries. 7. The required facilities include computers for all technical staff, secure buildings, secure storage facilities, access to adequate computer servers, the ability to back up data and adequate communication tools. 8. There is a permanent or full-time staff, but it is insufficient to meet the needs of the examination. There have been issues with carrying out examination activities, such as frequent errors in data processing. Additionally, there is poor training of test administrators and unclear instructions and guidelines for administering the examination. 9. The examinations are aligned to the curriculum through the development of test specification tables. What is not in place are assessment schemes (assessment syllabuses), which are an elaboration of the test specification intended for dissemination to schools. Assessment schemes are broad scopes within which all test items are derived. This includes the description of learning outcomes, competencies and skills expected within the scope of assessment and defined content areas. 10. Information on how to prepare for the examination is available, and there is a report on the strengths and weaknesses of the students' performance. 25 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 11. The courses, however, are not available to all teachers. Teachers are eligible to participate in the courses if they meet certain criteria. The courses offered are on examination setting, marking and analyzing results. 12. The examination-related tasks include administering the examination, scoring the examination, acting as a judge (i.e., in orals), supervising examination procedures and ensuring that candidates subject entries are properly completed. 13. Examples of the systematic mechanisms that are in place to ensure the quality of the examination include internal and external review or observers, pilot/field testing, translation verification and pre-testing that is conducted for all Grade 7 examination papers. 14. Inappropriate behavior surrounding the examination process has included leakage of the content of an examination paper or part of a paper prior to the examination; impersonation (when an individual other than the registered candidate takes the examination); copying from other candidates; using unauthorized materials such as prepared answers and notes; collusion among candidates via mobile phones, passing of paper or equivalent; substitution of scripts; and marker malpractice (specifically, the deliberate alteration of a mark designed to inflate or deflate a candidate's original mark). 15. For example, the examinations are used for selection to the next level of education, by employers for identifying who to employ and by parents to select the school in which to enroll their children. 16. Examinations fees at grade 7 were abolished and all students can take the examinations. Subsidized fees are charged for the Grade 9 and 12 examinations and all pupils can take the examination in those classes as well as out-of-school youths who wish to sit or repeat the examinations provided they are undergoing continuous tuition. 17. Examination results are used for certification and selection purposes to higher levels, for recruitment to employment, by Civil Society to assess the quality of education and by government to monitor the quality of education. 18. An individual student can access their results but the individual student's results are not public. 19. Students who do not perform well on the examination may retake the examination, attend remedial or preparatory courses in order to prepare to retake the examination, opt for less selective schools/universities/tracks, repeat the grade or they can leave the education system. 20. Predictive studies are commissioned by the Council from time to time. Additionally, there is funding for independent research on the impact of the examination, and expert review groups are organized. 26 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ZAMBIA National (or System-Level) Large-Scale Assessment (NLSA) 27 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this irregular basis. been operating regularly. dimension. There is no policy document pertaining There is an informal or draft policy There is a formal policy document that * This option does not apply to this 1 to NLSA. document that authorizes the NLSA. authorizes the NLSA. dimension. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. dimension. * There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the dimension. NLSA will take place. coming years. * ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the 2 the NLSA or are indifferent to it. NLSA. NLSA. NLSA. * (CONTINUED) 28 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the There is irregular funding allocated to There is regular funding allocated to the This option does not apply to this 3 NLSA. the NLSA. NLSA. dimension. * This option does not apply to this Funding covers some core NLSA Funding covers all core NLSA activities: This option does not apply to this dimension. activities: design, administration, analysis design, administration, analysis and dimension. 4 and reporting. reporting. * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and dimension. development activities. dimension. development activities. * ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this 5 team. group of people. institution or unit. dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes * Political considerations never hamper dimension. technical considerations. hamper technical considerations. technical considerations. * This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a This option does not apply to this 5 dimension. clearly recognized body. clearly recognized body. dimension. * (CONTINUED) 29 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed The NLSA office is adequately staffed to The NLSA office is adequately staffed to 6 NLSA. to effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no minimal issues. issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities to The country offers a wide range of that prepare individuals for work on dimension. prepare individuals for work on the opportunities to prepare individuals for 7 NLSA. NLSA. work on the NLSA. * 30 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this curriculum or learning standards. dimension. curriculum or learning standards. dimension. * What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned What the NLSA measures is largely questioned by stakeholder groups. dimension. by some stakeholder groups. accepted by stakeholder groups. * There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it dimension. measures what it is supposed to intended to measure. is intended to measure. measure. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on There are occasional courses or There are some courses or workshops on There are widely available high quality the NLSA. workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. * 31 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include Different options are offered to include 8 groups of students in the NLSA. dimension. all groups of students in the NLSA. all groups of students in the NLSA. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in 9 ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. place to ensure the quality of the NLSA. * There is no technical report or other There is some documentation about the There is a comprehensive technical There is a comprehensive, high quality 10 documentation about the NLSA. technical aspects of the NLSA, but it is report but with restricted circulation. technical report available to the general not in a formal report format. public. * ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA 11 NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way. dimension. * NLSA information is not used or is used This option does not apply to this NLSA results are used by some NLSA information is used by all in ways inconsistent with the purposes dimension. stakeholder groups in a way that is stakeholder groups in a way that is or the technical characteristics of the consistent with the purposes and consistent with the purposes and assessment. technical characteristics of the technical characteristics of the * 12 assessment. assessment. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA. place to monitor the consequences of the NLSA. * 32 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 National (or System-Level) Large Scale Assessment (NLSA): Development level rating justifications 1. The formal policy document is called Focus on Learning. 2. All of the stakeholder groups are represented on the National Steering Committee, and they support the NLSA. 3. Funding is from government sources. There is a budget line in the Ministry of Education for national assessment that provides the budget for the NLSA activities every year. 4. Funding also covers long- or medium-term planning of program milestones and staff training 5. The NLSA office is accountable to the Ministry of Education. 6. Although there is an adequate number of permanent or full-time staff, certain issues hamper carrying out the NLSA effectively. These issues include: delays in administering the NLSA due to issues with the design of the questions; poor training of test administrators; unclear instructions and guidelines for administering the assessment; errors in scoring the NLSA that have led to delays in the results being reported; frequent errors in data processing, sampling and advanced item analysis. 7. Courses specific to the NLSA in particular, and assessment and measurement in general, are not offered in public institutions. 8. Accommodations or alternative assessments are provided for students with disabilities. Additionally, special plans are made to ensure that the NLSA is administered to students in hard-to-reach areas. The NLSA is also offered in the language of instruction for almost all student groups. 9. Mechanisms to ensure the quality of the NLSA instrument include that all booklets are numbered, there is a standardized manual for large-scale assessment administrators, and all proctors or administrators are trained according to a protocol. Additionally, a pilot is conducted before the main data collection takes place. Optical Mark Reader forms are employed during the NLSA, which are then scanned. 10. The consultant provided a copy of the report. 11. Reports with results are made available for all stakeholder groups. Specifically, the main reports on the results contain information on the overall achievement levels and subgroups. Additionally, the main reports on the results contain information on trends over time, overall and for subgroups. Workshops or presentations for key stakeholders are held to communicate the NLSA results. 12. Ministry of Education, the donor community and civil society use the results to monitor the quality of education. Very few teachers, however, use NLSA results to inform their teaching or improve student learning. 33 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ZAMBIA International Large-Scale Assessment (ILSA) 34 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in The country/system has participated in 1 in an ILSA in the last 10 years. dimension. at least one ILSA in the last 10 years. two or more ILSA in the last 10 years. * The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this 2 concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. the next 5 years. the next 5 years. * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. dimension. 3 ILSA. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 4 dimension. the public. public. dimension. * ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, ILSA. donors. discretion. decree or norm. * This option does not apply to this Funding covers some core activities of Funding covers all core activities of the Funding covers all core activities of the 5 dimension. the ILSA. ILSA. ILSA. * Funding does not cover research and This option does not apply to this This option does not apply to this Funding covers research and 6 development activities. dimension. dimension. development activities. * (CONTINUED) 35 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. activities. activities. activities. * This option does not apply to this The national/system coordinator or The national/system coordinator is fluent This option does not apply to this dimension. other designated team member may not in the language of the assessment. dimension. be fluent in the language of the This option does not apply to this assessment. The ILSA office is inadequately staffed or * The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, effectively. with minimal issues. with no issues. * 36 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this international workshops or meetings. international workshops or meetings. workshops or meetings. dimension. * The country/system offers no This option does not apply to this The country/system offers some The country/system offers a wide range 7 opportunities to learn about ILSA. dimension. opportunities to learn about ILSA. of opportunities to learn about ILSA. * This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are dimension. dimension. available to the country's/system's ILSA available to a wide audience, in addition team members only. to the country's/system's ILSA team * members. 37 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical The country/system met all technical published. standards to have its data presented standards required to have its data standards required to have its data beneath the main display of the presented in the main displays of the presented in the main displays of the 8 international report or in an annex. international report. international report. The country/system has not contributed This option does not apply to this This option does not apply to this * The country/system has contributed new new knowledge on ILSA. dimension. dimension. knowledge on ILSA. * ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely 9 the country/system. in the country/system. the country/system. disseminated in the country/system. * Products to provide feedback to schools This option does not apply to this Products to provide feedback to schools Products to provide feedback to schools and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are 10 not made available. sometimes made available. systematically made available. * There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the There is wide media coverage of the ILSA results. ILSA results. ILSA results. results. * If any, country/system-specific results Results from the ILSA are used in a Results from the ILSA are used in some Results from the ILSA are used in a and information from the ILSA are not limited way to inform decision making in ways to inform decision making in the variety of ways to inform decision 11 used to inform decision making in the the country/system. country/system. making in the country/system. country/system. * It is not clear that decisions based on This option does not apply to this This option does not apply to this Decisions based on the ILSA results have ILSA results have had a positive impact dimension. dimension. had a positive impact on students' on students' achievement levels. achievement levels. * 38 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 International Large Scale Assessment (ILSA): Development level rating justifications 1. Zambia has participated in three SACMEQ surveys in the last ten years. 2. Zambia will participate in SACMEQ IV. 3. The formal policy document is SACMEQ's general policy that is established by the SACMEQ Assembly of Ministers, which meets every two years. In between the meetings of the Assembly of Ministers, the SACMEQ Managing Committee (elected by the Assembly of Ministers on a rotation basis every two years) guides the work of the SACMEQ Director and the SACMEQ Coordinating Center. 4. Funding covers international participation fees, implementation of the assessment exercise (e.g., printing booklets, travel to schools), processing and analyzing data collected from implementation of the assessment, reporting and disseminating the assessment results in Zambia and attendance at international expert meetings for the assessment. 5. At SACMEQ HQ level, donor support provides for research and development activities, but not at the country level. 6. Opportunities to learn about the ILSA include workshops or meetings on using international assessment databases as well as funding for attending international workshops or training on international assessments. 7. Opportunities include attendance at workshops and international meetings. 8. National-level dissemination of the results is conducted for all provinces and key stakeholders. 9. Copies of the national/system and international reports are distributed to key stakeholders; brochures and PowerPoint presentations with the country’s/system’s results are distributed to key stakeholders; and products providing feedback to schools or educators about the results have been made available. 10. Policy Briefs are produced and information is posted on the SACMEQ website. 11. The results of the ILSA have been used by policy makers or education leaders to improve education quality by tracking the impact of reforms on student achievement levels, as well as by informing curriculum improvement, teacher training programs, resource allocation and other assessment activities in the system (e.g., classroom assessment, examinations). 39 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Acknowledgements This report was prepared by The World Bank SABER- Student Assessment team in collaboration with Carla Bertoncino, World Bank Senior Human Development Economist and Task Team Leader for education projects in Zambia, and Emily Gardner, World Bank Education Specialist (HDNED). References African Development Bank. 2010. Republic of Zambia United Nations Development Programme (UNDP). 2011. 2011-2015 Country Strategy Paper (CPS). Data retrieved “Zambia Human Development Report.� Data retrieved from http://www.afdb.org/fileadmin/uploads/afdb/ from http://hdr.undp.org/en/reports/national/africa/ Documents/Project-and-Operations/ZAMBIA%20- zambia/NHDR_Zambia_2011_education_flyer.pdf on CSP%202011-2015.pdf on March 12, 2013. March 12, 2013. Clarke, M. 2012. “What Matters Most for Student United Nations Educational, Scientific, and Cultural Assessment Systems: A Framework Paper.� Organization (UNESCO) – Institute for Statistics. Zambia READ/SABER Working Paper Series. Washington, DC: Country Indicator Data. Montreal, QC: UNESCO. Data World Bank. retrieved from http://stats.uis.unesco.org/unesco on March 12, 2013. Organisation for Economic Co-operation and Development (OECD). Education at a Glance. 2011. World Bank Development Indicators. Zambia Country “Public expenditure on education.� Data retrieved from Indicator Data. Washington, DC: World Bank. Data http://www.oecd.org/education/skills-beyond- retrieved from http://databank.worldbank.org/dataon school/48631582.pdf on March 12, 2013. March 12, 2013. United Nations Children's Fund (UNICEF). Zambia Zambia. 2010. “Self-Diagnosis Report.� Report prepared Education. Data retrieved from http://www.unicef.org/ under the Russia Education Aid for Development Trust zambia/education.html on April 12, 2013. Fund program. 40 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ZAMBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policymakers and business people—an accessible, objective snapshot showing how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. The World This work is a product of the staff of The Bank with World Bank external contributions. with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. THE WORLD BANK 41 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS