79941 Mozambique SABER Country Report STUDENT ASSESSMENT 2009 Key Policy Areas for Student Assessment Status 1. Classroom Assessment The national curriculum provides some guidance for classroom assessment activities, although curricular expectations appear to be too high considering students' typical knowledge and skills. At the same time, limited pedagogical resources are available to support teachers in carrying out classroom assessment activities. While country-level mechanisms (inspection and supervision) that could help teachers develop classroom assessment competencies are in place, these do not currently focus on classroom assessment as a key area of consideration. 2. Examinations Examination results are used to make promotion decisions for students in grades 5, 7, 10, and 12. CNECE, which is part of the Ministry of Education, is responsible for overseeing the examinations. While examination results are credible overall, there is some concern about corruption and malpractice. There also is awareness of the need to further standardize the examinations. Examination results are mainly used by schools and teachers, and are not used to inform education policies. 3. National Large-Scale Assessment (NLSA) The assessment office in the Ministry of Education (INDE) has been in charge of three NLSA exercises, in 2000, 2006, and 2009. Funding for these exercises was provided mainly by international donors. The NLSAs have been aligned with the national curriculum, but the expectations of this curriculum are too high given students' typical knowledge and skills. The assessment office has provided supervision to ensure standardization of NLSA activities, although lack of resources and low capacity has threatened effective implementation. The intention to disseminate NLSA results has not translated into an effective communication plan, and results have been poorly disseminated, if at all. 4. International Large-Scale Assessment (ILSA) Mozambique has participated in two ILSA exercises: SACMEQ II and SACMEQ III. The SACMEQ exercises have been overseen by the assessment office in the Ministry of Education (INDE) and have benefited from stable funding that covers all core activities. While a national coordinator was appointed for each SACMEQ exercise, there is no full-time staff devoted to SACMEQ activities. INDE staff has benefited from the training provided by SACMEQ to build its assessment capacity. Despite the intent to disseminate and use SACMEQ results, there is concern about their poor dissemination and lack of impact on the school system. THE WORLD BANK MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Introduction SABER-Student Assessment methodology In 2009, Mozambique joined the Russia Education Aid The SABER-Student Assessment framework is built on for Development (READ) Trust Fund program, the goal the available evidence base for what an effective of which is to help countries improve their capacity to assessment system looks like. The framework provides design, carry out, analyze, and use assessments for guidance on how countries can build more effective improved student learning. As part of the READ Trust student assessment systems. The framework is Fund program, and in order to gain a better structured around two main dimensions of assessment understanding of the strengths and weaknesses of its systems: the types/purposes of assessment activities existing assessment system, Mozambique participated and the quality of those activities. in a formal exercise to benchmark this system under The World Bank’s Systems Approach for Better Assessment types and purposes Education Results (SABER) program. SABER is an evidence-based program to help countries Assessment systems tend to be comprised of three systematically examine and strengthen the main types of assessment activities, each of which performance of different aspects of their education serves a different purpose and addresses different systems. information needs. These three main types are: classroom assessment, examinations, and large-scale, What is SABER-Student Assessment? system level assessments. Classroom assessment provides real-time information SABER-Student Assessment is a component of the to support ongoing teaching and learning in individual SABER program that focuses specifically on classrooms. Classroom assessments use a variety of benchmarking student assessment policies and systems. formats, including observation, questioning, and paper- The goal of SABER-Student Assessment is to promote and-pencil tests, to evaluate student learning, generally stronger assessment systems that contribute to on a daily basis. improved education quality and learning for all. Examinations provide a basis for selecting or certifying National governments and international agencies are students as they move from one level of the education increasingly recognizing the key role that assessment of system to the next (or into the workforce). All eligible student learning plays in an effective education system. students are tested on an annual basis (or more often if The importance of assessment is linked to its role in: the system allows for repeat testing). Examinations (i) providing information on levels of student cover the main subject areas in the curriculum and learning and achievement in the system; usually involve essays and multiple-choice questions. (ii) monitoring trends in education quality over time; Large-scale, system-level assessments provide feedback (iii) supporting educators and students with real- on the overall performance of the education system at time information to improve teaching and particular grades or age levels. These assessments learning; and typically cover a few subjects on a regular basis (such as (iv) holding stakeholders accountable for results. every 3 to 5 years), are often sample based, and use multiple-choice and short-answer formats. They may be national or international in scope. Appendix 1 summarizes the key features of these main types of assessment activities. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 2 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Quality drivers of an assessment system Table 1: Framework for building an effective assessment system, with indicator areas The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or policy framework for assessment activities; institutional The indicators are identified based on a combination of and organizational structures for designing, carrying criteria, including: out, or using results from the assessment; the • professional standards for assessment; availability of sufficient and stable sources of funding; • empirical research on the characteristics of effective and the presence of trained assessment staff. assessment systems, including analysis of the characteristics that differentiate between the System alignment refers to the extent to which the assessment systems of low- versus high-performing assessment is aligned with the rest of the education nations; and system. This includes the degree of congruence • theory—that is, general consensus among between assessment activities and system learning experts that it contributes to effective goals, standards, curriculum, and pre- and in-service assessment. teacher training. Levels of development Assessment quality refers to the psychometric quality of the instruments, processes, and procedures for the The World Bank has developed a set of standardized assessment activity. It covers such issues as design and questionnaires and rubrics for collecting and evaluating implementation of assessment activities, analysis and data on the three assessment types and related quality interpretation of student responses to those activities, drivers. and the appropriateness of how assessment results are reported and used. The questionnaires are used to collect data on the characteristics of the assessment system in a particular Crossing the quality drivers with the different country. The information from the questionnaires is assessment types/purposes provides the framework then applied to the rubrics in order to judge the and broad indicator areas shown in Table 1. This development level of the country’s assessment system framework is a starting point for identifying indicators in different areas. that can be used to review assessment systems and plan for their improvement. The basic structure of the rubrics for evaluating data collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice in each area. For each indicator, the rubric displays four development levels—Latent, Emerging, Established, and Advanced. These levels are SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 3 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 artificially constructed categories chosen to represent key than for boys at both levels, with 88 percent of girls and stages on the underlying continuum for each indicator. 93 percent of boys enrolled at the primary level, and 14 Each level is accompanied by a description of what percent of girls and 16 percent of boys enrolled at the performance on the indicator looks like at that level. secondary level. • Latent is the lowest level of performance; it Following rapid expansion of access to primary represents absence of, or deviation from, the education since the mid-1990s, the country has faced a desired attribute. shortage of trained teachers and adequate • Emerging is the next level; it represents partial infrastructure. Additionally, many schools do not have presence of the attribute. adequate water and sanitation facilities, and many • Established represents the acceptable minimum classrooms lack desks and school materials. Double or standard. triple-shift teaching has been introduced to allow for • Advanced represents the ideal or current best separate intakes of students throughout the day, practice. allowing for increased use of facilities. The impact of poverty and AIDS has also placed significant A summary of the development levels for each responsibilities on schools, requiring them to take on assessment type is presented in Appendix 3. many of the functions that families traditionally performed in relation to children’s education and care, In reality, assessment systems are likely to be at such as providing psycho-social assistance to orphaned different levels of development in different areas. For and vulnerable children as well as health services. example, a system may be Established in the area of examinations, but Emerging in the area of large-scale, The Government of Mozambique has identified several system-level assessment, and vice versa. While intuition priority areas for increasing the quality of education in suggests that it is probably better to be further along in the country, including training educators in teaching as many areas as possible, the evidence is unclear as and school management, promoting the participation of to whether it is necessary to be functioning at parents in the education process, providing quality Advanced levels in all areas. Therefore, one might view learning and teaching materials, and rehabilitating the Established level as a desirable minimum outcome to classrooms in need. achieve in all areas, but only aspire beyond that in those areas that most contribute to the national vision or Detailed information on Mozambique’s student assessment priorities for education. In line with these considerations, system was obtained from the country’s self-diagnosis the ratings generated by the rubrics are not meant to be report that was prepared as part of the READ Trust Fund additive across assessment types (that is, they are not program in 2009. At the time of the review, SABER-Student meant to be added to create an overall rating for an Assessment questionnaires had not been developed. Self- assessment system; they are only meant to produce an diagnosis information was analyzed using the SABER- overall rating for each assessment type). The methodology Student Assessment rubrics. It is important to remember for assigning development levels is summarized in that the SABER-Student Assessment rubrics primarily focus Appendix 4. on benchmarking a country’s policies and arrangements for assessment activities at the system or macro level. Education in Mozambique Additional data would need to be collected to determine actual, on-the-ground practices in Mozambique, particularly by teachers and students in schools. The following sections Mozambique is a low income country in Southern discuss the findings by each assessment type, accompanied Africa. GDP per capita (current US$, 2009) is $423, with by suggested policy options. The suggested policy options annual growth of approximately 3.9 percent. were determined in collaboration with key local stakeholders based on Mozambique’s immediate interests At the primary level, net school enrollment is 90 and needs. Detailed, completed rubrics for each assessment percent; at the secondary level, it is 15 percent. type are provided in Appendix 5. Although similar, net enrollment for girls is slightly less SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 4 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Classroom Assessment Classroom assessment information is required to be Level of development disseminated to some key stakeholders. Specifically, classroom assessment results are made public, and schools invite parents to view classroom assessment In Mozambique, there are no official guidelines for results. classroom assessment. In addition, there are very few system-wide resources available to help teachers to While there is generally limited use of classroom engage in effective classroom assessment activities, assessment information, students’ marks are used as an such as materials to help align assessment practices to input to promotion and retention decisions. the new curriculum. Suggested policy options: A new curriculum was introduced between 2004 and 2007. This curriculum includes competencies to be 1. Strengthen inspection and supervision reached at the end of each school cycle, but does not mechanisms, putting a stronger focus on provide information on the difficulty level of the tasks classroom assessment activities. that students are expected to solve. Regardless, curricular expectations seem too high considering the 2. Provide schools and teachers with pedagogical typical achievement levels of students. resources that specify the appropriate difficulty level of the tasks that students are expected to Teacher training in classroom assessment is very scarce do. and weak. It is widely accepted that teachers do not have the required competencies to effectively engage in classroom assessment activities. In primary education, for example, approximately one-third of the teachers do not have formal training to teach. Mechanisms to monitor the quality of classroom assessment practices are ad hoc. The Ministry of Education, in coordination with the provincial and district level offices of education, has an inspection and supervision system with school visits. However, implementation is very weak and unlikely to include much focus on classroom assessment as an area of consideration. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 5 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Examinations The examinations assess the school curriculum. There are limited systematic mechanisms in place to ensure their quality. For example, while examination questions Level of development are reviewed from a content perspective, there is no psychometric analysis, and the level of standardization of the examinations is minimal. Some important steps In Mozambique, there are annual examinations with to improve standardization have included automatic formative purposes (grade 2) and to decide promotion data capturing and the use of multiple choice questions. (grades 5, 7, 10 and 12). There also are admission CNECE produces a technical report on the examinations, examinations into teacher training institutes at grade but does not publish it. 10, as well as a certification examination at the end of the training. All schools participate in the examinations. Inappropriate behavior occurs in less than 5 percent of Currently, there are no examinations in place for the cases and is concentrated in Maputo. For the grade university admissions. National and subject-specific 12 examinations, inappropriate behavior occurs among guidelines are in place for the examinations. These students and to a lesser extent among teachers. The guidelines cover all stages of the examination process, examination results are generally credible, although including the reporting of results. However, the their perceived credibility diminishes if they were guidelines are not available to the public. scored by teachers (this is the case for all of the examinations, excepting those at grade 12, which are Since 2008, the Ministry of Education has led a reform scored centrally). agenda to improve the examinations. Attitudes toward the examinations have improved although there is still Examination results are mainly used by schools and no clear vision for how the examinations can contribute teachers; they are rarely used to inform policy. At grade to informing policy decisions and pedagogical practices. 12, examination results are used to rank schools and In general, efforts to improve the examinations are provinces. Currently, there are no mechanisms in place welcomed by the leadership in charge of them. to monitor the consequences of the examinations. Conselho Nacional de Examens, Certificacao e Equivalencia, or CNECE, is the examination office in Suggested policy options: Mozambique that is mandated by decree to administer the examinations. CNECE is accountable to the Ministry 1. Critically review the consequences of the of Education. Every year, CNECE requests a budget for examinations, especially their potentially the examinations. The funding it receives covers all negative impact on student repetition and planned activities. The staff at CNECE is insufficient to dropping out of school. carry out all examination activities effectively. Although almost all of the personnel have bachelor degrees, none 2. Improve the design and standardization of the has training in evaluation. There also is no capacity to examinations. do research and development. 3. Improve the use of examination results to Teachers are involved in most examination-related inform policy at the national and provincial tasks, such as scoring the grades 2, 5, 7, and 10 levels. examinations, as well as in test design, administration, and supervision. This is important to promote ownership of the examinations. There are some up-to- date voluntary courses or workshops on examinations available to teachers. CNECE also prepares subject area specialists and trains examination administrators at the school level. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 6 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 National Large-Scale Assessment Currently, no mechanisms are in place to ensure that the NLSA accurately measures what it is supposed to (NLSA) measure. Level of development The three previously administered NLSAs have excluded special needs students and students from very small schools. In Mozambique, the national center known as Instituto Nacional de Desenvolvimento de Educação, or INDE, has Some mechanisms are in place to ensure the quality of carried out three NLSAs, in 2000, 2006, and 2009. the NLSA. For example, INDE has created an However, while INDE regulations mention educational administration manual and supervised the assessment assessment, there is no policy document pertaining administration. A special computer program is used for specifically to NLSAs. data capturing, which helps minimize errors. There also is some documentation about the technical aspects of Funding for NLSA projects has been ad hoc, and the the NLSA, but it is not in a formal report format. NLSAs carried out in 2000 and 2009 were funded by international donors. The government also has There is no clear evidence that assessment results have contributed some funds. been disseminated and used. Some assessment reports were distributed in the Ministry of Education. However, While INDE, which is part of the Ministry of Education, the reports did not reach the provincial offices of has a solid institutional base, there is no single unit at education, the schools, or the teachers. INDE fully devoted to assessment. In general, INDE is inadequately staffed to effectively carry out NLSA Currently, no mechanisms are in place to monitor the activities. INDE does not have full time personnel consequences of the NLSA. working in the area of assessment; most of the personnel have training in curricular subject areas, but Suggested policy options: not in assessment. 1. Implement a new NLSA that allows for the In general, opportunities that prepare individuals for institutionalization of these assessment work on NLSAs are offered on an ad hoc basis. Training activities at INDE. is not stable, and it has not been institutionalized. Limited training has been made available through 2. Strengthen the assessment skills of INDE staff as international cooperation, local universities, and the well as of staff at the provincial level. international assessment program, SACMEQ. Teachers participate in some assessment-related tasks. In 2000, school teachers from selected schools were trained to administer the assessment; however, no additional training on NLSA has been offered for teachers. A new school curriculum was introduced between 2004 and 2007. The curriculum does not clearly specify the difficulty level of the problems or tasks that students are expected to solve. Regardless, it is clear that curricular expectations are too high. While the assessment framework for each NLSA has been based on the curriculum, there is no information regarding the degree of alignment between the assessment and the curriculum. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 7 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 International Large-Scale Assessment opportunities to learn about ILSA are offered in Mozambique other than to SACMEQ team members. (ILSA) Mozambique met all technical standards required to have its data presented in the main displays of the Level of development international reports for both SACMEQ II and SACMEQ III. Some graduate students have been using the SACMEQ data in their dissertations. Mozambique has participated in two ILSAs: SACMEQ II (2000) and SACMEQ III (2007). The country also has Country-specific results and information have been taken concrete steps to participate in at least one more disseminated irregularly. INDE created a SACMEQ II ILSA in the next 5 years. Specifically, Mozambique is a (2000) report for Mozambique, and is working on the permanent member of SACMEQ’s executive committee. SACMEQ III (2007) report. The reports have been The Ministry of Education is willing and expected to disseminated, but are barely used for decision making. participate in future SACMEQ exercises. While SACMEQ results and information has been Mozambique became a member of SACMEQ in 2007. disseminated to pre-service teacher training programs There is a formal document nominating the country’s or IFPs, (Institucao de formação de professores), the national coordinator for SACMEQ. There also is an information has not reached the schools. official document recognizing Mozambique as a permanent member of SACMEQ; however, this There is some media coverage of the SACMEQ results. document is not available to the public. For example, results are disseminated in newspapers (including opinion letters), TV, and radio. However, To date, the SACMEQ exercises in Mozambique have there has not been a press release for the SACMEQ been overseen by INDE. There is a permanent budget results. for INDE, but not for ILSA in particular. In practice, budget has been always available for ILSA. Funding It is also not clear that decisions based on SACMEQ primarily supports core activities and does not support results have had a positive impact on students' research and development. achievement levels. In general, INDE has carried out these ILSAs effectively, Suggested policy options: with minimal issues. INDE does not have full-time personnel working on assessment. Personnel have 1. Ensure participation in and budget for future benefited from hands-on training from SACMEQ. SACMEQ (or other ILSA) exercises. Training has been replicated using the cascade model to reach all of the people involved in the implementation 2. Ensure effective dissemination and use of of SACMEQ in Mozambique, both at the central and SACMEQ (or other ILSA) results by creating and provincial levels. The assessment team has participated funding a clear communication plan. in all SACMEQ meetings and trainings. However, no SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 8 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations Surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select immediate feedback on feedback on the students as they students for feedback to overall health of comparative move from one further inform the system at performance of level of the educational classroom particular the education education system opportunities instruction grade/age system at to the next (or level(s), and to particular into the monitor trends in grade/age workforce) learning level(s) Frequency Daily For individual For individual Annually and Annually and subjects offered subjects offered more often more often on a regular on a regular where the system where the system basis (such as basis (such as allows for allows for every 3-5 years) every 3-5 years) repeats repeats Who is All students Sample or A sample of All eligible All eligible tested? census of students at a students students students at a particular grade particular grade or age level(s) or age level(s) Format Varies from Usually multiple Usually multiple Usually essay Usually essay observation to choice and short choice and short and multiple and multiple questioning to answer answer choice choice paper-and-pencil tests to student performances Coverage of All subject areas Generally Generally Covers main Covers main curriculum confined to a few confined to one subject areas subject areas subjects or two subjects Additional Yes, as part of Frequently Yes Seldom Seldom information the teaching collected from process students? Scoring Usually informal Varies from Usually involves Varies from Varies from and simple simple to more statistically simple to more simple to more statistically sophisticated statistically statistically sophisticated techniques sophisticated sophisticated techniques techniques techniques SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 9 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development Level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—ENABLING CONTEXT EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—SYSTEM ALIGNMENT SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—ASSESSMENT QUALITY AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 10 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Appendix 3: Summary of the Development Levels for Each Assessment Type Assessment Type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system- institutional capacity to wide institutional system-wide institutional wide institutional support and ensure the capacity to support and capacity to support and capacity to support and quality of classroom ensure the quality of ensure the quality of ensure the quality of assessment practices. classroom assessment classroom assessment classroom assessment practices. practices. practices. CLASSROOM ASSESSMENT There is no standardized There is a partially There is a stable There is a stable examination in place for stable standardized standardized standardized key decisions. examination in place, examination in place. examination in place and and a need to develop There is institutional institutional capacity and institutional capacity to capacity and some strong mechanisms to run the examination. The limited mechanisms to monitor it. The EXAMINATIONS examination typically is monitor it. The examination is of high of poor quality and is examination is of quality and is perceived perceived as unfair or acceptable quality and is as fair and free from corrupt. perceived as fair for corruption. most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA There is a stable NLSA place. NLSA in place and a in place. There is in place and institutional need to develop institutional capacity and capacity and strong institutional capacity to some limited mechanisms to monitor run the NLSA. mechanisms to monitor it. The NLSA is of high NATIONAL (OR SYSTEM- Assessment quality and it. The NLSA is of quality and its LEVEL) LARGE-SCALE impact are weak. moderate quality and its information is ASSESSMENT information is effectively used to disseminated, but not improve education. always used in effective ways. There is no history of Participation in an ILSA There is more or less There is stable participation in an ILSA has been initiated, but stable participation in an participation in an ILSA nor plans to participate there still is need to ILSA. There is and institutional capacity in one. develop institutional institutional capacity to to run the ILSA. The capacity to carry out the carry out the ILSA. The information from the INTERNATIONAL LARGE- ILSA. information from the ILSA is effectively used SCALE ASSESSMENT ILSA is disseminated, to improve education. but not always used in effective ways. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 11 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Appendix 4: Methodology for Assigning Development Levels 1. The country team or consultant collects information about the assessment system in the country. 2. Based on the collected information, a level of development and score is assigned to each dimension in the rubrics: • Latent = 1 score point • Emerging = 2 score points • Established = 3 score points • Advanced = 4 score points 3. The score for each quality driver is computed by aggregating the scores for each of its constituent dimensions. For example: The quality driver, ‘Enabling Context,’ in the case of ILSA, has 3 dimensions on which a hypothetical country receives the following scores: Dimension A = 2 points; Dimension B = 2 points; Dimension C = 3 points. The hypothetical country’s overall score for this quality driver would be: (2+2+3)/3 = 2.33 4. A preliminary level of development is assigned to each quality driver. 5. The preliminary development level is validated using expert judgment in cooperation with the country team and The World Bank Task Team Leader. For scores that allow a margin of discretion (i.e., to choose between two levels of development), a final decision has to be made based on expert judgment. For example, the aforementioned hypothetical country has an ‘Enabling Context’ score of 2.33, corresponding to a preliminary level of development of ‘Emerging or Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses ‘Emerging’ as the most appropriate level. 6. Scores for certain key dimensions under ‘Enabling Context’ (in the case of EXAM, NLSA, and ILSA) and under ‘System Alignment’ (in the case of CLASS) were set as ceiling scores, i.e., the overall mean score for the particular assessment type cannot be greater than the score for these key dimensions. These key variables include formal policy, regular funding, having a permanent assessment unit, and the quality of assessment practices. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 12 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Appendix 5: SABER-Student Assessment Rubrics for Mozambique This appendix provides the completed SABER-Student Assessment rubrics for each type of assessment activity in Mozambique. In each row of the rubric, the relevant selection is indicated by a thick border and an asterisk. The selection may include a superscript number that refers to the justification or explanation for the selection (as indicated by a thick border and an asterisk). The explanation or justification text can be located in the “Development level rating justifications� section at the end of each rubric. If a row includes a superscript, but not a thick border and an asterisk, this means that insufficient information was available to determine the relevant selection in the row. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 13 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 MOZAMBIQUE Classroom Assessment SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 14 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. 1 assessment. classroom assessment. assessment. * This option does not apply to this This option does not apply to this The availability of the document is The document is widely available 2 dimension. dimension. restricted. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide 3 teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment. resources available for teachers for classroom assessment. * There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or students are expected to learn, but the students are expected to learn and to to what level of performance. level of performance required is not what level of performance. 4 clear. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms This option does not apply to this There are some system-level There are a variety of system-level to ensure that teachers develop skills dimension. mechanisms to ensure that teachers mechanisms to ensure that teachers 5 and expertise in classroom assessment. develop skills and expertise in classroom develop skills and expertise in classroom * assessment. assessment. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 15 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are from widespread weaknesses or there is known to be weak. known to be of moderate quality. known to be generally of high quality. no information available on classroom * 6 assessment practices. There are no mechanisms to monitor the There are ad hoc mechanisms to monitor There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the 7 practices. practices. assessment practices. quality of classroom assessment * practices. ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key 8 stakeholders. stakeholders. stakeholders. * There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support classroom assessment to support classroom assessment to support 9 student learning. student learning, excluding its use as an student learning, including its use as an * input for external examination results. input for external examination results. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 16 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Classroom Assessment: Development level rating justifications 1. There is an official policy and guidelines for inspection/supervision, and training in classroom assessment. 2. Each school has access to the policy document, which is distributed at the beginning of the school year. The document is public. 3. There are very scarce resources to align pedagogical practices and classroom assessment to the new curriculum. Provincial and local offices of education do not have guidelines to ensure the quality of classroom assessment. 4. A new curriculum was introduced between 2004 and 2007; it specifies competencies to be reach at the end of each school cycle. Curricular expectations seem too high considering the current typical achievement levels of students. The curriculum does not specify the difficulty level of the tasks that students are expected to solve. 5. Teacher training in classroom assessment is very scarce and weak. It is widely accepted that teachers do not have the required competencies for effective classroom assessment. In primary education, approximately one-third of the teachers do not have formal training to teach. 6. It is widely accepted that classroom assessment practices are weak. 7. The Ministry of Education, in coordination with the provincial and district level offices of education, has an inspection/supervision system with schools visits. However, the system implementation is very weak. It is very unlikely that it has a focus on classroom assessment. Most of the current resources are targeted at expanding school enrolment. 8. Classroom assessment results are public. Schools call parents to come and see classroom assessment results. 9. There is limited use of classroom assessment information. However, students’ marks are used as an input to decide promotion/retention. A score is computed based on students’ marks (2/3 of the weight) and exam results (1/3). SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 17 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 MOZAMBIQUE Examinations SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 18 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this 1 place. operating on an irregular basis. has been operating regularly. dimension * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this 2 authorizes the examination. document that authorizes the authorizes the examination. dimension. examination. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public public. dimension. * This option does not apply to this This option does not apply to this The policy document addresses some The policy document addresses all key 3 dimension. dimension. key aspects of the examination. aspects of the examination. * ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the 4 the examination or are indifferent to it. examination. examination. examination. * There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder 5 groups. groups. * Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this not welcomed by the leadership in dimension. generally welcomed by the leadership in dimension. 6 charge of the examination charge of the examination. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 19 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this 7 examination. the examination. examination. dimension. * This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. 8 processing or reporting. processing and reporting * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and 9 dimension. development. dimension. development. ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or The examination office is newly The examination office is a stable This option does not apply to this 10 is newly established. established. organization. dimension. * The examination office is not This option does not apply to this The examination office is accountable to This option does not apply to this accountable to an external board or dimension. an external board or agency. dimension. agency. * Examination results are not recognized Examination results are recognized by Examination results are recognized by Examination results are recognized by by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection country. * another country. system in another country. The examination office does not have The examination office has some of the The examination office has all of the The examination office has state of the the required facilities to carry out the required facilities to carry out the required facilities to carry out the art facilities to carry out the examination. 11 examination. examination. examination. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 20 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment 12 examination, issues are pervasive. effectively, with minimal issues. effectively, with no issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities The country offers a wide range of that prepare for work on the dimension. that prepare for work on the opportunities that prepare for work on 13 examination. examination. the examination. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 21 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what This option does not apply to this 14 measures. dimension. the examination measures. dimension. * What the examination measures is This option does not apply to this What is measured by the examination is This option does not apply to this 15 questioned by some stakeholder groups. dimension. largely accepted by stakeholder groups. dimension. Material to prepare for the examination There is some material to prepare for the There is comprehensive material to * There is comprehensive material to is minimal and it is only accessible to examination that is accessible to some prepare for the examination that is prepare for the examination that is 16 very few students. students. accessible to most students. accessible to all students. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses There are up-to-date compulsory courses examinations available to teachers. workshops on examinations available to or workshops on examinations available or workshops on examinations for 17 teachers. to teachers. teachers. * Teachers are excluded from all Teachers are involved in very few Teachers are involved in some Teachers are involved in most 18 examination-related tasks. examination-related tasks. examination-related tasks. examination-related tasks. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 22 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical There is a comprehensive, high quality documentation. examination, but it is not in a formal report but with restricted circulation. technical report available to the general 19 report format. public. * There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the 20 examination. quality of the examination. * ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the 21 examination process is high. examination process is moderate. examination process is low. examination process is marginal. * The examination results lack credibility The examination results are credible for The examination results are credible for This option does not apply to this 22 for all stakeholder groups. some stakeholder groups. all stakeholder groups. dimension. * The majority of the students (over 50%) A significant proportion of students A small proportion of students (less than All students can take the examination; may not take the examination because of (10%-50%) may not take the examination 10%) may not take the examination there are no language, gender or other 23 language, gender, or other equivalent because of language, gender, or other because of language, gender, or other equivalent barriers. barriers. equivalent barriers. equivalent barriers. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 23 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all 24 proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way. 25 * Student names and results are public. This option does not apply to this Students’ results are confidential. This option does not apply to this dimension. dimension. * ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who There are very limited options for There are some options for students who There is a variety of options for students do not perform well on the examination, students who do not perform well on the do not perform well on the examination. who do not perform well on the 26 or students must leave the education examination. examination. system. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the 27 examination. examination. examination. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 24 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Examinations: Development level rating justifications 1. There are annual exams with formative purposes (grade 2) and to decide promotion/retention (grades 5, 7, 10 and 12). There also are admission exams into teacher training institutes at grade 10 and a certification exam at the end of the training. All schools participate in the exams. Exams are not used for university admission, but there is a plan to implement them. 2. There are national and subject-specific guidelines for administering the exams. 3. Guidelines cover all stages of the examination process, including reporting exam results. 4, 5, 6. Before 2008, there was a lot of fraud as well as communication problems with the schools. Since 2008, the Ministry of Education is leading a reform agenda to improve the exams. Attitudes toward the exams have improved. There is not a clear vision about how the exams can contribute to informing policy decisions and pedagogical practices at the school level. 7. CNECE requests an annual budget for the exams. 8. CNECE does not have capacity to run all important exam activities, but it does cover all planned activities (e.g., CNECE cannot do training or item analysis). 9. Not possible to judge. There is no capacity to do research and development. 10. CNECE is mandated by decree to administer the exams and is accountable to the Ministry of Education. 11. CNECE does not have its own building. 12. There is not enough personnel, but the problems with running the exams are rather specific. There are 35 persons working at CNECE, but it is not known how many work full time on the exams. Almost all of the personnel have bachelor degrees, but none has training in evaluation. 13. Mozambique does not offer evaluation courses. Nevertheless, CNECE has benefited from international cooperation in specific topics. CNECE basically depends on the skills of the local staff. 14. Exams assess the school curriculum. However, it is not clear to what extent the exams assess the new secondary school curriculum. 15. School teachers are very much involved in exam design, which is important to promote ownership of the exams. 16. CNECE publishes an assessment matrix and exam brochures, and sends them to schools. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 25 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 17. CNECE prepares subject area specialists and trains exam administrators at the school level. School teachers are also involved in exam design. 18. Exams at grades 2, 5, 7, and 10 are scored by school teachers. Exams at grade 12 and admission exams for technical schools are scored at the central level. School teachers participate in all exam steps, including test design, administration and supervision. 19. CNECE creates an exam report, but this is not published. 20. There are quality problems in test construction. Exams questions are reviewed from a content perspective, but there is no psychometric analysis. The level of standardization of the exams is minimal. Some important steps to improve standardization have been automatic data capturing and the use of multiple choice questions. 21. Inappropriate behavior occurs in less than 5 percent of the cases and is concentrated in Maputo. At grade 12, fraud occurs among students and to a lesser extent among teachers. 22. The credibility of the exams is diminished when teachers score them (which is the case for all of the exams, with the exception of exams at grade 12). 23. Adaptations are used in the exams of students with special needs (Braille, hearing impairment). Exams are translated for students in special needs schools. 24. Exam results are mainly used by teachers; however, they are barely used to inform policy. At grade 12 exam results are used to rank school and provinces. Rankings are not produced at the other grades, apparently because scoring is not standardized. 25. There is no information available. 26. In a context of limited enrolment capacity, exams are a meritocratic mechanism to select students for further education. Students who fail annual exams have a second chance to take them. If students fail in their second chance, they have to leave school. 17 percent of students fail or repeat a grade in primary school. Exams are available for people who do not attend school but would like to have a school certificate. 27. There are no validity studies about the exams. There is no information about how the exams affect dropout rates. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 26 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 . MOZAMBIQUE National (or System-Level) Large-Scale Assessment (NLSA) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 27 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this 1 irregular basis. been operating regularly. dimension. There is no policy document pertaining There is an informal or draft policy * There is a formal policy document that This option does not apply to this 2 to NLSA. document that authorizes the NLSA. authorizes the NLSA. dimension. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. dimension. 3 * There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the dimension. NLSA will take place. coming years. * ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the NLSA. 4 the NLSA or are indifferent to it. NLSA. NLSA. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 28 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the There is irregular funding allocated to There is regular funding allocated to the This option does not apply to this 5 NLSA. the NLSA. NLSA. dimension. * This option does not apply to this Funding covers some core NLSA Funding covers all core NLSA activities: This option does not apply to this dimension. activities: design, administration, analysis design, administration, analysis and dimension. and reporting. reporting. * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and dimension. development activities. dimension. development activities. * ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this 5 6 team. group of people. institution or unit. dimension. * This option does not apply to this Political considerations regularly hamper Political considerations sometimes Political considerations never hamper dimension. technical considerations. hamper technical considerations. technical considerations. * This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a This option does not apply to this 7 dimension. clearly recognized body. clearly recognized body. dimension. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 29 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed The NLSA office is adequately staffed to The NLSA office is adequately staffed to 8 NLSA. to effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no minimal issues. issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities to The country offers a wide range of that prepare individuals for work on dimension. prepare individuals for work on the opportunities to prepare individuals for 9 NLSA. NLSA. work on the NLSA. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 30 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this 10 curriculum or learning standards. dimension. curriculum or learning standards. dimension. * What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned What the NLSA measures is largely 11 questioned by stakeholder groups. dimension. by some stakeholder groups. accepted by stakeholder groups. * There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it dimension. measures what it is supposed to intended to measure. is intended to measure. measure. 12 * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on There are occasional courses or There are some courses or workshops on There are widely available high quality 13 the NLSA. workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 31 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include Different options are offered to include 14 groups of students in the NLSA. dimension. all groups of students in the NLSA. all groups of students in the NLSA. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in 15 ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. place to ensure the quality of the NLSA. * There is no technical report or other There is some documentation about the There is a comprehensive technical There is a comprehensive, high quality documentation about the NLSA. technical aspects of the NLSA, but it is report but with restricted circulation. technical report available to the general not in a formal report format. public. * ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA 16 NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way. dimension. * NLSA information is not used or is used This option does not apply to this NLSA results are used by some NLSA information is used by all in ways inconsistent with the purposes dimension. stakeholder groups in a way that is stakeholder groups in a way that is or the technical characteristics of the consistent with the purposes and consistent with the purposes and 17 assessment. technical characteristics of the technical characteristics of the * assessment. assessment. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in 18 monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA. place to monitor the consequences of the NLSA. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 32 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 National (or System-Level) Large Scale Assessment (NLSA): Development level rating justifications 1. In Mozambique, NLSA activities are not stable and have not been institutionalized. The national assessment center (INDE, Instituto Nacional de Desenvolvimento de Educação) of the Ministry of Education has been in charge of 3 NLSA, in 2000, 2006 and 2009. 2. INDE regulations mention educational assessment, but do not mention NLSA in specific. 3. With READ support, INDE elaborated an assessment plan for 2011-12. 4. Before READ, there was an interest in assessment, but no real demand from the Ministry of Education. There was no clear institutional leadership, coordination or plan for developing the assessment. There was no clear definition about the purpose of the assessment or who should lead it. Now there is a greater appreciation of the importance of assessment, and both an interest in, and a demand for, NLSA. 5. There is no stable budget. Funding is only available for specific NLSA projects. NLSA from 2000 and 2009 were funded by international donors. 6. INDE has a solid institutional base. However, there is no single unit at INDE fully devoted to assessment. Before READ there was no stable assessment team. 7. There was no clear definition of assessment functions. 8. INDE does not have full time personnel working in assessment. Most of the personnel has training in curricular subject areas but not in assessment. Before READ there was only one person working in assessment. 9. Training is not stable, and it has not been institutionalized. There is a strong need to develop assessment capacity at the provincial level. Some training has been available through international cooperation, local universities and the international assessment program, SAQMEC. Trained personnel face serious difficulties when trying to apply the new knowledge acquired in trainings. Other working demands threat their ability to apply the new knowledge. 10. A new school curriculum was introduced between 2004 and 2007. The curriculum does not clearly specify the difficulty level of the problems/tasks that students are expected to solve. Nevertheless, it is clear that curricular expectations are too high considering the starting point of the students. The assessment framework of NLSA has been based on the curriculum. However, there is no information available regarding the degree of alignment between the assessment and the curriculum. 11. Acceptance has not been an issue with previous NLSA. 12. No external review of the tests has been carried out. 13. In 2000, school teachers from the selected schools were trained to administer the assessment. No additional training has been offered for teachers. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 33 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 14. The 3 previous NLSA have excluded special need students and students from very small schools. 15. Very scarce resources put at risk the quality of the different assessment steps. INDE tries to ensure quality by creating an administration manual and supervising the assessment administration. A special program is used for data capturing, which allows for minimizing errors. 16. There is no clear evidence that assessment results had been disseminated and used. Assessment reports were distributed in the Ministry of Education. However, the reports did not reach the provincial offices of education, the schools and the teachers. The 2006 assessment report was created but not distributed. Another report only reached draft form. 17. Assessment results were practically not used by the Ministry of Education. There is no evidence of concrete actions taken as a consequence of the assessment. 18. There is no information available. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 34 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 MOZAMBIQUE International Large-Scale Assessment (ILSA) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 35 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in The country/system has participated in 1 in an ILSA in the last 10 years. dimension. at least one ILSA in the last 10 years. two or more ILSA in the last 10 years. * The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. 2 the next 5 years. the next 5 years. * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this 3 addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. dimension. ILSA. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 4 dimension. the public. public. dimension. * ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, 5 ILSA. donors. discretion. decree or norm. * This option does not apply to this Funding covers some core activities of Funding covers all core activities of the Funding covers all core activities of the 6 dimension. the ILSA. ILSA. ILSA. * Funding does not cover research and This option does not apply to this This option does not apply to this Funding covers research and 7 development activities. dimension. dimension. development activities. * (CONTINUED) SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 36 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. 8 activities. activities. activities. * This option does not apply to this The national/system coordinator or The national/system coordinator is fluent This option does not apply to this 9 dimension. other designated team member may not in the language of the assessment. dimension. be fluent in the language of the assessment. * This option does not apply to this The ILSA office is inadequately staffed or The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, 10 effectively. with minimal issues. with no issues. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 37 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this 11 international workshops or meetings. international workshops or meetings. workshops or meetings. dimension. * The country/system offers no This option does not apply to this The country/system offers some The country/system offers a wide range 12 opportunities to learn about ILSA. dimension. opportunities to learn about ILSA. of opportunities to learn about ILSA. * This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are dimension. dimension. available to the country's/system's ILSA available to a wide audience, in addition 13 team members only. to the country's/system's ILSA team * members. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 38 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical The country/system met all technical published. standards to have its data presented standards required to have its data standards required to have its data beneath the main display of the presented in the main displays of the presented in the main displays of the international report or in an annex. international report. international report. 14 * The country/system has not contributed This option does not apply to this This option does not apply to this The country/system has contributed new 15 new knowledge on ILSA. dimension. dimension. knowledge on ILSA. * ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely 16 the country/system. in the country/system. the country/system. disseminated in the country/system. * Products to provide feedback to schools This option does not apply to this Products to provide feedback to schools Products to provide feedback to schools and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are 17 not made available. sometimes made available. systematically made available. * There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the There is wide media coverage of the ILSA 18 results. ILSA results. ILSA results. results. * If any, country/system-specific results Results from the ILSA are used in a Results from the ILSA are used in some Results from the ILSA are used in a and information from the ILSA are not limited way to inform decision making in ways to inform decision making in the variety of ways to inform decision 19 used to inform decision making in the the country/system. country/system. making in the country/system. country/system. * It is not clear that decisions based on This option does not apply to this This option does not apply to this Decisions based on the ILSA results have ILSA results have had a positive impact dimension. dimension. had a positive impact on students' 20 on students' achievement levels. achievement levels. * SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 39 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 International Large Scale Assessment (ILSA): Development level rating justifications 1. Mozambique has participated in two ILSAs: SAQMEC II (2000) and SAQMEC III (2007). 2. Mozambique is a permanent member of SACMEQ’s executive committee. The Ministry of Education is willing and expected to participate in SACMEQ. 3. Mozambique became a member of SACMEQ in 2007. There is a formal document nominating the country’s national coordinator for SACMEQ. 4. There is an official document recognizing that Mozambique is a permanent member of SACMEQ. The document is not public. 5. There is a permanent budget for INDE, not for ILSA. In practice, budget has been always available for ILSA. 6. No information available. 7. Priority given to core activities. 8. No information available. 9. No information available. 10. INDE does not have full-time personnel working on assessment. Personnel have benefited from hands-on training from SACMEQ. Training has been replicated using the cascade model to reach all the people involved with the implementation of SACMEQ, both at the central and the provincial levels. 11. Assessment team has participated in all SACMEQ meetings and trainings. 12. Universities do not offer training courses to learn about international assessment. 13. SACMEQ has provided training to the INDE team directed involved in the ILSA implementation. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 40 MOZAMBIQUE ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 14. This has been the case in both SACMEQ exercises. 15. However, graduate students have been using SACMEQ data in their dissertations. 16. INDE created a SACMEQ II (2000) report and is working on the SACMEQ III (2007) report. The reports have been disseminated but are barely used for decision making. 17. ILSA information has been disseminated to pre-service teacher training programs (IFPs, (Institucao de formação de professores). However, the information has not reached the schools. 18. There has not been a press release for the SACMEQ results. However, results are disseminated in newspapers (including opinion letters), TV, and radio. 19. Results from ILSA were used to inform curricular review. 20. This is largely the consequence of limited dissemination and use of ILSA information. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 41 MOZAMBIQUE ǀ -STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 Acknowledgements This report was prepared by The World Bank SABER- Student Assessment team in collaboration with Sophie Naudeau, World Bank Senior Education Specialist and Task Team Leader for education projects in Mozambique. References African Development Bank. 2011. Mozambique Country United Nations Children’s Fund (UNICEF). “Mozambique Strategy 2011-2015. Data retrieved from - Basic Education Quality.� Available online: Data http://www.afdb.org/fileadmin/uploads/afdb/Documen retrieved from http://www.unicef.org/mozambique/ ts/Policy-Documents/Mozambique%20-%202011- education.html on April 11, 2013. 15%20CSP.pdf on April 11, 2013. U.S. Department of Labor. 2006. "Mozambique." 2005 Clarke, M. 2012. “What Matters Most for Student Findings on the Worst Forms of Child Labor. Bureau of Assessment Systems: A Framework Paper.� International Labor Affairs. READ/SABER Working Paper Series. Washington, DC: World Bank. World Bank. 2003. “Cost and Financing of Education - Opportunities and Obstacles for Expanding and Instituto Nacional de Desenvolvimento de Educacao Improving Education in Mozambique.� Washington DC: (INDE), Ministry of Education of Mozambique and J. The World Bank. Marshall. 2009. “Improving Learning Assessment Systems in READ Countries: Self-Diagnosis and Action World Bank Development Indicators. Mozambique Plan.� Report prepared under the Russia Education Aid Country Indicator Data. Washington, DC: World Bank. for Development Trust Fund program. Data retrieved from http://databank.worldbank.org/ data on March 12, 2013. United Nations Educational, Scientific, and Cultural Organization (UNESCO) Institute for Statistics. Mozambique Country Indicator Data. Montreal, QC: UNESCO. Data retrieved from http://stats.uis. unesco.org/unesco on March 12, 2013. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 42 MOZAMBIQUE ǀ -STUDENT ASSESSMENT SABER COUNTRY REPORT |2009 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policymakers and business people—an accessible, objective snapshot showing how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. THE WORLD BANK SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 43