79942 Serbia SABER Country Report STUDENT ASSESSMENT 2012 Key Policy Areas for Student Assessment Status 1. Classroom Assessment There are several formal policy documents that provide detailed guidelines on classroom assessment. Although these policy documents are available online, and required to be available in primary and secondary schools, teachers are usually familiar only with those provisions that relate to the number of assessments that are required to be carried out. There are limited required uses of classroom assessment information to support student learning. In general, classroom assessment practices provide little feedback to students and parents, tend not to be aligned with the pedagogical or curricular framework, and are considered weak. 2. Examinations The Final Exam in Primary School Education was administered for the first time in 2011 to grade 8 students. According to the Law on the Foundation of the Education System, the three main purposes of the Final Exam are: (i) student certification for primary school completion, (ii) student selection to secondary school, and (iii) monitoring of education quality levels. There are limited systematic mechanisms in place to ensure the quality of the Final Exam, and there are limited options for students who do not perform well on the exam. No mechanisms are in place to monitor the consequences of the Final Exam. 3. National Large-Scale Assessment (NLSA) The National Assessment of Fourth Grade Students was administered in 2006 to a sample of students. The main objective of the survey was to collect data on education practices and overall levels of student achievement in Serbian language and mathematics for the purpose of improving the education system. Limited resources did not allow for all student groups to be represented in the sample chosen for the national assessment. While Serbian policymakers support national assessment exercises in principle, there is no plan for, or budget allocated to, future national large-scale assessment exercises. 4. International Large-Scale Assessment (ILSA) Serbia has participated in a number of international assessment exercises, including TIMSS (2003, 2007, 2011) and PISA (2003, 2006, 2009). Serbia will participate in PISA 2012, and is planning to participate in PISA 2015 and TIMSS 2015. However, there is no formal policy document that addresses the country's participation in international assessments. Funding for Serbia's participation in international assessment exercises, including the recently concluded PISA 2009, has been provided by donors, including the World Bank. THE WORLD BANK SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Introduction SABER-Student Assessment methodology Serbia has focused on increasing student learning outcomes by improving the quality of education in the The SABER-Student Assessment framework is built on country. An effective student assessment system is an the available evidence base for what an effective important component to improving education quality assessment system looks like. The framework provides and learning outcomes as it provides the necessary guidance on how countries can build more effective information to meet stakeholders’ decision-making student assessment systems. The framework is needs. In order to gain a better understanding of the structured around two main dimensions of assessment strengths and weaknesses of its existing assessment systems: the types/purposes of assessment activities system, Serbia decided to benchmark this system using and the quality of those activities. standardized tools developed under The World Bank’s Systems Approach for Better Education Results (SABER) Assessment types and purposes program. SABER is an evidence-based program to help Assessment systems tend to be comprised of three countries systematically examine and strengthen the main types of assessment activities, each of which performance of different aspects of their education serves a different purpose and addresses different systems. information needs. These three main types are: classroom assessment, examinations, and large-scale, What is SABER-Student Assessment? system level assessments. SABER-Student Assessment is a component of the SABER program that focuses specifically on Classroom assessment provides real-time information benchmarking student assessment policies and systems. to support ongoing teaching and learning in individual The goal of SABER-Student Assessment is to promote classrooms. Classroom assessments use a variety of stronger assessment systems that contribute to formats, including observation, questioning, and paper- improved education quality and learning for all. and-pencil tests, to evaluate student learning, generally on a daily basis. National governments and international agencies are Examinations provide a basis for selecting or certifying increasingly recognizing the key role that assessment of students as they move from one level of the education student learning plays in an effective education system. system to the next (or into the workforce). All eligible The importance of assessment is linked to its role in: students are tested on an annual basis (or more often if (i) providing information on levels of student the system allows for repeat testing). Examinations learning and achievement in the system; cover the main subject areas in the curriculum and (ii) monitoring trends in education quality over usually involve essays and multiple-choice questions. time; (iii) supporting educators and students with real- Large-scale, system-level assessments provide feedback time information to improve teaching and on the overall performance of the education system at learning; and particular grades or age levels. These assessments (iv) holding stakeholders accountable for results. typically cover a few subjects on a regular basis (such as every 3 to 5 years), are often sample based, and use multiple-choice and short-answer formats. They may be national or international in scope. Appendix 1 summarizes the key features of these main types of assessment activities. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 2 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Quality drivers of an assessment system Table 1: Framework for building an effective assessment system, with indicator areas The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or policy framework for assessment activities; institutional and organizational structures for designing, carrying The indicators are identified based on a combination of out, or using results from the assessment; the criteria, including: availability of sufficient and stable sources of funding; and the presence of trained assessment staff. • professional standards for assessment; • empirical research on the characteristics of effective System alignment refers to the extent to which the assessment systems, including analysis of the assessment is aligned with the rest of the education characteristics that differentiate between the system. This includes the degree of congruence assessment systems of low- versus high-performing between assessment activities and system learning nations; and goals, standards, curriculum, and pre- and in-service • theory—that is, general consensus among teacher training. experts that it contributes to effective assessment. Assessment quality refers to the psychometric quality of the instruments, processes, and procedures for the Levels of development assessment activity. It covers such issues as design and implementation of assessment activities, analysis and The World Bank has developed a set of interpretation of student responses to those activities, standardized questionnaires and rubrics for collecting and the appropriateness of how assessment results are and evaluating data on the three assessment types reported and used. and related quality drivers. Crossing the quality drivers with the different The questionnaires are used to collect data on the assessment types/purposes provides the framework characteristics of the assessment system in a particular and broad indicator areas shown in Table 1. This country. The information from the questionnaires is framework is a starting point for identifying indicators then applied to the rubrics in order to judge the that can be used to review assessment systems and development level of the country’s assessment system plan for their improvement. in different areas. The basic structure of the rubrics for evaluating data collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice in each area. For each SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 3 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 indicator, the rubric displays four development levels— Education in Serbia Latent, Emerging, Established, and Advanced. These levels are artificially constructed categories chosen to Serbia is an upper-middle-income country in South represent key stages on the underlying continuum for Eastern Europe. GDP per capita (current US$) is each indicator. Each level is accompanied by a $6310.40, with annual growth of 2 percent in 2011. description of what performance on the indicator looks like at that level. The Serbian education system is comprised of pre- primary (for children between the ages of 0.5 and 6.5 • Latent is the lowest level of performance; it years); primary education from grades 1 through 8; and represents absence of, or deviation from, the secondary education, which consists of either four years desired attribute. gymnasium or three to four years of vocational. Primary • Emerging is the next level; it represents partial education is compulsory, and the preparatory pre- presence of the attribute. primary grade has been mandatory since the 2006-2007 • Established represents the acceptable minimum school year. The gross enrollment rate in primary standard. education is approximately 95 percent (in 2011) and the • Advanced represents the ideal or current best completion rate is approximately 94 percent. The gross practice. enrollment rate in secondary education is approximately 91 percent and the completion rate is A summary of the development levels for each approximately 99 percent. assessment type is presented in Appendix 3. Serbia has made important efforts to improve its In reality, assessment systems are likely to be at education system, particularly given the government’s different levels of development in different areas. For goal of acceding to the European Union. The example, a system may be Established in the area of government’s priorities for education include adoption examinations, but Emerging in the area of large- of new approaches to classroom teaching and scale, system-level assessment, and vice versa. While enhancement of the integrity and efficiency of intuition suggests that it is probably better to be education administration through development of further along in as many areas as possible, the education management information systems. evidence is unclear as to whether it is necessary to be functioning at Advanced levels in all areas. Detailed information was collected on Serbia’s student Therefore, one might view the Established level as a assessment system using the SABER-Student desirable minimum outcome to achieve in all areas, but Assessment questionnaires and rubrics. It is important only aspire beyond that in those areas that most to remember that these tools primarily focus on contribute to the national vision or priorities for benchmarking a country’s policies and arrangements for education. In line with these considerations, the ratings assessment activities at the system or macro level. generated by the rubrics are not meant to be additive Additional data would need to be collected to across assessment types (that is, they are not meant to determine actual, on-the-ground practices in Serbia, be added to create an overall rating for an assessment particularly by teachers and students in schools. system; they are only meant to produce an overall rating for each assessment type). The methodology for The following sections discuss the findings by each assigning development levels is summarized in assessment type, accompanied by suggested policy Appendix 4. options. The suggested policy options were determined in collaboration with key local stakeholders based on Serbia’s immediate interests and needs. Detailed, completed rubrics for each assessment type in Serbia are provided in Appendix 5. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 4 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Classroom Assessment respect to classroom assessment, taking into consideration these rulebooks. Level of development Classroom assessment information is required to be disseminated to students and parents. Parents and The Law on the Fundamentals of the Education System students receive information on the student's (2009), the Regulations on Student Assessment in performance at the end of the semester and school Primary Education (2011), and the Regulations on year in an official form (student book). If a student has a Student Assessment in Secondary Education (1999) negative assessment, the parent receives official provide detailed guidelines on classroom assessment. information at the end of the first or third trimester as Although they are available online, and required to be well. During the academic year, parents receive available in primary and secondary schools, teachers information on student performance during regular usually are familiar only with those provisions that parent-teacher meetings. relate to the number of assessments that are required to be carried out. There are limited required uses of classroom assessment information to support student learning. There are a variety of system-wide resources available These include providing feedback to students on their to teachers for classroom assessment. Education learning, informing parents about their child’s learning, standards describe what students are required to know and planning next steps in instruction. In reality, ‘on- and be able to do at the end of the two cycles of the-ground’ classroom assessment practices provide primary education as well as at the end of four years of little feedback to students and parents, are often not secondary education. Teachers are also provided with aligned with the pedagogical or curricular framework, textbooks or workbooks that assist them with and are considered weak. implementing classroom assessment as well as with scoring criteria/rubrics for evaluating students’ work. Suggested policy options: There are some system-level mechanisms to ensure 1. Introduce required courses for teachers on that teachers develop appropriate skills in classroom classroom assessment, especially on the topic assessment. For example, teachers are required to of formative assessment. Additionally, develop receive 120 hours of accredited training over five years minimum standards for teacher competencies and may elect to receive some of this on classroom in classroom assessment and make the assessment. Additionally, the Institute for Education minimum standards part of the core curriculum Quality and Evaluation implemented one-day trainings for pre-service teacher training. For teachers for about 12,000 teachers, principals, and school already in service, introduce professional counselors in 2010. Training topics have included development training on classroom assessment. implementation of education standards and student assessment. However, there are limited systematic 2. Introduce mechanisms to monitor the quality of mechanisms to monitor the quality of classroom classroom assessment practices into the assessment practices. Classroom assessment is a mandate of educational supervisors. This will required component of a teacher’s performance help ensure that classroom assessment is based evaluation and is a required component of school on standards, and is consistent and comparable inspection or teacher supervision. between schools. Several recently-adopted rulebooks describe the 3. Develop regular school reports that combine relationship between assessment and the competencies the results of the external examination and of teachers. Educational advisors and school inspectors classroom assessment to enable schools to monitor and evaluate teachers’ performance with better monitor the quality of their classroom assessment activities. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 5 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Examinations the Ministry of Education and Science, which is accountable to the Government of Serbia. Level of development Regular funding for the Final Exam is provided by the government and covers all core examination activities In Serbia, the Final Exam in Primary School Education (examination design, administration, data processing, and was administered for the first time in 2011 to grade 8 reporting), as well as research. Development activities students. According to the Law on the Foundation of the have been supported under the European Union Support Education System, which authorizes the examination, its for Quality Assurance Within the National Primary and three main purposes are for (i) student certification of Secondary Education Examination System project. primary school completion, (ii) student selection to secondary school, and (iii) monitoring of education Teachers are involved in most examination-related quality levels. tasks. The Institute establishes working groups of subject teachers for writing items and developing the Secondary schools enroll students using the “total examination scoring guide. Teachers are also involved in enrollment score� (school grades, which account for 60 scoring the examination. Additionally, teachers serve as percent + Final Exam score, which accounts for 40 test administrators and supervisors, but not in their percent). Students can apply to various schools with own school. At the same time, there are no general their school grades and Final Exam scores and select courses or workshops on the Final Exam available to which school to attend from the ones that accept them. teachers. Teachers who are involved in examination Students who do not perform well on the examination activities only receive instructions and guidelines for the may choose to enroll in less selective secondary schools. implementation of those activities. If a student is not satisfied with their Final Exam results and cannot enroll in a preferred secondary school, he or There is some documentation about the technical she can re-take the Final Exam in June of the following aspects of the Final Exam, but it is not in a formal report year. format. Additionally, limited systematic mechanisms are in place, such as internal review of the quality of items The material needed to prepare for the Final Exam is and test materials, and external review of exam widely accessible by over 90 percent of the students in implementation, to ensure the quality of the Final a variety of learning contexts (e.g., covered in public Exam. There are also no mechanisms in place to school, available for home schooling, available on line). monitor the consequences of the Final Exam. Materials include examples of the types of questions that are on the exam, information on how to prepare Suggested policy options: for the exam, and a framework document explaining what is measured by the exam. 1. Develop and implement additional mechanisms for assuring the quality of invigilation and There have been coordinated attempts to improve the marking of the Final Exam, such as ensuring that Final Exam by stakeholder groups, which have been all students take the exam under the same welcomed by the Ministry of Education and Science and conditions, and that test marking is objective, the Institute for Education Quality and Evaluation (the reliable, and valid. Institute). Both entities share the responsibilities for carrying out the Final Exam. The Institute prepares the 2. Develop and implement mechanisms for the exam framework, test materials, data analysis, and the improvement of the quality of the Final Exam data report. The Ministry of Education and Science is items and instruments. responsible for the implementation of the Final Exam, including the distribution of test materials, organizing 3. Establish a national commission for examinations exam rooms, test administration, exam supervision, test composed of key stakeholders with a mandate to marking, and data entry. The Institute is accountable to monitor the quality of the Final Exam. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 6 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 National Large-Scale Assessment (NLSA) non-graduate) specifically focused on educational measurement and evaluation. In addition, internships and short-term employment opportunities at the Level of development Institute provide hands-on NLSA experience. There are no courses or workshops on national large-scale The National Assessment of Fourth Grade Students was assessment available to teachers in the country. administered for the first time in 2006. The test was administered to a representative sample of students Mechanisms are in place to ensure the quality of the educated in the Serbian language who were able to take National Assessment of Fourth Grade Students, paper-and-pencil tests. The main objective of the survey including verification of the quality of the test items and was to collect data on existing education practices and instruments by external experts and psychometric overall levels of student achievement in Serbian language analysis, and use of a standardized manual for the and mathematics for the purpose of improving the assessment administration. education system. While there is some documentation about the technical While there are formal and informal policy documents aspects of the National Assessment of Fourth Grade that authorize the National Assessment of Fourth Grade Students, it is not in a formal report format. Students – including the Law on the Foundations of the Education System (2009), the Law on Amendments and Results from the National Assessment of Fourth Additions to the Law on the Fundamentals of the Grade Students are used by some stakeholder groups Education System (2011), and the draft of the Law on in a way that is consistent with the purposes and Primary Education (2012) – these policy documents are technical characteristics of the assessment. For available to the public. There is also no firm plan in place example, the Institute used the 2006 assessment for carrying out the assessment in the coming years. results as an input for developing standards of achievement for the first cycle of education as well as In Serbia, NLSA studies are financed on an ad hoc basis for developing criterion-referenced tests for schools. by donors due to the lack of financial resources in the At the same time, there are no mechanisms in place budget of the Ministry of Education and Science. The to monitor the consequences of the National funding provided covers all core NLSA activities (design, Assessment of Fourth Grade Students. administration, analysis, and reporting), as well as staff training. Funding does not cover research and Suggested policy options: development activities. The limited resources available meant that not all student groups were represented in 1. Adopt a policy document prescribing how NLSA data the sample chosen for the 2006 National Assessment of will be used to support monitoring and Fourth Grade Students. improvement of the quality and equity of education. The Institute for Education Quality and Evaluation (the 2. Ensure that all student groups are included in the Institute) is in charge of the National Assessment of sample for the NLSA (minority languages and Fourth Grade Students. While the knowledge and students with learning difficulties and/or competence of Institute staff is sufficient, the staff has disabilities). to work overtime when it needs to complete tasks for other assessment-related activities (such as the Final 3. Develop a multi-year plan for carrying out NLSA Exam in primary education). activities. There are some opportunities to prepare individuals for 4. Ensure that NLSA activities are part of the national work on the National Assessment of Fourth Grade multi-year budget planning, and include resources Students, including university graduate programs to allow for the inclusion of vulnerable student (masters or doctorate level) and courses (graduate and populations in the national assessment. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 7 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 International Large-Scale Assessment international assessments is offered at the University of Belgrade. (ILSA) To date, Serbia has met all technical standards required Level of development to have its data presented in the main displays of the international reports for each ILSA. The country also has contributed new knowledge on international large-scale Serbia has participated in a number of international assessment to the global evidence base. assessment exercises, including TIMSS (2003, 2007, 2011) and PISA (2003, 2006, 2009). Serbia will While media coverage of ILSAs is limited to a few small participate in PISA 2012, and is planning to participate articles, the discussion of (for example) PISA results in in PISA 2015 and TIMSS 2015. However, there is no the Serbian media has tended to focus on key issues, formal policy document that addresses the country’s including how PISA results can help improve education. participation in international assessments. Results from the ILSAs are used in some ways to inform Funding for Serbia’s participation in international decision making in Serbia. For example, the results and assessment exercises, including the recently concluded analysis of PISA 2009 data has informed the recently PISA 2009, has been provided by donors, including the announced National Strategy for Development of World Bank. Funding for research and development Education until 2020. Additionally, the results of PISA related to these assessment exercises and their results are used by the National Education Council for has been provided by the Institute of Psychology at monitoring the quality and equity of education in Belgrade University (which implements PISA) and by Serbia. PISA results have been used to inform teacher other institutes that request specific research and training programs and other assessment activities in the development studies. system (e.g., classroom assessment, examinations), for developing education standards for the end of There is a team and national coordinator to carry out compulsory and secondary education, and for informing the ILSA activities. For example, in addition to the the framework for the law on education and regulations National Project Manager, the team responsible for governing the assessment practices in schools. carrying out PISA consists of a core group of 4-5 people and a large number of associates who are included in Suggested policy options: specific phases of the study (administration, marking, data entry, etc.). The core staff is comprised of full time 1. Prepare a policy document that provides employees at the Institute of Psychology who have guidelines for Serbia’s participation in ILSAs, regular duties other than PISA activities. The team is specifying the rationale for Serbia’s sufficiently staffed, although the size and structure of participation in a select number of international the team has not always been aligned to the needs of large-scale assessments. the project. In general, the trend has been to increase the size of the core team. Due to financial constraints, 2. Increase local capacity on the use of ILSA the team was able to attend only some of the databases to inform policy decisions. international workshops and meetings for PISA. 3. Ensure that ILSA activities are part of the Opportunities to learn about international assessments national multi-annual budget planning, and are available to individuals working directly on a specific include the cost of participation in the chosen international assessment exercise, and to university number of ILSAs, as well as the follow up students studying assessment or a related area. For analytical work to inform policy making. example, a masters-level university course on educational assessment that includes the topic of SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 8 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select immediate feedback on feedback on the students as they students for feedback to overall health of comparative move from one further inform the system at performance of level of the educational classroom particular the education education system opportunities instruction grade/age system at to the next (or level(s), and to particular into the monitor trends in grade/age workforce) learning level(s) Frequency Daily For individual For individual Annually and Annually and subjects offered subjects offered more often more often on a regular on a regular where the system where the system basis (such as basis (such as allows for allows for every 3-5 years) every 3-5 years) repeats repeats Who is All students Sample or A sample of All eligible All eligible tested? census of students at a students students students at a particular grade particular grade or age level(s) or age level(s) Format Varies from Usually multiple Usually multiple Usually essay Usually essay observation to choice and short choice and short and multiple and multiple questioning to answer answer choice choice paper-and-pencil tests to student performances Coverage of All subject areas Generally Generally Covers main Covers main curriculum confined to a few confined to one subject areas subject areas subjects or two subjects Additional Yes, as part of Frequently Yes Seldom Seldom information the teaching collected from process students? Scoring Usually informal Varies from Usually involves Varies from Varies from and simple simple to more statistically simple to more simple to more statistically sophisticated statistically statistically sophisticated techniques sophisticated sophisticated techniques techniques techniques SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 9 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development Level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—ENABLING CONTEXT EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—SYSTEM ALIGNMENT SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—ASSESSMENT QUALITY AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 10 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 3: Summary of the Development Levels for Each Assessment Type Assessment Type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system- institutional capacity to wide institutional system-wide institutional wide institutional support and ensure the capacity to support and capacity to support and capacity to support and quality of classroom ensure the quality of ensure the quality of ensure the quality of assessment practices. classroom assessment classroom assessment classroom assessment practices. practices. practices. CLASSROOM ASSESSMENT There is no standardized There is a partially There is a stable There is a stable examination in place for stable standardized standardized standardized key decisions. examination in place, examination in place. examination in place and and a need to develop There is institutional institutional capacity and institutional capacity to capacity and some strong mechanisms to run the examination. The limited mechanisms to monitor it. The EXAMINATIONS examination typically is monitor it. The examination is of high of poor quality and is examination is of quality and is perceived perceived as unfair or acceptable quality and is as fair and free from corrupt. perceived as fair for corruption. most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA There is a stable NLSA place. NLSA in place and a in place. There is in place and institutional need to develop institutional capacity and capacity and strong institutional capacity to some limited mechanisms to monitor run the NLSA. mechanisms to monitor it. The NLSA is of high NATIONAL (OR SYSTEM- Assessment quality and it. The NLSA is of quality and its LEVEL) LARGE-SCALE impact are weak. moderate quality and its information is ASSESSMENT information is effectively used to disseminated, but not improve education. always used in effective ways. There is no history of Participation in an ILSA There is more or less There is stable participation in an ILSA has been initiated, but stable participation in an participation in an ILSA nor plans to participate there still is need to ILSA. There is and institutional capacity in one. develop institutional institutional capacity to to run the ILSA. The capacity to carry out the carry out the ILSA. The information from the INTERNATIONAL LARGE- ILSA. information from the ILSA is effectively used SCALE ASSESSMENT ILSA is disseminated, to improve education. but not always used in effective ways. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 11 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 4: Methodology for Assigning Development Levels 1. The country team or consultant collects information 5. The preliminary development level is validated using about the assessment system in the country. expert judgment in cooperation with the country team and The World Bank Task Team Leader. 2. Based on the collected information, a level of development and score is assigned to each dimension in For scores that allow a margin of discretion (i.e., to the rubrics: choose between two levels of development), a final decision has to be made based on expert judgment. For • Latent = 1 score point example, the aforementioned hypothetical country has • Emerging = 2 score points an ‘Enabling Context’ score of 2.33, corresponding to a • Established = 3 score points preliminary level of development of ‘Emerging or • Advanced = 4 score points Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses ‘Emerging’ as the most 3. The score for each quality driver is computed by appropriate level. aggregating the scores for each of its constituent dimensions. For example: 6. Scores for certain key dimensions under ‘Enabling Context’ (in the case of EXAM, NLSA, and ILSA) and The quality driver, ‘Enabling Context,’ in the case of under ‘System Alignment’ (in the case of CLASS) were ILSA, has 3 dimensions on which a hypothetical country set as ceiling scores, i.e., the overall mean score for the receives the following scores: Dimension A = 2 points; particular assessment type cannot be greater than the Dimension B = 2 points; Dimension C = 3 points. The score for these key dimensions. These key variables hypothetical country’s overall score for this quality include formal policy, regular funding, having a driver would be: (2+2+3)/3 = 2.33 permanent assessment unit, and the quality of assessment practices. 4. A preliminary level of development is assigned to each quality driver. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 12 SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 5: SABER-Student Assessment Rubrics for Serbia This appendix provides the completed SABER-Student Assessment rubrics for each type of assessment activity in Serbia. In each row of the rubric, the relevant selection is indicated by a thick border and an asterisk. The selection may include a superscript number that refers to the justification or explanation for the selection (as indicated by a thick border and an asterisk). The explanation or justification text can be located in the “Development level rating justifications� section at the end of each rubric. If a row includes a superscript, but not a thick border and an asterisk, this means that insufficient information was available to determine the relevant selection in the row. 13 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SERBIA Classroom Assessment 14 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. 1 assessment. classroom assessment. assessment. * 2 This option does not apply to this This option does not apply to this The availability of the document is The document is widely available. dimension. dimension. restricted. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment. resources available for teachers for 3 classroom assessment. * There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or students are expected to learn, but the students are expected to learn and to 4 to what level of performance. level of performance required is not what level of performance. clear. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms This option does not apply to this There are some system-level There are a variety of system-level to ensure that teachers develop skills dimension. mechanisms to ensure that teachers mechanisms to ensure that teachers and expertise in classroom assessment. develop skills and expertise in classroom develop skills and expertise in classroom * 5 assessment. assessment. 15 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are 6 from widespread weaknesses or there is known to be weak. known to be of moderate quality. known to be generally of high quality. no information available on classroom assessment practices. There are no mechanisms to monitor the * There are ad hoc mechanisms to monitor There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the 7 practices. practices. assessment practices. quality of classroom assessment * practices. ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key 8 stakeholders. stakeholders. stakeholders. * There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support classroom assessment to support classroom assessment to support 9 student learning. student learning, excluding its use as an student learning, including its use as an * input for external examination results. input for external examination results. 16 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Classroom Assessment: Development-level rating justifications 1. Three official documents authorized by the Ministry of Education and Science, provide the guidelines on classroom assessment: a. The Law on the Fundamentals of the Education System (2009) b. Regulations on student assessment in primary education (2011) c. Regulations on student assessment in secondary education (1999) 2. The documents are available online and are required to be available in primary and secondary schools. In practice, however, legal documents are often at the principal’s, pedagogue’s or psychologist’s office in schools and teachers do not always know the regulations well or are familiar with only the provisions that relate to the number of required assessments. 3. The resources that are typically available include a document that outlines what students are expected to learn and the level(s) of performance that students are expected to reach in different subject areas at different grade/age levels. In Serbia education standards of student achievement describe what students are required to know and be able to do at the end of two cycles of the primary education as well as the standards at the end of four years of secondary education. Teachers are also provided with textbooks or workbooks that assist them with implementing classroom assessment as well as scoring criteria/rubrics for students’ work. After the final exam at the end of primary education, the Institute for Education Quality and Evaluation provides schools with a report on the achievements of their students. This report also includes comparison of school achievement in the Serbian language or mother tongue and mathematics, with the average achievement of the municipality, county, school authority and the Republic and comparison of class achievements in the Serbian language or mother tongue and mathematics. All schools receive the Assessment from idea to practice (Ministry of Education and Sports, 2003) handbook during the training for applying “descriptive assessment� (narrative report on the standards that students achieved, as opposed to just school marks on the 1 to 5 scale). The handbook consists Assessment towards outcomes, written by Dragica Pavlovic-Babic and associates, and Assessment for student development written by Nenad Havelka, Emina Hebib and Aleksandar Baucal. 4. The two official documents are the Standards of student achievements at the end of the primary education document and the Standards for the end of the first cycle of the primary education document that addresses Serbian language, mathematics, nature, and society. 5. Teachers must receive 100 hours of accredited training every five years. The training can be selected from a list of accredited trainings. The Ministry can also require specific training to be offered to teachers in certain academic years, in which case teachers are required to participate in this training. For example, the Ministry of Education and Science has organized trainings for all teachers teaching first grade, and principals and school counselors in primary schools. The Institute for Education Quality and Evaluation has implemented an in-service teacher training seminar on assessment for primary and secondary school teachers, principals and school counselors since 2005. 17 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 The Institute for Education Quality and Evaluation has implemented one-day trainings for about 12,000 teachers, principals and school counselors in 2010. Training topics included implementation educational standards in planning class work and evaluating students. The Institute for Improving Education invites applications each year for professional development programs for school staff. Various institutions and organizations submit their teacher training programs for accreditation. Once a training program is accredited, teachers may choose to participate in it for their professional development. These programs are published each year in the catalogue which is available on-line and in a paper version. Some of the courses are systematic and some are ad-hoc. 6. Although teachers rarely rely mainly on multiple-choice, selection-type questions and it is not common to observe errors in the scoring or grading of students' work, classroom assessment practices are mainly about recalling information, provide little useful feedback to students, and are mainly used as administrative or control tool rather than as pedagogical resource. Additionally, it is very common to observe uneven application of standards for grading students’ work and grade inflation is a serious problem. It is common for teachers to not use explicit or a priori criteria for scoring or grading students' work and parents are often poorly informed about students' grades. It is also common for classroom assessment practices to not be aligned with the pedagogical or curricular framework. 7. Classroom assessment is a required component of a teacher’s performance evaluation and it is a required component of school inspection or teacher supervision. Several rulebooks that were recently adopted describe the relation between assessment and the competencies of teachers. Educational advisors and school inspectors monitor and evaluate teachers’ performance with respect to classroom assessment taking into consideration these rulebooks. These rulebooks include: - The Regulation on continuous professional development and gaining the title of a teacher, preschool teacher or school councilor - The Regulation on the working license for teachers, preschool teachers and school councilors - The Regulations on professional-pedagogical monitoring, - The Regulations on evaluating the quality of educational institutions is in the final phase of development. It will regulate the external evaluation of schools that will be based on the Quality standards for educational institutions 8. Teachers are required to report on individual student’s performance to students and parents. Parents and students receive information on student's performance at the end of the semester and school year in an official form (student book). If a student has a negative assessment, the parent receives official information at the end of the first or third trimester as well. During the academic year, parents receive information on student performance 18 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 informally during regular parent-teacher meetings. These meetings are held by the class-teacher or main subject-teacher. Parents can also obtain information on student's performance during individual consultations, usually on pre-scheduled dates. Parents can talk to the class-teacher or subject- teacher. While teachers should meet with parents during regular parent-teacher meetings, the frequency of these meetings is not strictly regulated. 9. The required uses of classroom assessment activities to promote and inform student learning include providing feedback to students on their learning, informing parents about their child’s learning, and planning the next steps in instruction. Regulation on assessing students was changed based on the changes in the education system. In more detail than before, the regulation describes the ways of assessing and the criteria that needs to be fulfilled for a particular assessment. It points out the need to respect the conditions in which the student learns, as well as the circumstances in which teaching takes place. The key innovations are also the obligatory initial testing by instruments created by teachers in order to assist in planning of teaching/learning. Formative classroom assessment is introduced in this way. The general criteria and the framework for assessing the quality of learning as well as the criteria for assessing student behavior are defined in this regulation. The innovations in the regulations refer to the relation between the assessment and education or achievement standards. Students with special needs or students in disadvantaged living conditions who have an individual educational plan (IEP) are evaluated based on the adjusted requirements and standards that are defined in that IEP. Under the Regulations, the assessment is supposed to be an objective and reliable measure of progress and development, and an indication of quality and efficiency of the work of teachers and schools. 19 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SERBIA Examinations 20 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this 1 place. operating on an irregular basis. has been operating regularly. dimension * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this 2 authorizes the examination. document that authorizes the authorizes the examination. dimension. examination. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 3 dimension. the public public. dimension. * This option does not apply to this This option does not apply to this The policy document addresses some The policy document addresses all key 4 dimension. dimension. key aspects of the examination. aspects of the examination. * ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the 5 the examination or are indifferent to it. examination. examination. examination. * There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder 6 groups. groups. Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this * not welcomed by the leadership in dimension. generally welcomed by the leadership in dimension. 7 charge of the examination charge of the examination. * 21 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 (CONTINUED) LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this 8 examination. the examination. examination. dimension. * This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. 9 processing or reporting. processing and reporting. * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and 10 dimension. development. dimension. development. * ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or The examination office is newly The examination office is a stable This option does not apply to this 11 is newly established. established. organization. dimension. * The examination office is not This option does not apply to this The examination office is accountable to This option does not apply to this 12 accountable to an external board or dimension. an external board or agency. dimension. agency. * Examination results are not recognized Examination results are recognized by Examination results are recognized by Examination results are recognized by by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection 13 country. another country. system in another country. * The examination office does not have The examination office has some of the The examination office has all of the The examination office has state of the the required facilities to carry out the required facilities to carry out the required facilities to carry out the art facilities to carry out the examination. 14 examination. examination. examination. * (CONTINUED) 22 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment 15 examination, issues are pervasive. effectively, with minimal issues. effectively, with no issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities The country offers a wide range of that prepare for work on the dimension. that prepare for work on the opportunities that prepare for work on 16 examination. examination. the examination. * 23 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what This option does not apply to this 17 measures. dimension. the examination measures. dimension. What the examination measures is This option does not apply to this * What is measured by the examination is This option does not apply to this questioned by some stakeholder groups. dimension. largely accepted by stakeholder groups. dimension. 18 * Material to prepare for the examination There is some material to prepare for the There is comprehensive material to There is comprehensive material to is minimal and it is only accessible to examination that is accessible to some prepare for the examination that is prepare for the examination that is 19 very few students. students. accessible to most students. accessible to all students. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses There are up-to-date compulsory courses 20 examinations available to teachers. workshops on examinations available to or workshops on examinations available or workshops on examinations for teachers. to teachers. teachers. Teachers are excluded from * all Teachers are involved in very few Teachers are involved in some Teachers are involved in most 21 examination-related tasks. examination-related tasks. examination-related tasks. examination-related tasks. * 24 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical There is a comprehensive, high quality documentation. examination, but it is not in a formal report but with restricted circulation. technical report available to the general 22 report format. public. * There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the 23 examination. quality of the examination. * ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the 24 examination process is high. examination process is moderate. examination process is low. examination process is marginal. * The examination results lack credibility The examination results are credible for The examination results are credible for This option does not apply to this 25 for all stakeholder groups. some stakeholder groups. all stakeholder groups. dimension. * The majority of the students (over 50%) A significant proportion of students A small proportion of students (less than All students can take the examination; may not take the examination because of (10%-50%) may not take the examination 10%) may not take the examination there are no language, gender or other 26 language, gender, or other equivalent because of language, gender, or other because of language, gender, or other equivalent barriers. barriers. equivalent barriers. equivalent barriers. * (CONTINUED) 25 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all 27 proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way. Student names and results are public. 28 This option does not apply to this Students’ results are confidential. This option does not apply to this * dimension. dimension. * ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who There are very limited options for There are some options for students who There is a variety of options for students do not perform well on the examination, students who do not perform \well on do not perform well on the examination. who do not perform well on the 29 or students must leave the education the examination. examination. system. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the 30 examination. examination. examination. * 26 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Examinations: Development-level rating justifications 1. The Final Exam in Primary School Education is a new examination program that started in 2011. Students in grade 8 are required to take the exam. The Law on the Foundation of Education System defines the three main purposes of the exam: student certification for grade or school cycle completion; student selection to secondary school; and monitoring education quality levels. This law also identifies the secondary purposes of the examination: planning education policy reforms; school or educator accountability; and self evaluation of schools. While the Final Exam is taken for selection for enrollment in secondary schools, there is no formal cut-off score. Different secondary schools enroll students with different total enrollment score (school grades + Final exam score). For example, students applying for secondary medical schools need to earn more than 95 points out of 100 to be enrolled. Students can apply to various schools with their school grades and exam scores and then they select which school to attend out of the ones that accept them. 2. Policy documents that authorize the Final Exam include: - The Law on the Fundamentals of the Education System (2009), authorized by the Serbian Parliament. - The Law on Amendments and Additions to the Law on the Fundamentals of the Education System (2011), authorized by the Serbian Parliament - The Regulations on the Program of the Final Exam in Elementary Education (2011), authorized by the Serbian Government - Manual for Implementation of the Final Exam (2011), authorized by the Ministry of Education 3. The Laws, Regulations, and Manual are available online. 4. The Law on the Fundamentals of the Education System describes the purpose of the examination and specifies who can sit for the examination. The Regulations on the Program of the Final Exam in Elementary Education describe the purpose of the examination, outline the procedures for special/disadvantaged students, specify who can sit for the examination, explain the alignment of the examination with curricula and standards, and explain the format of the examination questions. The Manual for Implementation of the Final Exam outlines the governance, distribution of power, and responsibilities among key entities, outlines the procedures to investigate and address security breaches, cheating, or other forms of inappropriate behavior, and identifies rules about preparation. The policy documents do not describe the authorized uses of results or the sources of funding. 5. Policy makers strongly support the Final Exam, particularly because of their need to have an instrument for monitoring the quality and equity of primary education. 27 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Teacher unions and educators support the introduction of the new Final Exam because it consists of items previously not available to students (which was not the case with the previously administered exam). Teacher unions and educators have expressed that cheating needs to be prevented through more effective test administration, and because students cannot fail (students are must take the Final Exam, but not to achieve certain minimal result), students may not be making a strong effort to prepare for the exam. Students and parents accept the Final Exam because they are used to having an exam at the end of the primary education. They also believe that it is fair to make enrolment decisions based on results on a national exam together with school marks than only based on school marks. The media follows the Final Exam only during its administration. The media provides information on the exam as opposed to criticism or praise. The European Commission has provided strong technical and financial support for the development of the Final Exam within the IPA 2008 project Support for quality assurance within the national primary and secondary education examination system. 6. Efforts to improve the Final exam are coordinated by the EU project that provides support to the Ministry of Education for the improvement of the implementation of the Final Exam. The project also supports the efforts of the Institute for Education Quality and Evaluation to improve the quality of test materials, data analysis, and data reporting. The Institute for Education Quality and Evaluation ("the Institute") is also planning a consultation process with teachers and students to identify areas for potential improvement. 7. Efforts to improve the Final Exam are very welcomed by the Ministry of Education and Science and the Institute. For example, the Ministry negotiated with the European Commission to receive technical and financial support through the IPA project and the Institute initiated independently activities aimed toward improving its assessment activities. 8. The implementation of the Final Exam is supported by regular government budget. The budget covers preparation of test materials, data analysis and reporting by the Institute, as well as the administration of the exam by the Ministry. Additional activities related to the Final exam (for example, secondary analysis of the final exam results) are co-financed through IPA. The main aim of the IPA project is to build the capacity of the Institute. 9. Funding for core examination activities is provided by the government. 10. Developmental activities are supported by the EU project “Support for quality assurance within the national primary and secondary education examination system.� 11. There is shared responsibility between the Ministry of Education and Science and the Institute for Education Quality and Evaluation for carrying out the Final Exam. The Institute prepares the exam framework, test materials, data analysis, and data report. The Ministry of Education and Science is responsible for implementation of the exam (distribution of test materials, organizing exam rooms, test administration, exam supervision, test marking, and data entry). The distribution of these responsibilities is identified in the Law on Foundations of Education System that was adopted in 2009. 28 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 12. The Institute is accountable to the Ministry of Education and Science, which is accountable to the Government. 13. Examination results are recognized by the certification or selection system in Serbia. 14. The Institute is accommodated in a secure building that has secure storage facilities and is equipped with computers. The Institute has access to computer servers, backup devices, and has adequate communication tools, however their quality and capacity are not aligned with the Institute needs – for example, current servers do not have enough capacity to support the Final Exam database, therefore the Institute must rely on external servers. 15. There is permanent or full-time staff, but it is insufficient to meet needs of the examination. Specifically, according to the strategy of the Institute as approved by the Government, the Institute should have 60 employees. However, because of budgetary restrictions the Institute has only 37 employees. For effective operation, at least 13 additional employees are needed. No systematic or serious issues have been identified with the performance of the human resources that are responsible for the Final exam. From time to time, however, some item turns out to be problematic and need to be excluded, marking criteria are applied somewhat differently across the country as teachers from different schools score the Final Exam papers, and/or some exam administrators do not strictly follow the rules and procedures for administrating the Final Exam. These issues, however, typically do not have serious impact on the quality and equity of the Final Exam. 16. The country offers university graduate programs (masters or doctorate level) and courses (graduate and non-graduate), for example at the University of Belgrade, that are specifically focused on educational measurement and evaluation. It is also possible for junior experts to do an internship at the Institute. 17. The Final Exam measures national school curriculum guidelines or standards. According to the Law on Foundation of Education System, the Final Exam is defined as an exam that serves to assess the level of attainment of general and specific standards of achievement in elementary school education. 18. The content of the Final Exam is defined by the Law on Foundation of Education System (general and specific standards). However, a key issue among educators is related to the content of items. Subject specialists argue for more items that assess the recall of key facts and knowledge from a given school subject, while education specialists argue for more items measuring subject specific and cross curricular competencies (items similar to those used in the PISA study). 19. The material needed to prepare for the Final Exam is widely accessible by all students (over 90 percent) in a variety of learning contexts (e.g., covered in public school, available for home schooling, available on line). Materials can be downloaded for free from the Internet site of the Institute. Materials include examples of the types of questions that are on the exam, information on how to prepare for the exam, and the framework document explaining what is measured on the exam, The framework and examples of items are published as a part of the Regulations on the Program of the Final Exam in Elementary Education, and information on how to prepare for the examination is published in the Manual for Implementation of the Final Exam. 29 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 While there is no provision for providing material to students who cannot access personal copies of the materials due to poverty or living in a distant rural area, during the month of May (prior to the administration of the Final Exam in June), 8th grade students have an opportunity to become familiar with the content and format of the exam by taking the trial version of the exam in Serbian language and mathematics. 20. There are no courses on the Final Exam for teachers. Teachers who are involved in certain examination activities only receive instructions and guidelines for the implementation of those activities. 21. The Institute establishes working groups of subject teachers for writing items and developing the scoring guide. Teachers are also involved in scoring the examination. Additionally, teachers serve as test administrators and supervisors, but not in their own school. 22. There is some documentation about the technical aspects of the Final Exam, but it is not in a formal report format. After implementation of the first Final Exam in 2011, the Institute prepared several analytical reports. These reports are only available to the audience for whom the reports were intended (schools or the Ministry). The report to the schools provided the comparison of a particular school in relation to the average achievement at the national, regional and municipality levels, by gender, subjects and items. The report for the Ministry of Education provided an analysis of Final Exam results by School Regional Administrations, region, municipality, and school level. The internal report on the quality of items and test instruments identified areas of improvement of the item writing and test development for future exams. The Institute is working on producing an additional report on the quality of scoring in order to identify potential issues and to develop instruments and procedures to improve marking. 23. Internal review (such as the internal review of the quality of items and test materials produced by the Institute), and external review (such as the supervision of exam implementation organized by the Ministry of Education and Science the (the implementation of the Final exam in 2011 was monitored in each school by one Ministry supervisor)), help ensure the quality of the Final Exam. 24. Copying from other candidates has been observed. In general, however, according to the Manual for Implementation of the Final Exam, schools are obligated to organize the Final Exam in one large room and to arrange the room in such way that the distance between students is at least 1.2 meters. Additionally, the administration of the Final Exam is supervised by an external supervisor who reports directly to the Ministry on the regularity of exam. 25. Although there is some indication that the administration of the Final Exam in some schools was not fully aligned with the rules, procedures, and guidelines set in the Manual for Implementation of the Final Exam, only few complaints were voiced by students and parents as these practices did not 30 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 have a significant impact on student chances to be enrolled in preferred secondary schools. The Ministry of Education and Science and the Institute are receiving support from the EU IPA 2008 project to improve the monitoring and quality assurance of the next Final Exam. 26. According to the Law on Foundation of Education System, all students are obligated to take the Final Exam at the end of grade 8. The Final Exam is free of charge for students therefore there are no financial barriers to take it. Additionally, students take the Final Exam in their own school and the exam is offered in the minority languages, which makes it accessible for all students. For students with special needs, the Final Exam can be delivered in different mediums (for example, Braille letters for students with vision impairment) or with accommodated content (for example, students with special needs receive test accommodations based on the personal Individual Education Program). In 2011, about 98.5 percent of students tool the Final exam in the first term (June 2011). The remaining 1.5 percent of students had an objective reason (for example, sickness) for not taking the Final Exam. 27. The results of the Final Exam are used only for placing students in secondary education programs and schools, and for monitoring of the quality and equity of the primary education by the Ministry of Education and Science. There is no systematic evidence of improper use of examination results. 28. Up to five days after the Final Exam, student results are publically displayed on a poster in the school. 29. Students who do not perform well on the examination may opt for less selective schools. Students take the Final exam at the end of June of the year when they complete grade 8. About 99 percent of students take the Final Exam in that term. If a student is unable to take at the end of June, he or she can do so in August of the same year. According to the Law on Foundation of Education System, students are obligated to only take the Final Exam, and not to achieve a certain minimum score. Therefore, students cannot fail the exam. However, because results from the Final Exam , together with school marks in Grade 6�8, constitute the total score (exam results 40 percent and school marks 60 percent) with which students apply for enrollment to secondary education programs and schools, students who do not perform well on the Final Exam can only be enrolled in a limited number of programs and schools. In the infrequent cases when a student is not satisfied with the exam results and cannot be enrolled in a preferred secondary school, he or she can take the Final Exam again the following June (the student can wait for the next Final Exam without repeating the grade, or can accept to be enrolled in some secondary school for which they have achieved the required entrance score). Students cannot repeat the grade or attend remedial/preparatory courses in order to prepare to retake the examination. 30. There are no mechanisms in place (such as funding for independent research on the impact of the examination, a permanent oversight committee, studies (e.g., predictive validity) that are updated regularly, regular focus groups or surveys of key stakeholders, and/or expert review groups) to monitor the consequences of the examination. 31 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SERBIA National (or System-Level) Large-Scale Assessment (NLSA) 32 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this 1 irregular basis. been operating regularly. dimension. There is no policy document pertaining There is an informal or draft policy * There is a formal policy document that This option does not apply to this 2 to NLSA. document that authorizes the NLSA. authorizes the NLSA. dimension. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 3 dimension. the public. public. dimension. 4 * There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the dimension. NLSA will take place. coming years. * ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the NLSA. 5 the NLSA or are indifferent to it. NLSA. NLSA. * (CONTINUED) 33 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the There is irregular funding allocated to There is regular funding allocated to the This option does not apply to this 6 NLSA. the NLSA. NLSA. dimension. * This option does not apply to this Funding covers some core NLSA Funding covers all core NLSA activities: This option does not apply to this dimension. activities: design, administration, analysis design, administration, analysis and dimension. 7 and reporting. reporting. This option does not apply to this Funding does not cover research and This option does not apply to this * Funding covers research and 8 dimension. development activities. dimension. development activities. * ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this 9 team. group of people. institution or unit. dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes * Political considerations never hamper 10 dimension. technical considerations. hamper technical considerations. technical considerations. * This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a This option does not apply to this 11 dimension. clearly recognized body. clearly recognized body. dimension. * (CONTINUED) 34 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed The NLSA office is adequately staffed to The NLSA office is adequately staffed to NLSA. to effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no 12 minimal issues. issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities to The country offers a wide range of that prepare individuals for work on dimension. prepare individuals for work on the opportunities to prepare individuals for 13 NLSA. NLSA. work on the NLSA. * 35 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this 14 curriculum or learning standards. dimension. curriculum or learning standards. dimension. * What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned What the NLSA measures is largely 15 questioned by stakeholder groups. dimension. by some stakeholder groups. accepted by stakeholder groups. * There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it dimension. 16 measures what it is supposed to intended to measure. is intended to measure. measure. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on There are occasional courses or There are some courses or workshops on There are widely available high quality 17 the NLSA. workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. * 36 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include Different options are offered to include 18 groups of students in the NLSA. dimension. all groups of students in the NLSA. all groups of students in the NLSA. There are no mechanisms in place to * This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in 19 ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. place to ensure the quality of the NLSA. There is no technical report or other There is some documentation about the There is a comprehensive technical There is a comprehensive, high quality * documentation about the NLSA. technical aspects of the NLSA, but it is report but with restricted circulation. technical report available to the general 20 not in a formal report format. public. * ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA 21 NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way. dimension. NLSA information is not used or is used This option does not apply to this * NLSA results are used by some NLSA information is used by all in ways inconsistent with the purposes dimension. stakeholder groups in a way that is stakeholder groups in a way that is or the technical characteristics of the consistent with the purposes and consistent with the purposes and assessment. technical characteristics of the technical characteristics of the 22 There are no mechanisms in place to This option does not apply to this assessment. There are some mechanisms in place to * assessment. There are a variety of mechanisms in 23 monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA. place to monitor the consequences of the NLSA. * 37 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 National (or System-Level) Large Scale Assessment (NLSA): Development-level rating justifications 1. National Assessment of Fourth Grade Students is used to monitor the education quality at the system level, support schools and teachers, and for policy design, evaluation, and decision making. The assessment has been administered once in 2006 in Serbian Language (Reading, Writing, Grammar, Literature) and Mathematics (Natural numbers, Operations with natural numbers, Geometry, Fractions, Solving everyday life problems) to a representative sample of students educated in the Serbian language and who were able to take paper-and-pencil tests. The specific objectives of the National Assessment of Fourth Grade Students were to determine the knowledge, competences and skills in Serbian Language and Mathematics that students acquired by the end of the first cycle of education, to determine factors that affect student achievement, to evaluate knowledge and proficiency of educators, as well as school condition. The ultimate objective of the survey was to collect relevant data on existing education practices, and to learn about the level of student achievement and the factors that affect student achievement, for the purpose of improving the education system. 2. There are formal and informal policy documents that authorize the National Assessment of Fourth Grade Students: - The Law on the Foundations of the Education System (2009), authorized by the Parliament of the Republic of Serbia. - The Law on Amendments and Additions to the Law on the Fundamentals of the Education System (2011), authorized by the Parliament of the Republic of Serbia. - Draft Law on Primary Education (2012), authorized by the Parliament of the Republic of Serbia, which will regulate national testing. It is submitted to the Parliament of Serbia by the Ministry of Education, and can be downloaded from the Parliament of the Republic of Serbia website. 3. The policy documents are available to the public. 4. There is currently no large-scale assessment plan for the coming years or future assessment rounds. 5. Policymakers support national assessments in principle, yet due to the lack of financial resources, there is no annual program of national assessment studies in place. Teacher Unions have not addressed national assessments which may be due to the fact that there have only been two (Grade Three National Assessment of Student Achievement and National Assessment of Fourth Grade Students), neither of which has affected teachers. Because the media is generally interested in reporting on interesting results, they support the national assessment. However, the news that the media reports on the national assessment is not analytical or in-depth. The media has generally been more interested in international assessment studies, such as PISA. 38 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Think thanks and NGOs support national assessment studies as they can use the results for analysis and reviews. University staff strongly support the National Assessment of Fourth Grade Students because they understand the role and importance of the assessment for improving education quality. Nevertheless, among education specialists at the universities, there is concern that the assessment focuses only on outcomes of education, and that it may cause policy makers to neglect the process of education. 6. National assessment studies are financed on ad hoc basis by donors due to the lack of financial resources in the budget of the Ministry of Education. 7. Funding covers all core NLSA activities: design, administration, analysis and reporting. 8. Funding does cover staff training. 9. The Institute for Education Quality and Evaluation ("the Institute") is in charge of the National Assessment of Fourth Grade Students. 10. The Institute receives the request from the Ministry to carry out certain assessment studies and determines when those studies will be carried out. The Institute has the responsibility and autonomy to carry out those studies. 11. The Institute is accountable to the Ministry of Education. The Institute also employs expert consultants to complete work on specific assessment-related tasks. These experts are accountable to the Institute. 12. While the quality of the Institute staff is sufficient, staff has to work overtime when it needs to complete tasks for other assessment-related activities (such as for the Final Exam in Primary School Education, and others). 13. Opportunities to prepare individuals for work on the National Assessment of Fourth Grade Students that are available on an annual basis include university graduate programs (masters or doctorate level) and courses (graduate and non-graduate) specifically focused on educational measurement and evaluation. For example, the University of Belgrade offers the Educational Psychology program (at undergraduate and master levels) that includes a course Evaluation of Knowledge and Educational Outcomes, which is specifically focused on education assessments and examinations. This program has a long tradition, and the course that specifically focuses on educational assessment and examination was introduced in 2006. The quality of this program is considered good as the content of the program and the assessment course is up-to-date. The professors of the program have both good theoretical knowledge and practical experience, and provide up to date information on the knowledge and practice in the area of national assessment studies. Training workshops are held at international education conference, and conference attendees can attend these workshops. Additionally, it is possible to gain hands-on experience through internships or short-term employment in the large-scale assessment office. 14. The National Assessment of Fourth Grade Students measures performance against curriculum or learning standards. 39 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 15. What the National Assessment of Fourth Grade Students measures is largely accepted by stakeholder groups. 16. Mechanisms include regular internal review as well as regular independent review by qualified experts of the alignment between the assessment instrument and what it is supposed to measure. Specifically, the Institute engages external experts to evaluate the quality and validity of test items. 17. There are no courses or workshops on the National Assessment of Fourth Grade Students. 18. The Institute does not have sufficient resources to translate the National Assessment of Fourth Grade Students into minority languages. 19. Mechanisms to ensure the quality of the National Assessment of Fourth Grade Students include that the quality of items and test instruments are verified by external experts and are based on psychometric analysis, all proctors or administrators are trained according to a protocol, there is a standardized manual for large-scale assessment administrators, a pilot is conducted before the main data collection takes place, all booklets are numbered, there is double data scoring, and scorers are trained and monitored to ensure high interrater reliability, and data entry is monitored as well. 20. There is some documentation about the technical aspects of the National Assessment of Fourth Grade Students, but it is not in a formal report format. 21. Results of the National Assessment of Fourth Grade Students are disseminated within twelve months after the assessment is administered, reports with results are made available for all stakeholder groups, and the main reports on the results contain information on overall achievement levels and subgroups. The main reports on the results do not, however, contain information on trends over time overall and for subgroups, do not contain standard errors (measures of uncertainty), there are no media briefings organized to discuss the results, there are no workshops or presentations for stakeholders on the results, and the results are not featured in the media (in newspapers, magazines, radio, or television). 22. Assessment information is used by the Institute as an input for developing standards of achievements for the first cycle of education as well as for developing criterion-referenced tests for schools. 23. There are no mechanisms in place to monitor the consequences of the National Assessment of Fourth Grade Students. 40 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SERBIA International Large-Scale Assessment (ILSA) 41 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in The country/system has participated in 1 in an ILSA in the last 10 years. dimension. at least one ILSA in the last 10 years. two or more ILSA in the last 10 years. * The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. 2 the next 5 years. the next 5 years. There is no policy document that There is an informal or draft policy There is a formal policy document that * This option does not apply to this 3 addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. dimension. ILSA. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. dimension. ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, 4 ILSA. donors. discretion. decree or norm. * This option does not apply to this Funding covers some core activities of Funding covers all core activities of the Funding covers all core activities of the 5 dimension. the ILSA. ILSA. ILSA. * Funding does not cover research and This option does not apply to this This option does not apply to this Funding covers research and 6 development activities. dimension. dimension. development activities. * (CONTINUED) 42 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. 7 activities. activities. activities. * This option does not apply to this The national/system coordinator or The national/system coordinator is fluent This option does not apply to this 8 dimension. other designated team member may not in the language of the assessment. dimension. be fluent in the language of the assessment. * This option does not apply to this The ILSA office is inadequately staffed or The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, 9 effectively. with minimal issues. with no issues. * 43 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this 10 international workshops or meetings. international workshops or meetings. workshops or meetings. dimension. The country/system offers no This option does not apply to this * The country/system offers some The country/system offers a wide range 11 opportunities to learn about ILSA. dimension. opportunities to learn about ILSA. of opportunities to learn about ILSA. * This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are dimension. dimension. available to the country's/system's ILSA available to a wide audience, in addition 12 team members only. to the country's/system's ILSA team * members. 44 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical The country/system met all technical published. standards to have its data presented standards required to have its data standards required to have its data beneath the main display of the presented in the main displays of the presented in the main displays of the * 13 international report or in an annex. international report. international report. The country/system has not contributed This option does not apply to this This option does not apply to this The country/system has contributed new 14 new knowledge on ILSA. dimension. dimension. knowledge on ILSA. * ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely 15 the country/system. in the country/system. the country/system. disseminated in the country/system. * Products to provide feedback to schools This option does not apply to this Products to provide feedback to schools Products to provide feedback to schools and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are 16 not made available. sometimes made available. systematically made available. * There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the There is wide media coverage of the ILSA 17 results. ILSA results. ILSA results. results. * If any, country/system-specific results Results from the ILSA are used in a Results from the ILSA are used in some Results from the ILSA are used in a and information from the ILSA are not limited way to inform decision making in ways to inform decision making in the variety of ways to inform decision 18 used to inform decision making in the the country/system. country/system. making in the country/system. country/system. * It is not clear that decisions based on This option does not apply to this This option does not apply to this Decisions based on the ILSA results have ILSA results have had a positive impact dimension. dimension. had a positive impact on students' 19 on students' achievement levels. achievement levels. * 45 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 International Large Scale Assessment (ILSA): Development-level rating justifications 1. Serbia has participated in a number of international large scale assessments, including TIMSS 2003, TIMSS 2007, TIMSS 2011, PISA 2003, PISA 2006, and PISA 2009. Serbia has consistently participated in PISA, and it most recently completed PISA 2009 which the remainder of the rubric refers to. 2. Serbia has taken concrete steps to participate in PISA 2012, and is planning to participate in PISA 2015 and TIMSS 2015. 3. There is no policy document that addresses participation in international assessments. 4. Funding for PISA 2009 is provided for by government funding that is supported by a World Bank loan. Funding for participation in other international large scale assessments is also provided for by government funding supported by loans (such as from the World Bank and the European Investment Bank). 5. Funding for PISA 2009 covered international participation fees, implementation of the assessment exercise in Serbia (e.g., printing booklets, travel to schools), attendance at international expert meetings for the assessment exercise, processing and analyzing data collected from implementation of the assessment exercise, and reporting and disseminating the assessment results in Serbia. 6. While donor funding does not cover research and development activities, the Institute of Psychology which implements PISA is a research institution. Additional research and secondary analyses are typically done by the Institute of Psychology and financed by the Institute itself by another organization at its request (e.g. Social Inclusion and Poverty Reduction unit of the Government of Serbia requested and financially supported the Institute of Psychology to make a targeted secondary analysis of PISA achievements and transition from compulsory to secondary education for students with lower SES. Results from this secondary analysis were used in the “First national report on poverty�). 7. Dragica Pavlović-Babić is the National Project Manager for PISA. The team responsible for carrying out PISA consist of a core group (4�5 people) and a large number of associates who are included in specific phases of the study(administration, marking, data entry, etc.). The core staff are full time employees at the Institute of Psychology (i.e. formally at the Faculty of Philosophy) as teaching and research staff with regular duties other than PISA. The network of associates are not employees of the Institute but are brought in on a temporary basis as the majority of them are teachers employed by schools or Faculties/Universities. Some of the associates are students or unemployed persons brought in to complete data entry or other temporary administrative tasks. TIMSS, on the other hand, is administered by the Institute of Pedagogical Research, which is a public institute. 8. The national coordinator is fluent in the language of the assessment. 9. The team is sufficiently staffed, however the size and structure of the team has not always been aligned to the needs of the project. In general, the trend is to increase the size of the core team. Some members of the core team are stretched by different PISA tasks and other duties they have as full time employees of the Institute of Psychology and the Faculty of Philosophy. At the same time, the core team's composition is interchangeable because different members of the core team can take on various additional tasks, which is particularly helpful during the implementation of many PISA activities. The Institute can extend the core team at its own discretion, but the size of the PISA budget received from the Ministry limits this expansion. Lack of financial resources 46 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 keeps the number at a minimum. The team has previous experience working on international assessments as well as the necessary training to carry out the required PISA activities effectively. The only issue identified with the carrying out of PISA in Serbia is that there have been errors or delays in the printing or layout of the test booklets. Specifically, the team in charge of the PISA exercise could not achieve the expected level of print quality (although the quality of the test booklets was still acceptable) due to lack of financial resources. 10. Team members attended most of the training organized by the OECD. Several trainings were not attended due to financial constraints (for example, if the same training was attended in previous cycles and if participation requires a high cost). 11. Limited opportunities to learn about international assessments include university courses on the topic of international assessments. A masters-level university course on educational assessment that includes international assessment studies is offered for students studying Education psychology at the University of Belgrade. The topic of the international studies will be also included in the new master program for educational policy at the University of Belgrade. Additionally, there is limited funding for attending international workshops or training on international assessments. 12. Individuals working directly on the specific international assessment exercise and university students studying assessment or a related area benefit from the opportunities to learn about international assessments. 13. Serbia met all technical standards required to have its data presented in the main displays of the international report. 14. Serbia has contributed new knowledge on PISA. 15. Results and information from the PISA exercise were regularly and widely disseminated in Serbia. Specifically, a national report was made available online, copies of the national and international report were distributed to key stakeholders, Serbia's results were communicated through a press release, brochures and PowerPoint presentations with Serbia's results were made available online or distributed to key stakeholders, and products providing feedback to the schools and educators about the results were made available. 16. Participating schools receive the school report as well as the National report (other schools can download the national report from Serbian PISA site). In addition, some schools have asked the National Center to visit the school and make a presentation and/or discuss PISA findings with teachers. 17. Media coverage has included editorials or columns commenting on the international assessment results, including, for example, how PISA results can help improve education. The quality of the discussion of international assessment results in the media has also become more professional. In 2012, media coverage of implementation of PISA was very high. For example, Politika, a reputable daily newspaper, published and analyzed the administered PISA items two times per week over three months. 47 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 18. The results and analysis of PISA 2009 data has informed the recently announced National Strategy for Development of Education until 2020. Additionally, the results of PISA are used by the National Education Council for monitoring the quality and equity of education in Serbia. PISA results were used in the First report on poverty reduction produced by the Government of Serbia. Additionally, PISA results have been used to inform teacher training programs and other assessment activities in the system (e.g., classroom assessment, examinations), for developing education standards for the end of compulsory and secondary education, and for informing the framework for the law on education and regulations governing the assessment practices in schools. The results of PISA 2009 have not been used to track the impact of reforms on student achievement levels, to inform curriculum improvement, or to inform resource allocation. Serbia is currently implementing a project of formulating education standards for the end of general secondary education. PISA results, as well as the PISA concept of literacy, informed the working groups responsible for developing the standards of students achievements. 19. There is evidence that the achievement of students in Serbia significantly increased in PISA 2009, in comparison to previous PISA cycles. However, during the same period, there were no systematic changes in the education system. At least in part, the growth of achievement could be explained by the changing teaching practices that resulted from specific teacher training programs, as well as by the increased general understanding of such concepts as education competence, quality of education, and functional literacy, all concepts which were informed by international assessments. 48 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Acknowledgements This report was prepared by The World Bank SABER- Student Assessment team in collaboration with Ivana Aleksic, World Bank Education Specialist and Task Team Leader for education projects in Serbia, and Aleksandar Baucal, Professor at the University of Belgrade, Department of Psychology. References Clarke, M. 2012. “What Matters Most for Student United Nations Educational, Scientific, and Cultural Assessment Systems: A Framework Paper.� Organization (UNESCO). Serbia Country Indicator Data. READ/SABER Working Paper Series. Washington, DC: Montreal, QC: UNESCO. Data retrieved from World Bank. http://stats.uis.unesco.org/unesco on March 14, 2013 Organisation for Economic Co-operation and World Bank. 2013. “Serbia Country Overview� Data Development (OECD). 2012. “Strengthening Integrity retrieved from http://www.worldbank.org/en/ and Fighting Corruption in Education: Serbia.� Paris: country/serbia/overview on March 16, 2013. OECD. ———. 2011. “Education at a Glance 2011: OECD ———. 2013. World Bank Development Indicators: Indicators.� Paris: OECD. Armenia Country Indicator Data. Washington, DC: World Bank. Data retrieved from United Nations Children’s Fund (UNICEF). 2008. http://databank.worldbank.org/data on March 12, “Country profile: Education in Serbia.� Data retrieved 2013. from http://www.unicef.org/ceecis/Serbia.pdfon March 12, 2013. 49 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policymakers and business people—an accessible, objective snapshot showing how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. THE WORLD BANK 50 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS