76462 Serbia SABER Country Report STUDENT ASSESSMENT 2012 Key Policy Areas for Student Assessment Status 1. Classroom Assessment There are several formal policy documents that provide detailed guidelines on classroom assessment. Although these policy documents are available online, and required to be available in primary and secondary schools, teachers are usually familiar only with those provisions that relate to the number of assessments that are required to be carried out. There are limited required uses of classroom assessment information to support student learning. In general, classroom assessment practices provide little feedback to students and parents, tend not to be aligned with the pedagogical or curricular framework, and are considered weak. 2. Examinations The Final Exam in Primary School Education was administered for the first time in 2011 to grade 8 students. According to the Law on the Foundation of the Education System, the three main purposes of the Final Exam are (i) student certification for primary school completion, (ii) student selection to secondary school, and (iii) monitoring of education quality levels. There are limited systematic mechanisms in place to ensure the quality of the Final Exam, and there are limited options for students who do not perform well on the exam. No mechanisms are in place to monitor the consequences of the Final Exam. 3. National Large-Scale Assessment (NLSA) The National Assessment of Fourth Grade Students was administered in 2006 to a sample of students. The main objective of the survey was to collect data on education practices and overall levels of student achievement in Serbian language and mathematics for the purpose of improving the education system. Limited resources did not allow for all student groups to be represented in the sample chosen for the national assessment. While Serbian policymakers support national assessment exercises in principle, there is no plan for, or budget allocated to, future national large-scale assessment exercises. 4. International Large-Scale Assessment (ILSA) Serbia has participated in a number of international assessment exercises, including TIMSS (2003, 2007, 2011) and PISA (2003, 2006, 2009). Serbia will participate in PISA 2012, and is planning to participate in PISA 2015 and TIMSS 2015. However, there is no formal policy document that addresses the country's participation in international assessments. Funding for Serbia's participation in international assessment exercises, including the recently concluded PISA 2009, has been provided by donors, including the World Bank. THE WORLD BANK Introduction SABER-Student Assessment methodology The SABER-Student Assessment framework is built on Serbia has focused on increasing student learning the available evidence base for what an effective outcomes by improving the quality of education in the assessment system looks like. The framework country. An effective student assessment system is an provides guidance on how countries can build more important component to improving education quality effective student assessment systems. The framework and learning outcomes as it provides the necessary is structured around two main dimensions of information to meet stakeholders’ decision-making assessment systems: the types/purposes of needs. In order to gain a better understanding of the assessment activities and the quality of those strengths and weaknesses of its existing assessment activities. system, Serbia decided to benchmark this system using standardized tools developed under The World Assessment types and purposes Bank’s Systems Approach for Better Education Results (SABER) program. SABER is an evidence-based Assessment systems tend to be comprised of three program to help countries systematically examine and main types of assessment activities, each of which strengthen the performance of different aspects of serves a different purpose and addresses different their education systems. information needs. These three main types are: classroom assessment, examinations, and large-scale, What is SABER-Student Assessment? system level assessments. Classroom assessment provides real-time information SABER-Student Assessment is a component of the to support ongoing teaching and learning in individual SABER program that focuses specifically on classrooms. Classroom assessments use a variety of benchmarking student assessment policies and formats, including observation, questioning, and systems. The goal of SABER-Student Assessment is to paper-and-pencil tests, to evaluate student learning, promote stronger assessment systems that contribute generally on a daily basis. to improved education quality and learning for all. Examinations provide a basis for selecting or certifying National governments and international agencies are students as they move from one level of the increasingly recognizing the key role that assessment education system to the next (or into the workforce). of student learning plays in an effective education All eligible students are tested on an annual basis (or system. The importance of assessment is linked to its more often if the system allows for repeat testing). role in: Examinations cover the main subject areas in the curriculum and usually involve essays and multiple- (i) providing information on levels of student choice questions. learning and achievement in the system; (ii) monitoring trends in education quality over time; Large-scale, system-level assessments provide (iii) supporting educators and students with real- feedback on the overall performance of the education time information to improve teaching and system at particular grades or age levels. These learning; and assessments typically cover a few subjects on a (iv) holding stakeholders accountable for results. regular basis (such as every 3 to 5 years), are often sample based, and use multiple-choice and short- answer formats. They may be national or international in scope. Appendix 1 summarizes the key features of these main types of assessment activities. 2 Quality drivers of an assessment system Table 1: Framework for building an effective assessment system, with indicator areas The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or policy framework for assessment The indicators are identified based on a combination activities; institutional and organizational structures of criteria, including: for designing, carrying out, or using results from the assessment; the availability of sufficient and stable  professional standards for assessment; sources of funding; and the presence of trained  empirical research on the characteristics of assessment staff. effective assessment systems, including analysis of the characteristics that differentiate System alignment refers to the extent to which the between the assessment systems of low- versus assessment is aligned with the rest of the education high-performing nations; and system. This includes the degree of congruence  theory—that is, general consensus among between assessment activities and system learning experts that it contributes to effective goals, standards, curriculum, and pre- and in-service assessment. teacher training. Levels of development Assessment quality refers to the psychometric quality of the instruments, processes, and procedures for the The World Bank has developed a set of assessment activity. It covers such issues as design standardized questionnaires and rubrics for and implementation of assessment activities, analysis collecting and evaluating data on the three and interpretation of student responses to those assessment types and related quality drivers. activities, and the appropriateness of how assessment results are reported and used. The questionnaires are used to collect data on the characteristics of the assessment system in a Crossing the quality drivers with the different particular country. The information from the assessment types/purposes provides the framework questionnaires is then applied to the rubrics in order and broad indicator areas shown in Table 1. This to judge the development level of the country’s framework is a starting point for identifying indicators assessment system in different areas. that can be used to review assessment systems and plan for their improvement. The basic structure of the rubrics for evaluating data collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice in each area. For each indicator, the rubric displays four development levels—Latent, Emerging, Established, 3 and Advanced. These levels are artificially constructed years); primary education from grades 1 through 8; categories chosen to represent key stages on the and secondary education, which consists of either underlying continuum for each indicator. Each level is four years gymnasium or three to four years of accompanied by a description of what performance vocational. Primary education is compulsory, and the on the indicator looks like at that level. preparatory pre-primary grade has been mandatory since the 2006-2007 school year. The gross enrollment rate in primary education is approximately  Latent is the lowest level of performance; it 95% (in 2011) and the completion rate is represents absence of, or deviation from, the approximately 94%. The gross enrollment rate in desired attribute. secondary education is approximately 91% and the  Emerging is the next level; it represents partial completion rate is approximately 99%. presence of the attribute.  Established represents the acceptable minimum standard. Serbia has made important efforts to improve its  Advanced represents the ideal or current best education system, particularly given the government’s practice. goal of acceding to the European Union. The government’s priorities for education include adoption of new approaches to classroom teaching A summary of the development levels for each and enhancement of the integrity and efficiency of assessment type is presented in Appendix 3. education administration through development of education management information systems. In reality, assessment systems are likely to be at different levels of development in different areas. Detailed information was collected on Serbia’s For example, a system may be Established in the area student assessment system using the SABER-Student of examinations, but Emerging in the area of Assessment questionnaires and rubrics. It is important large-scale, system-level assessment, and vice versa. to remember that these tools primarily focus on While intuition suggests that it is probably better to benchmarking a country’s policies and arrangements be further along in as many areas as possible, the for assessment activities at the system or macro level. evidence is unclear as to whether it is necessary Additional data would need to be collected to to be functioning at Advanced levels in all areas. determine actual, on-the-ground practices in Serbia, Therefore, one might view the Established level as a particularly by teachers and students in schools. desirable minimum outcome to achieve in all areas, but only aspire beyond that in those areas that most contribute to the national vision or priorities for The following sections discuss the findings by each education. In line with these considerations, the assessment type, accompanied by suggested policy ratings generated by the rubrics are not meant to be options. The suggested policy options were additive across assessment types (that is, they are not determined in collaboration with key local meant to be added to create an overall rating for an stakeholders based on Serbia’s immediate interests assessment system; they are only meant to produce and needs. Detailed, completed rubrics for each an overall rating for each assessment type). The assessment type in Serbia are provided in Appendix 5. methodology for assigning development levels is summarized in Appendix 4. Education in Serbia Serbia is an upper-middle-income country in South Eastern Europe. GDP per capita (current US$) is $6310.40, with annual growth of 2% in 2011. The Serbian education system is comprised of pre- primary (for children between the ages of 0.5 and 6.5 4 Classroom Assessment Classroom assessment information is required to be disseminated to students and parents. Parents and Level of development: EMERGING students receive information on the student's performance at the end of the semester and school year in an official form (student book). If a student has a negative assessment, the parent receives official The Law on the Fundamentals of the Education information at the end of the first or third trimester as System (2009), the Regulations on Student well. During the academic year, parents receive Assessment in Primary Education (2011), and the information on student performance during regular Regulations on Student Assessment in Secondary parent-teacher meetings. Education (1999) provide detailed guidelines on classroom assessment. Although they are available online, and required to be available in primary and There are limited required uses of classroom secondary schools, teachers usually are familiar only assessment information to support student learning. with those provisions that relate to the number of These include providing feedback to students on their assessments that are required to be carried out. learning, informing parents about their child’s learning, and planning next steps in instruction. In reality, ‘on-the-ground’ classroom assessment There are a variety of system-wide resources available practices provide little feedback to students and to teachers for classroom assessment. Education parents, are often not aligned with the pedagogical or standards describe what students are required to curricular framework, and are considered weak. know and be able to do at the end of the two cycles of primary education as well as at the end of four years of secondary education. Teachers are also provided with textbooks or workbooks that assist them with Suggested policy options: implementing classroom assessment as well as with scoring criteria/rubrics for evaluating students’ work. 1. Introduce required courses for teachers on classroom assessment, especially on the topic There are some system-level mechanisms to ensure of formative assessment. Additionally, that teachers develop appropriate skills in classroom develop minimum standards for teacher assessment. For example, teachers are required to competencies in classroom assessment and receive 120 hours of accredited training over five make the minimum standards part of the core years and may elect to receive some of this on curriculum for pre-service teacher training. classroom assessment. Additionally, the Institute for For teachers already in service, introduce Education Quality and Evaluation implemented one- professional development training on day trainings for about 12,000 teachers, principals, classroom assessment. and school counselors in 2010. Training topics have included implementation of education standards and student assessment. However, there are limited 2. Introduce mechanisms to monitor the quality systematic mechanisms to monitor the quality of of classroom assessment practices into the classroom assessment practices. Classroom mandate of educational supervisors. This will assessment is a required component of a teacher’s help ensure that classroom assessment is performance evaluation and is a required component based on standards, and is consistent and of school inspection or teacher supervision. comparable between schools. Several recently-adopted rulebooks describe the 3. Develop regular school reports that combine relationship between assessment and the the results of the external examination and competencies of teachers. Educational advisors and classroom assessment to enable schools to school inspectors monitor and evaluate teachers’ better monitor the quality of their classroom performance with respect to classroom assessment, assessment activities. taking into consideration these rulebooks. 5 Examinations Regular funding for the Final Exam is provided by the government and covers all core examination activities Level of development: EMERGING (examination design, administration, data processing, and reporting), as well as research. Development activities have been supported under the European Union Support for Quality Assurance Within the In Serbia, the Final Exam in Primary School Education National Primary and Secondary Education was administered for the first time in 2011 to grade 8 Examination System project. students. According to the Law on the Foundation of the Education System, which authorizes the Teachers are involved in most examination-related examination, its three main purposes are for (i) tasks. The Institute establishes working groups of student certification of primary school completion, (ii) subject teachers for writing items and developing the student selection to secondary school, and (iii) examination scoring guide. Teachers are also involved monitoring of education quality levels. in scoring the examination. Additionally, teachers serve as test administrators and supervisors, but not Secondary schools enroll students using the “total in their own school. At the same time, there are no enrollment score� (school grades, which account for general courses or workshops on the Final Exam 60% + Final Exam score, which accounts for 40%). available to teachers. Teachers who are involved in Students can apply to various schools with their examination activities only receive instructions and school grades and Final Exam scores and select which guidelines for the implementation of those activities. school to attend from the ones that accept them. Students who do not perform well on the examination There is some documentation about the technical may choose to enroll in less selective secondary aspects of the Final Exam, but it is not in a formal schools. If a student is not satisfied with their Final report format. Additionally, limited systematic Exam results and cannot enroll in a preferred mechanisms are in place, such as internal review of secondary school, he or she can re-take the Final the quality of items and test materials, and external Exam in June of the following year. review of exam implementation, to ensure the quality of the Final Exam. There are also no mechanisms in The material needed to prepare for the Final Exam is place to monitor the consequences of the Final Exam. widely accessible by over 90% of the students in a variety of learning contexts (e.g., covered in public school, available for home schooling, available on line). Materials include examples of the types of Suggested policy options: questions that are on the exam, information on how to prepare for the exam, and a framework document 1. Develop and implement additional explaining what is measured by the exam. mechanisms for assuring the quality of invigilation and marking of the Final Exam, There have been coordinated attempts to improve such as ensuring that all students take the the Final Exam by stakeholder groups, which have exam under the same conditions, and that been welcomed by the Ministry of Education and Science and the Institute for Education Quality and test marking is objective, reliable, and valid. Evaluation (the Institute). Both entities share the responsibilities for carrying out the Final Exam. The 2. Develop and implement mechanisms for the Institute prepares the exam framework, test improvement of the quality of the Final Exam materials, data analysis, and the data report. The items and instruments. Ministry of Education and Science is responsible for the implementation of the Final Exam, including the distribution of test materials, organizing exam rooms, 3. Establish a national commission for test administration, exam supervision, test marking, examinations composed of key stakeholders and data entry. The Institute is accountable to the with a mandate to monitor the quality of the Ministry of Education and Science, which is Final Exam. accountable to the Government of Serbia. 6 National Large-Scale Assessment (NLSA) and short-term employment opportunities at the Institute provide hands-on NLSA experience. There are no courses or workshops on national large-scale Level of development: EMERGING assessment available to teachers in the country. Mechanisms are in place to ensure the quality of the National Assessment of Fourth Grade Students, The National Assessment of Fourth Grade Students including verification of the quality of the test items was administered for the first time in 2006. The test and instruments by external experts and psychometric was administered to a representative sample of analysis, and use of a standardized manual for the students educated in the Serbian language who were assessment administration. able to take paper-and-pencil tests. The main objective of the survey was to collect data on existing While there is some documentation about the education practices and overall levels of student technical aspects of the National Assessment of achievement in Serbian language and mathematics for Fourth Grade Students, it is not in a formal report the purpose of improving the education system. format. While there are formal and informal policy documents Results from the National Assessment of Fourth Grade that authorize the National Assessment of Fourth Students are used by some stakeholder groups in a Grade Students – including the Law on the way that is consistent with the purposes and technical Foundations of the Education System (2009), the Law characteristics of the assessment. For example, the on Amendments and Additions to the Law on the Institute used the 2006 assessment results as an input Fundamentals of the Education System (2011), and for developing standards of achievement for the first the draft of the Law on Primary Education (2012) – cycle of education as well as for developing criterion- these policy documents are not available to the referenced tests for schools. At the same time, there public. There is also no firm plan in place for carrying are no mechanisms in place to monitor the out the assessment in the coming years. consequences of the National Assessment of Fourth Grade Students. In Serbia, NLSA studies are financed on an ad hoc basis by donors due to the lack of financial resources in the budget of the Ministry of Education and Science. The funding provided covers all core NLSA Suggested policy options: activities (design, administration, analysis, and reporting), as well as staff training. Funding does not cover research and development activities. The 1. Adopt a policy document prescribing how limited resources available meant that not all student NLSA data will be used to support monitoring and improvement of the quality and equity of groups were represented in the sample chosen for the education. 2006 National Assessment of Fourth Grade Students. 2. Ensure that all student groups are included in The Institute for Education Quality and Evaluation (the the sample for the NLSA (minority languages Institute) is in charge of the National Assessment of Fourth Grade Students. While the knowledge and and students with learning difficulties and/or competence of Institute staff is sufficient, the staff has disabilities). to work overtime when it needs to complete tasks for other assessment-related activities (such as the Final 3. Develop a multi-year plan for carrying out Exam in primary education). NLSA activities. 4. Ensure that NLSA activities are part of the There are some opportunities to prepare individuals national multi-year budget planning, and for work on the National Assessment of Fourth Grade include resources to allow for the inclusion of Students, including university graduate programs vulnerable student populations in the national (masters or doctorate level) and courses (graduate and non-graduate) specifically focused on educational assessment. measurement and evaluation. In addition, internships 7 International Large-Scale Assessment To date, Serbia has met all technical standards required to have its data presented in the main (ILSA) displays of the international reports for each ILSA. The country also has contributed new knowledge on Level of development: ESTABLISHED international large-scale assessment to the global evidence base. While media coverage of ILSAs is limited to a few Serbia has participated in a number of international small articles, the discussion of (for example) PISA assessment exercises, including TIMSS (2003, 2007, results in the Serbian media has tended to focus on 2011) and PISA (2003, 2006, 2009). Serbia will key issues, including how PISA results can help participate in PISA 2012, and is planning to participate improve education. in PISA 2015 and TIMSS 2015. However, there is no formal policy document that addresses the country’s Results from the ILSAs are used in some ways to participation in international assessments. inform decision making in Serbia. For example, the results and analysis of PISA 2009 data has informed Funding for Serbia’s participation in international the recently announced National Strategy for assessment exercises, including the recently Development of Education until 2020. Additionally, concluded PISA 2009, has been provided by donors, the results of PISA are used by the National Education including the World Bank. Funding for research and Council for monitoring the quality and equity of development related to these assessment exercises education in Serbia. PISA results have been used to and their results has been provided by the Institute of inform teacher training programs and other Psychology at Belgrade University (which implements assessment activities in the system (e.g., classroom PISA) and by other institutes that request specific assessment, examinations), for developing education research and development studies. standards for the end of compulsory and secondary education, and for informing the framework for the There is a team and national coordinator to carry out law on education and regulations governing the the ILSA activities. For example, in addition to the assessment practices in schools. National Project Manager, the team responsible for carrying out PISA consists of a core group of 4-5 people and a large number of associates who are included in specific phases of the study Suggested policy options: (administration, marking, data entry, etc.). The core staff is comprised of full time employees at the 1. Prepare a policy document that provides Institute of Psychology who have regular duties other guidelines for Serbia’s participation in ILSAs, than PISA activities. The team is sufficiently staffed, specifying the rationale for Serbia’s although the size and structure of the team has not participation in a select number of always been aligned to the needs of the project. In international large-scale assessments. general, the trend has been to increase the size of the core team. Due to financial constraints, the team was able to attend only some of the international 2. Increase local capacity on the use of ILSA workshops and meetings for PISA. databases to inform policy decisions. Opportunities to learn about international 3. Ensure that ILSA activities are part of the assessments are available to individuals working national multi-annual budget planning, and directly on a specific international assessment include the cost of participation in the chosen exercise, and to university students studying number of ILSAs, as well as the follow up assessment or a related area. For example, a masters- analytical work to inform policy making. level university course on educational assessment that includes the topic of international assessments is offered at the University of Belgrade. 8 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations surveys National International Exit Entrance Purpose Frequency Who is tested? Format Coverage of curriculum Additional information collected from students? Scoring 9 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development Level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—ENABLING CONTEXT EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—SYSTEM ALIGNMENT SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—ASSESSMENT QUALITY AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses 10 Appendix 3: Summary of the Development Levels for Each Assessment Type Assessment Type LATENT EMERGING ESTABLISHED ADVANCED CLASSROOM ASSESSMENT EXAMINATIONS NATIONAL (OR SYSTEM- LEVEL) LARGE-SCALE ASSESSMENT INTERNATIONAL LARGE- SCALE ASSESSMENT 11 Appendix 4: Methodology for Assigning under ‘System Alignment’ (in the case of CLASS) were Development Levels set as ceiling scores, i.e., the overall mean score for the particular assessment type cannot be greater than the score for these key dimensions. These key 1. The country team or consultant collects variables include formal policy, regular funding, information about the assessment system in the having a permanent assessment unit, and the quality country. of assessment practices. 2. Based on the collected information, a level of development and score is assigned to each dimension in the rubrics:  Latent = 1 score point  Emerging = 2 score points  Established = 3 score points  Advanced = 4 score points 3. The score for each quality driver is computed by aggregating the scores for each of its constituent dimensions. For example: The quality driver, ‘Enabling Context,’ in the case of ILSA, has 3 dimensions on which a hypothetical country receives the following scores: Dimension A = 2 points; Dimension B = 2 points; Dimension C = 3 points. The hypothetical country’s overall score for this quality driver would be: (2+2+3)/3 = 2.33 4. A preliminary level of development is assigned to each quality driver. 5. The preliminary development level is validated using expert judgment in cooperation with the country team and The World Bank Task Team Leader. For scores that allow a margin of discretion (i.e., to choose between two levels of development), a final decision has to be made based on expert judgment. For example, the aforementioned hypothetical country has an ‘Enabling Context’ score of 2.33, corresponding to a preliminary level of development of ‘Emerging or Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses ‘Emerging’ as the most appropriate level. 6. Scores for certain key dimensions under ‘Enabling Context’ (in the case of EXAM, NLSA, and ILSA) and 12 SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Appendix 5: SABER-Student Assessment Rubrics for Serbia SERBIA Classroom Assessment 13 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. 1 assessment. classroom assessment. assessment. * 2 This option does not apply to this This option does not apply to this The availability of the document is The document is widely available. dimension. dimension. restricted. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment. resources available for teachers for 3 classroom assessment. * There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or students are expected to learn, but the students are expected to learn and to 4 to what level of performance. level of performance required is not what level of performance. clear. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms This option does not apply to this There are some system-level There are a variety of system-level to ensure that teachers develop skills dimension. mechanisms to ensure that teachers mechanisms to ensure that teachers and expertise in classroom assessment. develop skills and expertise in classroom develop skills and expertise in classroom 5 assessment. assessment. * 14 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are 6 from widespread weaknesses or there is known to be weak. known to be of moderate quality. known to be generally of high quality. no information available on classroom assessment practices. There are no mechanisms to monitor the * There are ad hoc mechanisms to monitor There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the 7 practices. practices. assessment practices. quality of classroom assessment * practices. ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key 8 stakeholders. stakeholders. stakeholders. * There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support classroom assessment to support classroom assessment to support 9 student learning. student learning, excluding its use as an student learning, including its use as an input for external examination results. input for external examination results. * 15 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Classroom Assessment: Development-level rating justifications 1. Three official documents authorized by the Ministry of Education and Science, provide the guidelines on classroom assessment: a. The Law on the Fundamentals of the Education System (2009) b. Regulations on student assessment in primary education (2011) c. Regulations on student assessment in secondary education (1999) 2. The documents are available online and are required to be available in primary and secondary schools. In practice, however, legal documents are often at the principal’s, pedagogue’s or psychologist’s office in schools and teachers do not always know the regulations well or are familiar with only the provisions that relate to the number of required assessments. 3. The resources that are typically available include a document that outlines what students are expected to learn and the level(s) of performance that students are expected to reach in different subject areas at different grade/age levels. In Serbia education standards of student achievement describe what students are required to know and be able to do at the end of two cycles of the primary education as well as the standards at the end of four years of secondary education. Teachers are also provided with textbooks or workbooks that assist them with implementing classroom assessment as well as scoring criteria/rubrics for students’ work. After the final exam at the end of primary education, the Institute for Education Quality and Evaluation provides schools with a report on the achievements of their students. This report also includes comparison of school achievement in the Serbian language or mother tongue and mathematics, with the average achievement of the municipality, county, school authority and the Republic and comparison of class achievements in the Serbian language or mother tongue and mathematics. All schools receive the Assessment from idea to practice (Ministry of Education and Sports, 2003) handbook during the training for applying “descriptive assessment� (narrative report on the standards that students achieved, as opposed to just school marks on the 1 to 5 scale). The handbook consists Assessment towards outcomes, written by Dragica Pavlovic-Babic and associates, and Assessment for student development written by Nenad Havelka, Emina Hebib and Aleksandar Baucal. 4. The two official documents are the Standards of student achievements at the end of the primary education document and the Standards for the end of the first cycle of the primary education document that addresses Serbian language, mathematics, nature, and society. 5. Teachers must receive 100 hours of accredited training every five years. The training can be selected from a list of accredited trainings. The Ministry can also require specific training to be offered to teachers in certain academic years, in which case teachers are required to participate in this training. For example, the Ministry of Education and Science has organized trainings for all teachers teaching first grade, and principals and school counselors in primary schools. 16 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 The Institute for Education Quality and Evaluation has implemented an in-service teacher training seminar on assessment for primary and secondary school teachers, principals and school counselors since 2005. The Institute for Education Quality and Evaluation has implemented one-day trainings for about 12,000 teachers, principals and school counselors in 2010. Training topics included implementation educational standards in planning class work and evaluating students. The Institute for Improving Education invites applications each year for professional development programs for school staff. Various institutions and organizations submit their teacher training programs for accreditation. Once a training program is accredited, teachers may choose to participate in it for their professional development. These programs are published each year in the catalogue which is available on-line and in a paper version. Some of the courses are systematic and some are ad-hoc. 6. Although teachers rarely rely mainly on multiple-choice, selection-type questions and it is not common to observe errors in the scoring or grading of students' work, classroom assessment practices are mainly about recalling information, provide little useful feedback to students, and are mainly used as administrative or control tool rather than as pedagogical resource. Additionally, it is very common to observe uneven application of standards for grading students’ work and grade inflation is a serious problem. It is common for teachers to not use explicit or a priori criteria for scoring or grading students' work and parents are often poorly informed about students' grades. It is also common for classroom assessment practices to not be aligned with the pedagogical or curricular framework. 7. Classroom assessment is a required component of a teacher’s performance evaluation and it is a required component of school inspection or teacher supervision. Several rulebooks that were recently adopted describe the relation between assessment and the competencies of teachers. Educational advisors and school inspectors monitor and evaluate teachers’ performance with respect to classroom assessment taking into consideration these rulebooks. These rulebooks include: - The Regulation on continuous professional development and gaining the title of a teacher, preschool teacher or school councilor - The Regulation on the working license for teachers, preschool teachers and school councilors - The Regulations on professional-pedagogical monitoring, - The Regulations on evaluating the quality of educational institutions is in the final phase of development. It will regulate the external evaluation of schools that will be based on the Quality standards for educational institutions 17 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 8. Teachers are required to report on individual student’s performance to students and parents. Parents and students receive information on student's performance at the end of the semester and school year in an official form (student book). If a student has a negative assessment, the parent receives official information at the end of the first or third trimester as well. During the academic year, parents receive information on student performance informally during regular parent-teacher meetings. These meetings are held by the class-teacher or main subject-teacher. Parents can also obtain information on student's performance during individual consultations, usually on pre-scheduled dates. Parents can talk to the class-teacher or subject- teacher. While teachers should meet with parents during regular parent-teacher meetings, the frequency of these meetings is not strictly regulated. 9. The required uses of classroom assessment activities to promote and inform student learning include providing feedback to students on their learning, informing parents about their child’s learning, and planning the next steps in instruction. Regulation on assessing students was changed based on the changes in the education system. In more detail than before, the regulation describes the ways of assessing and the criteria that needs to be fulfilled for a particular assessment. It points out the need to respect the conditions in which the student learns, as well as the circumstances in which teaching takes place. The key innovations are also the obligatory initial testing by instruments created by teachers in order to assist in planning of teaching/learning. Formative classroom assessment is introduced in this way. The general criteria and the framework for assessing the quality of learning as well as the criteria for assessing student behavior are defined in this regulation. The innovations in the regulations refer to the relation between the assessment and education or achievement standards. Students with special needs or students in disadvantaged living conditions who have an individual educational plan (IEP) are evaluated based on the adjusted requirements and standards that are defined in that IEP. Under the Regulations, the assessment is supposed to be an objective and reliable measure of progress and development, and an indication of quality and efficiency of the work of teachers and schools. 18 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SERBIA Examinations 19 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this 1 place. operating on an irregular basis. has been operating regularly. dimension * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this 2 authorizes the examination. document that authorizes the authorizes the examination. dimension. examination. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 3 dimension. the public public. dimension. * This option does not apply to this This option does not apply to this The policy document addresses some The policy document addresses all key 4 dimension. dimension. key aspects of the examination. aspects of the examination. * ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the 5 the examination or are indifferent to it. examination. examination. examination. * There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder 6 groups. groups. Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this * not welcomed by the leadership in dimension. generally welcomed by the leadership in dimension. 7 charge of the examination charge of the examination. * 20 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 (CONTINUED) LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this 8 examination. the examination. examination. dimension. * This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. 9 processing or reporting. processing and reporting. * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and 10 dimension. development. dimension. development. * ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or The examination office is newly The examination office is a stable This option does not apply to this 11 is newly established. established. organization. dimension. * The examination office is not This option does not apply to this The examination office is accountable to This option does not apply to this 12 accountable to an external board or dimension. an external board or agency. dimension. agency. * Examination results are not recognized Examination results are recognized by Examination results are recognized by Examination results are recognized by by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection 13 country. another country. system in another country. * The examination office does not have The examination office has some of the The examination office has all of the The examination office has state of the the required facilities to carry out the required facilities to carry out the required facilities to carry out the art facilities to carry out the examination. 14 examination. examination. examination. * (CONTINUED) 21 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment 15 examination, issues are pervasive. effectively, with minimal issues. effectively, with no issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities The country offers a wide range of that prepare for work on the dimension. that prepare for work on the opportunities that prepare for work on 16 examination. examination. the examination. * 22 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what This option does not apply to this 17 measures. dimension. the examination measures. dimension. What the examination measures is This option does not apply to this * What is measured by the examination is This option does not apply to this questioned by some stakeholder groups. dimension. largely accepted by stakeholder groups. dimension. 18 * Material to prepare for the examination There is some material to prepare for the There is comprehensive material to There is comprehensive material to is minimal and it is only accessible to examination that is accessible to some prepare for the examination that is prepare for the examination that is 19 very few students. students. accessible to most students. accessible to all students. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses There are up-to-date compulsory courses 20 examinations available to teachers. workshops on examinations available to or workshops on examinations available or workshops on examinations for teachers. to teachers. teachers. Teachers are excluded from * all Teachers are involved in very few Teachers are involved in some Teachers are involved in most 21 examination-related tasks. examination-related tasks. examination-related tasks. examination-related tasks. * 23 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical There is a comprehensive, high quality documentation. examination, but it is not in a formal report but with restricted circulation. technical report available to the general 22 report format. public. * There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the 23 examination. quality of the examination. * ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the 24 examination process is high. examination process is moderate. examination process is low. examination process is marginal. * The examination results lack credibility The examination results are credible for The examination results are credible for This option does not apply to this 25 for all stakeholder groups. some stakeholder groups. all stakeholder groups. dimension. * The majority of the students (over 50%) A significant proportion of students A small proportion of students (less than All students can take the examination; may not take the examination because of (10%-50%) may not take the examination 10%) may not take the examination there are no language, gender or other 26 language, gender, or other equivalent because of language, gender, or other because of language, gender, or other equivalent barriers. barriers. equivalent barriers. equivalent barriers. * (CONTINUED) 24 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all 27 proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way. 28 * Student names and results are public. This option does not apply to this Students’ results are confidential. This option does not apply to this dimension. dimension. * ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who There are very limited options for There are some options for students who There is a variety of options for students do not perform well on the examination, students who do not perform well on the do not perform well on the examination. who do not perform well on the 29 or students must leave the education examination. examination. system. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the 30 examination. examination. examination. * 25 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Examinations: Development-level rating justifications 1. The Final Exam in Primary School Education is a new examination program that started in 2011. Students in grade 8 are required to take the exam. The Law on the Foundation of Education System defines the three main purposes of the exam: student certification for grade or school cycle completion; student selection to secondary school; and monitoring education quality levels. This law also identifies the secondary purposes of the examination: planning education policy reforms; school or educator accountability; and self evaluation of schools. While the Final Exam is taken for selection for enrollment in secondary schools, there is no formal cut-off score. Different secondary schools enroll students with different total enrollment score (school grades + Final exam score). For example, students applying for secondary medical schools need to earn more than 95 points out of 100 to be enrolled. Students can apply to various schools with their school grades and exam scores and then they select which school to attend out of the ones that accept them. 2. Policy documents that authorize the Final Exam include: - The Law on the Fundamentals of the Education System (2009), authorized by the Serbian Parliament. - The Law on Amendments and Additions to the Law on the Fundamentals of the Education System (2011), authorized by the Serbian Parliament - The Regulations on the Program of the Final Exam in Elementary Education (2011), authorized by the Serbian Government - Manual for Implementation of the Final Exam (2011), authorized by the Ministry of Education 3. The Laws, Regulations, and Manual are available online. 4. The Law on the Fundamentals of the Education System describes the purpose of the examination and specifies who can sit for the examination. The Regulations on the Program of the Final Exam in Elementary Education describe the purpose of the examination, outline the procedures for special/disadvantaged students, specify who can sit for the examination, explain the alignment of the examination with curricula and standards, and explain the format of the examination questions. The Manual for Implementation of the Final Exam outlines the governance, distribution of power, and responsibilities among key entities, outlines the procedures to investigate and address security breaches, cheating, or other forms of inappropriate behavior, and identifies rules about preparation. The policy documents do not describe the authorized uses of results or the sources of funding. 26 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 5. Policy makers strongly support the Final Exam, particularly because of their need to have an instrument for monitoring the quality and equity of primary education. Teacher unions and educators support the introduction of the new Final Exam because it consists of items previously not available to students (which was not the case with the previously administered exam). Teacher unions and educators have expressed that cheating needs to be prevented through more effective test administration, and because students cannot fail (students are must take the Final Exam, but not to achieve certain minimal result), students may not be making a strong effort to prepare for the exam. Students and parents accept the Final Exam because they are used to having an exam at the end of the primary education. They also believe that it is fair to make enrolment decisions based on results on a national exam together with school marks than only based on school marks. The media follows the Final Exam only during its administration. The media provides information on the exam as opposed to criticism or praise. The European Commission has provided strong technical and financial support for the development of the Final Exam within the IPA 2008 project Support for quality assurance within the national primary and secondary education examination system. 6. Efforts to improve the Final exam are coordinated by the EU project that provides support to the Ministry of Education for the improvement of the implementation of the Final Exam. The project also supports the efforts of the Institute for Education Quality and Evaluation to improve the quality of test materials, data analysis, and data reporting. The Institute for Education Quality and Evaluation ("the Institute") is also planning a consultation process with teachers and students to identify areas for potential improvement. 7. Efforts to improve the Final Exam are very welcomed by the Ministry of Education and Science and the Institute. For example, the Ministry negotiated with the European Commission to receive technical and financial support through the IPA project and the Institute initiated independently activities aimed toward improving its assessment activities. 8. The implementation of the Final Exam is supported by regular government budget. The budget covers preparation of test materials, data analysis and reporting by the Institute, as well as the administration of the exam by the Ministry. Additional activities related to the Final exam (for example, secondary analysis of the final exam results) are co-financed through IPA. The main aim of the IPA project is to build the capacity of the Institute. 9. Funding for core examination activities is provided by the government. 10. Developmental activities are supported by the EU project “Support for quality assurance within the national primar y and secondary education examination system.� 27 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 11. There is shared responsibility between the Ministry of Education and Science and the Institute for Education Quality and Evaluation for carrying out the Final Exam. The Institute prepares the exam framework, test materials, data analysis, and data report. The Ministry of Education and Science is responsible for implementation of the exam (distribution of test materials, organizing exam rooms, test administration, exam supervision, test marking, and data entry). The distribution of these responsibilities is identified in the Law on Foundations of Education System that was adopted in 2009. 12. The Institute is accountable to the Ministry of Education and Science, which is accountable to the Government. 13. Examination results are recognized by the certification or selection system in Serbia. 14. The Institute is accommodated in a secure building that has secure storage facilities and is equipped with computers. The Institute has access to computer servers, backup devices, and has adequate communication tools, however their quality and capacity are not aligned with the Institute needs – for example, current servers do not have enough capacity to support the Final Exam database, therefore the Institute must rely on external servers. 15. There is permanent or full-time staff, but it is insufficient to meet needs of the examination. Specifically, according to the strategy of the Institute as approved by the Government, the Institute should have 60 employees. However, because of budgetary restrictions the Institute has only 37 employees. For effective operation, at least 13 additional employees are needed. No systematic or serious issues have been identified with the performance of the human resources that are responsible for the Final exam. From time to time, however, some item turns out to be problematic and need to be excluded, marking criteria are applied somewhat differently across the country as teachers from different schools score the Final Exam papers, and/or some exam administrators do not strictly follow the rules and procedures for administrating the Final Exam. These issues, however, typically do not have serious impact on the quality and equity of the Final Exam. 16. The country offers university graduate programs (masters or doctorate level) and courses (graduate and non-graduate), for example at the University of Belgrade, that are specifically focused on educational measurement and evaluation. It is also possible for junior experts to do an internship at the Institute. 17. The Final Exam measures national school curriculum guidelines or standards. According to the Law on Foundation of Education System, the Final Exam is defined as an exam that serves to assess the level of attainment of general and specific standards of achievement in elementary school education. 18. The content of the Final Exam is defined by the Law on Foundation of Education System (general and specific standards). However, a key issue among educators is related to the content of items. Subject specialists argue for more items that assess the recall of key facts and knowledge from a given school subject, while education specialists argue for more items measuring subject specific and cross curricular competencies (items similar to those used in the PISA study). 28 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 19. The material needed to prepare for the Final Exam is widely accessible by all students (over 90%) in a variety of learning contexts (e.g., covered in public school, available for home schooling, available on line). Materials can be downloaded for free from the Internet site of the Institute. Materials include examples of the types of questions that are on the exam, information on how to prepare for the exam, and the framework document explaining what is measured on the exam, The framework and examples of items are published as a part of the Regulations on the Program of the Final Exam in Elementary Education, and information on how to prepare for the examination is published in the Manual for Implementation of the Final Exam. While there is no provision for providing material to students who cannot access personal copies of the materials due to poverty or living in a distant rural area, during the month of May (prior to the administration of the Final Exam in June), 8th grade students have an opportunity to become familiar with the content and format of the exam by taking the trial version of the exam in Serbian language and mathematics. 20. There are no courses on the Final Exam for teachers. Teachers who are involved in certain examination activities only receive instructions and guidelines for the implementation of those activities. 21. The Institute establishes working groups of subject teachers for writing items and developing the scoring guide. Teachers are also involved in scoring the examination. Additionally, teachers serve as test administrators and supervisors, but not in their own school. 22. There is some documentation about the technical aspects of the Final Exam, but it is not in a formal report format. After implementation of the first Final Exam in 2011, the Institute prepared several analytical reports. These reports are only available to the audience for whom the reports were intended (schools or the Ministry). The report to the schools provided the comparison of a particular school in relation to the average achievement at the national, regional and municipality levels, by gender, subjects and items. The report for the Ministry of Education provided an analysis of Final Exam results by School Regional Administrations, region, municipality, and school level. The internal report on the quality of items and test instruments identified areas of improvement of the item writing and test development for future exams. The Institute is working on producing an additional report on the quality of scoring in order to identify potential issues and to develop instruments and procedures to improve marking. 23. Internal review (such as the internal review of the quality of items and test materials produced by the Institute), and external review (such as the supervision of exam implementation organized by the Ministry of Education and Science the (the implementation of the Final exam in 2011 was monitored in each school by one Ministry supervisor)), help ensure the quality of the Final Exam. 29 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 24. Copying from other candidates has been observed. In general, however, according to the Manual for Implementation of the Final Exam, schools are obligated to organize the Final Exam in one large room and to arrange the room in such way that the distance between students is at least 1.2 meters. Additionally, the administration of the Final Exam is supervised by an external supervisor who reports directly to the Ministry on the regularity of exam. 25. Although there is some indication that the administration of the Final Exam in some schools was not fully aligned with the rules, procedures, and guidelines set in the Manual for Implementation of the Final Exam, only few complaints were voiced by students and parents as these practices did not have a significant impact on student chances to be enrolled in preferred secondary schools. The Ministry of Education and Science and the Institute are receiving support from the EU IPA 2008 project to improve the monitoring and quality assurance of the next Final Exam. 26. According to the Law on Foundation of Education System, all students are obligated to take the Final Exam at the end of grade 8. The Final Exam is free of charge for students therefore there are no financial barriers to take it. Additionally, students take the Final Exam in their own school and ithe exam is offered in the minority languages, which makes it accessible for all students. For students with special needs, the Final Exam can be delivered in different mediums (for example, Braille letters for students with vision impairment) or with accommodated content (for example, students with special needs receive test accommodations based on the personal Individual Education Program). In 2011, about 98.5% of students tool the Final exam in the first term (June 2011). The remaining 1.5% of students had an objective reason (for example, sickness) for not taking the Final Exam. 27. The results of the Final Exam are used only for placing students in secondary education programs and schools, and for monitoring of the quality and equity of the primary education by the Ministry of Education and Science. There is no systematic evidence of improper use of examination results. 28. Up to five days after the Final Exam, student results are publically displayed on a poster in the school. 29. Students who do not perform well on the examination may opt for less selective schools. Students take the Final exam at the end of June of the year when they complete grade 8. About 99% of students take the Final Exam in that term. If a student is unable to take at the end of June, he or she can do so in August of the same year. According to the Law on Foundation of Education System, students are obligated to only take the Final Exam, and not to achieve a certain minimum score. Therefore, students cannot fail the exam. However, because results from the Final Exam , together with school marks in Grade 6�8, constitute the total score (exam results 40% and school marks 60%) with which students apply for enrollment to secondary education programs and schools, students who do not perform well on the Final Exam can only be enrolled in a limited number of programs and schools. In the infrequent cases when a student is not satisfied with the exam results and cannot be enrolled in a preferred secondary school, he or she can take the Final Exam again the following June (the student can wait for the next Final Exam without repeating the grade, or can accept to be enrolled in some secondary school for which they have achieved the required entrance score). Students cannot repeat the grade or attend remedial/preparatory courses in order to prepare to retake the examination. 30. There are no mechanisms in place (such as funding for independent research on the impact of the examination, a permanent oversight committee, studies (e.g., predictive validity) that are updated regularly, regular focus groups or surveys of key stakeholders, and/or expert review groups) to monitor the consequences of the examination. 30 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SERBIA National (or System-Level) Large-Scale Assessment (NLSA) 31 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this 1 irregular basis. been operating regularly. dimension. There is no policy document pertaining There is an informal or draft policy * There is a formal policy document that This option does not apply to this 2 to NLSA. document that authorizes the NLSA. authorizes the NLSA. dimension. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 3 dimension. the public. public. dimension. 4 * There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the dimension. NLSA will take place. coming years. * ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the NLSA. 5 the NLSA or are indifferent to it. NLSA. NLSA. * (CONTINUED) 32 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the There is irregular funding allocated to There is regular funding allocated to the This option does not apply to this 6 NLSA. the NLSA. NLSA. dimension. * This option does not apply to this Funding covers some core NLSA Funding covers all core NLSA activities: This option does not apply to this dimension. activities: design, administration, analysis design, administration, analysis and dimension. 7 and reporting. reporting. This option does not apply to this Funding does not cover research and This option does not apply to this * Funding covers research and 8 dimension. development activities. dimension. development activities. * ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this 9 team. group of people. institution or unit. dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes * Political considerations never hamper 10 dimension. technical considerations. hamper technical considerations. technical considerations. * This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a This option does not apply to this 11 dimension. clearly recognized body. clearly recognized body. dimension. * (CONTINUED) 33 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed The NLSA office is adequately staffed to The NLSA office is adequately staffed to NLSA. to effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no 12 minimal issues. issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities to The country offers a wide range of that prepare individuals for work on dimension. prepare individuals for work on the opportunities to prepare individuals for 13 NLSA. NLSA. work on the NLSA. * 34 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this 14 curriculum or learning standards. dimension. curriculum or learning standards. dimension. * What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned What the NLSA measures is largely 15 questioned by stakeholder groups. dimension. by some stakeholder groups. accepted by stakeholder groups. * There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it dimension. 16 measures what it is supposed to intended to measure. is intended to measure. measure. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on There are occasional courses or There are some courses or workshops on There are widely available high quality 17 the NLSA. workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. * 35 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include Different options are offered to include 18 groups of students in the NLSA. dimension. all groups of students in the NLSA. all groups of students in the NLSA. There are no mechanisms in place to * This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in 19 ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. place to ensure the quality of the NLSA. There is no technical report or other There is some documentation about the There is a comprehensive technical There is a comprehensive, high quality * documentation about the NLSA. technical aspects of the NLSA, but it is report but with restricted circulation. technical report available to the general 20 not in a formal report format. public. * ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA 21 NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way. dimension. NLSA information is not used or is used This option does not apply to this * NLSA results are used by some NLSA information is used by all in ways inconsistent with the purposes dimension. stakeholder groups in a way that is stakeholder groups in a way that is or the technical characteristics of the consistent with the purposes and consistent with the purposes and assessment. technical characteristics of the technical characteristics of the 22 assessment. assessment. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to * There are a variety of mechanisms in 23 monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA. place to monitor the consequences of the NLSA. * 36 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 National (of System-Level) Large Scale Assessment (NLSA): Development-level rating justifications 1. National Assessment of Fourth Grade Students is used to monitor the education quality at the system level, support schools and teachers, and for policy design, evaluation, and decision making. The assessment has been administered once in 2006 in Serbian Language (Reading, Writing, Grammar, Literature) and Mathematics (Natural numbers, Operations with natural numbers, Geometry, Fractions, Solving everyday life problems) to a representative sample of students educated in the Serbian language and who were able to take paper-and-pencil tests. The specific objectives of the National Assessment of Fourth Grade Students were to determine the knowledge, competences and skills in Serbian Language and Mathematics that students acquired by the end of the first cycle of education, to determine factors that affect student achievement, to evaluate knowledge and proficiency of educators, as well as school condition. The ultimate objective of the survey was to collect relevant data on existing education practices, and to learn about the level of student achievement and the factors that affect student achievement, for the purpose of improving the education system. 2. There are formal and informal policy documents that authorize the National Assessment of Fourth Grade Students: - The Law on the Foundations of the Education System (2009), authorized by the Parliament of the Republic of Serbia. - The Law on Amendments and Additions to the Law on the Fundamentals of the Education System (2011), authorized by the Parliament of the Republic of Serbia. - Draft Law on Primary Education (2012), authorized by the Parliament of the Republic of Serbia, which will regulate national testing. It is submitted to the Parliament of Serbia by the Ministry of Education, and can be downloaded from the Parliament of the Republic of Serbia website. 3. The policy documents are available to the public. 4. There is currently no large-scale assessment plan for the coming years or future assessment rounds. 5. Policymakers support national assessments in principle, yet due to the lack of financial resources, there is no annual program of national assessment studies in place. Teacher Unions have not addressed national assessments which may be due to the fact that there have only been two (Grade Three National Assessment of Student Achievement and National Assessment of Fourth Grade Students), neither of which has affected teachers. 37 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Because the media is generally interested in reporting on interesting results, they support the national assessment. However, the news that the media reports on the national assessment is not analytical or in-depth. The media has generally been more interested in international assessment studies, such as PISA. Think thanks and NGOs support national assessment studies as they can use the results for analysis and reviews. University staff strongly support the National Assessment of Fourth Grade Students because they understand the role and importance of the assessment for improving education quality. Nevertheless, among education specialists at the universities, there is concern that the assessment focuses only on outcomes of education, and that it may cause policy makers to neglect the process of education. 6. National assessment studies are financed on ad hoc basis by donors due to the lack of financial resources in the budget of the Ministry of Education. 7. Funding covers all core NLSA activities: design, administration, analysis and reporting. 8. Funding does cover staff training. 9. The Institute for Education Quality and Evaluation ("the Institute") is in charge of the National Assessment of Fourth Grade Students. 10. The Institute receives the request from the Ministry to carry out certain assessment studies and determines when those studies will be carried out. The Institute has the responsibility and autonomy to carry out those studies. 11. The Institute is accountable to the Ministry of Education. The Institute also employs expert consultants to complete work on specific assessment-related tasks. These experts are accountable to the Institute. 12. While the quality of the Institute staff is sufficient, staff has to work overtime when it needs to complete tasks for other assessment-related activities (such as for the Final Exam in Primary School Education, and others). 13. Opportunities to prepare individuals for work on the National Assessment of Fourth Grade Students that are available on an annual basis include university graduate programs (masters or doctorate level) and courses (graduate and non-graduate) specifically focused on educational measurement and evaluation. For example, the University of Belgrade offers the Educational Psychology program (at undergraduate and master levels) that includes a course Evaluation of Knowledge and Educational Outcomes, which is specifically focused on education assessments and examinations. This program has a long tradition, and the course that specifically focuses on educational assessment and examination was introduced in 2006. The quality of this program is considered good as the content of the program and the assessment course is up-to-date. The professors of the program have both good theoretical knowledge and practical experience, and provide up to date information on the knowledge and practice in the area of national assessment studies. Training workshops are held at 38 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 international education conference, and conference attendees can attend these workshops. Additionally, it is possible to gain hands-on experience through internships or short-term employment in the large-scale assessment office. 14. The National Assessment of Fourth Grade Students measures performance against curriculum or learning standards. 15. What the National Assessment of Fourth Grade Students measures is largely accepted by stakeholder groups. 16. Mechanisms include regular internal review as well as regular independent review by qualified experts of the alignment between the assessment instrument and what it is supposed to measure. Specifically, the Institute engages external experts to evaluate the quality and validity of test items. 17. There are no courses or workshops on the National Assessment of Fourth Grade Students. 18. The Institute does not have sufficient resources to translate the National Assessment of Fourth Grade Students into minority languages. 19. Mechanisms to ensure the quality of the National Assessment of Fourth Grade Students include that the quality of items and test instruments are verified by external experts and are based on psychometric analysis, all proctors or administrators are trained according to a protocol, there is a standardized manual for large-scale assessment administrators, a pilot is conducted before the main data collection takes place, all booklets are numbered, there is double data scoring, and scorers are trained and monitored to ensure high interrater reliability, and data entry is monitored as well. 20. There is some documentation about the technical aspects of the National Assessment of Fourth Grade Students, but it is not in a formal report format. 21. Results of the National Assessment of Fourth Grade Students are disseminated within twelve months after the assessment is administered, reports with results are made available for all stakeholder groups, and the main reports on the results contain information on overall achievement levels and subgroups. The main reports on the results do not, however, contain information on trends over time overall and for subgroups, do not contain standard errors (measures of uncertainty), there are no media briefings organized to discuss the results, there are no workshops or presentations for stakeholders on the results, and the results are not featured in the media (in newspapers, magazines, radio, or television). 22. Assessment information is used by the Institute as an input for developing standards of achievements for the first cycle of education as well as for developing criterion-referenced tests for schools. 23. There are no mechanisms in place to monitor the consequences of the National Assessment of Fourth Grade Students. 39 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SERBIA International Large-Scale Assessment (ILSA) 40 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in The country/system has participated in 1 in an ILSA in the last 10 years. dimension. at least one ILSA in the last 10 years. two or more ILSA in the last 10 years. * The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. 2 the next 5 years. the next 5 years. There is no policy document that There is an informal or draft policy There is a formal policy document that * This option does not apply to this 3 addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. dimension. ILSA. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. dimension. ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, 4 ILSA. donors. discretion. decree or norm. * This option does not apply to this Funding covers some core activities of Funding covers all core activities of the Funding covers all core activities of the 5 dimension. the ILSA. ILSA. ILSA. * Funding does not cover research and This option does not apply to this This option does not apply to this Funding covers research and 6 development activities. dimension. dimension. development activities. * 41 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 (CONTINUED) LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. 7 activities. activities. activities. * This option does not apply to this The national/system coordinator or The national/system coordinator is fluent This option does not apply to this 8 dimension. other designated team member may not in the language of the assessment. dimension. be fluent in the language of the assessment. * This option does not apply to this The ILSA office is inadequately staffed or The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, 9 effectively. with minimal issues. with no issues. * 42 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this 10 international workshops or meetings. international workshops or meetings. workshops or meetings. dimension. The country/system offers no This option does not apply to this * The country/system offers some The country/system offers a wide range 11 opportunities to learn about ILSA. dimension. opportunities to learn about ILSA. of opportunities to learn about ILSA. * This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are dimension. dimension. available to the country's/system's ILSA available to a wide audience, in addition 12 team members only. to the country's/system's ILSA team * members. 43 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical The country/system met all technical published. standards to have its data presented standards required to have its data standards required to have its data beneath the main display of the presented in the main displays of the presented in the main displays of the 13 The country/system has not contributed international report or in an annex. This option does not apply to this international report. This option does not apply to this international report. * The country/system has contributed new 14 new knowledge on ILSA. dimension. dimension. knowledge on ILSA. * ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely 15 the country/system. in the country/system. the country/system. disseminated in the country/system. * Products to provide feedback to schools This option does not apply to this Products to provide feedback to schools Products to provide feedback to schools and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are 16 not made available. sometimes made available. systematically made available. * There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the There is wide media coverage of the ILSA 17 results. ILSA results. ILSA results. results. * If any, country/system-specific results Results from the ILSA are used in a Results from the ILSA are used in some Results from the ILSA are used in a and information from the ILSA are not limited way to inform decision making in ways to inform decision making in the variety of ways to inform decision 18 used to inform decision making in the the country/system. country/system. making in the country/system. country/system. * It is not clear that decisions based on This option does not apply to this This option does not apply to this Decisions based on the ILSA results have ILSA results have had a positive impact dimension. dimension. had a positive impact on students' 19 on students' achievement levels. achievement levels. * 44 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 International Large Scale Assessment (ILSA): Development-level rating justifications 1. Serbia has participated in a number of international large scale assessments, including TIMSS 2003, TIMSS 2007, TIMSS 2011, PISA 2003, PISA 2006, and PISA 2009. Serbia has consistently participated in PISA, and it most recently completed PISA 2009 which the remainder of the rubric refers to. 2. Serbia has taken concrete steps to participate in PISA 2012, and is planning to participate in PISA 2015 and TIMSS 2015. 3. There is no policy document that addresses participation in international assessments. 4. Funding for PISA 2009 is provided for by government funding that is supported by a World Bank loan. Funding for participation in other international large scale assessments is also provided for by government funding supported by loans (such as from the World Bank and the European Investment Bank). 5. Funding for PISA 2009 covered international participation fees, implementation of the assessment exercise in Serbia (e.g., printing booklets, travel to schools), attendance at international expert meetings for the assessment exercise, processing and analyzing data collected from implementation of the assessment exercise, and reporting and disseminating the assessment results in Serbia. 6. While donor funding does not cover research and development activities, the Institute of Psychology which implements PISA is a research institution. Additional research and secondary analyses are typically done by the Institute of Psychology and financed by the Institute itself by another organization at its request (e.g. Social Inclusion and Poverty Reduction unit of the Government of Serbia requested and financially supported the Institute of Psychology to make a targeted secondary analysis of PISA achievements and transition from compulsory to secondary education for students with lower SES. Results from this secondary analysis were used in the “First national report on poverty�). 7. Dragica Pavlović-Babić is the National Project Manager for PISA. The team responsible for carrying out PISA consist of a core group (4�5 people) and a large number of associates who are included in specific phases of the study(administration, marking, data entry, etc.). The core staff are full time employees at the Institute of Psychology (i.e. formally at the Faculty of Philosophy) as teaching and research staff with regular duties other than PISA. The network of associates are not employees of the Institute but are brought in on a temporary basis as the majority of them are teachers employed by schools or Faculties/Universities. Some of the associates are students or unemployed persons brought in to complete data entry or other temporary administrative tasks. TIMSS, on the other hand, is administered by the Institute of Pedagogical Research, which is a public institute. 8. The national coordinator is fluent in the language of the assessment. 9. The team is sufficiently staffed, however the size and structure of the team has not always been aligned to the needs of the project. In general, the trend is to increase the size of the core team. Some members of the core team are stretched by different PISA tasks and other duties they have as full time employees of the Institute of Psychology and the Faculty of Philosophy. At the same time, the core team's composition is interchangeable because different members of the core team can take on various additional tasks, which is particularly helpful during the implementation of many PISA activities. The Institute 45 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 can extend the core team at its own discretion, but the size of the PISA budget received from the Ministry limits this expansion. Lack of financial resources keeps the number at a minimum. The team has previous experience working on international assessments as well as the necessary training to carry out the required PISA activities effectively. The only issue identified with the carrying out of PISA in Serbia is that there have been errors or delays in the printing or layout of the test booklets. Specifically, the team in charge of the PISA exercise could not achieve the expected level of print quality (although the quality of the test booklets was still acceptable) due to lack of financial resources. 10. Team members attended most of the training organized by the OECD. Several trainings were not attended due to financial constraints (for example, if the same training was attended in previous cycles and if participation requires a high cost). 11. Limited opportunities to learn about international assessments include university courses on the topic of international assessments. A masters-level university course on educational assessment that includes international assessment studies is offered for students studying Education psychology at the University of Belgrade. The topic of the international studies will be also included in the new master program for educational policy at the University of Belgrade. Additionally, there is limited funding for attending international workshops or training on international assessments. 12. Individuals working directly on the specific international assessment exercise and university students studying assessment or a related area benefit from the opportunities to learn about international assessments. 13. Serbia met all technical standards required to have its data presented in the main displays of the international report. 14. Serbia has contributed new knowledge on PISA. 15. Results and information from the PISA exercise were regularly and widely disseminated in Serbia. Specifically, a national report was made available online, copies of the national and international report were distributed to key stakeholders, Serbia's results were communicated through a press release, brochures and PowerPoint presentations with Serbia's results were made available online or distributed to key stakeholders, and products providing feedback to the schools and educators about the results were made available. 16. Participating schools receive the school report as well as the National report (other schools can download the national report from Serbian PISA site). In addition, some schools have asked the National Center to visit the school and make a presentation and/or discuss PISA findings with teachers. 17. Media coverage has included editorials or columns commenting on the international assessment results, including, for example, how PISA results can help improve education. The quality of the discussion of international assessment results in the media has also become more professional. In 2012, media coverage of implementation of PISA was very high. For example, Politika, a reputable daily newspaper, published and analyzed the administered PISA items two times per week over three months. 46 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 18. The results and analysis of PISA 2009 data has informed the recently announced National Strategy for Development of Education until 2020. Additionally, the results of PISA are used by the National Education Council for monitoring the quality and equity of education in Serbia. PISA results were used in the First report on poverty reduction produced by the Government of Serbia. Additionally, PISA results have been used to inform teacher training programs and other assessment activities in the system (e.g., classroom assessment, examinations), for developing education standards for the end of compulsory and secondary education, and for informing the framework for the law on education and regulations governing the assessment practices in schools. The results of PISA 2009 have not been used to track the impact of reforms on student achievement levels, to inform curriculum improvement, or to inform resource allocation. Serbia is currently implementing a project of formulating education standards for the end of general secondary education. PISA results, as well as the PISA concept of literacy, informed the working groups responsible for developing the standards of students achievements. 19. There is evidence that the achievement of students in Serbia significantly increased in PISA 2009, in comparison to previous PISA cycles. However, during the same period, there were no systematic changes in the education system. At least in part, the growth of achievement could be explained by the changing teaching practices that resulted from specific teacher training programs, as well as by the increased general understanding of such concepts as education competence, quality of education, and functional literacy, all concepts which were informed by international assessments. 47 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 Acknowledgements References This report was prepared by The World Bank SABER- Clarke, M. 2012. “What Matters Most for Student Student Assessment team in collaboration with Ivana Assessment Systems: A Framework Paper.� Aleksic, World Bank Education Specialist and Task Team READ/SABER Working Paper Series. Washington, DC: Leader for education projects in Serbia, and Aleksandar World Bank. Baucal, Professor at the University of Belgrade, Department of Psychology. Organisation for Economic Co-operation and Development (OECD). 2012. “Strengthening Integrity and Fighting Corruption in Education: Serbia.� Paris: OECD. ———. 2011. “Education at a Glance 2011: OECD Indicators.� Paris: OECD. United Nations Children’s Fund (UNICEF). 2008. “Country profile: Education in Serbia.� Data retrieved from http://www.unicef.org/ceecis/Serbia.pdfon March 12, 2013. United Nations Educational, Scientific, and Cultural Organization (UNESCO). Serbia Country Indicator Data. Montreal, QC: UNESCO. Data retrieved from http://stats.uis.unesco.org/unesco on March 14, 2013 World Bank. 2013. “Serbia Country Overview� Data retrieved from http://www.worldbank.org/en/country/serbia/overvi ew on March 16, 2013. ———. 2013. World Bank Development Indicators: Armenia Country Indicator Data. Washington, DC: World Bank. Data retrieved from http://databank.worldbank.org/data on March 12, 2013. 48 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS SERBIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2012 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policymakers and business people—an accessible, objective snapshot showing how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. THE WORLD BANK 2 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS