ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 79936 Armenia SABER Country Report STUDENT ASSESSMENT 2011 Key Policy Areas for Student Assessment Status 1. Classroom Assessment In Armenia, an official curriculum/standards document specifies what students are expected to learn in different subject areas at different grade/age levels and to what level of performance. Varied and systematic mechanisms, such as an external moderation system that reviews the difficulty of classroom assessment activities and the appropriateness of scoring criteria, are in place to monitor the quality of classroom assessment practices. There also are some mechanisms to ensure that teachers develop skills and expertise in classroom assessment. 2. Examinations The United Examinations, which are used for high school graduation and university entrance, started to be administered less than five years ago. The examinations receive regular funding from the government and from student fees. Funding covers all core examination activities, as well as research and development. There is a clear understanding of what the United Examinations measure, and comprehensive material to prepare for the examinations is accessible to all students. 3. National Large-Scale Assessment (NLSA) An NLSA program was introduced in Armenia in the last five years, and one NLSA has been carried out (in 2010). The program receives regular funding from the government; this funding covers all core NLSA activities as well as staff training and research and development. Armenia offers some opportunities to prepare individuals for work on the NLSA. However, the Assessment and Testing Center is inadequately staffed to effectively carry out the NLSA due to the staff's limited experienced in conducting NLSAs. 4. International Large-Scale Assessment (ILSA) Armenia has participated in three ILSAs in the last ten years (TIMSS 2003, 2007, and 2011), and has taken concrete steps to participate in two more ILSAs in the next five years (TIMSS 2015 and PIRLS 2016). A formal policy document addresses Armenia's participation in ILSAs, and regular funding for participation in ILSAs is provided by the government. Results from the ILSAs are regularly and widely disseminated, especially through widespread media coverage. However, there are no opportunities available in Armenia to learn about ILSAs. THE WORLD BANK ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Introduction SABER-Student Assessment methodology In 2011, Armenia joined the Russia Education Aid for The SABER-Student Assessment framework is built on Development (READ) Trust Fund program, the goal of the available evidence base for what an effective which is to help countries improve their capacity to assessment system looks like. The framework provides design, carry out, analyze, and use assessments for guidance on how countries can build more effective improved student learning. As part of the READ Trust student assessment systems. The framework is Fund program, and in order to gain a better structured around two main dimensions of assessment understanding of the strengths and weaknesses of its systems: the types/purposes of assessment activities existing assessment system, Armenia participated in a and the quality of those activities. formal exercise to benchmark its student assessment system under The World Bank’s Systems Approach for Assessment types and purposes Better Education Results (SABER) program. SABER is an evidence-based program to help countries Assessment systems tend to be comprised of three systematically examine and strengthen the main types of assessment activities, each of which performance of different aspects of their education serves a different purpose and addresses different systems. information needs. These three main types are: classroom assessment, examinations, and large-scale, What is SABER-Student Assessment? system level assessments. SABER-Student Assessment is a component of the Classroom assessment provides real-time information SABER program that focuses specifically on to support ongoing teaching and learning in individual benchmarking student assessment policies and systems. classrooms. Classroom assessments use a variety of The goal of SABER-Student Assessment is to promote formats, including observation, questioning, and paper- stronger assessment systems that contribute to and-pencil tests, to evaluate student learning, generally improved education quality and learning for all. on a daily basis. National governments and international agencies are Examinations provide a basis for selecting or certifying increasingly recognizing the key role that assessment of students as they move from one level of the education student learning plays in an effective education system. system to the next (or into the workforce). All eligible The importance of assessment is linked to its role in: students are tested on an annual basis (or more often if (i) providing information on levels of student the system allows for repeat testing). Examinations learning and achievement in the system; cover the main subject areas in the curriculum and (ii) monitoring trends in education quality over usually involve essays and multiple-choice questions. time; (iii) supporting educators and students with real- Large-scale, system-level assessments provide feedback time information to improve teaching and on the overall performance of the education system at learning; and particular grades or age levels. These assessments (iv) holding stakeholders accountable for results. typically cover a few subjects on a regular basis (such as every 3 to 5 years), are often sample based, and use multiple-choice and short-answer formats. They may be national or international in scope. Appendix 1 summarizes the key features of these main types of assessment activities. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 2 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Quality drivers of an assessment system Table 1: Framework for building an effective assessment system, with indicator areas The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or policy framework for assessment activities; institutional The indicators are identified based on a combination of and organizational structures for designing, carrying criteria, including: out, or using results from the assessment; the • professional standards for assessment; availability of sufficient and stable sources of funding; • empirical research on the characteristics of effective and the presence of trained assessment staff. assessment systems, including analysis of the characteristics that differentiate between the System alignment refers to the extent to which the assessment systems of low- versus high-performing assessment is aligned with the rest of the education nations; and system. This includes the degree of congruence • theory — that is, general consensus among between assessment activities and system learning experts that it contributes to effective goals, standards, curriculum, and pre- and in-service assessment. teacher training. Levels of development Assessment quality refers to the psychometric quality of the instruments, processes, and procedures for the The World Bank has developed a set of assessment activity. It covers such issues as design and standardized questionnaires and rubrics for collecting implementation of assessment activities, analysis and and evaluating data on the three assessment types interpretation of student responses to those activities, and related quality drivers. and the appropriateness of how assessment results are reported and used. The questionnaires are used to collect data on the characteristics of the assessment system in a particular Crossing the quality drivers with the different country. The information from the questionnaires is assessment types/purposes provides the framework then applied to the rubrics in order to judge the and broad indicator areas shown in Table 1. This development level of the country’s assessment system framework is a starting point for identifying indicators in different areas. that can be used to review assessment systems and plan for their improvement. The basic structure of the rubrics for evaluating data collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice in each area. For each indicator, the rubric displays four development levels—Latent, Emerging, Established, and Advanced. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 3 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 These levels are artificially constructed categories levels. Armenia spent 3.2 percent of GDP on education chosen to represent key stages on the underlying in 2010, compared to an average 6 percent of GDP by continuum for each indicator. Each level is accompanied OECD countries. Armenia’s spending on education is in by a description of what performance on the indicator part reflective of the government’s cuts in spending in looks like at that level. 2009 in response to the economic crisis. The education sector was one of the most affected sectors, with a 15 • Latent is the lowest level of performance; it percent decrease from the original budget. The Medium represents absence of, or deviation from, the Term Expenditure Framework 2012-2014 indicates that desired attribute. spending on education will continue to decline over the • Emerging is the next level; it represents partial next three years. As of 2011, gross enrollment at the presence of the attribute. primary level is 84 percent and the completion rate is • Established represents the acceptable minimum 83 percent. Gross enrollment at the secondary level is standard. 89 percent. • Advanced represents the ideal or current best practice. Armenia has committed to reform the education system. Reforms introduced in the past 10 years have A summary of the development levels for each included new curriculum for grades 1 through 9 and assessment type is presented in Appendix 3. state standards for secondary education. These changes were accompanied by large-scale teacher training In reality, assessment systems are likely to be at programs to support the new curriculum. The Ministry different levels of development in different areas. For of Education also invested in improving management example, a system may be Established in the area of efficiency through the development of its Education examinations, but Emerging in the area of large-scale, Management Information System (EMIS). Further, system-level assessment, and vice versa. While intuition large-scale assessments to evaluate and monitor the suggests that it is probably better to be further along in quality of learning in schools were introduced. Recently, as many areas as possible, the evidence is unclear as the Armenian general education system has been to whether it is necessary to be functioning at extended from 10 to 12 years with the introduction of Advanced levels in all areas. Therefore, one might view high schools with specialized streams. All schools in the Established level as a desirable minimum outcome to Armenia will have access to the internet by the end of achieve in all areas, but only aspire beyond that in those 2011, which will allow students and teachers, especially areas that most contribute to the national vision or in rural schools, to be connected to the most up-to-date priorities for education. In line with these considerations, information. the ratings generated by the rubrics are not meant to be additive across assessment types (that is, they are not Detailed information was collected on Armenia’s meant to be added to create an overall rating for an student assessment system using the SABER-Student assessment system; they are only meant to produce an Assessment questionnaires and rubrics in 2011. It is overall rating for each assessment type). The methodology important to remember that these tools primarily focus for assigning development levels is summarized in on benchmarking a country’s policies and arrangements Appendix 4. for assessment activities at the system or macro level. Additional data would need to be collected to Education in Armenia determine actual, on-the-ground practices in Armenia, particularly by teachers and students in schools. The following sections discuss the findings by each Armenia is a lower-middle income country in the Eastern assessment type, accompanied by suggested policy Europe and Central Asia region. GDP per capita (current options. The suggested policy options were determined US$ 2011) is $3305, with annual growth of approximately in collaboration with key local stakeholders based on 4 percent. The annual growth reflects a rebound from Armenia’s immediate interests and needs. Detailed, 2009 levels and can be attributed to improved conditions completed rubrics for each assessment type in Armenia in mining, manufacturing, and services such as tourism, as are provided in Appendix 5. well as a rebound in agriculture from pre-financial crisis SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 4 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Classroom Assessment Generally, classroom assessment practices in Armenia are known to be of moderate quality. Specifically, while classroom assessment activities are mainly about Level of development recalling information (as opposed to assessing higher- order thinking skills), classroom assessment activities are nevertheless aligned with pedagogical and curricular In Armenia, formal system-level documents provide frameworks. Teachers use explicit criteria for scoring or guidelines for classroom assessment. These documents, grading students' work. Parents are well informed which are available to the public, are the General about students' grades and teachers provide substantial Education State order, and the Students’ Progress useful feedback to students. Assessment Procedure document. Additionally, there are official curriculum and standards documents Classroom assessment information is required to be available for all subjects that specify what students are disseminated to all key stakeholders, including students, expected to learn in different subject areas at different parents, and school district and Ministry of Education grade/age levels and to what level of performance. officials. Additionally, classroom assessment information is required to be used in various ways in There are some system-wide resources available to Armenia to support student learning. These required teachers for conducting classroom assessment uses include diagnosing student learning issues, activities. Along with the official curriculum and providing feedback to students on their learning, standards documents, there are textbooks and informing parents about their child’s learning, planning workbooks that provide support for classroom the next steps in instruction, grading students for assessment activities. Scoring criteria and rubrics for internal classroom uses, and providing input to an evaluating students’ work, as well as item banks or external examination program. pools with examples of selection/multiple-choice or supply/open-ended questions are also made available Additionally, varied and systematic mechanisms are in to teachers; so too are brochures with sample questions place to monitor the quality of classroom assessment for different subjects. practices, including an external moderation system that reviews the difficulty of classroom assessment activities Some mechanisms are in place to ensure that teachers and the appropriateness of scoring criteria. Classroom develop the necessary skills and expertise in classroom assessment is also a required component of a teacher's assessment. For example, teachers are offered the performance evaluation, as well as of school inspection opportunity to participate in conferences and and teacher supervision. workshops, as well as in item development for, or scoring of, large-scale assessments or examinations. Furthermore, school inspection and teacher supervision Suggested policy options: includes a component that focuses on classroom assessment. Additionally, the World Bank Education 1. Ensure that classroom assessment activities Quality and Relevance Project (EQRP) I and EQRP II promote student learning by incorporating introduced and supported teacher training programs, assessment activities that evaluate students' which included training on classroom assessment. higher-order thinking skills. However, training on classroom assessment has not been institutionalized in pre-service teacher training 2. Introduce institutionalized pre- and in-service institutions (such as pedagogical universities and teacher training courses on classroom institutes) or in the in-service teacher training assessment. programs. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 5 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Examinations Some technical documentation on the examinations is developed, but it generally consists of statistical analysis of examination results and is not in a formal report Level of development format. There are also limited systematic mechanisms in place to ensure the quality of the examinations, The United Examinations, which are used for both high including internal and external review or observers, and school graduation and university entrance, started to be pilot or field testing. During the administration of the administered less than five years ago. The formal policy examinations, the ATC and the Ministry of Education document that authorizes the examination, the Concept representatives observe the administration procedures of Assessment System (authorized by the Republic of in every examination center. Cameras are also placed in Armenia, 2005), describes the official purposes of the examination centers, and parents can observe the examinations and the authorized uses of the results. examination process on video monitors. The Assessment and Testing Center (ATC) has been Inappropriate behavior surrounding the examination responsible for running the examinations since 2006. process is marginal, and the examination results are ATC is adequately staffed to carry out the examinations viewed as credible by all stakeholder groups. All effectively, with minimal issues. Armenia offers some students can take the examinations; there are no opportunities that prepare individuals for work on the language, gender, or other equivalent barriers. examinations, including non-university training courses and workshops on educational measurement and There are very limited options for students who do not evaluation. perform well on the examinations. For example, students may retake the examination, but only for the Regular funding for the examination is provided by the graduation certificate and not for university entrance. government and by student fees. Funding covers all Instead, students may opt for less selective universities. core examination activities (design, administration, data processing, and reporting) as well as research and There is a committee responsible for reviewing development, staff training, and ATC facilities. students’ appeals and errors in examinations. Currently, there are no mechanisms in place to monitor the Efforts to improve the examination are generally consequences of the examinations. welcomed by ATC, and the rules of the examinations and methods of administration are reviewed every year on the basis of the previous years' experience. Suggested policy options: There is a clear understanding of what the examinations 1. Introduce a variety of mechanisms to monitor measure, and comprehensive material to prepare for the consequences of the examinations, the examinations is made available to all students. including providing funding for independent Preparatory material includes examples of examination research on the impact of the examinations, questions, information on how to prepare for the conducting regular focus groups of key examinations, and the framework document that stakeholders, and organizing expert review explains what the examinations measure. Guidelines for groups. taking the examinations, as well as samples of test materials, are annually distributed to all schools. 2. Introduce varied and systematic mechanisms to ensure the quality of the examinations, such as external certification or audit, and making available comprehensive, high-quality technical reports on the examinations to the general public. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 6 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 National Large-Scale Assessment (NLSA) There are ad hoc reviews of the NLSA to ensure that it measures what it is intended to measure. Although there are comparisons of the results of the NLSA with Level of development the results of other assessment programs, as well as reviews of the test materials and statistical analysis of results, there are limited internal reviews of the NLSA to An NLSA program was introduced in Armenia in the last ensure that it measures what it is intended to measure. five years. An NLSA exercise, assessing grade 8 student achievement levels in Armenian language and Armenian To better ensure the quality of the NLSA, efforts are history, was carried out in 2010. There is regular made to ensure a representative sample of students. funding allocated to the NLSA program, which is For example, students from all Marzes of Armenia are provided by the Government of Armenia to the selected to participate in the NLSA. Additionally, there Assessment and Testing Center (ATC). The ATC is are various mechanisms in place to ensure the technical responsible for all NLSA activities. The funding covers quality of the NLSA, including that all proctors or core NLSA activities (design, administration, analysis, administrators are trained according to a protocol. and reporting) as well as research and development, There is a standardized manual for NLSA administrators, and staff training. and a pilot is conducted before the main data collection takes place. All booklets are numbered, there is double Although ATC employs full-time staff as well as scoring of data, and scorers are trained to ensure high temporary specialists, it is inadequately staffed to interrater reliability. effectively carry out NLSA activities due to a lack of staff with extensive experience in carrying out this particular NLSA results are used by some stakeholder groups in a type of assessment activity. Specifically, the staff way that is consistent with the purposes and technical responsible for the NLSA program is far more familiar characteristics of the assessment. Specifically, the with graduation and university entrance examinations. information from the NLSA is used to review the It has been recognized that ATC staff needs to build content that is being taught in schools, to provide capacity specifically in designing NLSAs. Currently, suggestions to teachers on teaching methods, and to Armenia offers some opportunities to prepare understand the level of students' knowledge in the individuals for work on NLSAs, including non-university tested subject areas. training courses on educational measurement and evaluation, and funding for attending international Although conferences to discuss research on the programs or workshops on educational measurement consequences of the NLSA and expert review groups are and evaluation. However, funding for attending being planned, there are currently no mechanisms in international programs and workshops is limited; place to monitor the consequences of the NLSA. therefore, only a few people have been able to participate in such international programs. Suggested policy options: The NLSA measures performance against national curriculum or learning standards. Although what the 1. Conduct regular internal reviews of the NLSA to NLSA measures is largely accepted by stakeholder ensure that it is aligned with learning goals and groups, there is sometimes a need to explain that the measures what it is intended to measure. NLSA is not a high-stakes assessment. 2. Build the capacity of the staff of the NLSA office by introducing university graduate programs (at the masters or doctorate level) and courses (graduate and non-graduate) that are focused on educational measurement and evaluation, and specifically on conducting NLSAs. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 7 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 International Large-Scale Assessment Armenia met all technical standards required to have its data presented in the main displays of the international (ILSA) report. Additionally, Armenia has contributed new knowledge on ILSAs, and country-specific results and Level of development information have been regularly and widely disseminated in the country. As part of dissemination activities, copies of the national and international Armenia has participated in three ILSAs in the last ten reports were distributed to key stakeholders and the years – TIMSS 2003, TIMSS 2007, and TIMSS 2011 – and key findings were communicated through a press has taken concrete steps to participate in two ILSAs in release. Additionally, ILSA results received wide the next five years, specifically in TIMSS 2015 and PIRLS coverage on the television, radio, and newspapers, and 2016. brochures and PowerPoint presentations with Armenia’s ILSA results were made available online and A formal, publically-available policy document, the distributed to key stakeholders. However, products to Concept of Assessment System (authorized by the provide feedback directly to schools and educators Republic of Armenia, 2005), addresses Armenia's about the ILSA results are only sometimes made participation in ILSAs. Regular funding for participation available. in ILSAs is provided by the government. The funding covers international participation fees, implementation Results from ILSAs are used in a variety of ways to of the assessment exercise in Armenia, processing and inform decision making in Armenia. For example, ILSA analyzing data collected from implementation of the results have been used by policy makers at the Ministry assessment exercise, reporting and disseminating the of Education as well as by education leaders to improve assessment results in the country, and attendance at education quality in Armenia. Uses of ILSA results have international expert meetings. The funding does not included tracking the impact of reforms on student cover research and development activities. achievement levels, informing curriculum improvement, informing teacher training programs, and informing In Armenia, a National Research Coordinator and a other assessment activities in the country (e.g., team are responsible for carrying out ILSA activities in classroom assessment and examinations). the country. The team is composed of staff from the Assessment and Testing Center (ATC) as well as temporary specialists assigned to work on the ILSA. The Suggested policy options: team is sufficiently staffed, and has previous experience working on international assessments as well as the 1. Provide opportunities to learn about the ILSA to necessary training to carry out the required assessment those individuals working directly on the activities effectively. While several team members have specific ILSA exercise as well as university attended some of the international workshops and students studying educational assessment or a meetings, it was primarily the National Research related area and professionals and university Coordinator who attended the workshops and staff interested in assessment of student meetings. There are no opportunities to learn about learning. ILSA in Armenia other than for those individuals who are directly involved in the assessment exercise. 2. Such opportunities might include organizing meetings or workshops on using ILSA databases, university courses on the topic of ILSAs, on-line courses on international assessments, and funding for attending international workshops or training on ILSAs. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 8 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations Surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select immediate feedback on feedback on the students as they students for feedback to overall health of comparative move from one further inform the system at performance of level of the educational classroom particular the education education opportunities instruction grade/age system at system to the level(s), and to particular next (or into the monitor trends grade/age workforce) in learning level(s) Frequency Daily For individual For individual Annually and Annually and subjects offered subjects offered more often more often on a regular on a regular where the where the basis (such as basis (such as system allows system allows every 3-5 years) every 3-5 years) for repeats for repeats Who is tested? All students Sample or A sample of All eligible All eligible census of students at a students students students at a particular grade particular grade or age level(s) or age level(s) Format Varies from Usually multiple Usually multiple Usually essay Usually essay observation to choice and short choice and short and multiple and multiple questioning to answer answer choice choice paper-and-pencil tests to student performances Coverage of All subject areas Generally Generally Covers main Covers main curriculum confined to a confined to one subject areas subject areas few subjects or two subjects Additional Yes, as part of Frequently Yes Seldom Seldom information the teaching collected from process students? Scoring Usually informal Varies from Usually involves Varies from Varies from and simple simple to more statistically simple to more simple to more statistically sophisticated statistically statistically sophisticated techniques sophisticated sophisticated techniques techniques techniques SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 9 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development Level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—ENABLING CONTEXT EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—SYSTEM ALIGNMENT SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—ASSESSMENT QUALITY AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 10 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Appendix 3: Summary of the Development Levels for Each Assessment Type Assessment Type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system- institutional capacity to wide institutional system-wide wide institutional support and ensure the capacity to support and institutional capacity to capacity to support and quality of classroom ensure the quality of support and ensure the ensure the quality of assessment practices. classroom assessment quality of classroom classroom assessment CLASSROOM ASSESSMENT practices. assessment practices. practices. There is no standardized There is a partially There is a stable There is a stable examination in place for stable standardized standardized standardized key decisions. examination in place, examination in place. examination in place and a need to develop There is institutional and institutional institutional capacity to capacity and some capacity and strong run the examination. limited mechanisms to mechanisms to monitor EXAMINATIONS The examination monitor it. The it. The examination is of typically is of poor examination is of high quality and is quality and is perceived acceptable quality and is perceived as fair and as unfair or corrupt. perceived as fair for free from corruption. most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA in There is a stable NLSA in place. NLSA in place and a place. There is place and institutional need to develop institutional capacity capacity and strong institutional capacity to and some limited mechanisms to monitor NATIONAL (OR SYSTEM- run the NLSA. mechanisms to monitor it. The NLSA is of high LEVEL) LARGE-SCALE Assessment quality and it. The NLSA is of quality and its ASSESSMENT impact are weak. moderate quality and its information is information is effectively used to disseminated, but not improve education. always used in effective ways. There is no history of Participation in an ILSA There is more or less There is stable participation in an ILSA has been initiated, but stable participation in an participation in an ILSA nor plans to participate there still is need to ILSA. There is and institutional in one. develop institutional institutional capacity to capacity to run the ILSA. capacity to carry out the carry out the ILSA. The The information from INTERNATIONAL LARGE- ILSA. information from the the ILSA is effectively SCALE ASSESSMENT ILSA is disseminated, but used to improve not always used in education. effective ways. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 11 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Appendix 4: Methodology for Assigning 6. Scores for certain key dimensions under ‘Enabling Development Levels Context’ (in the case of EXAM, NLSA, and ILSA) and under ‘System Alignment’ (in the case of CLASS) were set as ceiling scores, i.e., the overall mean score for the 1. The country team or consultant collects information particular assessment type cannot be greater than the about the assessment system in the country. score for these key dimensions. These key variables include formal policy, regular funding, having a 2. Based on the collected information, a level of permanent assessment unit, and the quality of development and score is assigned to each dimension in assessment practices. the rubrics: • Latent = 1 score point • Emerging = 2 score points • Established = 3 score points • Advanced = 4 score points 3. The score for each quality driver is computed by aggregating the scores for each of its constituent dimensions. For example: The quality driver, ‘Enabling Context,’ in the case of ILSA, has 3 dimensions on which a hypothetical country receives the following scores: Dimension A = 2 points; Dimension B = 2 points; Dimension C = 3 points. The hypothetical country’s overall score for this quality driver would be: (2+2+3)/3 = 2.33 4. A preliminary level of development is assigned to each quality driver. 5. The preliminary development level is validated using expert judgment in cooperation with the country team and The World Bank Task Team Leader. For scores that allow a margin of discretion (i.e., to choose between two levels of development), a final decision has to be made based on expert judgment. For example, the aforementioned hypothetical country has an ‘Enabling Context’ score of 2.33, corresponding to a preliminary level of development of ‘Emerging or Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses ‘Emerging’ as the most appropriate level. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 12 ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Appendix 5: SABER-Student Assessment Rubrics for Armenia This appendix provides the completed SABER-Student Assessment rubrics for each type of assessment activity in Armenia. In each row of the rubric, the relevant selection is indicated by a thick border and an asterisk. The selection may include a superscript number that refers to the justification or explanation for the selection (as indicated by a thick border and an asterisk). The explanation or justification text can be located in the “Development level rating justifications� section at the end of each rubric. If a row includes a superscript, but not a thick border and an asterisk, this means that insufficient information was available to determine the relevant selection in the row. 13 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ARMENIA Classroom Assessment 14 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. 1 assessment. classroom assessment. assessment. * 2 This option does not apply to this This option does not apply to this The availability of the document is The document is widely available. dimension. dimension. restricted. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide 3 teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment. resources available for teachers for classroom assessment. * There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or students are expected to learn, but the students are expected to learn and to to what level of performance. level of performance required is not what level of performance. clear. * ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms This option does not apply to this There are some system-level There are a variety of system-level to ensure that teachers develop skills dimension. mechanisms to ensure that teachers mechanisms to ensure that teachers and expertise in classroom assessment. develop skills and expertise in classroom develop skills and expertise in classroom * 4 assessment. assessment. 15 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are 5 from widespread weaknesses or there is known to be weak. known to be of moderate quality. known to be generally of high quality. no information available on classroom assessment practices. * There are no mechanisms to monitor the There are ad hoc mechanisms to monitor There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the practices. practices. assessment practices. quality of classroom assessment 6 practices. * ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key 7 stakeholders. stakeholders. stakeholders. * There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support classroom assessment to support classroom assessment to support student learning. student learning, excluding its use as an student learning, including its use as an 8 input for external examination results. input for external examination results. * 16 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Classroom Assessment: Development level rating justifications 1. The two relevant documents are the General Education State order, and the Students Progress Assessment Procedure. 2. The documents are available to the public online, in the public libraries, as well as during in-service courses for teachers. They are also published and distributed to schools. 3. A variety of system-wide resources are available for teachers to carry out classroom assessment activities. For example, there is a document that outlines what students are expected to learn in different subject areas at different grade/age levels and to what performance level. Additionally, textbooks or workbooks are made available that provide support for classroom assessment activities. Furthermore, there are scoring criteria or rubrics for students’ work as well as item banks or pools with examples of selection/multiple-choice or supply/open-ended question. There are also many brochures by various authors with examples of questions, but teachers are usually not required to use them. However, the system-wide resources are not available online. 4. The system-level mechanisms include in-service teacher training and on-line resources on classroom assessment. Additionally, all teacher training programs include a required course on classroom assessment. Armenia also offers opportunities for teachers to participate in conferences and workshops, as well as in item development for, or scoring of, large-scale assessments or exams. School inspection or teacher supervision includes a component focused on classroom assessment. The Assessment and Testing Center serves to ensure that teachers develop skills and expertise in classroom assessment. Under EQRPI and EQRPII (Bank projects), massive training programs were and are implemented for all teachers in Armenia (on-going for more than 5 years now) and all training programs include classroom assessment. However, training on assessment in not institutionalized in teacher pre-service institutions (Pedagogical Universities and Institutes) as well as in the in-service training programs run under the state funding on the regular basis. 5. Although classroom assessment activities are mainly about recalling information (as opposed to assessing higher-order thinking skills), classroom assessment activities are aligned with pedagogical and curricular frameworks. Teachers use explicit criteria for scoring or grading students' work, and it is not common to observe errors in teachers' scoring or grading. Parents are well informed about students' grades and teachers provide substantial useful feedback to students. 6. Specifically, classroom assessment is a required component of a teacher's performance evaluation and of school inspection and teacher supervision. Additionally, there exists an external moderation system that reviews the difficulty of classroom assessment activities, appropriateness of scoring criteria, etc. System-wide reviews of the quality of education include a focus on classroom assessment, and government funding is available for research on the quality of classroom assessment activities and how to improve classroom assessment. There are also annual classroom assessment programs that are organized by the Assessment and Testing Center. 7. The key stakeholders include students, parents, and school district/Ministry of Education officials. 8. The required uses include diagnosing student learning issues, providing feedback to students on their learning, informing parents about their child’s learning, planning next steps in instruction, grading students for internal classroom uses, and providing input to an external examination program (e.g., school-based assessment with moderation and quality audit). 17 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ARMENIA Examinations 18 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this 1 place. operating on an irregular basis. has been operating regularly. dimension * There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this 2 authorizes the examination. document that authorizes the authorizes the examination. dimension. examination. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 3 dimension. the public public. dimension. * This option does not apply to this This option does not apply to this The policy document addresses some The policy document addresses all key 4 dimension. dimension. key aspects of the examination. aspects of the examination. * ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the 5 the examination or are indifferent to it. examination. examination. examination. There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to * examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder 6 groups. groups. * Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this not welcomed by the leadership in dimension. generally welcomed by the leadership in dimension. 7 charge of the examination charge of the examination. * (CONTINUED) 19 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this 8 examination. the examination. examination. dimension. * This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. 9 processing or reporting. processing and reporting. * This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and dimension. development. dimension. development. * ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or The examination office is newly The examination office is a stable This option does not apply to this 10 is newly established. established. organization. dimension. * The examination office is not This option does not apply to this The examination office is accountable to This option does not apply to this 11 accountable to an external board or dimension. an external board or agency. dimension. agency. * Examination results are not recognized Examination results are recognized by Examination results are recognized by Examination results are recognized by by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection 12 country. * another country. system in another country. The examination office does not have The examination office has some of the The examination office has all of the The examination office has state of the the required facilities to carry out the required facilities to carry out the required facilities to carry out the art facilities to carry out the 13 examination. examination. examination. examination. (CONTINUED) * 20 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment 14 examination, issues are pervasive. effectively, with minimal issues. effectively, with no issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities The country offers a wide range of that prepare for work on the dimension. that prepare for work on the opportunities that prepare for work on 15 examination. examination. the examination. * 21 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what This option does not apply to this 16 measures. dimension. the examination measures. dimension. What the examination measures is This option does not apply to this * What is measured by the examination is This option does not apply to this questioned by some stakeholder groups. dimension. largely accepted by stakeholder groups. dimension. * Material to prepare for the examination There is some material to prepare for the There is comprehensive material to There is comprehensive material to is minimal and it is only accessible to examination that is accessible to some prepare for the examination that is prepare for the examination that is 17 very few students. students. accessible to most students. accessible to all students. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses There are up-to-date compulsory courses examinations available to teachers. workshops on examinations available to or workshops on examinations available or workshops on examinations for 18 teachers. to teachers. teachers. * Teachers are excluded from all Teachers are involved in very few Teachers are involved in some Teachers are involved in most 19 examination-related tasks. examination-related tasks. examination-related tasks. examination-related tasks. * 22 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical There is a comprehensive, high quality documentation. examination, but it is not in a formal report but with restricted circulation. technical report available to the general 20 report format. public. * There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the 21 examination. quality of the examination. * ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the examination process is high. examination process is moderate. examination process is low. examination process is marginal. * The examination results lack credibility The examination results are credible for The examination results are credible for This option does not apply to this 22 for all stakeholder groups. some stakeholder groups. all stakeholder groups. dimension. * The majority of the students (over 50%) A significant proportion of students A small proportion of students (less than All students can take the examination; may not take the examination because of (10%-50%) may not take the examination 10%) may not take the examination there are no language, gender or other language, gender, or other equivalent because of language, gender, or other because of language, gender, or other equivalent barriers. barriers. equivalent barriers. equivalent barriers. * (CONTINUED) 23 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way. 23 * Student names and results are public. This option does not apply to this Students’ results are confidential. This option does not apply to this dimension. dimension. * ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who There are very limited options for There are some options for students who There is a variety of options for students do not perform well on the examination, students who do not perform well on the do not perform well on the examination. who do not perform well on the 24 or students must leave the education examination. examination. system. * There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the 25 examination. examination. examination. * 24 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 Examinations: Development level rating justifications 1. The program has been operating for less than five years. 2. The policy document is called "Concept of Assessment System", Government of RA, 2005 3. The policy document is published and distributed. 4. The policy document outlines governance, distribution of power, responsibilities among key entities, describes the purpose of the examination, and it describes authorized uses of results. 5. The Ministry of Education supports the examinations the most. 6. The Government provides funding sources, the Ministry of Education and Science provides support in organizing, administrating and using the examinations. Teachers, educators and parents suggest new ideas. 7. The rules of the examinations, their structures, and methods of administration are revised each year on the basis of previous years' experience. 8. Funding comes from the government as well as from the fees that students pay to take the examinations. 9. Funding also covers long- or medium-term planning of program milestones and staff training, printing of materials, pens, computers, scanners, etc. 10. The Assessment and Testing Center has had the responsibility for running the examinations since 2006. 11. The examination office is accountable to an external board that is comprised of representatives from various stakeholder groups. 12. Students receive certificates with their scores, which they can use to apply to the university of their choice. 13. This includes computers for all technical staff, a secure building, secure storage facilities, access to adequate computer servers, the ability to backup data, and adequate communication tools [phone, email, internet]. 14. During the preparation and administration of the examinations there is some need for outsourced human resources. Some errors in the examination questions have been identified as a result of using outsourced human resources. These errors are mostly typos. 15. These include non-university training courses or workshops on educational measurement and evaluation and funding for attending international programs, courses or workshops on educational measurement and evaluation. 25 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 16. The examinations measure student knowledge against the national school curriculum guidelines and standards for the purposes of graduating from high school and entering university. 17. Material that is needed to prepare for the examinations is widely accessible by all students (over 90%) in a variety of learning contexts. This material is published and made publicly available. It includes examples of the types of questions that are on the examination, information on how to prepare for the examination and the framework document that explains what the examinations measure. Guidelines and examples of test material are annually distributed to all schools. 18. The Assessment and Testing Center organizes voluntary courses for teachers in selected schools. Additionally, there is compulsory annual training and courses for teachers working on the examinations. The National Institute of Education organizes courses for all teachers as well. 19. Teachers are involved in selecting or creating examination questions, administering the examination, acting as judges (i.e., in orals) and supervising the examination procedures. 20. The documentation usually consists of statistical analysis of examination results. 21. The systematic mechanisms include internal and external review or observers, and pilot or field testing. Additionally, the Assessment and Testing Center and the Ministry of Education have their representatives in every examination center to observe the administration. Parents can also observe the examination process on special monitors. 22. The scoring of the examination answers is done by scanner which compares the student’s answers with the correct answer keys. Each student has a copy of his answers and has the opportunity to obtain his score (answer keys are available on the internet). 23. The names of students and their results are posted in the examination center after each examination. 24. Students may retake the examination only for the graduation certificate, but not for university entrance. Students may opt for less selective universities. Students may also repeat a grade. 25. There is a committee the role of which is to consider student's appeals and errors in test items. 26 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ARMENIA National (or System-Level) Large-Scale Assessment (NLSA) 27 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this 1 irregular basis. been operating regularly. dimension. There is no policy document pertaining There is an informal or draft policy * There is a formal policy document that This option does not apply to this 2 to NLSA. document that authorizes the NLSA. authorizes the NLSA. dimension. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 3 dimension. the public. public. dimension. * There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the dimension. NLSA will take place. coming years. * ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholders groups support the All stakeholder groups support the NLSA. 4 the NLSA or are indifferent to it. NLSA. NLSA. * (CONTINUED) 28 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the There is irregular funding allocated to There is regular funding allocated to the This option does not apply to this 5 NLSA. the NLSA. NLSA. dimension. * This option does not apply to this Funding covers some core NLSA Funding covers all core NLSA activities: This option does not apply to this dimension. activities: design, administration, analysis design, administration, analysis and dimension. 6 and reporting. reporting. This option does not apply to this Funding does not cover research and This option does not apply to this * Funding covers research and dimension. development activities. dimension. development activities. * ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this 7 team. group of people. institution or unit. dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes * Political considerations never hamper dimension. technical considerations. hamper technical considerations. technical considerations. * This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a This option does not apply to this 8 dimension. clearly recognized body. clearly recognized body. dimension. * (CONTINUED) 29 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed The NLSA office is adequately staffed to The NLSA office is adequately staffed to 9 NLSA. to effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no minimal issues. issues. * The country does not offer opportunities This option does not apply to this The country offers some opportunities to The country offers a wide range of that prepare individuals for work on dimension. prepare individuals for work on the opportunities to prepare individuals for 10 NLSA. NLSA. work on the NLSA. * 30 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this curriculum or learning standards. dimension. curriculum or learning standards. dimension. * What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned What the NLSA measures is largely 11 questioned by stakeholder groups. dimension. by some stakeholder groups. accepted by stakeholder groups. * There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it dimension. 12 measures what it is supposed to intended to measure. is intended to measure. measure. * SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on There are occasional courses or There are some courses or workshops on There are widely available high quality 13 the NLSA. workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. * 31 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include Different options are offered to include 14 groups of students in the NLSA. dimension. all groups of students in the NLSA. all groups of students in the NLSA. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to * There are a variety of mechanisms in 15 ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. place to ensure the quality of the NLSA. * There is no technical report or other There is some documentation about the There is a comprehensive technical There is a comprehensive, high quality 16 documentation about the NLSA. technical aspects of the NLSA, but it is report but with restricted circulation. technical report available to the general not in a formal report format. public. * ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this 17 effective way. dimension. * NLSA information is not used or is used This option does not apply to this NLSA results are used by some NLSA information is used by all in ways inconsistent with the purposes dimension. stakeholder groups in a way that is stakeholder groups in a way that is or the technical characteristics of the consistent with the purposes and consistent with the purposes and assessment. technical characteristics of the technical characteristics of the * 18 assessment. assessment. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in 19 monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA. place to monitor the consequences of the NLSA. * 32 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 National (or System-Level) Large Scale Assessment (NLSA): Development level rating justifications 1. The program has been operating for less than five years. One NLSA has been carried out (in 2010). 2. The policy document is called "Concept of Assessment System", Government of RA, 2005. 3. Yes, in the form of a booklet. 4. Policymakers, teachers unions, students, and the media are supportive of the NLSA, while the educators, parents, employers, think-tanks and universities are neutral to it. The NLSA is a focus of the Ministry of Education and Science, as well as teachers, parents, and students, and is highlighted by the media. 5. The funding is regular and provided by the government and is part of the annual budget of the Assessment and Testing Center. 6. Funding also includes staff training, printing of booklets, and trips to regional schools. 7. It is the Assessment and Testing Center with permanent staff as well as temporary specialists. 8. The Assessment and Testing Center is accountable to its board and to the government. 9. There is the full-time staff of the Assessment and Testing Center and some temporarily assigned specialists. However, the office is inadequately staffed because there are some issues due to the lack of extensive experience of the staff in carrying out the NLSA. Specifically, the staff responsible for the NLSA is more familiar with graduation exams and initially designed items using a similar approach as the one used for exams. The staff needs to build capacity in specifically designing NLSAs. 10. Opportunities include non-university training courses or workshops on educational measurement and evaluation, and funding for attending international programs or courses or workshops on educational measurement and evaluation. Training courses are considered very productive because they are conducted on a regular basis and are available to more people. Funding for attending international programs and workshops is rare, and only a few people have participated in them. 11. Sometimes there is a need to explain that the NLSA is not high-stakes. 12. Although there are comparisons of the results of the NLSA with the results of other assessment programs, there are limited internal reviews of the NLSA to ensure that it measures what it is intended to measure. Specifically, there are reviews of the test materials, and statistical analysis of results in conducted. 13. There are regularly available and occasional courses or workshops, as well as occasional presentations on the NLSA. Most teachers have access to the live courses or workshops. Trainings and seminars to present the results of the NLSA also have been planned. 14. Schools from all Marzes of Armenia are selected. 33 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 15. Such mechanisms include: all proctors or administrators are trained according to a protocol; there is a standardized manual for NLSA administrators; a pilot is conducted before the main data collection takes place; all booklets are numbered; there is double data scoring; scorers are trained to ensure high interrater reliability; and the Coordinator of the NLSA program has considered and approved all disputable situations with scores. The results of the statistical analyses is used to improve the quality of the test items and to make decisions regarding testing subjects. 16. While the current version of the report has restricted circulation, a comprehensive, high-quality technical report is being planned. 17. Specifically, results are disseminated within twelve months after the NLSA is administered; reports with results are made available for all stakeholder groups; the main reports on the results contain information on overall achievement levels and subgroups; the main reports on the results contain information on trends over time, overall and for subgroups; the main reports on the results contain standard errors (measure of uncertainty); there is a media briefing organized to discuss results; there are workshops or presentations for key stakeholders on the results; and results are featured in newspapers, magazines, radio and television. Currently, a report is being prepared to be published. 18. Assessment information is used to review the content of the subjects being tested, to provide suggestions to teachers on teaching methods, and to understand the level of students' knowledge in the tested subject areas. 19. Themed conferences that provide a forum to discuss research and other data on the consequences of the NLSA and expert review groups are being planned. 34 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ARMENIA International Large-Scale Assessment (ILSA) 35 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in The country/system has participated in 1 in an ILSA in the last 10 years. dimension. at least one ILSA in the last 10 years. two or more ILSA in the last 10 years. * The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. 2 the next 5 years. the next 5 years. There is no policy document that There is an informal or draft policy There is a formal policy document that * This option does not apply to this 3 addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. dimension. ILSA. * This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this 4 dimension. the public. public. dimension. * ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, ILSA. donors. discretion. decree or norm. * This option does not apply to this Funding covers some core activities of Funding covers all core activities of the This option does not apply to this 5 dimension. the ILSA. ILSA. dimension. Funding does not cover research and This option does not apply to this This option does not apply to this * Funding covers research and development activities. dimension. dimension. development activities. * (CONTINUED) 36 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. 6 activities. activities. activities. * This option does not apply to this The national/system coordinator or The national/system coordinator is fluent This option does not apply to this 7 dimension. other designated team member may not in the language of the assessment. dimension. be fluent in the language of the assessment. * This option does not apply to this The ILSA office is inadequately staffed or The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, 8 effectively. with minimal issues. with no issues. * 37 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this 9 international workshops or meetings. international workshops or meetings. workshops or meetings. dimension. The country/system offers no This option does not apply to this * The country/system offers some The country/system offers a wide range opportunities to learn about ILSA. dimension. opportunities to learn about ILSA. of opportunities to learn about ILSA. * This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are dimension. dimension. available to the country's/system's ILSA available to a wide audience, in addition team members only. to the country's/system's ILSA team members. 38 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical The country/system met all technical published. standards to have its data presented standards required to have its data standards required to have its data beneath the main display of the presented in the main displays of the presented in the main displays of the international report or in an annex. international report. international report. * The country/system has not contributed This option does not apply to this This option does not apply to this The country/system has contributed new 10 new knowledge on ILSA. dimension. dimension. knowledge on ILSA. * ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely 11 the country/system. in the country/system. the country/system. disseminated in the country/system. Products to provide feedback to schools This option does not apply to this Products to provide feedback to schools * Products to provide feedback to schools and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are 12 not made available. sometimes made available. systematically made available. * There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the There is wide media coverage of the ILSA 17 13 results. ILSA results. ILSA results. results. * If any, country/system-specific results Results from the ILSA are used in a Results from the ILSA are used in some Results from the ILSA are used in a and information from the ILSA are not limited way to inform decision making in ways to inform decision making in the variety of ways to inform decision 14 used to inform decision making in the the country/system. country/system. making in the country/system. country/system. * It is not clear that decisions based on This option does not apply to this This option does not apply to this Decisions based on the ILSA results have ILSA results have had a positive impact dimension. dimension. had a positive impact on students' 15 on students' achievement levels. achievement levels. * 39 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 International Large Scale Assessment (ILSA): Development level rating justifications 1. Armenia has participated in TIMSS 2003, 2007, and 2011. 2. The country has taken concrete steps to participate in PIRLS 2016 and TIMSS 2015. Specifically, the Assessment and Testing Center has requested funding from the government for participation in these ILSAs. 3. The policy document is called "Concept of Assessment System", Government of RA, 2005. 4. The policy document is available in the form of a booklet. 5. Funding also covers attendance at international expert meetings for the assessment exercise. 6. By the order of the Minister of Education (in 1999), there is a national coordinator responsible for the ILSA. The staff of the Assessment and Testing Center is responsible for carrying out the ILSA in Armenia, along with some temporary specialists assigned to work on the ILSA. 7. The team has participated in four rounds of TIMSS. 8. The team is sufficiently staffed, and has previous experience working on international assessments as well as the necessary training to carry out the required assessment activities effectively. 9. While team members have attended some of the international workshops/meetings, it was mostly the National Research Coordinator who attended the workshops/meetings. 10. There are printed and distributed brochures on the research of TIMSS. 11. Dissemination activities included: copies of the national report were distributed to key stakeholders; copies of the international report were distributed to key stakeholders and the key findings communicated through a press release; results received coverage on the television, radio or newspapers, and brochures and PowerPoint presentations with the country’s/system’s results were made available online or distributed to key stakeholders. 12. However, not all educators receive the results. 13. There are editorials or columns commenting on the international assessment results, and the assessment results have been reported on in newspapers, radio and on television. 14. ILSA results have been used by policy makers in the Ministry of Education and education leaders to improve education quality in Armenia by tracking the impact of reforms on student achievement levels, informing curriculum improvement, informing teacher training programs and informing other assessment activities in the country (e.g., classroom assessment, examinations). 15. The student achievement levels for TIMSS 2007 were significantly better than for TIMSS 2003. The student achievement levels for TIMSS 2011 will be released in 2012. 40 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 United Nations Children’s Fund (UNICEF). 2013. Acknowledgements Education in Armenia. Data retrieved from This report was prepared by The World Bank SABER- http://www.unicef.org/armenia/education.html on Student Assessment team in collaboration with Cristian March 12, 2013. Aedo, World Bank Senior Education Specialist and Task Team Leader for education projects in Armenia, and ———. 2010. “Country profile: Education in Armenia Arsen Baghdasaryan, Deputy Director of the 2010.� Data retrieved from http://www.unicef.org/ Assessment and Testing Center in Armenia. ceecis/Armenia_2010.pdf on March 12, 2013. United Nations Educational, Scientific, and Cultural References Organization (UNESCO) - Institute for Statistics. Armenia Country Indicator Data. Montreal, QC: UNESCO. Data Clarke, M. 2012. “What Matters Most for Student retrieved from http://stats.uis.unesco.org/unesco on Assessment Systems: A Framework Paper.� March 12, 2013. READ/SABER Working Paper Series. Washington, DC: World Bank. World Bank. 2013. World Bank Development Indicators: Armenia Country Indicator Data. Washington, DC: World Organisation for Economic Co-operation and Bank. Data retrieved from http://databank.worldbank.org/ Development (OECD). 2011. “Education at a Glance data on March 12, 2013. 2011: OECD Indicators.� Paris: OECD. ———. 2009. Country Partnership Strategy for the Republic of Armenia for the Period FY09-FY12. Report No. 48222-AM. Washington, DC: World Bank. 41 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS ARMENIA ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2011 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policymakers and business people—an accessible, objective snapshot showing how well the policies of their country's education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. THE WORLD BANK 42 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS