JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Jordan SABER Country Report STUDENT ASSESSMENT 2014 Key Policy Areas for Student Assessment Status 1. Classroom Assessment Several system-level documents provide guidelines for carrying out classroom assessment activities. Additionally, there are system-wide resources and materials available to teachers to support them in conducting classroom assessment activities, including in-service training offered by supervisors in directorates of education. Nonetheless, classroom assessment practices by teachers continue to be weak, and there is a recognized need to implement more systematic mechanisms to monitor and ensure their quality. 2. Examinations The General Secondary Examination (Tawjihi) has been administered to grade 12 students since 1962. The main purposes of the examination are to certify student completion of secondary school and determine selection to higher-education institutions. Regular funding for the examination is provided by the government and through student fees, and covers all core activities. The Department of Examinations and Tests in the Ministry of Education is responsible for running the Tawjihi and has all of the required facilities to carry out the examination. An issue that needs to be addressed is increasing the number of options for students who do not perform well on the examination so that they can stay engaged in education or training and continue to develop their potential and ability to contribute to society. 3. National Large-Scale Assessment (NLSA) There are two important NLSA programs in Jordan. The National Test for Quality Control of Education was administered for the first time in 2004. Since then, it has been administered every year in math, science, Arabic language, and English language to all students in (depending on the year) grade 4, 8, or 10. The National Assessment for the Knowledge Economy (NAfKE), has been administered three times since 2006 to students in grades 5, 9, and 11 in math, science, and literacy. Regular funding for NLSA activities is provided by the government (there is also some external funding for NAfKE) and covers all core activities. However, the funding does not cover long- or medium-term planning of program milestones, research and development activities, or staff training. With regard to the National Test in particular, more staff are needed to help carry out the assessment activities. 4. International Large-Scale Assessment (ILSA) In the last ten years, Jordan has participated in TIMSS (2003, 2007, 2011) and PISA (2006, 2009, 2012) and taken concrete steps to participate in TIMSS 2015 and PISA 2015. The ILSA office is adequately staffed to carry out international assessments effectively, and opportunities to learn about ILSAs, such as workshops and online courses, are widely available. At the same time, funding for ILSA activity in Jordan needs to be stabilized and institutionalized, including moving away from a reliance on external funding and ensuring that funding is sufficient to cover all core activities. Data collection on Jordan's assessment system was completed in 2014. Consequently, the findings in this report reflect the status of the country’s assessment system at that time. THE WORLD BANK JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Introduction Jordan has focused on increasing student learning outcomes by improving the quality of education in the country. An effective student assessment system is an important component to improving education quality and learning outcomes as it provides the necessary information to meet stakeholders’ decision-making needs. In order to gain a better understanding of the strengths and weaknesses of its existing assessment system, Jordan decided to benchmark this system using standardized tools developed under the World Bank’s Systems Approach for Better Education Results (SABER) program. SABER is an evidence-based program to help countries systematically examine and strengthen the performance of different aspects of their education systems. What Is SABER-Student Assessment? SABER-Student Assessment is a component of the SABER program that focuses specifically on benchmarking student assessment policies and systems. The goal of SABER-Student Assessment is to promote stronger assessment systems that contribute to improved education quality and learning for all. National governments and international agencies are increasingly recognizing the key role that assessment of student learning plays in an effective education system. The importance of assessment is linked to its role in: (i) providing information on levels of student learning and achievement in the system; (ii) monitoring trends in education quality over time; (iii) supporting educators and students with real-time information to improve teaching and learning; and (iv) holding stakeholders accountable for results. SABER-Student Assessment methodology The SABER-Student Assessment framework is built on the available evidence base for what an effective assessment system looks like. The framework provides guidance on how countries can build more effective student assessment systems. The framework is structured around two main dimensions of assessment systems: the types/purposes of assessment activities and the quality of those activities. Assessment Types and Purposes Assessment systems tend to be comprised of three main types of assessment activities, each of which serves a different purpose and addresses different information needs. These three main types are: classroom assessment, examinations, and large-scale, system level assessments. Classroom assessment provides real-time information to support ongoing teaching and learning in individual classrooms. Classroom assessments use a variety of formats, including observation, questioning, and paper-and-pencil tests, to evaluate student learning, generally on a daily basis. Examinations provide a basis for selecting or certifying students as they move from one level of the education system to the next (or into the workforce). All eligible students are tested on an annual basis (or more often if the system allows for repeat testing). Examinations cover the main subject areas in the curriculum and usually involve essays and multiple- choice questions. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 59 JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Large-scale, system-level assessments provide feedback on the overall performance of the education system at particular grades or age levels. These assessments typically cover a few subjects on a regular basis (such as every 3 to 5 years), are often sample based, and use multiple-choice and short-answer formats. They may be national or international in scope. Appendix 1 summarizes the key features of these main types of assessment activities. Quality Drivers of an Assessment System The key considerations when evaluating a student assessment system are the individual and combined quality of assessment activities in terms of the adequacy of the information generated to support decision making. There are three main drivers of information quality in an assessment system: enabling context, system alignment, and assessment quality. Enabling context refers to the broader context in which the assessment activity takes place and the extent to which that context is conducive to, or supportive of, the assessment. It covers such issues as the legislative or policy framework for assessment activities; institutional and organizational structures for designing, carrying out, or using results from the assessment; the availability of sufficient and stable sources of funding; and the presence of trained assessment staff. System alignment refers to the extent to which the assessment is aligned with the rest of the education system. This includes the degree of congruence between assessment activities and system learning goals, standards, curriculum, and pre- and in-service teacher training. Assessment quality refers to the psychometric quality of the instruments, processes, and procedures for the assessment activity. It covers such issues as design and implementation of assessment activities, analysis and interpretation of student responses to those activities, and the appropriateness of how assessment results are reported and used. Crossing the quality drivers with the different assessment types/purposes provides the framework and broad indicator areas shown in Table 1. This framework is a starting point for identifying indicators that can be used to review assessment systems and plan for their improvement. Table 1: Framework for Building an Effective Assessment System, with Indicator Areas 3 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 The indicators are identified based on a combination of criteria, including: x professional standards for assessment; x empirical research on the characteristics of effective assessment systems, including analysis of the characteristics that differentiate between the assessment systems of low- and high-performing nations; and x theory — that is, general consensus among experts that it contributes to effective assessment. Levels of Development The World Bank has developed a set of standardized questionnaires and rubrics for collecting and evaluating data on the three assessment types and related quality drivers. The questionnaires are used to collect data on the characteristics of the assessment system in a particular country. The information from the questionnaires is then applied to the rubrics in order to judge the development level of the country’s assessment system in different areas. The basic structure of the rubrics for evaluating data collected using the standardized questionnaires is summarized in Appendix 2. The goal of the rubrics is to provide a country with some sense of the development level of its assessment activities compared to best or recommended practice in each area. For each indicator, the rubric displays four development levels—Latent, Emerging, Established, and Advanced. These levels are artificially constructed categories chosen to represent key stages on the underlying continuum for each indicator. Each level is accompanied by a description of what performance on the indicator looks like at that level. x Latent is the lowest level of performance; it represents absence of the desired attribute. x Emerging is the next level; it represents partial presence of the attribute. x Established represents the acceptable minimum standard. x Advanced represents the ideal or current best practice. A summary of the development levels for each assessment type is presented in Appendix 3. In reality, assessment systems are likely to be at different levels of development in different areas. For example, a system may be Established in the area of examinations, but Emerging in the area of large-scale, system-level assessment, and vice versa. While intuition suggests that it is probably better to be further along in as many areas as possible, the evidence is unclear as to whether it is necessary to be functioning at Advanced levels in all areas. Therefore, one might view the Established level as a desirable minimum outcome to achieve in all areas, but only aspire beyond that in those areas that most contribute to the national vision or priorities for education. In line with these considerations, the ratings generated by the rubrics are not meant to be additive across assessment types (that is, they are not meant to be added to create an overall rating for an assessment system; they are only meant to produce an overall rating for each assessment type). The methodology for assigning development levels is summarized in Appendix 4. 4 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Education in Jordan Jordan is an upper-middle-income country in the Middle East & North Africa region. Gross domestic product per capita was $4,945 in 2012, and growth was 2.8 percent. Jordan has made significant progress in increasing school enrollments and education quality. The vast majority of children go to primary school, with a net enrollment rate of 91 percent in 2010. There was impressive growth in the transition rate to secondary school, with an increase from 63 percent in 1994 to 97 percent in 2006. Jordan also ensures a high level of gender parity in access to basic services. The country reached gender parity at the primary level in 1999 (earlier than most other countries in the region) and has now achieved full parity in secondary and tertiary enrollment and 90 percent parity in adult literacy. In the period between 1999 and 2007, Jordan also made significant gains on international surveys of student achievement, with a particularly impressive gain of almost 30 points on the science portion of the Trends in International Mathematics and Science Study (TIMSS). In 2003 Jordan implemented the first phase of the Education Reform for the Knowledge Economy Program (ERfKE I) and is now implementing the second phase of this program (ERfKE II 2009–2015), which aims to provide students enrolled in pretertiary education institutions in Jordan with increased levels of skills to participate in the knowledge economy. In particular, the program focuses on school-level implementation and teacher quality. It aims to strengthen the institutional capacity of the Ministry of Education in policy, strategic planning, monitoring, and evaluation and to improve teacher employment, utilization, and professional development policies and implementation. The program also seeks to fine-tune the curriculum and student assessment to ensure alignment with the knowledge-based economy. Detailed information was collected on Jordan’s student assessment system using the SABER–Student Assessment questionnaires and rubrics. It is important to remember that these tools primarily focus on benchmarking a country’s policies and arrangements for assessment activities at the system or macrolevel. Additional data would need to be collected to determine actual, on-the-ground practices in Jordan, particularly by teachers and students in schools. The following sections discuss the findings by each assessment type, accompanied by suggested policy options. The suggested policy options were determined in collaboration with key local stakeholders based on Jordan’s immediate interests and needs. Detailed, completed rubrics for each assessment type in Jordan are provided in Appendix 5. 5 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Classroom Assessment Level of Development: EMERGING In Jordan, several documents provide guidance on classroom assessment. The Directorate of Tests in the Department of Examinations and Tests authorized the Guidance Brochures on Classroom Assessment and Student Assessment Reports in 2011 to provide guidelines for classroom assessment in grades 1 through 12. In addition, the Achievement Tests Guide, which applies to grades 1 through 12, and the Classroom Assessment Strategy document, which applies generally to all grade levels, provide guidelines for classroom assessment. Some system-wide resources are available to teachers for engaging in classroom assessment activities. For example, a document that outlines what students are expected to learn in all subject areas at each grade level between 1 and 12 and a document that outlines the levels of performance students are expected to reach in math, science, Arabic language, and English language at each grade level between 1–10 and grade 12 are available through the Ministry of Education. In addition, textbooks and workbooks that provide support for classroom assessment and Performance Standards and Indicator Guides are available. System-level mechanisms are in place to ensure that teachers develop skills and expertise in classroom assessment. For example, teachers have opportunities to participate in conferences and workshops, and in-service teacher training includes a focus on classroom assessment. The in-service training is offered by supervisors in directorates of education who have been trained by experts at the Ministry of Education. The supervisors train teachers to use the Performance Standards and Indicator Guides. Classroom assessment is also a required component of teacher supervision. Teacher supervision, which is conducted by the Department of Examinations and Tests, includes field visits to evaluate the implementation of classroom assessment standards. At the end of each term, the Department of Examinations and Tests prepares a report based on a sample of schools visited, detailing issues identified, along with recommendations for improvement. Despite these resources and supports, classroom assessment practices are considered to be weak, overly relying on multiple-choice, selection-type questions and overemphasizing the recall of information. The uneven application of standards for grading students’ work and grade inflation also have been identified as problems. Classroom assessment information is required to be disseminated to all key stakeholders. For example, teachers distribute performance reports to students, who are required to share these with their parents. In addition, dissemination of classroom assessment information to school district or Ministry of Education officials is required. Suggested policy options: 1. Review existing pre- and in-service training opportunities and improve their quality, breadth, and duration so that teachers become more fluent and confident in using a variety of classroom assessment activities in their practice. 2. Ensure the system-wide availability of a variety of high-quality resources to help teachers engage in classroom assessment. For example, develop and make available scoring criteria and rubrics for grading students’ work and encourage school principals to promote their use. 6 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Examinations Level of Development: ESTABLISHED The General Secondary Examination (Tawjihi) is administered to students in grade 12 in core and elective subjects. The examination was first administered in 1962. Its main purposes are certification of secondary school completion and student selection to higher education. The Ministry of Education most recently authorized the use of the Tawjihi for these purposes in its 2011 General Secondary Certificate Instructions document. Regular funding for the Tawjihi is provided by the government and through student fees. Funding covers all core examination activities, including examination design and administration, data analysis and reporting, and medium- to long-term planning of program milestones. However, funding does not cover staff training or research and development activities. The Department of Examinations and Tests, which is responsible for the Tawjihi, is an office within the Ministry of Education that has been running the examination since it was first administered in the 1960s. The Department of Examinations and Tests has the required facilities to carry out the examination. The permanent staff in the Department of Examinations and Tests is supported by staff from the Ministry as needed, including for supervision or correction activities. Occasionally the Ministry seeks external temporary support to assist with the design of questions. For all other functions, the Ministry has sufficient staff to run the examination. Up-to-date compulsory workshops on the Tawjihi are available for teachers. These workshops are organized by supervisors in the directorates of education in the governorates. The workshops address what grade 12 teachers are expected to cover and include information on any updates to the examination. Teachers are involved in some examination-related tasks, including administering and scoring the examination, acting as a judge, and supervising examination procedures. Only one systematic mechanism, internal review or observers, is in place to ensure the quality of the examination. External review or observers, external certification or audits, and pilot or field testing are not in place. Inappropriate behavior surrounding the examination process is not widespread but does occur. Policies are in place to address breaches of examination-taking rules. For example, impersonation (when an individual other than the registered candidate takes the examination), copying from other candidates, the use of unauthorized materials such as prepared answers and notes, and collusion among candidates via mobile phones sometimes occur during the examination process. To address these forms of inappropriate behavior, strict identification policies are put in place, observers monitor test takers from both the front and back of examination rooms, and students are prohibited from bringing mobile phones or any paper into the examination room. If students are suspected of bringing paper or phones into the examination room, they are thoroughly searched and may be prohibited from taking the exam. An important issue to be addressed is increasing the number of options for students who do not perform well on the examination. Currently the options are very limited. Although students may retake the examination, or parts of it, up to four times in three years, they cannot continue their formal education or training at the next level of the system, nor can they repeat grade 12. Suggested policy options: 1. Continue the ongoing work to integrate new knowledge economy skills into the Tawjihi. 7 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 2. Ensure the availability of a greater variety of options for students who do not pass the Tawjihi. For example, encourage dialogue around devising alternative “avenues of success” for students and make this a priority in the examination reform. 3. Ensure the fairness of the Tawjihi by implementing additional preventive and reactive measures to reduce inappropriate behaviors during the examination process. For example, train monitors to effectively react to inappropriate behaviors exhibited by test takers. 8 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 National Large-Scale Assessment (NLSA) — National Test for Quality Control of Education Level of Development: EMERGING The National Test for Quality Control of Education (National Test) is administered every year, alternating between grades 4, 8, and 10 (each of the three grades is assessed once in a three-year span). The assessment was first administered in 2004. Students are assessed in Arabic language, mathematics, science, and English language. All students at the tested grade level are assessed. The main purposes are to monitor education quality at the system level; support schools and teachers; and inform policy design, evaluation, and decision making. A formal document, the Ministry of Education’s Statistical Report, authorizes the National Test and is available to the public. In addition, a written plan is publicly available that specifies who will be tested and in which subject areas. Regular funding is allocated by the government and covers all core NLSA activities, including assessment design and administration as well as data analysis and reporting. However, funding does not cover long- or medium-term planning of program milestones, research and development activities, or staff training. The Directorate of Examinations and Tests, which is responsible for carrying out the National Test, is a permanent unit in the Ministry of Education. Although there is a permanent and full-time staff, it is insufficient in terms of quality and quantity. Regular reviews are conducted, both internal and by qualified external experts, of the alignment between the assessment instrument and what it is supposed to measure. Mechanisms are in place to ensure the quality of the NLSA. For example, a pilot is conducted before the main data collection to ensure the quality of the NLSA, all proctors and administrators are trained according to a protocol, there are internal reviewers, a standardized manual is available for assessment administrators, and discrepancies are required to be reported on a standard sheet. However, scorers are not trained to ensure high interrater reliability, and no requirements are in place for double scoring or processing of data, for booklets to be numbered, or for external reviewers or audits. NLSA results are disseminated in an effective way. For example, reports are disseminated within 12 months after the assessment is administered, reports with results are made available to all stakeholder groups, and a media briefing is held to discuss the results. In addition, the main reports on the results contain information on overall achievement levels and subgroups, trends over time, and standard errors. However, no workshops or presentations are given for key stakeholders on the results, and results are not featured in newspapers and magazines or on radio and television. Very limited mechanisms are in place to monitor the consequences of the NLSA. Remedial plans are developed to address students’ weaknesses in basic learning skills, with follow-up on implementation of these plans conducted by field supervisors. However, there are no expert review groups, regular focus groups of key stakeholders, themed conferences that provide a forum to discuss research and other data on the assessment, or funding for independent research. Suggested policy options: 1. Strengthen quality assurance mechanisms for the National Test by implementing key procedures such as training for scorers to ensure high interrater reliability. 9 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 2. Increase the number of staff in the Directorate of Examinations and Tests working on the National Test in order to address issues of insufficient human resources. In addition, provide additional opportunities for staff working on the National Test to improve their skills related to carrying out assessment activities. 10 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 National Large-Scale Assessment (NLSA) – National Assessment for the Knowledge Economy (NAfKE) Level of Development: EMERGING The National Assessment for the Knowledge Economy (NAfKE) has been administered three times, in 2006, 2008, and 2011, to a representative random sample of students in grades 5, 9, and 11 in the subjects of math, science, and literacy. NAfKE is conducted alongside the Education Reform for the Knowledge Economy (ERfKE I and II) projects. ERfKE is a sector- level intervention supported by the World Bank and other donors, which aims to transform the education system at the early childhood, basic, and secondary levels to produce graduates with the necessary skills for the knowledge economy. NAfKE serves to provide the Ministry of Education in Jordan with data on strengths and weaknesses in students’ skills with respect to the knowledge economy to inform decision making. In addition, the assessment serves to measure changes in student performance related to the implementation of ERfKE. According to the appraisal document for ERfKE, the National Center for Human Resources Development (NCHRD) is authorized to carry out external monitoring and evaluation for the project, including the implementation of NAfKE. The NCHRD is a permanent research center created to carry out studies and assessments across sectors. In addition, the Monitoring & Evaluation Framework document for ERfKE includes NAfKE as a major input to assess progress and outcomes. This document also serves as a written plan for NAfKE, specifying who will be assessed and in which subject areas. A combination of funding from nongovernment sources and NCHRD’s internal budget are used to cover core NAfKE activities, including assessment design and administration as well as data analysis and reporting. The NCHRD is adequately staffed to effectively carry out the assessment. However, funding does not cover long- or medium-term planning of program milestones, staff training, or research and development activities. A variety of mechanisms are in place to ensure the quality of NAfKE. For example, a standardized manual is available for assessment administrators, double scoring and processing of data are done, and a pilot is conducted before the main data collection takes place. In addition, all proctors or administrators are trained according to a protocol, discrepancies must be recorded on a standard sheet, and all booklets are numbered. Scorers are trained to ensure high interrater reliability, and internal reviewers or observers are in place. NAfKE results are disseminated within 12 months after the assessment is administered through national reports, workshops, and presentations to key stakeholders. Reports containing average and disaggregated results are made available to all stakeholder groups. The main reports also present trends over time, overall and for subgroups, and standard errors for reported statistics. Workshops or presentations are also given for key stakeholders on the results. Suggested policy options: 1. Redouble efforts within the Ministry of Education and NCHRD to ensure better alignment and consistency of the NAfKE with the ongoing curriculum reform, the introduction of innovative classroom-based student assessment practices, and the integration of information and communication technology in the curriculum. 2. Establish and make widely available opportunities to learn about NAfKE in teacher training courses, workshops, or presentations so that teachers are knowledgeable about the assessment and so that NAfKE is aligned with the overall education system. For example, offer workshops to teachers on the domains measured in the assessment and how to read and use results. 11 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 3. Consider institutionalizing NAfKE as a measure of education system quality post-ERfKE by introducing formal policy documents authorizing the assessment and by allocating regular government funding for NAfKE activities. 12 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 International Large-Scale Assessment (ILSA) Level of Development: EMERGING Jordan has participated in multiple ILSAs in the last 10 years, including TIMSS at the grade 8 level (2003, 2007, and 2011) and PISA (2006, 2009, and 2012). Jordan has also taken concrete steps to participate in PISA 2015 and TIMSS 2015. The available funding for ILSA participation covers some of the core activities, including international participation fees, implementation of the assessment exercise in Jordan, and attendance at international expert meetings. To date, Jordan has covered participation fees for the international studies through a World Bank loan, and costs related to meetings and implementation of the study are covered by a grant from USAID. Data analysis and research are usually carried out by the NCHRD from its own budget and through a USAID-funded capacity building project supporting NCHRD researchers. The available funding for ILSAs covers research and development when there is a specific need for such activities. The ILSA office is adequately staffed and trained to carry out ILSA activities effectively, with minimal issues. The number of staff members is sufficient, including four permanent staff and some temporary staff. Permanent staff members work on data analysis and data management, and temporary staff are hired for data collection and data entry. Some members of the team have previous experience working on ILSAs and have attended some of the international meetings related to these assessments. Country-specific ILSA results and information are regularly disseminated. A national report is made available online, and copies are distributed to key stakeholders. Copies of the international report are also distributed. In addition to these reports, a policy brief is produced that targets policy makers and other key stakeholders. Recently, because of an apparent decline in TIMSS scores, another report was produced to analyze the results and possible factors contributing to this decline. Results from ILSAs are used in a variety of ways to inform decision making in Jordan. For example, results are used to track the impact of reforms on student achievement levels and to inform interventions such as ERfKE. In addition, results are used to inform curriculum improvement, teacher training programs, and other assessment activities in the system. ILSA results are not used to inform resource allocation. Decisions based on ILSA results have had a positive impact on students’ achievement levels. A study published in 2011 by H. Abdul-Hamid and colleagues found that ILSA results contributed to feedback between researchers and policy makers, improving student achievement levels in Jordan. Suggested policy options: 1. Identify ways to further engage the Department of Examinations and Tests in the analysis and utilization of TIMSS and PISA data, in collaboration with the NCHRD, to raise awareness among supervisors and teachers of the objective of those assessments and their potential value-added in the improvement of school and classroom practice. 2. Continue to build the capacity of NCHRD so that National Reports for TIMSS and PISA are issued in a timely manner and with sufficient analytical quality. In addition, collaborate with the national media to better disseminate information and results from Jordan’s participation in ILSAs. 13 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 3. Introduce government funding for ILSA activities, including research and development activities, to ensure the stability and continuity of the country’s ILSA engagement moving forward. 14 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Appendix 1: Assessment Types and Their Key Differences Classroom Large-scale assessment Examinations surveys National International Exit Entrance Purpose To provide To provide To provide To certify To select students immediate feedback on overall feedback on the students as they for further feedback to health of the system comparative move from one educational inform at particular performance of level of the opportunities classroom grade/age level(s), the education education system instruction and to monitor system at to the next (or into trends in learning particular the workforce) grade/age level(s) Frequency Daily For individual For individual Annually and Annually and more subjects offered on subjects offered more often where often where the a regular basis on a regular basis the system allows system allows for (such as every 3-5 (such as every 3-5 for repeats repeats years) years) Who is All students Sample or census of A sample of All eligible All eligible tested? students at a students at a students students particular grade or particular grade or age level(s) age level(s) Format Varies from Usually multiple Usually multiple Usually essay and Usually essay and observation choice and short choice and short multiple choice multiple choice to answer answer questioning to paper-and- pencil tests to student performances Coverage of All subject Generally confined Generally Covers main Covers main curriculum areas to a few subjects confined to one or subject areas subject areas two subjects Additional Yes, as part Frequently Yes Seldom Seldom information of the collected from teaching students? process Scoring Usually Varies from simple Usually involves Varies from Varies from simple informal and to more statistically statistically simple to more to more statistically simple sophisticated sophisticated statistically sophisticated techniques techniques sophisticated techniques techniques 15 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Appendix 2: Basic Structure of Rubrics for Evaluating Data Collected on a Student Assessment System Development level LATENT ESTABLISHED (Absence of, or EMERGING (Acceptable deviation from, (On way to meeting minimum ADVANCED Dimension attribute) minimum standard) standard) (Best practice) Justification EC—Enabling Context EC1—Policies EC2—Leadership, public engagement EC3—Funding EC4—Institutional arrangements EC5—Human resources SA—System Alignment SA1—Learning/quality goals SA2—Curriculum SA3—Pre-, in-service teacher training AQ—Assessment Quality AQ1—Ensuring quality (design, administration, analysis) AQ2—Ensuring effective uses 16 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Appendix 3: Summary of the Development Levels for Each Assessment Type Assessment type LATENT EMERGING ESTABLISHED ADVANCED Absence of, or deviation On way to meeting Acceptable minimum Best practice from, the attribute minimum standard standard There is no system-wide There is weak system- There is sufficient There is strong system-wide institutional capacity to wide institutional system-wide institutional institutional capacity to support support and ensure the capacity to support and capacity to support and and ensure the quality of Classroom Assessment quality of classroom ensure the quality of ensure the quality of classroom assessment practices. assessment practices. classroom assessment classroom assessment practices. practices. There is no standardized There is a partially There is a stable There is a stable standardized examination in place for stable standardized standardized examination in place and key decisions. examination in place, examination in place. institutional capacity and strong and a need to develop There is institutional mechanisms to monitor it. The institutional capacity to capacity and some examination is of high quality and run the examination. The limited mechanisms to is perceived as fair and free from Examinations examination typically is monitor it. The corruption. of poor quality and is examination is of perceived as unfair or acceptable quality and is corrupt. perceived as fair for most students and free from corruption. There is no NLSA in There is an unstable There is a stable NLSA There is a stable NLSA in place place. NLSA in place and a in place. There is and institutional capacity and need to develop institutional capacity and strong mechanisms to monitor it. institutional capacity to some limited The NLSA is of high quality and run the NLSA. mechanisms to monitor its information is effectively used National (or System-Level) Assessment quality and it. The NLSA is of to improve education. Large-Scale Assessment impact are weak. moderate quality and its information is disseminated, but not always used in effective ways. There is no history of Participation in an ILSA There is more or less There is stable participation in an participation in an ILSA has been initiated, but stable participation in an ILSA and institutional capacity to nor plans to participate there still is need to ILSA. There is run the ILSA. The information in one. develop institutional institutional capacity to from the ILSA is effectively used capacity to carry out the carry out the ILSA. The to improve education. International Large-Scale ILSA. information from the Assessment ILSA is disseminated, but not always used in effective ways. 17 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Appendix 4: Methodology for Assigning Development Levels 1. The country team or consultant collects information about the assessment system in the country. 2. Based on the collected information, a level of development and score is assigned to each dimension in the rubrics: x Latent = 1 score point x Emerging = 2 score points x Established = 3 score points x Advanced = 4 score points 3. The score for each quality driver is computed by aggregating the scores for each of its constituent dimensions. For example: The quality driver, ‘Enabling Context,’ in the case of ILSA, has 3 dimensions on which a hypothetical country receives the following scores: Dimension A = 2 points; Dimension B = 2 points; Dimension C = 3 points. The hypothetical country’s overall score for this quality driver would be: (2+2+3)/3 = 2.33 4. A preliminary level of development is assigned to each quality driver. 5. The preliminary development level is validated using expert judgment in cooperation with the country team and the World Bank Task Team Leader. For scores that allow a margin of discretion (i.e., to choose between two levels of development), a final decision has to be made based on expert judgment. For example, the aforementioned hypothetical country has an ‘Enabling Context’ score of 2.33, corresponding to a preliminary level of development of ‘Emerging or Established.’ Based on qualitative information not captured in the rubric, along with expert judgment, the country team chooses ‘Emerging’ as the most appropriate level. 6. Scores for certain key dimensions under ‘Enabling Context’ (in the case of EXAM, NLSA, and ILSA) and under ‘System Alignment’ (in the case of CLASS) were set as ceiling scores, i.e., the overall mean score for the particular assessment type cannot be greater than the score for these key dimensions. These key variables include formal policy, regular funding, having a permanent assessment unit, and the quality of assessment practices. 18 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Appendix 5: SABER-Student Assessment Rubrics for Jordan JORDAN Classroom Assessment 19 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ENABLING CONTEXT AND SYSTEM ALIGNMENT Overall policy and resource framework within which classroom assessment activity takes place in a country or system, and the degree to which classroom assessment activity is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT AND SYSTEM ALIGNMENT 1: Setting clear guidelines for classroom assessment There is no system-level document that There is an informal system-level There is a formal system-level document This option does not apply to this provides guidelines for classroom document that provides guidelines for that provides guidelines for classroom dimension. assessment. classroom assessment. assessment.1 This option does not apply to this This option does not apply to this The availability of the document is The document is widely available.2 dimension. dimension. restricted. ENABLING CONTEXT AND SYSTEM ALIGNMENT 2: Aligning classroom assessment with system learning goals There are no system-wide resources for There are scarce system-wide resources There are some system-wide resources There are a variety of system-wide teachers for classroom assessment. for teachers for classroom assessment. for teachers for classroom assessment.3 resources available for teachers for classroom assessment. There is no official curriculum or There is an official curriculum or There is an official curriculum or There is an official curriculum or standards document. standards document, but it is not clear standards document that specifies what standards document that specifies what what students are expected to learn or to students are expected to learn, but the students are expected to learn and to what level of performance. level of performance required is not clear. what level of performance.4 ENABLING CONTEXT AND SYSTEM ALIGNMENT 3: Having effective human resources to carry out classroom assessment activities There are no system-level mechanisms to This option does not apply to this There are some system-level mechanisms There are a variety of system-level ensure that teachers develop skills and dimension. to ensure that teachers develop skills and mechanisms to ensure that teachers expertise in classroom assessment. expertise in classroom assessment.5 develop skills and expertise in classroom assessment. 20 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ASSESSMENT QUALITY Quality of classroom assessment design, administration, analysis, and use. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of classroom assessment Classroom assessment practices suffer Classroom assessment practices are Classroom assessment practices are Classroom assessment practices are from widespread weaknesses or there is known to be weak.6 known to be of moderate quality. known to be generally of high quality. no information available on classroom assessment practices. There are no mechanisms to monitor the There are ad hoc mechanisms to monitor There are limited systematic mechanisms There are varied and systematic quality of classroom assessment the quality of classroom assessment to monitor the quality of classroom mechanisms in place to monitor the practices. practices. assessment practices.7 quality of classroom assessment practices. ASSESSMENT QUALITY 2: Ensuring effective uses of classroom assessment Classroom assessment information is not This option does not apply to this Classroom assessment information is Classroom assessment information is required to be disseminated to key dimension. required to be disseminated to some key required to be disseminated to all key stakeholders. stakeholders. stakeholders.8 There are no required uses of classroom There are limited required uses of There are adequate required uses of There are adequate required uses of assessment to support student learning. classroom assessment to support student classroom assessment to support student classroom assessment to support student learning. learning, excluding its use as an input for learning, including its use as an input for external examination results.9 external examination results. 21 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Classroom Assessment: Development-Level Rating Justifications 1. The Department of Examinations and Tests–Directorate of Tests authorized the Guidance Brochures on Classroom Assessment and Student Assessment Report in 2011 to provide guidelines for classroom assessment for grades 1 through 12. In addition, the Achievement Tests Guide, which applies to grades 1 through 12, and the Classroom Assessment Strategy document, which applies generally to all grade levels, provide guidelines for classroom assessment. 2. The Guidance Brochures on Classroom Assessment and Student Assessment Report, the Achievement Tests Guide, and the Classroom Assessment Strategy documents are available to the public through school libraries. These documents are also disseminated to directorates of education, as well as public, military, UNRWA, and private schools. The Achievement Tests Guide and the Classroom Assessment Strategy documents are also available online. 3. Some system-wide resources are available to teachers for engaging in classroom assessment activities. For example, a document that outlines what students are expected to learn in all subject areas at each grade between 1 and 12 and a document which outlines the levels of performance students are expected to reach in mathematics, science, Arabic language, and English language at each grade between levels 1–10 and grade 12 are available through the Ministry of Education. In addition, textbooks or workbooks that provide support for classroom assessment and Performance Standards and Indicator Guides are available. However, scoring criteria or rubrics for students’ work, item banks or pools with examples of selection/multiple-choice questions or supply/open-ended questions, online assessment resources, or computer based testing with instant reports on students’ performance are not available to teachers. 4. Performance Standards and Indicator Guides specify what students from grades 1 through 12 are expected to learn and to what level of performance across all subject areas. 5. Some system-level mechanisms are in place to ensure that teachers develop skills and expertise in classroom assessment. For example, teachers have opportunities to participate in conferences and workshops, and in-service teacher training focused on classroom assessment. In addition, school inspection or teacher supervision includes a component focused on classroom assessment. In-service teacher training is offered to teachers by supervisors in directorates of education who have been trained by experts at the Ministry of Education. Teachers are trained to use the Performance Standards and Indicator Guides during these in-service training opportunities. However, no preservice teacher training opportunities or online resources are available on classroom assessment. Additionally, not all teacher training programs include a required course on classroom assessment, and teachers do not have the opportunity to participate in item development for, or scoring of, large-scale assessments or exams. 6. Classroom assessment practices are considered to be weak. Classroom assessment activities commonly rely on multiple-choice, selection-type questions and are mainly about recalling information. Although it is not common to observe errors in the scoring or grading of students’ work, it is common for teachers to not use explicit or a priori criteria for scoring or grading students’ work. The uneven application of standards for grading students’ work and grade inflation have also been identified as problems. However, it is rare for parents to be poorly informed about students’ grades, for classroom assessment activities to provide little useful feedback to students, and for classroom assessment activities not to be aligned with a pedagogical or curricular framework. At the same time, it is not common for classroom assessment activities to be mainly used as administrative or control tools rather than as pedagogical resources. 22 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 7. Classroom assessment is a required component of a teacher’s performance evaluation and school inspection or teacher supervision. Teacher supervision is conducted by the Department of Examination and Tests, which include field visits to follow up on implementation of standards and criteria. At the end of each term, the department prepares a report based on a sample of schools visited detailing issues identified, along with recommendations for improvement. No external moderation system is in place to review the difficulty of classroom assessment activities, and national reviews of the quality of education do not include a focus on classroom assessment. In addition, no government funding is available for research on the quality of classroom assessment activities. 8. Classroom assessment information is required to be disseminated to all key stakeholders. For example, teachers distribute performance reports to students, who are required to share these with their parents. In addition, classroom assessment information is required to be disseminated to school district or Ministry of Education officials. 9. Classroom assessment activities are used for diagnosing student learning issues, providing feedback to students on their learning, informing parents about their child’s learning, and planning next steps in instruction. For example, classroom assessment activities are used to determine whether a student has passed or failed and whether they should repeat a grade. In addition, it also determines a student’s stream (Scientific, Literary, Information Technology, etc.) in secondary education, because a minimum grade must be achieved for each stream. However, classroom assessment activities are not used for grading students for internal classroom uses or for providing input to an external examination program. 23 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 JORDAN Examinations 24 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which assessment activity takes place in a country or system and the extent to which that framework is conducive to, or supportive of, the assessment activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies No standardized examination has taken The standardized examination has been The examination is a stable program that This option does not apply to this place. operating on an irregular basis. has been operating regularly.1 dimension There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this authorizes the examination. document that authorizes the authorizes the examination.2 dimension. examination. This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public.3 dimension. This option does not apply to this This option does not apply to this The policy document addresses some key The policy document addresses all key dimension. dimension. aspects of the examination.4 aspects of the examination. ENABLING CONTEXT 2: Having strong leadership All stakeholder groups strongly oppose Most stakeholder groups oppose the Most stakeholder groups support the All stakeholder groups support the the examination or are indifferent to it. examination. examination.5 examination. There are no attempts to improve the This option does not apply to this There are independent attempts to There are coordinated attempts to examination by stakeholder groups. dimension. improve the examination by stakeholder improve the examination by stakeholder groups. groups.6 Efforts to improve the examination are This option does not apply to this Efforts to improve the examination are This option does not apply to this not welcomed by the leadership in charge dimension. generally welcomed by the leadership in dimension. of the examination charge of the examination.7 25 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 (CONTINUED) LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding There is no funding allocated for the There is irregular funding allocated for There is regular funding allocated for the This option does not apply to this examination. the examination. examination.8 dimension. This option does not apply to this Funding covers some core examination Funding covers all core examination This option does not apply to this dimension. activities: design, administration, data activities: design, administration, data dimension. processing or reporting. processing and reporting.9 This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and dimension. development.10 dimension. development. ENABLING CONTEXT 4: Having strong organizational structures The examination office does not exist or is The examination office is newly The examination office is a stable This option does not apply to this newly established. established. organization.11 dimension. The examination office is not accountable This option does not apply to this The examination office is accountable to This option does not apply to this to an external board or agency.12 dimension. an external board or agency. dimension. Examination results are not recognized by Examination results are recognized by a Examination results are recognized by Examination results are recognized by any certification or selection system. certification or selection system in the one certification or selection system in two or more certification or selection country.13 another country. systems in another country.13 The examination office does not have the The examination office has some of the The examination office has all of the The examination office has state of the art required facilities to carry out the required facilities to carry out the required facilities to carry out the facilities to carry out the examination. examination. examination. examination.14 (CONTINUED) 26 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources There is no staff to carry out the The examination office is inadequately The examination office is adequately The examination office is adequately examination. staffed to effectively carry out the staffed to carry out the examination staffed to carry out the assessment examination; issues are pervasive. effectively, with minimal issues. effectively, with no issues.15 The country does not offer opportunities This option does not apply to this The country offers some opportunities The country offers a wide range of that prepare for work on the dimension. that prepare for work on the opportunities that prepare for work on examination. examination. 16 the examination. 27 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 SYSTEM ALIGNMENT Degree to which the assessment is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning examinations with learning goals and opportunities to learn It is not clear what the examination This option does not apply to this There is a clear understanding of what the This option does not apply to this measures. dimension. examination measures.17 dimension. What the examination measures is This option does not apply to this What is measured by the examination is This option does not apply to this questioned by some stakeholder groups. dimension. largely accepted by stakeholder groups.18 dimension. Material to prepare for the examination is There is some material to prepare for the There is comprehensive material to There is comprehensive material to minimal, and it is only accessible to very examination that is accessible to some prepare for the examination that is prepare for the examination that is few students. students. accessible to most students.19 accessible to all students. SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the examination There are no courses or workshops on There are no up-to-date courses or There are up-to-date voluntary courses or There are up-to-date compulsory courses examinations available to teachers. workshops on examinations available to workshops on examinations available to or workshops on examinations for teachers. teachers. teachers.20 Teachers are excluded from all Teachers are involved in very few Teachers are involved in some Teachers are involved in most examination-related tasks. examination-related tasks. examination-related tasks.21 examination-related tasks. 28 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ASSESSMENT QUALITY Degree to which the assessment meets quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring quality There is no technical report or other There is some documentation on the There is a comprehensive technical report There is a comprehensive, high-quality documentation. examination, but it is not in a formal but with restricted circulation.22 technical report available to the general report format. public. There are no mechanisms in place to This option does not apply to this There are limited systematic mechanisms There are varied and systematic ensure the quality of the examination. dimension. in place to ensure the quality of the mechanisms in place to ensure the quality examination.23 of the examination. ASSESSMENT QUALITY 2: Ensuring fairness Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the Inappropriate behavior surrounding the examination process is high. examination process is moderate.24 examination process is low. examination process is marginal. The examination results lack credibility The examination results are credible for The examination results are credible for This option does not apply to this for all stakeholder groups. some stakeholder groups. all stakeholder groups.25 dimension. The majority of the students (over 50%) A significant proportion of students (10– A small proportion of students (less than All students can take the examination; may not take the examination because of 50%) may not take the examination 10%) may not take the examination there are no language, gender, or other language, gender, or other equivalent because of language, gender, or other because of language, gender, or other equivalent barriers.26 barriers. equivalent barriers. equivalent barriers. (CONTINUED) 29 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 3: Using examination information in a fair way Examination results are not used in a Examination results are used by some Examination results are used by most Examination results are used by all proper way by all stakeholder groups. stakeholder groups in a proper way. stakeholder groups in a proper way. stakeholder groups in a proper way.27 Student names and results are public. This option does not apply to this Students’ results are confidential.28 This option does not apply to this dimension. dimension. ASSESSMENT QUALITY 4: Ensuring positive consequences of the examination There are no options for students who do There are very limited options for There are some options for students who There is a variety of options for students not perform well on the examination, or students who do not perform well on the do not perform well on the examination. who do not perform well on the students must leave the education examination.29 examination. system. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There is a variety of mechanisms in place monitor the consequences of the dimension. monitor the consequences of the to monitor the consequences of the examination. examination.30 examination. 30 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Examinations: Development-Level Rating Justifications 1. The General Secondary Examination (Tawjihi) is administered to students in grade 12 in core and elective subjects. Core subjects include Arabic language, Islamic culture, English language, general culture, and computers. Elective subjects depend on a student’s stream, which are chosen from the following specializations: science, literature, Islamic studies, information technology, health, and vocational. The examination was first administered in 1962, and its main purposes are for student certification for school cycle completion and student selection to higher education. 2. The Ministry of Education authorized the Tawjihi with the General Secondary Certificate Instructions document in 2011. 3. The General Secondary Certificate Instructions document is available to the public through the Ministry of Education’s website. In addition, it is available at directorates of education and at schools. 4. The General Secondary Certificate Instructions document addresses some key aspects of the examination. It outlines governance, distribution of power, responsibilities among key entities, procedures to investigate and address security breaches, cheating, and other forms of inappropriate behavior, and procedures for special or disadvantaged students. In addition, the policy document describes the purpose of the examination, specifies who can sit for the examination, identifies rules about preparation for the examination, and explains alignment of the examination with curricula and standards. However, the policy document does not describe authorized uses of results and the funding sources or explain the format of the examination questions. 5. Universities strongly support the Tawjihi, and policy makers, teacher unions, and educators support it as well. Because of the high-stakes nature of the exam, students, parents, think tanks, and NGOs oppose it. It is unclear whether media and employers support the examination. 6. Coordinated efforts are made to improve the examination by stakeholder groups. Ongoing efforts are being made to reform the Tawjihi, with particular attention to minimizing pressure faced by students for their performance. 7. Leadership in charge of the examination generally welcomes efforts to improve the examination. 8. Regular funding is provided by the government and by student fees. 9. Funding covers all core examination activities, including examination design and administration, data analysis and reporting, and long- or medium-term planning of program milestones. However, funding does not cover staff training. 10. Funding for the Tawjihi does not cover research and development activities. 11. The Department of Examinations and Tests, which is responsible for the Tawjihi, is an office within the Ministry of Education that has been running the examination since it was first administered in the 1960s. 31 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 12. The Department of Examinations and Tests is not accountable to an external board or agency. 13. The examination results are officially recognized by certification and selection systems in Jordan. 14. The Department of Examinations and Tests has all of the required facilities to carry out the examination, including computers for all technical staff, a secure building, secure storage facilities, access to adequate computer servers, an ability to backup data, and adequate communication tools. 15. The permanent staff in the department is supported by staff from the ministry as needed, including teachers for supervision, or for correction. Occasionally, the ministry seeks external support (temporary) to help in the design of questions. Otherwise, for all other functions the ministry has sufficient staff to run the examination. 16. Jordan offers some opportunities that prepare for work on the examination. For example, some graduate programs are specifically focused on educational measurement and evaluation at the University of Jordan, Amman Arab University, and Yarmouk University. In addition, non-university courses on educational measurement and evaluation are offered. However, there are no university courses on educational measurement and evaluation, internships in the examination office, or funding available for attending international programs, courses, or workshops on educational measurement and evaluation. 17. There is a clear understanding that the examination measures the national school curriculum guidelines and standards. 18. What is measured by the examination is largely accepted by stakeholder groups. 19. Comprehensive material is accessible to most students online and through schools and teachers. Available material includes examples of the types of questions that are on the examination as well as information on how to prepare for the examination. In addition, the general outcomes framework is available, which explains what is measured by the examination. However, a report on the strengths and weaknesses in student performance are not available. 20. Up-to-date compulsory courses or workshops on the Tawjihi are available for teachers. These workshops are organized by supervisors in directorates of education in governorates and address what grade 12 teachers are expected to deliver and updates to the examination. 21. Teachers are involved in some examination-related tasks. Teachers are involved in administering and scoring the examination, acting as a judge, and supervising examination procedures; however, they are not involved in selecting or creating examination questions and scoring guides, or resolving inconsistencies between examination scores and school grades. 22. A comprehensive technical report on the Tawjihi is available but with restricted circulation. 23. Only one systematic mechanism, internal review or observers, is in place to ensure the quality of the examination. External review or observers, external certification or audits, pilot or field testing, and translation verification are not in place. 32 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 24. Inappropriate behavior surrounding the examination process is moderate. For example, impersonation when an individual other than the registered candidate takes the examination, copying from other candidates, the use of unauthorized materials such as prepared answers and notes, and collusion among candidates via mobile phones and passing of paper do occur during the examination process. To address these forms of inappropriate behavior, strict identification policies are put in place, observers monitor test takers from both the front and back of exam rooms, and students are prohibited from bringing mobile phones or any paper into the exam room. If students are suspected of bringing phones or paper into the exam room, they are thoroughly searched and may be prohibited from taking the exam. Other inappropriate behaviors, such as leakage of the content of an examination paper and intimidation of examination supervisors, do not occur during the examination process. 25. All stakeholder groups perceive the examination results as credible. 26. All students can take the Tawjihi; there are no language, gender, or other equivalent barriers. 27. All stakeholder groups use examination results in a proper way. 28. Student results are confidential. Although results are not publicly available, they can be accessed online by entering a student’s unique seat number or through the school. 29. There are very limited options for students who do not perform well on the examination. Although students may retake the examination (or parts of it) up to four times in three years, they cannot opt for less selective schools, universities, or tracks or repeat the grade. Students also do not have the option to attend remedial courses to prepare to retake the examination. 30. Only one mechanism is in place to monitor the consequences of the examination: regular focus groups or surveys of key stakeholders. Other mechanisms, such as a permanent oversight committee, studies that are updated regularly, and expert review groups, are not in place. 33 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 JORDAN National (or System-Level) Large-Scale Assessment (NLSA)—National Test for Quality Control of Education 34 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system, and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this irregular basis. been operating regularly.1 dimension. There is no policy document pertaining to There is an informal or draft policy There is a formal policy document that This option does not apply to this NLSA. document that authorizes the NLSA. authorizes the NLSA.2 dimension. This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public.3 dimension. There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the dimension. NLSA will take place. coming years.4 ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholder groups support the All stakeholder groups support the NLSA. the NLSA or are indifferent to it. 5 NLSA. NLSA. (CONTINUED) 35 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the NLSA. There is irregular funding allocated to the There is regular funding allocated to the This option does not apply to this NLSA. NLSA.6 dimension. This option does not apply to this Funding covers some core NLSA activities: Funding covers all core NLSA activities: This option does not apply to this dimension. design, administration, analysis, and design, administration, analysis, and dimension. reporting. reporting.7 This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and dimension. development activities.8 dimension. development activities. ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this team. group of people. institution or unit.9 dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes Political considerations never hamper dimension. technical considerations. hamper technical considerations.10 technical considerations. This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a clearly This option does not apply to this dimension. clearly recognized body.11 recognized body. dimension. (CONTINUED) 36 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed to The NLSA office is adequately staffed to The NLSA office is adequately staffed to NLSA. effectively carry out the assessment.12 carry out the NLSA effectively, with carry out the NLSA effectively, with no minimal issues. issues. The country does not offer opportunities This option does not apply to this The country offers some opportunities to The country offers a wide range of that prepare individuals for work on dimension. prepare individuals for work on the opportunities to prepare individuals for NLSA. NLSA.13 work on the NLSA. 37 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this curriculum or learning standards. dimension. curriculum or learning standards.14 dimension. What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned by What the NLSA measures is largely questioned by stakeholder groups. dimension. some stakeholder groups.15 accepted by stakeholder groups. There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it is dimension. measures what it is supposed to measure. intended to measure. intended to measure.16 SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on the There are occasional courses or There are some courses or workshops on There are widely available high quality NLSA.17 workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. 38 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include all Different options are offered to include all groups of students in the NLSA. dimension. groups of students in the NLSA.18 groups of students in the NLSA. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in ensure the quality of the NLSA. dimension. ensure the quality of the NLSA.19 place to ensure the quality of the NLSA. There is no technical report or other There is some documentation about the There is a comprehensive technical report There is a comprehensive, high-quality documentation about the NLSA. technical aspects of the NLSA, but it is not but with restricted circulation. technical report available to the general in a formal report format. public.20 ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way.21 dimension. NLSA information is not used or is used in This option does not apply to this NLSA results are used by some NLSA information is used by all ways inconsistent with the purposes or dimension. stakeholder groups in a way that is stakeholder groups in a way that is the technical characteristics of the consistent with the purposes and consistent with the purposes and assessment. technical characteristics of the technical characteristics of the assessment. assessment.22 There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in monitor the consequences of the NLSA. dimension. monitor the consequences of the NLSA.23 place to monitor the consequences of the NLSA. 39 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 National (of System-Level) Large-Scale Assessment (NLSA): Development-Level Rating Justifications 1. The National Test for Quality Control of Education is administered every year, alternating between Grades 4, 8, and 10 (each of the three grades is assessed once in three years). The National Test for Quality Control of Education was first administered in 2004, and students are assessed in the following subjects: Arabic language, mathematics, science, and English language. All students at the given grade levels are assessed, and the main purposes are to monitor education quality at the system level, support schools and teachers, and inform policy design, evaluation, and decision making. 2. The Ministry of Education’s Statistical Report authorizes the National Test for Quality Control of Education annually. 3. The Statistical Report is available to the public. 4. There is a publicly available written plan specifying who will be tested and in which subject areas. 5. It is not possible to tell the stakeholder engagement for the National Test for Quality Control of Education. 6. Regular funding is allocated by the government to the National Examination. 7. Funding covers all core NLSA activities, including assessment design and administration, and data analysis and reporting. However, funding does not cover long- or medium-term planning of program milestones or staff training. 8. Funding for the NLSA program does not cover research and development activities. 9. The Directorate of Examination and Testing, which is responsible for carrying out the National Examination, is a permanent unit in the Ministry of Education. 10. Although political considerations sometimes hamper technical considerations, assessment results have never been withheld from publication because of political reasons. 11. The Directorate of Examination and Testing is not accountable to a clearly recognized body. 12. The Directorate of Examination and Testing is inadequately staffed to effectively carry out the assessment. Although there is permanent and full-time staff, it is insufficient in terms of quality and quantity. Although staffing is insufficient, no issues have been identified related to the performance of the human resources responsible for the National Examination. 40 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 13. Jordan offers some opportunities to prepare individuals for work on the National Examination. There are graduate programs and courses on educational measurement and evaluation at the University of Jordan, Amman Arab University, and Yarmouk University. In addition, non-university courses or workshops on educational measurement and evaluation are available. However, there are no internships or short-term employment opportunities in the large-scale assessment office or funding for attending international programs, courses, or workshops on educational measurement and evaluation. 14. The National Examination measures performance against internationally recognized and national curriculum guidelines or standards. 15. What the National Examination measures is questioned by some stakeholder groups. 16. Regular independent reviews are conducted by qualified experts as well as regular internal reviews and ad hoc reviews of the alignment between the assessment instrument and what it is supposed to measure. 17. Jordan offers no courses or workshops on the NLSA. 18. At least one option is offered to include all groups of students in the NLSA. For example, the large-scale assessment is offered in the language of instruction for almost all student groups. However, accommodations are not provided for students with disabilities, and no special plans are made to ensure that the large-scale assessment is administered to students in hard-to-reach areas. 19. Some mechanisms are in place to ensure the quality of the NLSA. For example, a pilot is conducted before the main data collection occurs to ensure the quality of the NLSA, all proctors and administrators are trained according to a protocol, there are internal reviewers, there is a standardized manual for large-scale assessment administrators, and discrepancies are required to be reported on a standard sheet. However, scorers are not trained to ensure high interrater reliability, and no requirements are set for double scoring or processing of data, booklets to be numbered, external reviewers, or external certification or audits. 20. A comprehensive, high-quality technical report on the National Examination is available to the general public. 21. NLSA results are disseminated in an effective way. For example, reports are disseminated within 12 months after the large-scale assessment is administered, reports with results are made available for all stakeholder groups, and a media briefing is organized to discuss results. In addition, the main reports on the results contain information on overall achievement levels and subgroups, trends over time overall and for subgroups, and standard errors. However, no workshops or presentations are held for key stakeholders on the results, and results are not featured in newspapers and magazines or on radio and television. 22. Information on the National Examination is used by all stakeholder groups in a way that is consistent with the purposes and technical characteristics of the assessment. 41 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 23. Very limited mechanisms are in place to monitor the consequences of the NLSA. Remedial plans are developed to address students’ weaknesses in basic learning skills, with follow-up on implementation of plans conducted by field supervisors. However, there are no expert review groups, regular focus groups of key stakeholders, themed conferences that provide a forum to discuss research and other data on the consequences of the large-scale assessment, funding for independent research on the impact of the large-scale assessment, or a permanent oversight committee in place. 42 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 JORDAN National (or System-Level) Large-Scale Assessment (NLSA)—National Assessment for the Knowledge Economy (NAfKE) 43 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which NLSA activity takes place in a country or system, and the extent to which that framework is conducive to, or supportive of, the NLSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for NLSA No NLSA exercise has taken place. The NLSA has been operating on an The NLSA is a stable program that has This option does not apply to this irregular basis.1 been operating regularly. dimension. There is no policy document pertaining to There is an informal or draft policy There is a formal policy document that This option does not apply to this NLSA. document that authorizes the NLSA.2 authorizes the NLSA. dimension. This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public. 3 dimension. There is no plan for NLSA activity. This option does not apply to this There is a general understanding that the There is a written NLSA plan for the dimension. NLSA will take place. coming years.4 ENABLING CONTEXT 2: Having strong public engagement for NLSA All stakeholder groups strongly oppose Some stakeholder groups oppose the Most stakeholder groups support the All stakeholder groups support the NLSA.5 the NLSA or are indifferent to it. NLSA. NLSA. (CONTINUED) 44 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having regular funding for NLSA There is no funding allocated to the NLSA. There is irregular funding allocated to the There is regular funding allocated to the This option does not apply to this NLSA.6 NLSA. dimension. This option does not apply to this Funding covers some core NLSA activities: Funding covers all core NLSA activities: This option does not apply to this dimension. design, administration, analysis, and design, administration, analysis, and dimension. reporting. reporting.7 This option does not apply to this Funding does not cover research and This option does not apply to this Funding covers research and dimension. development activities.8 dimension. development activities. ENABLING CONTEXT 4: Having strong organizational structures for NLSA There is no NLSA office, ad hoc unit or The NLSA office is a temporary agency or The NLSA office is a permanent agency, This option does not apply to this team. group of people. institution or unit.9 dimension. This option does not apply to this Political considerations regularly hamper Political considerations sometimes Political considerations never hamper dimension. technical considerations. hamper technical considerations.10 technical considerations. This option does not apply to this The NLSA office is not accountable to a The NLSA office is accountable to a clearly This option does not apply to this dimension. clearly recognized body.11 recognized body. dimension. (CONTINUED) 45 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 5: Having effective human resources for NLSA There is no staff allocated for running an The NLSA office is inadequately staffed to The NLSA office is adequately staffed to The NLSA office is adequately staffed to NLSA. effectively carry out the assessment. carry out the NLSA effectively, with carry out the NLSA effectively, with no minimal issues. issues.12 The country does not offer opportunities This option does not apply to this The country offers some opportunities to The country offers a wide range of that prepare individuals for work on dimension. prepare individuals for work on the NLSA. opportunities to prepare individuals for NLSA. work on the NLSA.13 46 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 SYSTEM ALIGNMENT Degree to which the NLSA is coherent with other components of the education system. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Aligning the NLSA with learning goals It is not clear if the NLSA is based on This option does not apply to this The NLSA measures performance against This option does not apply to this curriculum or learning standards. dimension. curriculum or learning standards.14 dimension. What the NLSA measures is generally This option does not apply to this What the NLSA measures is questioned by What the NLSA measures is largely questioned by stakeholder groups. dimension. some stakeholder groups. accepted by stakeholder groups.15 There are no mechanisms in place to There are ad hoc reviews of the NLSA to There are regular internal reviews of the This option does not apply to this ensure that the NLSA accurately ensure that it measures what it is NLSA to ensure that it measures what it is dimension. measures what it is supposed to measure. intended to measure. intended to measure.16 SYSTEM ALIGNMENT 2: Providing teachers with opportunities to learn about the NLSA There are no courses or workshops on the There are occasional courses or There are some courses or workshops on There are widely available high-quality NLSA.17 workshops on the NLSA. the NLSA offered on a regular basis. courses or workshops on the NLSA offered on a regular basis. 47 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ASSESSMENT QUALITY Degree to which the NLSA meets technical standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of the NLSA No options are offered to include all This option does not apply to this At least one option is offered to include all Different options are offered to include all groups of students in the NLSA.18 dimension. groups of students in the NLSA. groups of students in the NLSA. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in ensure the quality of the NLSA. dimension. ensure the quality of the NLSA. place to ensure the quality of the NLSA.19 There is no technical report or other There is some documentation about the There is a comprehensive technical report There is a comprehensive, high-quality documentation about the NLSA. technical aspects of the NLSA, but it is not but with restricted circulation.20 technical report available to the general in a formal report format. public. ASSESSMENT QUALITY 2: Ensuring effective uses of the NLSA NLSA results are not disseminated. NLSA results are poorly disseminated. NLSA results are disseminated in an This option does not apply to this effective way.21 dimension. NLSA information is not used or is used in This option does not apply to this NLSA results are used by some NLSA information is used by all ways inconsistent with the purposes or dimension. stakeholder groups in a way that is stakeholder groups in a way that is the technical characteristics of the consistent with the purposes and consistent with the purposes and assessment. technical characteristics of the technical characteristics of the assessment.22 assessment. There are no mechanisms in place to This option does not apply to this There are some mechanisms in place to There are a variety of mechanisms in monitor the consequences of the NLSA.23 dimension. monitor the consequences of the NLSA. place to monitor the consequences of the NLSA. 48 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 National (of System-Level) Large-Scale Assessment (NLSA): Development-Level Rating Justifications 1. The National Assessment for the Knowledge Economy (NAfKE) has been administered three times, in 2006, 2008, and 2011. It was introduced to monitor education quality at the system level and inform policy design, evaluation, and decision making and is administered to a representative random sample of students. The NAfKE is conducted alongside the Education Reform for Knowledge Economy (ERfKE I and II) projects. ERfKE is a sector-level intervention supported by the World Bank and other donors that aims to transform the education system at the early childhood, basic, and secondary levels to produce graduates with the skills needed for the knowledge economy. The NAfKE serves to provide the Ministry of Education in Jordan with data on strengths and weaknesses in students’ skills with respect to the knowledge economy to inform decision making. In addition, the assessment serves to measure changes in student performance related to implementation of the ERfKE. 2. Policy documents are available pertaining to NAfKE in Jordan. The appraisal document for the ERfKE states that the NCHRD is authorized to carry out external monitoring and evaluation for the project. In addition, the Monitoring and Evaluation Framework document for ERfKE includes NAfKE as a major input to assess progress and outcomes. In addition, terms of reference are available regarding the NAfKE. 3. The policy documents are available at the Ministry of Education, the National Library, and the Library of the National Center for the Development of Human Resources. 4. A publicly available written plan, the Monitoring & Evaluation Framework document for the ERfKE, specifies who will be tested and in which subject areas. The plan is available to, and accessible by, only certain selected groups. 5. Based on the information provided, a rating cannot be established for this category. 6. In addition to funding from nongovernment sources, NCHRD’s internal budget covers some assessment-related activities. 7. Funding covers all core NLSA activities, including assessment design and administration as well as data analysis and reporting. However, funding does not cover long- or medium-term planning of program milestones or staff training. 8. Funding for the NLSA program does not cover research and development activities. 9. The NCHRD is a permanent research center created to carry out studies and assessments across sectors and runs the NAfKE. 10. Political considerations sometimes hamper technical considerations. For example, debate surrounded which year’s assessment results would be considered the baseline. Since performance declined in 2011 compared to 2006 and 2008, discussion was held about using the 2011 results as the baseline so as not to highlight a decline in performance over time. 49 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 11. The NCHRD is not accountable to a clearly recognized body. 12. The NCHRD is adequately staffed with permanent or full-time staff to effectively carry out the assessment. 13. Jordan offers some opportunities to prepare individuals for work on the NAfKE such as university graduate programs on educational measurement and evaluation. In addition, there are nonuniversity courses or workshops on educational measurement and evaluation, internships or short-term employment opportunities in the large-scale assessment office for students and researchers, and funding for attending international programs, courses, or workshops on educational measurement and evaluation. 14. The NAfKE measures performance against internationally recognized and national curriculum guidelines or standards. 15. What the NAfKE measures is largely accepted by stakeholder groups. 16. Regular internal reviews are conducted of the alignment between the NAfKE and what it is supposed to measure. 17. Jordan offers no courses or workshops on the NLSA. 18. No options are offered to include all groups of students in the NLSA. 19. A variety of mechanisms are in place to ensure the quality of the NAfKE. For example, there is a standardized manual for large-scale assessment administrators, double scoring and processing of data, and a pilot conducted before the main data collection takes place. In addition, all proctors or administrators are trained according to a protocol, discrepancies must be recorded on a standard sheet, and all booklets are numbered. At the same time, scorers are trained to ensure high interrater reliability, and internal reviewers or observers are in place. However, no external reviewers or observers or external certifications or audits are in place. 20. A comprehensive technical report on the NAfKE is available, but with restricted circulation. 21. NLSA results are disseminated in an effective way. For example, reports are disseminated within 12 months after the large-scale assessment is administered, and reports with results are made available for all stakeholder groups. In addition, the main reports on the results contain information on overall achievement levels and subgroups, trends over time overall and for subgroups, and standard errors. Although workshops or presentations are given for key stakeholders on the results, results are not featured in newspapers and magazines or on radio and television. At the same time, no media briefing is organized to discuss results. 22. Information on the NAfKE is used by some stakeholder groups in a way that is consistent with the purposes and technical characteristics of the assessment. 50 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 23. No mechanisms are in place to monitor the consequences of the NLSA. There are no expert review groups, regular focus groups of key stakeholders, themed conferences that provide a forum to discuss research and other data on the consequences of the large-scale assessment, funding for independent research on the impact of the large-scale assessment, or a permanent oversight committee in place. 51 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 JORDAN International Large-Scale Assessment (ILSA) 52 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ENABLING CONTEXT Overall framework of policies, leadership, organizational structures, fiscal and human resources in which ILSA takes place in a country or system and the extent to which that framework is conducive to, or supportive of, ILSA activity. LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 1: Setting clear policies for ILSA The country/system has not participated This option does not apply to this The country/system has participated in at The country/system has participated in in an ILSA in the last 10 years. dimension. least one ILSA in the last 10 years. two or more ILSA in the last 10 years.1 The country/system has not taken This option does not apply to this The country/system has taken concrete This option does not apply to this concrete steps to participate in an ILSA in dimension. steps to participate in at least one ILSA in dimension. the next five years. the next five years.2 There is no policy document that There is an informal or draft policy There is a formal policy document that This option does not apply to this addresses participation in ILSA. document that addresses participation in addresses participation in ILSA. dimension. ILSA.3 This option does not apply to this The policy document is not available to The policy document is available to the This option does not apply to this dimension. the public. public.4 dimension. ENABLING CONTEXT 2: Having regular funding for ILSA There is no funding for participation in There is funding from loans or external There is regular funding allocated at There is regular funding approved by law, ILSA. donors.5 discretion. decree or norm. This option does not apply to this Funding covers some core activities of the Funding covers all core activities of the This option does not apply to this dimension. ILSA.6 ILSA. dimension. Funding does not cover research and This option does not apply to this This option does not apply to this Funding covers research and development activities. dimension. dimension. development activities.7 53 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 (CONTINUED) LATENT EMERGING ESTABLISHED ADVANCED ENABLING CONTEXT 3: Having effective human resources for ILSA There is no team or national/system There is a team or national/system There is a team and national/system This option does not apply to this coordinator to carry out the ILSA coordinator to carry out the ILSA coordinator to carry out the ILSA dimension. activities. activities. activities.8 This option does not apply to this The national/system coordinator or other The national/system coordinator is fluent This option does not apply to this dimension. designated team member may not be in the language of the assessment.9 dimension. fluent in the language of the assessment. This option does not apply to this The ILSA office is inadequately staffed or The ILSA office is adequately staffed or The ILSA office is adequately staffed and dimension. trained to carry out the assessment trained to carry out the ILSA effectively, trained to carry out the ILSA effectively, effectively. with minimal issues.10 with no issues. 54 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 SYSTEM ALIGNMENT Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED SYSTEM ALIGNMENT 1: Providing opportunities to learn about ILSA The ILSA team has not attended The ILSA team attended some The ILSA team attended all international This option does not apply to this international workshops or meetings. international workshops or meetings.11 workshops or meetings. dimension. The country/system offers no This option does not apply to this The country/system offers some The country/system offers a wide range opportunities to learn about ILSA. dimension. opportunities to learn about ILSA.12 of opportunities to learn about ILSA. This option does not apply to this This option does not apply to this Opportunities to learn about ILSA are Opportunities to learn about ILSA are dimension. dimension. available to the country’s/system’s ILSA available to a wide audience, in addition team members only. to the country’s/system’s ILSA team members.13 55 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 ASSESSMENT QUALITY Degree to which the ILSA meets technical quality standards, is fair, and is used in an effective way. LATENT EMERGING ESTABLISHED ADVANCED ASSESSMENT QUALITY 1: Ensuring the quality of ILSA Data from the ILSA has not been The country/system met sufficient The country/system met all technical This option does not apply to this published. standards to have its data presented standards required to have its data dimension. beneath the main display of the presented in the main displays of the international report or in an annex. international report.14 The country/system has not contributed This option does not apply to this This option does not apply to this The country/system has contributed new new knowledge on ILSA.15 dimension. dimension. knowledge on ILSA. ASSESSMENT QUALITY 2: Ensuring effective uses of ILSA If any, country/system-specific results Country/system-specific results and Country/system-specific results and Country/system-specific results and and information are not disseminated in information are disseminated irregularly information are regularly disseminated in information are regularly and widely the country/system. in the country/system. the country/system.16 disseminated in the country/system. Products to provide feedback to schools This option does not apply to this Products to provide feedback to schools Products to provide feedback to schools and educators about the ILSA results are dimension. and educators about the ILSA results are and educators about ILSA results are not made available. sometimes made available.17 systematically made available. There is no media coverage of the ILSA There is limited media coverage of the There is some media coverage of the ILSA There is wide media coverage of the ILSA results. ILSA results.18 results. results. If any, country/system-specific results Results from the ILSA are used in a limited Results from the ILSA are used in some Results from the ILSA are used in a variety and information from the ILSA are not way to inform decision making in the ways to inform decision making in the of ways to inform decision making in the used to inform decision making in the country/system. country/system. country/system.19 country/system. It is not clear that decisions based on ILSA This option does not apply to this This option does not apply to this Decisions based on the ILSA results have results have had a positive impact on dimension. dimension. had a positive impact on students’ students’ achievement levels. achievement levels.20 56 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 International Large Scale Assessment (ILSA): Development-Level Rating Justifications 1. Jordan has participated in multiple ILSAs in the last 10 years, including in TIMSS (2003, 2007, and 2011) and PISA (2006, 2009, and 2012). However, Jordan has participated in TIMSS only at the grade 8 level. 2. Jordan has taken concrete steps to participate in PISA 2015 and TIMSS 2015. 3. The Monitoring and Evaluation Framework document for the ERfKE Program, agreed upon by the government and all the donors, addresses Jordan’s participation in ILSAs. 4. The Monitoring and Evaluation Framework document is publicly available. 5. A regular funding program is available for international assessment participation allocated at discretion as well as funding provided by loans or external donors. For example, Jordan covers participation fees for the international studies through a World Bank loan, while costs related to meetings and the international study are covered by a grant from USAID. 6. Funding covers some core activities of the ILSAs, including international participation fees, implementation of the assessment exercise in Jordan, and attendance at international expert meetings for the assessment exercise. Data analysis and research is usually carried out by the NCHRD from their own budget and through a USAID-funded capacity-building project supporting NCHRD researchers. 7. Funding for ILSAs covers research and development when there is a need for such activities. 8. There are two national coordinators (one for TIMSS and one for PISA) and a team to carry out the ILSA activities. 9. The national ILSA coordinators are fluent in the language of the assessment. 10. The ILSA office is adequately staffed and trained to carry out ILSAs effectively, with minimal issues. There are sufficient staff members, who include four permanent staff and temporary staff; permanent staff members work on data analysis and data management, and temporary staff are hired for data collection and data entry. In addition, some members of the team have previous experience working on international assessments and have attended some international meetings related to the assessment. 11. Jordan’s ILSA team attended some international workshops or meetings. 12. Jordan offers some opportunities to learn about ILSAs, such as workshops or meetings on using international assessment databases and funding for attending international workshops or training on international assessments. 57 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 13. Opportunities to learn about ILSAs are available to Jordan’s ILSA team members, Ministry of Education employees working in the Departments of Examinations, Curricula, and Supervision and Training, and professionals or university staff interested in assessment. 14. Jordan met all technical standards required to have its data presented in the main displays of the international report. 15. Jordan has not contributed new knowledge on ILSAs. 16. Country-specific results and information are regularly disseminated in Jordan. A national report was made available online, and copies were distributed to key stakeholders. In addition, copies of the international report were distributed to key stakeholders. In addition to these reports, a policy brief was developed targeting policy makers and other key stakeholders. Recently, because of an apparent decline in TIMSS scores relative to previous years, another report was produced to analyze results and possible factors contributing to decline. Jordan’s results were communicated through a press release, and brochures and PowerPoint presentations with the country’s results were made available online and distributed to key stakeholders. Results have regularly received coverage on television and radio and in newspapers. All information related to the international study, its reports, and its results are usually copied to CDs and disseminated to universities and other interested stakeholders. However, products providing feedback to the schools and educators about the results have not regularly been made available. 17. Products providing feedback to schools and educators about the ILSA results are sometimes made available. 18. Media coverage of the ILSA results is limited. Although media coverage has been limited to a few small articles for most cycles of the assessment exercises, for TIMSS 2003 wider media coverage was organized, which included a large press conference. 19. Results from ILSAs are used in a variety of ways to inform decision making in Jordan. For example, results are used to track the impact of reforms on student achievement levels, informing interventions such as ERfKE. In addition, results are used to inform curriculum improvement, teacher training programs, and other assessment activities in the system. However, results from the ILSA are not used to inform resource allocation. 20. Decisions based on ILSA results have had a positive impact on students’ achievement levels. A study published in 2011 by H. Abdul-Hamid et al. found that assessment results contributed to feedback between researchers and policy makers, improving student achievement levels in Jordan. 58 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 Acknowledgments This report, part of a 16-country benchmarking exercise in the Middle East & North Africa and Africa regions, was prepared by the World Bank SABER–Student Assessment team, in partnership with the Arab League Educational, Cultural, and Scientific Organization (ALECSO) and the United Nations Educational, Scientific, and Cultural Organization (UNESCO), which led data collection efforts. This effort is part of the Arab Regional Agenda for Improving Education Quality (ARAIEQ) led by ALECSO in partnership with the World Bank. It benefited from feedback and review from Juan Manuel Moreno, World Bank Lead Education Specialist and Task Team Leader for education projects in Jordan, as well as comments received during a national validation workshop held in Jordan. References Abdul-Hamid, H., K. Abu-Lebdeh, and H. Patrinos. 2011. “Assessment Testing Can Be Used to Inform Policy Decisions: The Case of Jordan.” Policy Research Working Paper. World Bank, Washington, DC. Clarke, M. 2012. “What Matters Most for Student Assessment Systems: A Framework Paper.” READ/SABER Working Paper Series. World Bank, Washington, DC. Ministry of Education. 2012. “Education Reform for knowledge Economy- Second Phase (ERfKE II).” The Hashemite Kingdom of Jordan, Amman. Data retrieved from http://www.moe.gov.jo/Files/ (23-7-2012) (12-30-05%20PM).pdf on November 19, 2013. UNESCO. 2012. “Fact Sheet: Education in Jordan.” EFA Global Monitoring Report 2012. UNESCO, Paris. World Bank. 2009. “Project Appraisal Document for a Second Education Reform for the Knowledge Economy Project.” Report No. 46824-JO. World Bank, Washington, DC. ———. “Jordan Country Indicator Data.” World Bank, Washington, DC. Data retrieved from http://databank.worldbank.org/data on November 19, 2013. SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS 59 JORDAN ǀ SABER-STUDENT ASSESSMENT SABER COUNTRY REPORT |2014 www.worldbank.org/education/saber The Systems Approach for Better Education Results (SABER) initiative produces comparative data and knowledge on education policies and institutions, with the aim of helping countries systematically strengthen their education systems. SABER evaluates the quality of education policies against evidence-based global standards, using new diagnostic tools and detailed policy data. The SABER country reports give all parties with a stake in educational results—from administrators, teachers, and parents to policy makers and business people—an accessible, objective snapshot showing how well the policies of their country’s education system are oriented toward ensuring that all children and youth learn. This report focuses specifically on policies in the area of student assessment. This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. 2 SYSTEMS APPROACH FOR BETTER EDUCATION RESULTS