Using Student and Teacher Assessments to Design More Pertinent In-Service Teacher Training: The Case of Ecuador

The development of pertinent and effective in-service teacher training remains a policy challenge for many countries in Latin America and the Caribbean. Ecuador stands out as a country in the region that has made significant investments in teacher training in the past decade. However, most in-service training provision has been designed without enough elements to properly address teachers'skills gaps. This paper proposes a roadmap for improving the design of in-service teacher training in Ecuador using available data from student and teacher assessments. Although countries in the region have made important efforts to carry out periodic evaluations of student and teacher performance, the data resulting from these evaluations are rarely used to guide teacher development programs. The analysis presented in this paper suggests that doing so has the potential to raise program pertinence while allowing the prioritization of investments in teachers and students with the greatest needs.


Policy Research Working Paper 8970
The development of pertinent and effective in-service teacher training remains a policy challenge for many countries in Latin America and the Caribbean. Ecuador stands out as a country in the region that has made significant investments in teacher training in the past decade. However, most in-service training provision has been designed without enough elements to properly address teachers' skills gaps. This paper proposes a roadmap for improving the design of in-service teacher training in Ecuador using available data from student and teacher assessments. Although countries in the region have made important efforts to carry out periodic evaluations of student and teacher performance, the data resulting from these evaluations are rarely used to guide teacher development programs. The analysis presented in this paper suggests that doing so has the potential to raise program pertinence while allowing the prioritization of investments in teachers and students with the greatest needs. This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at dangelurdinola@worldbank.org.

I. Introduction
In-service teacher training is an essential component of Ecuador's teachers' professional development strategy. However, available training interventions offered by the state do not necessarily respond to the teachers' skills gaps. This occurs because available training programs do not systematically rely on diagnostics of the teachers' needs. This paper proposes a roadmap for improving the design of in-service teacher training in Ecuador using available data from student and teacher assessments. Current teacher assessments in Ecuador have an active cognitive component capable of assessing the level of knowledge of teachers in the field they teach. They provide valuable information for determining the specific areas of knowledge that teachers have problems with. Coupling such information with available data on student academic achievement, the state could develop not only more pertinent in-service training programs, but also target interventions to teachers that serve the student populations with the greatest needs.
Many countries in Latin America and the Caribbean (LAC) face shortages of good teachers, which undermines the capacity of their systems to improve student achievement in the short, medium, and long term (Rivkin, Hanushek, and Kain 2005;Hanushek and Woessmann 2006;Figlio and Kenny 2007;Chetty, Friedman, and Rockoff 2014). Individuals who become teachers in LAC, most of whom are female, tend to be socioeconomically vulnerable and to underperform in university entrance exams compared to the average university student (Bruns and Luque 2014). Many structural factors explain this phenomenon, such as the lack of incentives to choose teaching as a profession and high turnover of teachers (Ingersoll 2004;Woessmann 2011;Muralidharan and Sundararaman 2011). Moreover, salaries for teachers in the LAC region tend to be substantially lower than in the average profession (Moura- Castro and Ioschpe 2007), which limits the capacity of the sector to attract the best professionals.
Despite remarkable progress in teachers' educational attainment (at present, most teachers in the LAC region have completed a tertiary degree), graduates from the teaching profession enter classrooms with significant cognitive and pedagogical gaps (Bruns and Luque 2014). 2 These deficiencies weaken education systems in the region, as teachers' cognitive skills influence the achievement of their students (Hanushek, Piopiunik, and Wiederhold forthcoming). 3 In response to the cognitive gaps of teachers, countries in LAC have relied on in-service teacher training as part of their teachers' professional development strategies (Levy and Schady 2013;Avalos 2007). In-service teacher training is a significant element of non-salary spending in most LAC education systems (Bruns and Luque 2014). In some countries in the region, teachers participate in training between 21 and 35 days per year (OECD 2009).
A primary weakness of in-service teacher training programs is that they seldom respond to the actual skills gaps of teachers (Bruns and Luque 2014;Popova et al. 2018). Ávalos (2007) conducted a review of several in-service teacher training programs in LAC. The study concludes that in-service teacher training in the region should focus more on strengthening teachers' cognitive deficiencies, given the fragility of their initial training. Although teacher effectiveness involves several factors (such as adequate mastery of the curricula, high motivation, and proper use of pedagogy), the author highlights the importance of prioritizing training efforts in the region as a corrective measure to help teachers reach an adequate mastery of the curricula.
Ecuador has made substantial efforts to improve teacher quality in recent years. This paper proposes a roadmap for improving the design of in-service teacher training in Ecuador using available data from student and teacher assessments. Even though Ecuador has made substantial efforts to periodically evaluate student and teacher performance, the data resulting in these evaluations have not been used systematically to guide teacher development programs. The results of this study indicate that systematically using the data has the potential to help design more relevant programs that respond to observed skills gaps and allow prioritizing investments in teachers and students with the most substantial needs.
The study also intends to raise awareness about the potential gains of capitalizing on available information on student and teacher assessments. Section III provides a brief overview of the information collected by the national teacher's assessment (Ser Maestro) and discusses how the information included in this assessment could guide the design of in-service training courses. Section IV presents different policy scenarios that use data from student and teacher evaluations to better design in-service teacher training. A brief conclusion follows.

II. In-service teacher training in Ecuador
In 2011, Ecuador enacted a new education law (Ley Orgánica de Educación Intercultural) that provided the education sector with a renewed vision of the country's educational processes and actors. The law recognizes and gives high value to the work of teachers and introduces important elements to promote and develop the teaching career. For instance, Chapter 4 of the law specifies that teachers in Ecuador shall have free access to professional development opportunities, such as in-service training, to continuously update their knowledge and pedagogical practices.
Chapter 10 establishes the need for the creation of a National University of Education, whose objective would be to support and develop the teaching profession and to better train new teachers.
The law also establishes a more transparent teaching career path, whereby promotions are based on obtaining satisfactory results in teaching evaluations and on active participation in continuing education programs. an average of over 10,000 teachers a year, with an average yearly investment of US$2.7 million. 5 The cost of in-service teacher training is between US$28.5 and US$38.5 per hour of training (MINEDUC 2016).
In-service training under the SIPROFE program is mandatory for teachers who underperform (that is, teachers with a score lower than 700 points in the test) on the teacher assessment Ser Maestro, a test implemented periodically by the National Institute of Evaluation (INEVAL). However, this requirement has not been systematically enforced. Participation in training is optional for all other teachers and school administrators in the public education system.
Teachers' participation in these training programs is among one of many criteria considered for career promotion.
In terms of design features, most training programs take place on school premises and outside the teachers' working time (that is, holidays, evenings, and weekends). However, in recent years, training providers have introduced some flexibility in the curricula by providing some content (and thus training hours) online. Also, teachers can take training courses during working hours if they obtain permission from school district authorities. To complete a course, participating teachers need to attend all the required course hours and pass a course evaluation. Also, to promote attendance and completion, the MINEDUC requires that teachers who sign up for a training program and fail the course (due to lack of attendance or if they fail the course exam) pay for the training cost out-of-pocket. Training opportunities are normally communicated to teachers by email when new courses open. The content and curricula of the courses are generally designed between MINEDUC administrators and training providers based on the ministry's identified priorities and the provider's capacity to deliver courses (that is, courses are largely supply-driven).
Neither the MINEDUC nor training providers conduct a systematic diagnostic to identify teacher learning needs.
To date, the effectiveness of the SIPROFE program remains largely unassessed. The MINEDUC has not conducted a systematic evaluation of the program and its effects on teacher and student outcomes. There have been some attempts to collect qualitative information to assess the program's effectiveness through interviews and focus groups with teachers and school 7 principals. Nevertheless, most of the information available relies on small samples and cannot be assumed to be representative for statistical inference.
Results from qualitative interviews conducted by Portaluppi Castro (2012) and Fabara-Garzon (2013) show, for instance, that teachers identify several problems with the training courses offered by the SIPROFE program. Focus group participants indicated that the training programs offered under SIPROFE lacked relevance and that courses were theoretical and provided little space for dialogue, analysis, and contextualization. Participants also indicated that there was little follow-up (for example, mentoring, coaching, feedback mechanisms) after training courses were completed. Some participants complained about the quality and pedagogical approaches of the trainers.
On the positive side, participants agreed that the efforts made through the SIPROFE program are a clear signal of the interest of the state in teachers' professional development.
Participants also indicated that some teachers participated in the program to be promoted. Finally, participants suggested that the program could be improved by (a) having a better diagnostic of teachers' training needs and offering progress to address them; (b) complementing training courses with more follow-up support, such as coaching and mentoring; (c) making it easier to obtain permission to attend training during working hours; and (d) making courses more interactive, flexible, and relevant to local contexts.

III. Teacher assessments in Ecuador
The Ser Maestro assessment, the government's main instrument to quantify the level of knowledge of Ecuadoran teachers, is implemented by Ecuador's National Institute of Evaluation and is an important pillar of the country's teacher evaluation model. The assessment measures the level of knowledge of teachers in the field they teach, including specific topics they must master to promote an adequate learning process in their subjects. The assessment is applied to all teachers in active service (that is, it is census-based). The last assessment was conducted in 2016.
The assessment includes 19 knowledge areas (Saberes) that include subjects spanning Ecuador's three main educational cycles: Early Education (pre-K), Basic General Education (BGE) (grades 1 to 10), and Unified General Baccalaureate (UGB), equivalent to high-school (grades 1 to 3). Each subject includes several "curricular areas," which assess the level of knowledge of the teacher in specific aspects of the national curriculum within the subject matter.

8
For each operational definition, in turn, the assessment includes a series of questions that make it possible to determine exactly which areas of knowledge teachers have problems with. For example, the information provided by the assessment identifies whether a teacher of mathematics (subject area) in BGE (education cycle) has knowledge gaps on set theory or geometry (curricular areas).
The Ser Maestro assessment is applied to teachers according to the subject and grade they teach. Questions in the assessment relate to the main subject teachers teach in the classroom in the academic year of the assessment. Teachers are only assessed on one subject. Generalist teachers in primary schools (grades 2 to 7) are assessed on basic general education. The Ser Maestro database includes a variable that provides teachers' total scores in the assessment (scores oscillate between zero and 1,000 points). Teachers with an overall score below 700 points are considered underperformers (that is, they fail the exam). For these teachers, in-service training is mandatory, although, as mentioned, this policy is not systematically enforced. As illustrated in figure 1, a significant share of teachers in BGE and UGB score below the 700 threshold in the assessment. Based on their scores, teachers can be further classified into four different groups: (a) lowperformance teachers, which are those with scores lower than one standard deviation below the mean; (b) low-standard-performance teachers, which are those with scores between the 9 distribution mean and one standard deviation; (c) high-standard-performance teachers, which are those with scores between the mean and one standard deviation above the mean; and (d) highperformance teachers, which are those with scores above one standard deviation above the mean.
Based on these definitions, 6.28 percent of all teachers in Ecuador are classified as being in the low-performance group, 44.45 percent are classified as being in the low-standard-performance group, 38.26 percent are classified as being in the high-standard-performance group, and 11.01 percent are classified as being in the high-performance group.
Assessment results by knowledge topic indicate that a large majority of teachers (50 to 70 percent) underperform in mathematics, Basic General Education, and natural sciences (BGE level), and in mathematics, physics, and biology (UGB level). Overall teacher performance is generally better in subjects that pertain to social sciences and literature (table 1). A most interesting feature of the Ser Maestro assessment, as mentioned, is that teachers' knowledge is tested against a series of "curricular areas" within each subject. Such a level of granularity of information enables the identification of very specific knowledge areas where teachers display cognitive gaps. Data from the Ser Maestro identify the share of correct and incorrect teacher responses in the assessments. The specificity of this information enables personalized teacher development programs to be designed based on actual needs and knowledge gaps (table 2).

Identification of More Pertinent Training Content
Based on available funding and preestablished yearly training targets, administrators of the SIPROFE program reach out to universities (or other training providers) to have them propose training content. As mentioned, the training curricula for these courses are not based on a systematic assessment of needs, but rather on the university's expertise, the supply of courses, the availability of trainers, and general delivery capacity (that is, courses are supply-driven). programs, which are largely focused on building the capacity of teachers to revise and update the curricula; and (b) remedial programs, which aim to address teachers' knowledge gaps. Table 3 lists the remedial courses most recurrently offered by the SIPROFE program. Results indicate that courses offered are highly focused on pedagogy and general education, with a limited focus on improving cognitive skills. Program administrators and participating training providers could use the information of the Ser Maestro assessments to make courses more demand-driven and responsive to teachers' observed cognitive skills gaps. As mentioned, the assessment identifies both the subjects and curricular areas where teachers display cognitive deficits. Administrators, for instance, could develop reports using the Ser Maestro results that show the observed performance of teachers by 13 subject and curricular areas at the district level. For example, figure 2 displays (for a selected school district in Ecuador) the average performance of teachers in the district in curricular areas related to the teaching subject literature and language for UGB. A similar exercise could be replicated in other subject areas, such as mathematics for UGB, physics for UGB, and so forth.
The figure plots the average percentage of correct answers in a given curricular area for all teachers in the district who currently teach this subject. The red, yellow, and green bars display the subjects on which teachers, on average, correctly answered less than 30 percent of the subject-related questions, between 30 and 35 percent of the subject-related questions, and between 40 to 60 percent of the subject-related questions, respectively. This information could be used by program managers and universities to develop a more pertinent training curriculum that responds to actual knowledge gaps of teachers in a given school district.
A similar exercise using the results of the Ser Maestro assessments could be conducted at the school level. The MINEDUC could provide school directors with individualized teacher reports that contain similar information for all schoolteachers. Based on such information, teachers and directors could develop individualized training plans. Since at the district level the supply of services (in partnership with local universities and pedagogical institutes) would also consider the training needs of teachers in the district, teachers and directors could choose from the available courses offered at the district level the best ones that suit the teacher's individual training needs.
Of course, adapting the supply of services to the actual needs of teachers would take time. Also, for the process to work, the MINEDUC probably would need to provide technical assistance to school directors to be able to interpret teacher reports and to develop individualized training plans.
The MINEDUC could start by testing this model in some school districts (and learn from the experience) before scaling up the model nationally.

Figure 2. Average Percentage of Teacher's Correct Answers by Curricular Area in a Selected School District in Ecuador for Teachers of Literature and Language, UGB
Source: Author using Ser Maestro 2016 data. Note: The red, yellow, and green bars display the subjects on which teachers, on average, correctly answered less than 30 percent of the subject-related questions, between 30 and 35 percent of the subject-related questions, and between 40 to 60 percent of the subject-related questions, respectively.

Can the results of the Ser Maestro assessments be trusted?
Using the Ser Maestro assessments to guide the content of in-service teacher training could be problematic if the knowledge areas measured by the exam are not necessarily related to the curricula imparted in the classroom. If this were the case, investing in training programs to solve the cognitive skills gaps of teachers may not necessarily translate into better student learning.
Ecuador uses the Ser Bachiller assessment, a census-based instrument, to assess student learning. The Ser Bachiller assessment is a compulsory exam students take to graduate from the From a list of verses (independent) or a verse, recognize the metrics.
Recognize the subgenre to which the fragment of a literary work belongs.
From a text, list, or UGB. The assessment is used by most higher education institutions as the entry exam to access higher education. 6 Figure 3 plots school-level results of the Ser Bachiller (y-axis) and the Ser Maestro (xaxis) assessments in the areas of language (panel A) and mathematics (panel B). The panels indicate that schools with better-performing teachers in language and mathematics tend to also have better-performing students in language and mathematics. Moreover, estimates from a simple Ordinary Least Squares regression model indicate that schools having teachers with 10 percent higher average scores in language and mathematics in the Ser Maestro assessments are associated with schools with students with 15 percent higher scores in the language and mathematics Ser Bachiller assessment (table 4). While these results do not imply causality, they indicate a positive correlation between teacher and student performance in language and mathematics in both assessments. These results provide prima facie evidence that improving teachers' cognitive skills in language and mathematics (proxied by the Ser Maestro assessment) is expected to improve the cognitive skills in language and mathematics of the students in the schools they serve (proxied by the Ser Bachiller assessment).

Improving Program Targeting
In-service training opportunities in Ecuador under the SIPROFE program are made available to teachers on a first-come, first-served basis. Once these training opportunities become available, program administrators contact teachers by e-mail or through local campaigns organized  Thus, collecting information from the worst-performing schools and school districts in the Ser Bachiller and the Ser Maestro assessments could be used to target the school districts with the greatest needs. As an illustration, table 5 provides a list of the school districts that display the worst performance in both assessments. The MINEDUC could consider targeting interventions under the SIPROFE program to these districts. Other targeting criteria and additional filters could be added in the process (such as proximity to training providers, local poverty rates, and school size).
In any case, the main idea behind these results is that Ecuadoran policy makers could use their available assessment data to target in-service teacher training interventions to the student and teacher populations with the greatest needs.

V. Conclusion
Teacher performance constitutes a critical determinant of student learning. As such, continuous contact with poor-performing teachers over several years could result in significant learning gaps. In Ecuador, evidence indicates that children in poor localities face significant risks of being taught by teachers who do not master the curricula of the class they teach. Results from teacher assessments indicate that a large majority of teachers underperform in mathematics, Basic General Education, and the natural sciences (BGE level), and in mathematics, physics, and biology (UGB level). Given the fact that many teachers in Ecuador enter the education system with important cognitive gaps, in-service teacher training could be an important tool to strengthen teacher performance.
Ecuador in recent years has made important investments in improving teacher development and quality. Since 2008, the MINEDUC has been promoting in-service teacher training through its Comprehensive System for the Development of Education Professionals (SIPROFE) program.
Since 2010, the SIPROFE program has delivered in-service teacher training to about 75 percent of Ecuador's total teacher population. Nevertheless, training programs delivered under the SIPROFE program have been supply-driven, focused on improving pedagogy, and not necessarily designed to respond to the observed teacher's cognitive skills gaps. Moreover, their effectiveness has been largely unassessed, and program targeting has not prioritized school districts where teacher development programs are needed the most.
Ecuador currently uses teacher and student assessments that can identify in detail the cognitive deficiencies of students and teachers. These data, however, have not been used to guide the design of in-service teacher training programs. Ecuador's teacher assessment (Ser Maestro) evaluates teachers on the subjects they teach in the classroom and identifies curricular areas within a teaching subject where teachers display cognitive gaps. The specificity of this information enables personalized development programs to be designed based on teachers' actual needs.
Program administrators and participating training providers could use this information to develop a more pertinent training curriculum that addresses the knowledge gaps of teachers across different school districts.
Furthermore, combining available information on student and teacher assessments would allow policy makers in Ecuador to identify the worst-performing schools. Given that resources to provide in-service training opportunities are limited, the government could use this information to 20 target interventions to schools with the greatest needs. SIPROFE administrators could use these assessment results plus other information; such as proximity to training providers, local poverty rates, and school size; in their yearly programming, which targets selected territories nationwide.
The results of the Ser Maestro assessments could also serve to develop diagnostics for schools and teachers. The MINEDUC could provide school directors, teachers, and parents with individualized reports that contain information about observed cognitive gaps. Based on such information, teachers and school principals could develop individualized training plans for schools and teachers. Using available information from student and teacher assessments, thus, could become a key input for policy makers in countries that use such information, such as Ecuador, to develop more pertinent and effective teacher development programs. Since more countries are periodically conducting student and teacher assessments, the recommendations presented here for Ecuador could well be taken into consideration regionally and globally.
Of course, the process of adapting the supply of training services to the actual needs of teachers (and to local needs) takes time. However, Ecuador, as well as many other countries in the region, could consider using available evaluations to design and implement in-service training programs of higher relevance and impact while allowing the prioritization of investments in teachers and students with the greatest needs.