Report No. 31842-MX Mexico Determinants of Learning Policy Note July 21, 2005 Colombia and Mexico Country Management Unit Education Unit, Human Development Department Latin America and the Caribbean Region Document of the World Bank Draft for Internal Circulation OnlyPlease do not cite or quote Acknowledgments This report was prepared by a team consisting of Harry Anthony Patrinos (Task Team Leader), Raja Bentaouet-Kattan, Heather Marie Layton, Claudia Macias, Michelle Mesen and Vicente Paqueo, with significant contributions from Husein Abdul-Hamid, Vicente Garcia, Martha BelemHernandez, Gwang-Jo Kim, Yoshiko Koda, Jorge Moreno and Joseph Shapiro. The team received very useful feedback from Isabel Guerrero, Ariel Fiszbein, Mark Hagerstrom, Juan Prawda and Eduardo Velez. Peer reviewers for this task are Rafael Freyre Martinez (Secretario Tkcnico del Gabinete de Desarrollo Humano y Social), Vincent Greaney, Emmanuel Jimenez, Pablo Latapi (distinguished researcher, Mexico) and Emmanuel Skoufias. The report has benefited from work and insight of the Mexico-World Bank AAA Committee, which consists of Rafael Freyre Martinez, Felicia h a u l (Coordinadora General para la Modernizacidn Administrativa de la Educacio'n), Francisco Miranda (Coordinador de Asesores, Subsecretaria de Educacidn Bdsica y Normal) and Sylvia Schmelkes (Coordinadora General de Educacidn Cultural Bilingiie), under the overall guidance of Jose Maria Fraustro Siller (Oficial Mayor, Secretariat of Public Education). The overall AAA work received thoughtful comments at the Concept Paper stage from the peer reviewers: Todd Crawford, David Gould, Gladys Lopez-Acevedo, Keta Ruiz, Eduardo Velez and Michael Walton. The report was discussed during a meeting chaired by Isabel Guerrero (Country Director for Mexico and Colombia). Written comments were received by the peer reviewers, Emmanuel Jimenez and Emmanuel Skoufias. Written comments were also received from Felicia Knaul, Furio Rosati (ILO/World BanWnicef Understanding Children's Work), Felipe Martinez Rizo and Rafael Vidal (National Institute for Educational Evaluation). i TABLE OF CONTENTS ................................................................................... ........................................................................ Evidence from Mexico. CHAPTER2. THE STAT EducationinMexico................... ........................................................................... 15 State-levelOutcomes................ IndigenousPeoples' Education ................................................... ................................................................................... 26 INTERNATIONALASSESSMENTS .......................................................................................... 28 ........................................... 28 Conclusions ....................................................................................... CHAPTER4. AN AGENDA FOR ACTION ....................................... ...................... IncreaseSchool Autonomy at Public Schools... ................................................................... 50 ContinueLearningfrom Assessments .............................................................................. 56 Areas for Further Research. ............................... 62 TABLES Table 2.la Students by Level of Math, selectedcountries(percent), PISA 2003 ....................................................... 20 Table 2.lb Students by Level of Reading, selectedcountries (percent), PISA 2003 ........................................... 20 Table 3.1 Average Scores and Dispersion ................................................................................................... 34 Table 3.2 Summary of Determinantsof LearninginMexico, PISA 2003 .................................................................. 46 Table 3.3 Implicationsfor SchoolQualitybased on analysis of PISA........................ Table 3.4 Simulating Improvements,PISA2000..... Annex Table lb. Levels of MathCompetencies.... Annex Table la. Levels of ReadingCompetencies................................................... Annex Table 2. Analysis of meansand variancesi Annex Table 3. Variables usedinthe analysis, bas Annex Table 4. ExplainingEducationalAchieve Annex Table 5. Variables used in the analyses and descriptivesta Annex Table 6. EducationProductionFunction(based on GLS with School FixedEffects) Annex Table 7a. Quantile Analysis Model (Science). ........................................................................... 77 Annex Table 7b. Quantile Analysis Model (Math).............................. ........................... 78 Annex Table 7c. QuantileAnalysis Model (Readin .. 11 FIGURES Figure 1.1Results by SchoolType, PISA 2003.......................................................................................................... 11 Figure 2.1 Mexico: Average Returnsto Schoolingover Time.................................................................................... 13 Figure 2.2 Mexico Returnsto Schoolingby Income Quantile, 2002 ....................................................... Figure 2.3 Absolute Enrollment by Level and Year (`000s) .... ..................................... Figure 2.4 Performancein Reading, PISA 2003 ...................................................................................... Figure 2.5 Performancein Mathematics,PISA 2003.................................................................................................. 18 Figure 2.6 Performancein Science, PISA 2003 .......................................................................................................... 19 Figure 2.7 Readingand Math Performance of Mexicoby Level, PISA...................................................................... 21 Figure 2.8 Percentageof students at each level of performancein Readingby country, PISA 2003.......................... 21 Figure 2.9 Percentageof students at each level of performancein Mathematicsby country, PI Figure 2.10 Math Performanceand Net Enrollment Rate in Secondary, PISA 2003................. Figure 2.1 1Math Performanceand Expenditureper Student, PISA 2003....................................... Figure 2.12 Performancein Mathematicsand GDP per Capita, PISA 2003.............................. Figure 3.1 Mean StandardizedTest Scores by Region...... .................................. Figure 3.3 Math Scores and Dispersionacross Countries,PISA 2003......................................... Figure 3.2 Mean StandardizedTest Scoresby Region............................................................... Figure 3.4 Performanceby SocioeconomicLevel Using StandardizedScoresby Level, PISA Figure 3.5 Performanceof Public and Private Schools in Math, PISA 2003, across performan Figure 3.6 Performanceadvantage of school type vs. telesecundaria........................................ Figure 3.7 Effect of Teacher Morale at School among low achievers (Reading)........................... Figure 3.8 Effects of BeingFemaleon ReadingScores by AchievementLevel....................... 40 Figure 3.10 Effect of Teacher Morale on Performanceby Achievement Level, PISA 2003 ......... Figure 3.9 Effect of Equipment and Labs on Science Scores, PISA 2003, across achievementlevels....................... ..............41 Figure 3.11Effect of Student Use of Computersat Schoolby Achievement Level, PISA 2003 ................................ Figure 3.12 Effect of Motivating Students on Value of Subject in Market by Achievement Level, PISA 2003 .......43 43 Figure 3.13 Effect of Memorization on Performanceby Achievement Level, PISA 2003 ......................................... 44 Figure 3.14 Effect of Enjoying Readingby AchievementLevel, PISA 2000................................... 44 Figure 3.15 Effect of Homework by Math AchievementLevel, PISA 2003 ............................................................. 45 Figure 3.16 Effect of Mother with Secondary Educationby Math Achievement Level, PISA 2003................ 45 Figure 3.17 Effect of HomeEducationalResourcesby Math Achievement Level, PISA 2003 ................................. 46 BOXES Box 1.1 Compensatory Education............................................................................................................................... 10 Box 2.1 Basic Education Box 3.1 National Evaluat Box 3.2 Korea's equaliza Box 4.1 An Agenda for Action.. .................................................... Box 4.2 EscuelaNueva............................................................................................................................ Box 4.3 EducationTechnology ........................... 54 Box 4.4 Using InternationalA Box 4SSecondarySchool Reform ... 111 ExecutiveSummary The main objective of this report is to identify the determinants of learningin Mexico, focusing on equity, cross-country comparisons and national studies. The present report provides new analytical work on the determinants of learning using international and national assessments, as well as new work on the returns to quality of education and an assessment of media coverage of international assessments. While Mexico has made significant progress in expanding access to education, much remains to be done to improve the quality of education and to bring Mexican students to a comparable level with other OECD countries. Low but increasing secondary enrollments and low overall learning achievement levels, especially among newer students entering the system, are among the problems that plague Mexico's education system. Efforts to increase enrollments overall have been successful, although more needs to be done, especially at the secondary level. However, there i s reason to remain optimistic that the situation in Mexico can improve. For more than a decade, the education system has been experiencing a dynamic process of change. Recent initiatives at all levels of government have been undertaken to address the challenge of education equity and quality. Labor Market Outcomes. Education continues to be a good investment for increasing productivity. The rates of return to education remain high. There i s evidence that education will contribute to reduced inequality in Mexico as returns to education are highest for people at the bottom of the wage distribution, implying that education i s a good investment, and would particularly benefit those with lower skill levels. Preliminary findings show that those educated in states with higher quality schools, as measured, for instance, by teacher-student ratios, teachers' schooling and test scores, have higher earnings. Thus, there i s empirical evidence that the quality of education increases the returns to education inMexico. Indigenous Peoples. Despite significant improvements in education over the past several decades, considerable educational differences persist between the indigenous and non-indigenous populations. Illiteracy rates and dropout rates continue to be substantially higher among the indigenous population. Average years of schooling, on the other hand, are much lower for indigenous peoples. Indigenous peoples' returns to education are lower than for non-indigenous people. Part of the reason for the lower returns that indigenous peoples experience i s the quality of schooling that they receive. Indigenous schools consistently obtain lower reading and mathematics scores than all other types of schools, regardless o f area of residence. The degree o f implementation of bilingual education policies i s low, even for indigenous schools, and varies widely from one type of school to another. International Evidence. It i s well known that family background and socioeconomic status play a major role in determining learning outcomes. Yet there is evidence that some factors associated with school quality are amenable to policy change and play a significant role in reducing learning inequalities and improving learning outcomes. Specifically, changes in school environment have a positive correlation with improved school quality and improved education iv outcomes. Increased school autonomy, combined with central exams and school accountability mechanisms, also help to raise achievement levels. Other important school and institutional factors include: strong accountability mechanisms (usually punitive), parental and social participation and use of assessmentresults. National Evidence. Evidence from Mexico shows that, as elsewhere in the world, socioeconomic background plays an important role in levels of student achievement. However, recent research has emphasized that school factors also explain variance in outcomes: weak curriculum implementation and an over-emphasis on memorization contribute to low results; inadequate supervision has a negative effect; the school environment i s important for improving outcomes; resources dedicated to improving infrastructure and raising the textbook-student ratio can raise levels of student achievement; and time on task has a significant impact. A few studies have also shown the importance of institutional factors such as school management. Inorder to raise schooling outcomes, devolution of decision-making to the school level i s crucial. Flexibility of curriculum and school format i s just one example of how school autonomy has been useful inextending schooling to hard-to-reach populations. Analyses using Mexico's national assessments find that a number of individual and household factors, as well as school and teaching characteristics, are associated with significant improvements in learning outcomes. Findings indicate that school factors are important for student achievement. Good pedagogical practices and school security play a significant role in student outcomes, controlling for student and family characteristics. About three-fourths of learning inequality between indigenous and non-indigenous students can be explained by different levels of observed factors. Inaddition to socioeconomic background, school resources and teacher experience help explain the gap. Most notable, active pedagogical instruction i s an effective way of increasing student achievement and points to the need for teachers to expose students to behaviors such as problem-solving and critical thinkingfor basic education quality to improve. Analysis of PISA 2000 and 2003. Using econometric research methods (quantile regression analysis and taking into account multi-level influences), the study analyzes the impact of factors related to institutions, schools, students, parents and teachers on student outcomes. The analyses reaffirm what i s known about the impacts of socioeconomic status on learning and the limited role of physical investments. More importantly, they shed new light on the importance of school climate, expectations, participation, autonomy, accountability and the need to use assessments to inform policy decisions. Telesecundarias. Of particular interest among the PISA 2003 results are those for telesecundarias. A Mexican educational innovation, telesecundaria is a public, formal- education service that delivers televised lessons to students in rural areas. It was designed to specifically reach and meet the needs o f the most disadvantaged students. Early results were positive and the model has grown within Mexico to represent more than 20 percent of enrollments and i s the fasting growing segment o f secondary education in Mexico. However, using the PISA 2003 results, it has been documented that telesecundarias perform worse than almost all other types of secondary schools after controlling for other factors. It i s necessary to investigate the reasons for the particularly poor performance of telesecundarias. V An Agenda for Action Ingeneral more Autonomy,Accountability andAssessment -the three A's - will help improve the quality of education, through improvements in learning outcomes. This will also allow other policy actions to be more effective. That is, more autonomous schools can implement appropriate language policy, thus fulfilling national guidelines for indigenous students. A more accountable system will in turn encourage more active participation by parents and others, which i s instrumental for improving learning outcomes. Finally, a system that i s based on constant assessment and participation in international benchmarking exercises will improve cost- effectiveness. Increase School Autonomy at Public Schools. Results from the analysis of national and international assessments show that teacher morale, teacher-student relations, teacher behavior, teacher expectations of student performance, student's awareness o f value of school for future earnings and active learningheaching styles are all associated with better learning outcomes. To improve education quality, Mexico needs to continue efforts to move decision-making from the state level education secretariats to the local school level, thus increasing school autonomy. Increased autonomy may give schools the flexibility they need to empower teachers, thus improving the school climate as well as the relationship between students and teachers. Analysis of PISA 2003 suggest that both students and schools perform best in a climate characterized by high expectations that are supported through strong teacher-student relations, students who are ready to invest effort and who show interest and lower levels of anxiety with mathematics and a positive disciplinary climate. In most of the countries that performed well on the PISA, local authorities and schools have substantial responsibility for educational content and/or the use of resources, and many set out to teach heterogeneous groups of learners. In a simulation using PISA 2003, it is shown that changes and improvements in the school climate and current schooling practices increase the overall score of the bottom achievers by about 20 percent. There i s wide variation of results by state, socioeconomic status, indigenous groups and types of schools, highlighting the need to move decision-making to the school level. For example, although there i s some variation, overall private schools perform better than do public schools. This may be in part attributed to the high level of autonomy over school resources and educational content that private schools enjoy. In addition, there i s a differentiated effect of technology and school inputs. For instance, while computers have a positive effect on math scores among high achievers, calculators have a larger effect on math scores among low- achievers than do computers. These examples illustrate that there are many unobserved factors that contribute to learning outcomes and it i s difficult for a centralized authority to determine appropriate school interventions. Increasing autonomy would allow schools to determine locally appropriate policies, particularly in the case o f schools that serve indigenous students and other disadvantaged schools. Two current government programs designed to increase school autonomy are improving school climate and showing signs of positive impacts on learning outcomes. A compensatory education program implemented by CONAFE builds an environment in which a high level of satisfaction and learning at schools exists, and there i s some evidence that the compensatory program has v i improved equity inoutcomes. Evaluations show that the program reduced learning gaps between similarly disadvantaged groups. The Quality Schools Program (PEC) also aims to increase parental participation and the empowerment of the school community. The findings from on- going evaluations of these two programs should shed light on the effectiveness of the programs as well as the impact of increased autonomy on learning outcomes. Another example of a program that increases autonomy and learningoutcomes in schools i s Colombia's Escuela Nueva program, which may be particularly relevant for indigenous schools because it gives them a high degree of autonomy in order to adapt the program for local needs. Government policies and programs, as well as general social demand, have lead to significant increases in enrollments. However, many of the new students entering the system are not prepared and are entering poorly performing schools. There i s a need to acknowledge this problem and responsibility must be assigned to ensure that in addition to demand-side interventions (Oportunidades), the learning needs of such children are catered to through, for example, compensatory programs. Improve Accountability. Time spent on homework and student interest in the subject matter are both positively associated with learning outcomes, whereas memorization as a way of learning negatively impacts achievement. The effects of computers and lab equipment have mixed effects, with computers benefiting mostly high achievers as well as overall only improving reading scores and lab equipment only improving science scores. While these factors are not necessarily amenable to changes inpolicy at the federal level, they would likely be influenced b y stronger accountability mechanisms at the school and community level. L schools and their communities were heldresponsible for results, they would be more apt to make sure that students are responsible for doing their homework, that teachers are more enthusiastic about subject matters and using active pedagogic methods, and that administrators are acquiring the appropriate mix o f technology and equipment for their schools. The poor overall results in Mexico highlight the need to assign responsibility for improved learning outcomes. In order to improve learning outcomes it is necessary that school empowerment be accompanied by a strengthened accountability framework that enhances parental participation. Accountability can improve school quality by helping set clear goals and visions for the school system. This i s particularly true for indigenous schools, for which decisions regarding curriculum are made at the federal level with little accountability. If accountability was delegated to the school level, with community participation, school quality could improve. Accountability mechanisms that put people at the center of service provision can go a long way in making services work and improving outcomes by facilitating participation in service delivery. More specifically, focusing on people enables them to monitor and discipline service providers and amplifies their voice in policymaking, and strengthens the incentives for providers to serve them. There may be a need to look for strategies to strengthen the short-route to accountability, the direct influence of beneficiaries on service providers. Increased parental participation i s just one example of a mechanism that may increase the short route to accountability. vii It is important to note that while accountability is important, it must go hand in hand with increased autonomy. Holding schools and communities accountable for improved results necessitates devolution of decision-making to the school level so that they can initiate and carry out appropriate policies. In addition, the system needs to be constantly assessed so that schools can benchmark themselves and modify policies as needed. Continue Learningfrom Assessments. If increased autonomy and improved accountability are both to lead to policies for achieving higher learning outcomes, then national and international learning assessments are needed. In order to improve learning outcomes, countries must first have the capacity to measure levels of achievement. First and foremost, assessment testing can be used to inform policy decisions. As other OECD countries have shown, PISA results can be usedtojustify education reform initiatives. PISA provides an excellent opportunity for countries to evaluate their education systems, establish benchmarks for future tracking and inform policy responses. Analysis of assessments can foster public and civil society involvement in education reform. A review of the published media in PISA participating countries suggests that the assessment results had an impact on educational policy making. Media coverage was extensive, especially in countries where the public was not happy with the results, but also in countries where the results were much better. However, governments must be proactive in encouraging public debate. Without proper use of the results, media coverage, public debate, sector review and policy reform, it i s unlikely that outcomes will improve over time. But countries that acknowledge the results, engage in public debate, conduct serious analysis of the results, and launch programs to improve outcomes, may see improvements over time. Mexico has made remarkable efforts to improve assessment of the education system, and i s encouraged to continue participating in international tests, as well as improve and expand the national system. The assessment results need to be analyzed continuously and the results used to inform policy decisions. More information flowing to policy makers, the general public, communities, parents and schools will contribute to improving overall quality and learning outcomes. It i s recommended that Mexico undertake constant analysis of assessments in order to inform policymakers and guide reforms. The expanded PISA 2003 sample in Mexico, which is representative at the state level and by school type, i s an excellent tool for deeper analysis. This information can help decision makers design appropriate policies. States, too, can conduct state-specific analysis for future tailoring of local education policies. National and international assessments could be used to inform the secondary school reform process currently in progress. Mexico's net secondary enrollment rate ranks among the lowest of upper-middle income Latin American countries and falls below the average for Latin America (62 percent compared to 65 percent). The government needs to address the challenge of improving access to upper secondary, while improving its quality and relevance. Expansion of coverage will help improve quality - or the yield - as more people complete their education in an improving system. Secondary education, however, presents specific structural problems that point to the need for profound reforms of the curriculum and organization of secondary ... Vlll education, and assessments can be used to identify strengths and weaknesses of the system. Moreover, given the poor results of certain types of secondary schools - particularly telesecundurias, but also all types of lower secondary - there i s a need to investigate the reasons for poor performance. Stronger efforts to increase capacity in Mexico to analyze learning outcomes and evaluate programs are needed. Not only should Mexico continue its important task of disseminating and reporting outcome data, but more rigorous analysis of the relationships and determinants could be undertaken. Mexico has many important programs that need to be evaluated on a continuous basis. To do this, it i s recommended that permanent capacity within SEP be built for analytical work, including impact evaluations on a periodic basis. i x FOREWORD 1. The World Bank i s undertaking a multi-year program of analytical and advisory services in the area of quality of education to support the Government of Mexico in improving its programs and policies. The program seeks to: (a) provide sound policy advice and analytical work on pertinent topics in response to client demands, drawing on international experience and; (b) contribute to the evaluation of important school quality programs. The program was developed in collaboration with Mexico's Secretariat of Public Education (SEP), and after consultations with officials involved inthe implementation of programs. 2. The study is focused on the following questions, developed incollaboration with SEP: 0 What are the determinants of learning? 0 How can international achievement results and school rankings be used to guide education policy? 0 How can difficult to reach populations be incorporatedin quality education? 0 What are the lessons learned from key compensatory programs? 3. The program adopts a variety of instruments to respond to the Government's need to: (a) improve the quality of schooling; (b) set domestic and international benchmarks on key indicators; (c) evaluate the effectiveness of public programs designed to improve school quality; and (d) design improvements to current policies and programs. These instruments include: (a) research and literature reviews; (b) a set of policy briefs; (c) traditional analysis; and (d) seminars. 4. The work i s carried out in three phases. Phase 1 (2004-2005) focuses largely on the determinants of learning in Mexico. Phase 2 (2005-2006) will focus on program impacts, with emphasis on: (a) incorporating difficult to reach populations; and (b) compensatory and targeted programs. Phase 3 (2006-2007) will focus on: (a) finalizing and disseminating the research on learning outcomes and policy briefs; (b) generating new analytical work on program impacts, equity issues and secondary education; (c) providing technical assistance on areas to be determined with SEP and the AAA Committee; (d) consolidating the full set of outputs into a comprehensive review of education quality issues in Mexico (with much broader dissemination); and (e) preparingpolicy notes on education quality for the new administration. 5. The present report provides new analytical work on the determinants of learning using data from the OECD's Programme for International Student Assessment (PISA) 2000 and 2003 and background papers that were previously produced using Mexico's national assessment, as well as new work on the returns to quality and the impacts of bilingual education, and an assessment of media coverage on PISA. It also builds on previous work including Mexico: Transforming Schools into EfSective and Eficient Learning Centers (World Bank 2000a), which recommended that in order to improve learning outcomes it i s necessary that the education systemhas a clear vision, empowers the school, strengthens teacher support, improves incentives and accountability and enhances social/parental participation. The main objective of this report i s to identify the determinants of learning in Mexico, focusing on 1 equity, distributional issues, cross-regional issues, cross-country comparisons and national studies. However, it i s important to first present an overview and key background information on Mexico's education system. 6. It is well known that family background and socioeconomic status play a major role in determining school outcomes. Yet there i s evidence that some factors associated with school quality are amenable to policy change. School and other institutional factors can help reduce learning inequalities, as i s evidenced in cross-national, international andnational research (see, for example, Woessmann 2003; Hanushek and Luque 2003; Hanushek and Kimko 2000). This study builds on this rich knowledge base, updates the research on education quality, and contributes to a better understanding of the institutional arrangements that can improve learning outcomes. 7. Usingeconometric researchmethods, the study analyzes the impact of factors related to institutions, schools, students, parents and teachers on student outcomes. PISA 2000 and 2003 data i s used to conduct a comprehensive national analysis combined with a comparative international analysis. The analysis examines the distribution of scores across quantiles and levels of comprehension and compares it to international averages. The study analyzes the evidence and develops policy options for ensuring that Mexico i s able to improve learning outcomes. This study coincides with government efforts to reform and improve the education system and comes at a time of substantial interest ineducation quality. Structureof this Report 8. Chapter 1reviews the existing international and Mexican literature on education quality and the determinants of learning. Chapter 2 presents an overview of the state of education in Mexico. Chapter 3 discusses the analysis of the National Assessments (Esta`ndures Nucionales) from 1998 to 2000 and the results of the analysis of PISA 2000 and PISA 2003 data. Finally, Chapter 4 concludes the main report with an agenda for action, which responds to the education situation inMexico. The MainMessages of the Report 9. Education continues to be a good investment for increasing productivity. Mexico has made great strides in expanding its education system over the past decades, as reflected in the educational attainment of the population. The rates of return to education remain high. However, while educational coverage has increased over time, the quality of education remains low and secondary and tertiary enrollment rates are lower than expected given Mexico's income. Mexico underperforms in terms of quality and i s a long way from reaching OECD standards. 10. Analyses using Mexico's National Assessments find that a number of individual and household factors, as well as school and teaching characteristics, are associated with significant improvements in learning outcomes. Findings indicate that school factors are important for student achievement. Good pedagogical practices and school security play a significant role in student outcomes, controlling for student and family characteristics. About three-fourths of learning inequality between indigenous and non-indigenous students can be explained by 2 different levels o f observed factors. In addition to socioeconomic factors (including mother's education), school resources and teacher experience contribute greatly to the gap. 11. Supply-side interventions (for example, SEP's compensatory programs) have also helped reduce learning gaps in Mexico, especially among the most disadvantaged. However, given the fact that schooling opportunities have been extended to the poorest and most disadvantaged segments of society, much more needs to be done on the demand-side. Yet, while the significant gains of demand-side interventions (PROGRESA, now Oportunidudes) are well known, efforts could continue and be expanded to higher levels of schooling and to urban areas. 12. Analysis using the PISA 2000 data reveals that instructional practices and quality of teachers, and whether students are oriented to be effective learners, are important determinants of achievement. The analysis also reveals that school climate i s of measurable importance and has been found to play a significant role in the performance of both low and high achievers. The implication from these findings i s that Mexico needs to focus on improving the learning environment to achieve better learning outcomes. 13. The recently released PISA 2003 results reconfirm the poor performance of Mexican schools. Infact, there has been some slippage in test scores since 2000. Some of this i s due to increasing enrollments: there was a 5 percentage point increase in school participation by 15- year-olds from 2000 to 2003. Yet, test scores declined by 0.5 percentage points in math and about 5 percentage points in science and reading. This underscores the need to focus on improving school outcomes, while at the same time increasing the relatively low secondary school enrollment rate. 14. Mexico needs to increase secondary school enrollment rates. The country continues to do well at maintaining equity in terms of the impact of family background on scores and the very low dispersion between top and bottom achievers, which i s unusual for a Latin American country participating in international achievement tests. To improve quality, Mexican schools need to improve the school climate and continue efforts to move decision-making from the state level education secretariats to the school level, thus increasing school autonomy. Also, accountability needs to be further strengthened b y involving parents and the community more and setting clear goals and clear vision for the school system. Finally, Mexico's remarkable efforts to improve assessment of the system should continue, with continued participation in international achievement tests, as well as improvements of the national assessment system. The assessment results need to be analyzed continuously and used to inform policy decisions. The findings of this report point to the need for Mexico to address some of these policy- amenable factors that are linkedto school achievement in order to improve results and be able to compete globally. 3 CHAPTER 1. WHAT DOWE KNOWFROMTHE INTERNATIONALAND MEXICANLITERATURE? 1.1 This chapter reviews the existing literature on education quality and determinants of learning. Measuring the impact of educational policies i s hindered by the econometric problems inherent in conventional estimates of educational production functions and the complexity of the education process, which includes the wide variation in schools, teachers and students across countries. Due to variations in similar policies across countries, it i s difficult to make generalizations (Glewwe 2002). For example, while some studies reviewed below establish causality, many are associations. However, the abundance of these associations provide convincing evidence for the factors that impact student learning achievement, as in many cases we rely on a preponderance of evidence. (Most of the studies cited are listed and described inAppendix l a and lb.) InternationalEvidence 1.2 In 1966, the Coleman Report (Coleman 1966) first documented the important role of family background on student achievement. Research on the differential effect of education inputs on student achievement has grown extensively, especially since Hanushek's (1986) seminal work on education production functions. Early literature on learning achievement concluded that the socioeconomic characteristics of children are the dominant determinants of student academic performance and that differences in school quality have little influence on educational success. More recent research, however, emphasizes that the influence of institutionalfactors and school characteristics can be substantial (Woessmann 2003). 1.3 Changes in school environment have a positive correlation with improved school quality and, subsequently, improved education outcomes. School effectiveness studies statistically estimate the effects o f socioeconomic and school-related factors on student learning achievement tests. Heneveld and Craig (1996), Patrinos and Psacharopoulos (1995) and Lockheed and Verspoor (1991) present several factors that are important for the development of effective schools in developing countries. Necessary basic inputs include: (i) instructional materials such as textbooks, supplementary teachers' guides and materials and library books; (ii) a curriculum with appropriate scope and sequence and content related to pupil experience; (iii)time for learning (the number and length of school days); and (iv) appropriate teaching practices (such as active student learning, including discussion and group work). Facilitating conditions include: (i)community involvement and support, which includes both good school-community relations and parental involvement in the school; (ii) school-based professionalism, including leadership by the school head, teacher collegiality and commitment and accountability through assessment, supervision, and support; (iii) flexibility relevant to pupil curricula and adjustments in level and pace and organizational flexibility to include school clusters and active teaching; (iv) pedagogical flexibility to allow for teaching innovations; and (v) implementing decentralized, school-based solutions to problems. 1.4 Ina study of accountability mechanisms in the United States, Hanushek (2004a) found that accountability systems raise levels of student achievement. However, impact is minimal 4 when schools are just requiredto report scores. Tying incentives or disciplinary consequences to school performance has been shown to have a greater impact. 1.5 Significant research (see, for example, Greenberg 2004) highlights the value of school climate on achievement. On the relationship between school climate and performance, Greenberg (2004) uses the United States' National Assessment of Educational Progress (NAEP) 2000 to show that students in schools with the highest student behavior values had higher mean mathematics scores than students in schools in the middle or at the bottom of the student behavior distribution. Similar relationships existed between parental involvement and mathematics achievement and between school morale and mathematics achievement. 1.6 Researchers have begunto use international assessments to analyze the determinants of learning. Hanushek and Luque (2003) indicate that focus on the quality of human capital in different countries naturally leads to concerns about how school policies relate to student performance. Using the Trends in International Mathematics and Science Study (TIMSS), the results of their analyses of the educational production functions within a range of developed and developing countries show general problems with the efficiency of resource usage. These effects did not appear to be dictated by variations related to income level of the country or level of resources in the schools. 1.7 Educational quality has a consistent, stable and strong influence on economic growth. Using the TIMSS data at the aggregate cross-country level, Hanushek and Kimko (2000) analyze the quality of the labor force as measured by comparative tests of mathematics and scientific skills. The estimated impact of educational quality on growth indicated that a one standard deviation improvement in mathematics and science skills translates into more than a one percentage point increase in average annual real growth. The growth model results, however, implied that the externalities must be significantly stronger for quality than for quantity. The estimated growth effect of one standard deviation of quality i s larger than would be obtained from over nine years in average schooling. They concluded that labor force quality differences are important for growth; that these quality differences are related to schooling (but not necessarily the resources devoted b y a country to schooling); and that quality of schooling has a causal impact on growth. 1.8 In a recent article, Barro (2001), also using TIMSS data, looks at the impact of education on growth. In his analysis, he distinguishes between quantity of education (measured by years of school attainment) and quality of education (measured by scores on internationally comparable examinations). Barro (2001) finds that the quantity of schooling, measured as the school attainment of males at secondary and higher levels, has a positive and statistically significant relationship with growth. On the quality of education, Barro (2001) finds that science scores have a statistically significant positive effect on growth. The implication i s that a one-standard-deviation increase in scores i s associated with a growth rate of 1percent a year. B y contrast, a one-standard-deviation increase in school attainment would increase the growth rate by 0.2 percent a year. Thus, he concludes that quality and quantity of education matter, but quality matters much more. On what level of education matters, Wolff and Gittleman (1993) report that while the data seem to be consistent with the proposition that increases in higher education increase growth rates in high-income countries, increases in secondary education are more important for middle-income countries. 5 1.9 Family inputs and school resources are closely related to school outcomes. Lee and Barro (2001), again using TIMSS, investigated the determinants of educational quality in a panel data set that includes output and input measures for a broad number of countries. The results show that family inputs and school resources are closely related to school outcomes, as measured by internationally comparable test scores, repetition rates and drop-out rates. Family characteristics (income and education of parents) have strong effects on student performance. The findings also indicate that more school resources - especially smaller class sizes - may enhance educational outcomes. However, Hanushek (2004b) emphasizes that existing research on the relationship between school resources and improved educational achievement i s inconclusive and that outcomes may in fact depend on interactions between resources, teacher quality and other inputs, making i t difficult to identify best practices. 1.10 A detailed econometric analysis of the association between reading test scores and individual and family background information, and with characteristics o f the school and class of the 15-year-old respondents to the survey i s provided in Fertig and Schmidt (2002), based on PISA 2000 for Germany. They show that, overall, family background and school characteristics play a more important role for success inPISA 2000 than previously recognized in the debate. Furthermore, from a policy perspective the results indicate that countries directly improve the performance of their school system by investing in tangible aspects of the system. Inparticular, school conditions including teacher characteristics account for a sizeable fraction of student's individual success in PISA 2000. Moreover, it seems to be that the students in the bottom of the performance distribution are those who suffer most if their education environment i s sub-standard. 1.11 In one of the few if not the only study of a developing country, Abdul-Hamid (2003) investigated the factors that affected student performance in Jordan using TIMSS 1999 data. He found evidence of the positive impact of the home, family and demographics in determining student achievement. Parent's education, especially for those who finished university, has played a significant role in achievement. Parents making education materials available in the home has also been found to be correlated with achievement. It has also been noted that school governance and demographic factors play an important role in determining achievement. These factors matter not only for achievement but also for exposure to certain teaching methods, such as problem solving and critical thinking to the advantage of private and urban schools. 1.12 Differences in educational institutions accountfor the large international differences in student achievement. In an important paper, Woessmann (2003), using TIMSS data, suggests that differences in educational institutions explain the large international differences in student performance in cognitive achievement tests. An econometric student-level estimation basedon data for more than 260,000 students from 39 countries reveals that positive effects on student performance stem from centralized examinations and control mechanisms, school autonomy in personnel and process decisions, competition from private educational institutions, scrutiny of achievement and teacher influence on teaching methods. A large influence of teacher unions on curriculum scope has negative effects on student Performance. The findings imply that international differences in student performance are not caused by differences in schooling resources but are mainly due to differences in educational institutions. 6 Taking all countries into consideration, he finds that the following factors positively impact science and mathematics learning: Central examinations Centralized control of curriculum and budget matters School autonomy inprocess and personnel Teaching methodology Limited influence of unions Scrutiny of student performance Parental interest Intermediatelevel of administration Competition from private sector 1.13 Ina more recent study, Fuchs and Woessmann (2004a) found that school autonomy greatly improves student achievement in school systems that have central exit exams. They also found that institutional factors explain one quarter o f the variation in test scores between countries. Woessmann (2004) confirmed the role of central exams and school autonomy in raising levels of student achievement. Usingdata from TIMSS 1995 and 1999 and PISA 2000, he found that students in schools with central exams and autonomy over teacher salaries and course content, as well as schools that allowed for teacher influence over resource funding, outperformed students from schools with no autonomy and no central exams. They also scored higher than students from schools with no autonomy but with central exams. 1.14 Evidence from Finland's experience with PISA lends itself to highlighting the importance of school autonomy in terms of flexible curriculum options. Among OECD countries, Finland earned the highest scores in reading literacy. In addition, Finland scored relatively high in terms of equality. Student's engagement and interest in reading together explain 40 percent of the variance in reading scores. The authors of one report attribute Finland's success to flexible school curriculum and offering of optional subjects (Valijarvi and others 2002). In 2003, Finland repeated its success in PISA, ranking highest among OECD countries in terms of math scores, and coming in second among all participant countries, behind Hong Kong. Finland i s tied with Japan for first place among all countries for reading scores. 1.15 One can conclude from the discussion above that while family background and socioeconomic status play an important role in determining school outcomes, institutional and school factors also play a significant role in reducing learning inequalities and improving learning outcomes as evidenced in cross-national, international and national research. Specifically, changes in school environment have a positive correlation with improved school quality and improved education outcomes, which have a consistent, stable and strong influence on economic growth. Increased school autonomy, combined with central exams and school accountability mechanisms, also help to raise achievement levels. Evidence from Mexico 1.16 Available empirical evidence systematically reveals low levels of education achievement. Over the years, there has been an accumulation of data from tests given to 7 samples of students at all educational levels, especially in primary schools. Unfortunately, these tests are generally not comparable over time, and there has been little effort to analyze the results and, until recently, very little information was disseminated. Today researchers have access to standardized tests that have been conducted by the General Directorate for Evaluation (DEG) in Mexico's Secretariat of Public Education (SEP) since 1998. Since then significant amounts of data have been collected and technical capacity for measuring learning achievement has been developed. In 2005, a new external evaluation unit, the National Institute for Education Evaluation (NEE), will administer its first competency based learning and assessment system to measure learning outcomes and assess the quality of education. 1.17 School effectiveness research i s relatively recent in Mexico. Only a few studies have been conducted in Mexico to statistically quantify the effects of input, process and organizational variables on student learning achievement. Data from four schools in Leon, Guanajuato i s used to test a production function for primary school education (Anderson 2000), finding that teaching hours and increased student-teacher interaction, coupled with improved facilities and libraries are correlated with improved reading scores. Ontiveros (1998) uses data from the Primary Education Project (Programa para Abatir el Rezago Educativo, PARE), carried out in four states (Oaxaca, Guerrero, Chiapas and Hidalgo), using a quasi- experimental design, to test a production function. He finds that expenditure devoted to improve school facilities and to increasing the average number of textbooks per student has a positive effect on student performance. Increasing teacher salary or hiringteachers with higher education and more experience does not improve student achievement. Indeed, preliminary empirical evidence from the Teacher Career Program (Carrera Magisterial), a teacher career scheme in which individual teacher salary increases are tied to student performance as well as to other measures of teacher quality, using regression discontinuities to create a control group, shows no robust evidence that Mexican teachers faced with stronger incentives actually improved student achievement in the year in which they were assessed (McEwan and Santibaiiez 2004). 1.18 Inequality in educational achievement. A consistent finding in evaluations of student learning i s the difference in results among students according to school characteristics and the socioeconomic and cultural backgrounds of their families (Schmelkes 1997, 2000). Usingdata from Mexico's National Assessments (Esthndares Nacionales), Fernandez (2003) found that a wide range of family and individual characteristics are correlated with student achievement in mathematics and reading. Factors that had a positive impact include: attending pre-school, aspirations to achieve higher level education, the interaction between family capital and educational aspirations, and being female (for reading scores). Negative factors included: being female (for math scores), the interaction between being female and working, and repeating a grade in primary school. Fernandez (2003) also found evidence of the impact of school characteristics on student achievement. Schools located in localities that ranked highon the marginality index, were located in an indigenous area, had a supervisor who monitored teachers' actions daily and large schools all had negative impacts on student scores in mathematics and reading. Private schools, increased attention given to students, high institutional expectations of student achievement and schools where supervisors carried out regular evaluations of teachers hadpositive impacts on student achievement. 8 1.19 Reducing learning gaps. Holding socioeconomic characteristics constant, Schmelkes' (1997) case study found that there were significant differences in learning outcomes between the best and the worst schools within regions, but not always across regions. For example, for the fourth grade, there were no significant differences between the best schools in a disadvantaged urban region, an indigenous region and an urban middle-class region. This finding suggests that there are school improvements that could lead to reduction, if not elimination, of the barriers to greater equity in basic learning achievement. The existence of differences within regions indicates the importance of school management and of the dynamics and culture of the school. 1.20 Moreover, evidence from Mexico's compensatory programs (Box 1.1) shows that learning achievement of students in rural and indigenous schools can be raised substantially through interventions designed to improve the quality of teachers, principals and supervisors, and through the availability and proliferation of learning materials, according to two studies that use quasi-experimental designs and propensity score matching (Paqueo and Lopez- Acevedo 2003; Shapiro and Moreno 2004). PARE was effective in raising Spanish test scores inrural and indigenous schools (Paqueo and Lopez-Acevedo 2003). Student scores increased by 95 percent in indigenous treatment schools compared to only 17 percent in indigenous control schools, thus eliminating the gap in scores between the two groups. The gains were less, though still sizeable in rural schools: by 56 percent in treatment schools and 40 percent in control schools. The program appears to have negatively impacted scores in urban schools, though the authors surmise that it could have been due to poor implementation of program components. Factors that help explain the change in test scores include: school attendance, parent participation and the performance of teachers and principles. They found that school and educational system factors had large and significant impacts on student tests scores, especially in indigenous schools. After controlling for individual, family, community and school characteristics, it was found that, if implemented correctly, the PARE program could cause scores for the average rural students to increase by 19 to 38 percent, and by 45 to 90 percent for the average indigenous student. Demand-side educational grants have also been found to reduce the gap between poor and non-poor students, with the greatest impacts exhibited by students who face greater barriers to education, such as having uneducated parents and living far from school (Raymond and Sadoulet 2003). A review of compensatory programs in four Latin American countries, including Mexico, found that classroom libraries and distribution of textbooks and food are positively correlated with student learning, in both poor and non-poor neighborhoods (Anderson 2002). 9 Box 1.1CompensatoryEducation As early as 1971, Mexico began to address the challenge of including its most disadvantaged children by creating the National Council for Educational Development (CONAFE). In the 1990s the Secretariat of Public Education (SEP) created the compensatory education programs (CPs) seeking to support the most disadvantaged schools and nearly all indigenous schools. CONAFE implements the CPs, which support more than three million students in pre-primary and primary education, including about one million indigenous primary school students. CONAFE selects schools for support based on the average income of the school's community, the school's isolation and access to public infrastructure, the school's education indicators and other indicators of poverty. For schools that enroll indigenous students, CONAFE supports development of curricula, didactic materials and textbooks in an indigenous language and Spanish to facilitate bilingual education. I n most beneficiary schools, a group of community parents and leaders receive a grant that can be spent on the educational purpose selected by the group. If compensatory programs are well designed and properly targeted, compensatory programs can significantly reduce the "advantage" gap. A recent evaluation of the impact of SEP's compensatory programs implementedby CONAFE finds that these programs are effective in improving primary school math learning and secondary school reading. In addition, the program has resulted in lower repetition and failure rates. The communities in which supported schools are located have significantly lower levels of literacy, access to public services and industrial development than do the communities of non- CONAFE schools. Yet, an evaluation found that indigenous students supported by the CONAFE schools were catching up to their non-indigenous peers in test scores by about 10percent per year. Further analysis shows that the compensatory education program also contributes to equity in test scores. That is, there is less dispersion in test scores within CONAFE-supported compensatory schools, in both urban and rural areas. This appears to be larger in reading than in mathematics. Therefore, in addition to reducing the overall test score gap between supported and non-supported schools, compensatory education seems to contribute to enhanced equity as well. Source: Shapiro and Moreno 2004 1.21 Telesecundariaeducation and bilingual education for indigenous students are designed for difficult-to-reach populations. Telesecundaria i s a public, formal-education service that delivers televised lessons to students in primarily rural areas. It was designed to specifically reach and meet the needs of the most poor and marginalized students. Since 1968, when it was implemented, it has grown from 6,500 students to 1.2 million students by 2003. As of the same year, telesecundaria enrollment represented 1/5th of total secondary school enrollment in the country. Additionally, enrollment rates for telesecundaria have grown faster than enrollment rates for traditional and technical secondary schools (Shapiro and Moreno 2004; Torres and Tenti 2000). Early evidence showed that telesecundaria schools are effective in increasing math and reading test scores but more recently there is concern that such schools are not achieving their potential (Figure 1.1). 10 Figure 1.1 Resultsby School Type, PISA 2003 I1500 1 450 I 400 350 300 250 200 0Math , 150 100 Science Source: PISA 2003 1.22 In summary, evidence from Mexico shows that, as elsewhere in the world, socio- economic background and parental schooling play an important role in levels of student achievement. However, recent research has emphasized that school factors also explain variance inoutcomes: weak curriculum implementation and an over emphasis on memorization and rote learning contribute to low results; inadequate supervision has a negative effect; the school environment i s important for improving outcomes; resources dedicated to improving infrastructure and raising the textbook-student ratio can raise levels of student achievement; and time on task has a significant impact. Flexibility o f curriculum and school format can also be useful in extending schooling to hard-to-reach populations and in eliminating education gaps. Evidence i s mixed on the effects of teacher training and salary incentives. A few studies have also shown the importance of institutional factors such as school management. Conclusion 1.23 A review of the international and national quality of schooling and determinants of learning literature provides evidence of a need for increased autonomy at the local school level, improved accountability, and use of assessments for informing policy decisions. For example, international evidence shows that increased school autonomy, combined with central exams and school accountability mechanisms, help raise achievement levels. Other important school and institutional factors include: strong accountability mechanisms, parental and social participation and use of assessment results. National evidence finds that in order to raise schooling outcomes, devolution of decision-making to the school level i s crucial. Flexibility of curriculum and school format i s just one example of how school autonomy has been useful in 11 extending schooling to hard-to-reach populations and in eliminating education gaps. However, assessments need to be usedto inform policy decisions and to determine whether or not current policies are effective. 12 CHAPTER 2. THE STATE OFEDUCATIONINMEXICO TODAY 2.1 Mexico ranks as the ninth largest economy in the world and the largest in Latin America. However, Mexico's education indicators are significantly poorer, especially when compared with Mexico's main trading partners. The average educational attainment of the Mexican population aged 15 and over i s a disappointing 7.2 years, as compared with 7.6 in Chile, Uruguay and Peru; 8.8 in Argentina; and 10to 12 years for other, more advanced OECD countries. About 52 percent of the adult population in Mexico lacks the minimumbasic skills and knowledge required for Mexico to remain a competitive economy (World Bank 2003). Benefits of Schooling 2.2 There i s a shift in labor demand in Mexico towards advanced skills. Until recently the returns to schooling were increasing, especially at the tertiary level. The increasing demand for skilled workers i s out-pacing supply (Lopez-Acevedo 2001). Education i s a major determinant of earnings and lack of it or low quality education contributes greatly to earnings inequality. There was a significant decline in the economic rate of return to investments in schooling from the mid-1990s onwards, with a slight increase again in 2002 (Patrinos and Metzger 2004). Overall, returns to schooling in Mexico tend to fluctuate, but have remained remarkably high over time as compared to other middle income countries (Psacharopoulos and Patrinos 2004). Figure 2.1 presents estimates of returns to schooling between 1984 and 2002, though the point estimates are not strictly comparable. Figure 2.1 Mexico: Average Returns to Schooling over Time 16 - q % :": 14 1 8 2 0 1 I , I , , 1 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 Source: Patrinos and Metzger 2004 2.3 It i s important to note that returns to the average individual may not be usefulfor policy makers, especially if one is interested in looking at the impact of education on the margin, or for individuals with fewer unobserved skills and abilities. Those at the bottom of the wage distribution are liable to have not only little education but also a limited endowment of unobservable skills. Thus, it is interesting to ask whether the effects of education are 13 independent of these unobservable skills or whether it compensates for them or complements them. If the effect i s independent of unobservable skills, then we should find the effect of education i s the same throughout the wage distribution. On the other hand, if education compensates for low skill levels, then we should find a larger effect at the bottom of the wage distribution than at the top; or a larger effect at the top of the wage distribution if education complements the unobservable skills. In most other countries, higher returns for higher abilities have been observed (Buchinsky 1998; Mwabu and Schultz 1996; Patrinos and Sakellariou 2004; Walker and Zhu 2001). In Mexico, however, education appears to compensate for lack of skills (Figure 2.2); that is, returns are highest for people at the bottom of the wage distribution (Patrinos and Metzger 2004; Zamudio 2001). Evidence indicates, therefore, that education in Mexico i s a good investment for increasing productivity, particularly for those with lower skill levels of skill, and further investments in education will lead to reductions in inequality. Figure2.2 Mexico Returnsto Schoolingby Income Quantile, 2002 12 ` 0 ~ 8 % 6 ` 4 2 I O 10th Bottom 25th 50th 75th 90th Top Percentiles I Source: Patrinos and Metzger 2004 2.4 Benefits to education go beyond economic growth. The potential effects of schooling that are not reflectedin estimates of market returns are extensive, and involve both non-market effects that are private (in the sense of being captured by individuals) and social effects involving the public goods or "spillover'' effects of schooling. These effects may be large, and under certain assumptions may be as large as the market-based effects of education. A review of data from developing countries shows that there are many social effects of education: including such relationships as a likely positive link between one's own schooling and the schooling received by one's children; a positive association between schooling and health status; a positive relationship between education and the efficiency of consumer choices; a relationship between schooling and fertility choices; and a relationship between schooling in one's neighborhood and youth decisions regarding education, non-marital childbearinge and participation incriminal activities (Wolfe and Haveman 2001). 14 2.5 Countries face significant challenges in their effort to enable young people to become active citizens and to prepare youth to enter constantly changing workplace environments. Education forms the foundation of democratic societies and plays a critical role in preparing children and youth to meet these challenges by facilitating learners' creative and emotional development; in supporting objectives of peace, citizenship and security; in promoting equality and in sharing global and cultural values with future generations. Ultimately, education builds what Amartya Sen (1999) calls "human capabilities"-the essential and individual power to reflect, make choices, seek a voice in society and enjoy a better life. The school environment i s not limited to imparting academic learning. The school environment also provides a crucial forum for children and youth to gain life skills, including citizenship, participation ineconomic activity and the ability to make informed decisions throughout their lives. Education also has powerful synergistic effects on other development objectives, including protection of the environment, better health and good governance. It i s also one of the most effective preventive weapons against HIVIAIDS (Sen 1999). 2.6 A key question for consideration is this: do the positive externalities associated with primary schooling occur just because the child i s in school, or does it matter what kind of school the child i s in? Some have suggested that quality should not be a primary focus of educational development efforts since schools produce a "credentialing" effect rather than a "learning" effect. Working primarily in Nepal and Venezuela, Levine and others (2001) argue that, in fact, quality does matter. Looking at maternal literacy, his studies found schooling to be of benefit throughout adulthood, and good quality schooling was of greater benefit than lower quality schooling. Levine and others (2001) developed and tested a theoretical model to show how women's schooling contributes to social and demographic change. Their research proposes that schooling leads to social change by imparting skills and fostering other individual changes that alter women's patterns of social participation. They maintain that girls learn an "academic register" that i s the official language of all bureaucracies. The evidence supports the hypothesis that the literacy and language skills that women acquire in school provide the tools that can access better health care. The research also shows that lower infant mortality and maternal mortality are associated with the content of schooling, and interaction patterns in the classroom. Indeed there are many linkages between education and health. Better education also improves returns to investment in health. Gaining access to health services and programs often i s dependent on basic levels of literacy. In addition, training health professionals requires substantial investment in education. In particular, improved education for girls i s vital to improving health outcomes for future generations (Glewwe 1999). EducationinMexico 2.7 In the last decades, Mexico has made substantial progress in expanding access to primary and secondary education, especially in rural areas and for the poor. The lower secondary school completion rate, for example, has increased from 55 percent in 1994 to 67 percent in 2002. In rural areas, the net enrollment rate has increased fromjust 25 percent to 48 percent during the same period of time. Primary education completion rates are very high in all areas and practically universal overall. At the tertiary level, the proportion o f the population aged 15 and over with some university education has increased from only 6 percent in 1980 to 11 percent in 2000. Based on this evidence it i s clear that Mexico has made significant progress in terms of access to education (Figure 2.3). 15 P Figure2.3 Absolute Enrollmentby Level and Year (`000s) 18,000 16,0001 i Primary 14,000 - 12,000 - 10,000 - 8,000 - 4,000 2,000 1950 1960 1970 1980 1990 2000 I Source: SEP 2.8 Despite Mexico's notable achievements in expanding education coverage, many challenges remain (see, for example, INEE 2004a). While coverage at the primary level i s almost universal, investments in secondary education are lower than expected given Mexico's income level as compared with other Latin American countries (de Ferranti and others 2003). Incontrast with highenrollments for primary education, net enrollment insecondary education i s only 62 percent. This i s particularly low when compared with economies of similar size and with other Latin American countries such as Argentina and Chile, with net enrollment rates of 79 and 75 percent. Significant inequalities persist, mainly in remote or disadvantaged areas and in schools serving vulnerable groups (Lopez-Acevedo nd). This i s in fact the first challenge outlined inMexico's National EducationProgram (SEP 2001). 2.9 Another challenge i s to raise the quality of education. School quality has not kept pace with enrollment increases and increasingly more children, especially poor children and those living in rural areas, are being educated in low quality schools. The consequences of low quality schooling include grade repetition and low achievement. At the secondary and tertiary education levels, a key challenge i s to increase enrollment rates and improve quality in an equitable and sustainable manner. 2.10 Measuring relevant learning results, defined as basic competencies in communication, solution of mathematical problems related to daily life and preservation o f individual and collective health, Schmelkes (1997) reached the conclusion that primary education i s not impartingfunctional literacy to its graduates. 2.11 Indeed, the quality of education is low as measured by international achievement tests. In 1997, Mexico participated in UNESCO's Latin American Laboratory for Educational Quality Assessment (Laboratorio Latinoamericano de Evaluacidn de la Calidad de la Educacidn) (OREALC 1998). The survey compared indicators of educational achievement of 11 Latin American and Caribbean countries in third and fourth grade of primary school for mathematics and reading. Assessment of Latin American primary school students' learning achievement (third grade) shows that Mexico i s not lagging significantly behind other Latin 16 American countries, with the exception of Cuba. Mexico also participated in the Trends in International Mathematics and Science Study (TIMSS) in 1995, but the results were not made public and were not disseminated until very recently. Mexico re-administered the test in 2000 andfound improvements inthe results (SEP2001). However, available data show that student achievement in Latin American countries participating in TIMMS 1995 lags far behind Indonesia, not to mention Korea, Hong Kong and Singapore. 2.12 These poor results are further confirmed by Mexico's participation in the OECD's first Programme for International Student Assessment (PISA) in 2000. PISA provides a measure of reading, mathematics and science achievement for a nationally representative sample, comparable across countries. PISA's assessment focuses on young people's ability to apply their knowledge and skills to real-life problems and situations, rather than on how much curriculum-based knowledge they possess. Students from Mexico and other Latin American countries were among the worst performers. Overall, Mexican students, on average, perform as well as students from other Latin American (LAC) countries that participate in PISA. However, much improvement i s needed to reach the performance level of the other OECD participating countries. Furthermore, there has not been much improvement since the first PISA test in 2000. The results of PISA 2003 confirm these poor results. Mexico's performance in all three subjects declined, though this may be associated with the fact that enrollments increased during the same period of time by about 5 percentage points. 2.13 In comparison to other participating countries, Mexico outperformed only three countries: Indonesia, FYR Macedonia and Albania. Although students in high-income countries generally perform better than low and middle-income countries, wide variation remains. Mexico's overall performance inPISA 2000 was as follows: 0 Reading: 36th out of 43 countries and 1st among 5 L A C countries (followed by Argentina, Chile, Brazil and Peru) 0 Math: 37th out of 41 and 2nd in L A C (preceded by Argentina and followed by Chile, Brazil and Peru) 0 Science: 36th out of 41 and 1st inL A C (followed by Chile, Argentina, Brazil and Peru) 2.14 In PISA 2003, overall, scores on all three subjects fell significantly. From Latin America, only Mexico and Brazil repeated the test, with the addition of Uruguay. In comparison to other participating countries, Mexico outperformed only Indonesia, Tunisia and Brazil in mathematics; Indonesia and Tunisia in reading; and Indonesia, Brazil and Tunisia in science. Mexico's overall performance was as follows: 0 Reading: 38th of 40 and 3rd in L A C (preceded by Uruguay and Brazil) (Figure 2.4) 0 Math: 37th of 40 and 2nd in L A C (preceded by Uruguay and followed by Brazil) (Figure 2.5) 0 Science: 37th of 40 and 2nd in L A C (preceded by Uruguay and followed by Brazil) (Figure 2.6) However, more students participated in PISA 2003: 30,000, as compared to 4,600 in 2000. 17 among the middle groups). In Brazil, on the other hand, there are significant differences between means and variances between each pair of the socioeconomic groups, and this i s also the case for the UnitedStates. One country among the top achievers with high equity i s Korea, with a difference in means between the lowest socioeconomic group and the rest, and no significant difference in variation (Table 2.1a and 2.lb). Table 2.la Students by Level of Math, selected countries (percent), PISA 2003 Country Below Level 1 Level 1 Level 2 Level 3 Level 4 Level 5 Level 6 Brazil 53.3 21.9 14.1 6.8 2.7 0.9 0.3 Canada 2.4 7.7 18.3 26.2 25.1 14.8 5.5 Italy 13.2 18.7 24.7 22.9 13.4 5.5 1.5 Korea 2.5 7.1 16.6 24.1 25 16.7 8.1 Mexico 38.1 27.9 20.8 10.1 2.7 0.4 0.0 Spain 8.1 14.9 24.7 26.7 17.7 6.5 1.4 United States 10.2 15.5 23.9 23.8 16.6 8 2 Uruguay 26.3 21.8 24.2 16.8 8.2 2.3 0.5 OECD average 8.2 13.2 21.1 23.7 19.1 10.6 4 Table 2.l b Students by Level of Reading, selected countries (percent), PISA 2003 Country Below Level 1 Level 1 Level 2 Level 3 Level 4 Level 5 Brazil 26.9 23.1 25.2 16.5 6.3 1.9 Canada 2.3 7.3 18.3 31 28.6 12.6 Italy 9.1 14.8 24.9 28.3 17.8 5.2 Korea 1.4 5.4 16.8 33.5 30.8 12.2 Mexico 24.9 27.1 27.5 15.6 4.3 0.5 Spain 7.4 13.7 26.1 29.6 18.2 5 UnitedStates 6.5 12.9 22.7 27.8 20.8 9.3 Uruguay 20.2 19.6 23.9 19.8 11.2 5.3 OECD averape 6.7 12.4 22.8 28.7 21.3 8.3 20 Figure 2.7 Readingand Math Performance of Mexico by Level, PISA Reading - Math Math Level 5 Reading Literacy Level 5 Students can develop and work with models for Evaluating information and building complex situations, can select, compare andevaluate hpotheses, drawing on specialized appropriateproblem-solving strategies for dealing with knowledge, accommodating complex problemsrelated to these models, can work concepts contrary to expectations strategically using broad, well-developed thinlung and reasoningslulls, appropnately linked representations, symbolic and formal charactenzations.and insight 2'7% pertuning to these situdhons,canreflect on their actionr andcan formulate and communicatetheu intemretationsand reasoninn 15.6% 10.1% Math Level 1 Studentscan anbwer questions involving farmliar contexts where all relevant information 17 present questions are clearly defined able to identify tion andto carry out routine procedures connections ing to direct in~tructionsin explicit situations, form actions that are obvious and follow Below Level 1 lately from the given stimuli These students may be able to read but have not acquired the skills to usereading for learning Figure 2.8 Percentage of students at each level of performance in Reading by country, PISA 2003 E3 Level 4 0Level3 0Level2 @Level 1 Below Level 1 21 Figure 2.9 Percentageof students at each level of performancein Mathematicsby country, PISA 2003 1(K) 90 80 70 60 W Level 5 50 0Level 3 B L e v e l 2 Level I 40 30 20 I O n ComparisonsBasedonNationalIncome,EnrollmentandExpendituresonEducation 2.18 In PISA 2000, Mexico was above the trend-line in math, reading and science performance when controlling for net enrollment in secondary education. However, Mexico has one of the lowest secondary school enrollment rates. In all non-OECD countries for which data are available (except Bulgaria, Israel and Latvia), less than 80 per cent of the 15-year-olds are enrolled in school. Within the OECD, this i s true only for Ireland and Mexico. Net enrollment rates of 15-year-olds below 75 percent are observed in Albania, Brazil, Chile, Hong Kong-China, Indonesia, Mexico, Peru and Thailand. On other measures, Mexico under- performs in comparison to countries outside of Latin America when controlling for GDP per capita as a proxy for wealth, or public expenditure on education per student (Figures 2.10 through 2.12). Countries such as Thailand, Russia, Latvia and Bulgaria, with GDP per capita lower than Mexico's, performed significantly better. In 2003, when other countries with similar or lower performance dropped out, Mexico came below the trend line but close to it. 22 Figure 2.10 MathPerformanceandNet EnrollmentRatein Secondary, PISA2003 HongKong + Nrlhetland\ Finland c I 1 0 50 55 60 65 70 75 80 85 90 95 IW Net EnrollmentRate Source: PISA; EdStars Figure2.11MathPerformanceandExpenditureper Student, PISA 2003 . Rep of &ora Finland 0'. .. Auslraiia , Suiueiland b USA 0 2.m 4.000 6.000 8.ooO I0.WO 12.000 Expenditureper student Source: PISA; EdSrats 23 Figure 2.12 Performance in Mathematics and GDP per Capita, PISA 2003 0 jc * * MCXlCD lndoncsli j $e: ! xx) i ~ 200 --1 ~~- I 1w - 1 / ~ 0 , 0 1 m 2 m 3MMo 4 m 5 m 60000 State-level Outcomes 2.19 Colima outperforms all Mexican states in math, reading and science, with the Federal District and Aguascalientes trailing not far behind. These three states score roughly 150 to 200 points higher than the three lowest scoring states (Guerrero, Tabasco and Oaxaca). While Mexico as a whole scores well below the OECD average, its higher performing states, such as Colima and the Federal District, actually fall close to the OECD average. For example, Colima's mean science score i s close to those of Portugal and Denmark, and well above those of Turkey and Brazil. In reading, Colima and the Federal District outperform Russia, Turkey, Brazil and Indonesia, with scores similar to those of the Slovak Republic, Italy and Greece. Colima's math scores are on par with those for Greece and Serbia, while it ranks higher than Thailand, Brazil, Uruguay and Turkey. Additionally, dispersion is low across all Mexican states for all three subjects. Indigenous Peoples' Education and Outcomes 2.20 In 2002, an individual that lived in a municipality where 10 to 40 percent of the population i s indigenous had an average income equivalent to 46 percent of the income of a person in a non-indigenous municipality (Ramirez 2005). Similarly, an individual in a predominantly indigenous municipality (above 40 percent indigenous) had an income equivalent to only 26 percent of the income of a person in a non-indigenous municipality. Even with similar education levels, indigenous peoples have a much higher probability of being poor than do non-indigenous people. The indigenous population with no education is more than twice as likely to be extremely poor than the non-indigenous population without education. Rather than narrowing with increased education, the poverty gap grows as higher educational levels are attained. For individuals with incomplete secondary schooling, the probability of being extremely poor i s five times larger if you are indigenous compared to non- 24 indigenous, and almost ten times higher if they have completed secondary education (Ramirez 2005). 2.21 Despite significant improvements in education over the past several decades, considerable educational differences persist between the indigenous and non-indigenous populations. Illiteracy rates and dropout rates continue to be substantially higher among the indigenous population. Average years of schooling, on the other hand, are much lower for indigenous peoples. Educational gaps, however, are closing rapidly among the younger generations. The average non-indigenous person born before 1939 has 120 percent more years of schooling than an indigenous person of the same age group. A non-indigenous person born between 1979 and 1985, by contrast, has 26 percent more years of schooling than an indigenous person born in the same period. And among 7-14 year-old children the indigenoushon-indigenouseducational gap is 8 percent (Ramirez 2005). 2.22 Indigenous peoples' returns to education are lower than for non-indigenous people. In addition, indigenous peoples' returns across the earnings distribution are relatively flat (Garcia and Patrinos 2005), unlike the situation for non-indigenous people, which favors the less able. Therefore, despite advances in indigenous peoples' schooling attainment over the last several decades, indigenous peoples will continue to lag behind the non-indigenous because their schooling will result in lower labor market earnings. Part of the reason for the lower returns that indigenous peoples experience is the quality of schooling that they receive. Indigenous schools consistently obtain lower reading and mathematics scores than all other types of schools, regardless of area of residence. As evidenced by the scores, not only do indigenous schools obtain significantly lower scores than urban public and private schools, they also exhibit poorer test performance than rural public schools and CONAFE community schools. In 2002, a sixth grader from an indigenous school had 15 percent lower reading scores and 8 percent lower mathematics scores than the average sixth grader nationwide. When compared with students from urban public schools, sixth graders in indigenous schools had test results that were 16 percent lower in reading and 9 percent lower in mathematics. Even when compared with students from public rural schools, indigenous students performed poorly. The average sixth grader in an indigenous school had 10 percent lower reading test results and 6 percent lower results in mathematics than the average sixth grader in a public rural school (Ramirez 2005). Furthermore, indigenous students are about ten percentage points more likely to work than non-indigenous students. The indigenoushon-indigenousgap in reading and math scores is about 0.7 standard deviations in both exams. About three-fourths of learning inequality between indigenous and non-indigenous students can be explained by different levels of observed factors (parents' education and teachers' experience) and the language barrier. 2.23 A recent and on-going qualitative and quantitative study of intercultural bilingual education (BE) in Mexico showed that the degree of implementation of B E policies i s low, even in indigenous schools, and varies widely from one type of school to another (Yonker 2004). For example, only 1 to 6 percent of indigenous schools use indigenous language texts. Further, while the majority of teachers in all schools feel that it is important to teach indigenous languages, only about 1 percent o f non-indigenous schools teach an indigenous language, and 7 to 41 percent of the indigenous schools teach an indigenous language. It has also been found that most teachers have no knowledge o f IBE policies. In non-indigenous 25 schools, only 3 percent of the teachers are aware of IBE policies. In indigenous schools, only between 45 to 59 percent of the teachers have some type of knowledge o f IBE policies. Implementation o f IBE policies is most common in Mayan-language schools, which also exhibit the worst results on the 6thgrade national exams. Conclusion 2.24 One can conclude from the discussion above that while Mexico has made significant progress in expanding access to education, much remains to be done to improve the quality of education and to bring Mexican students to a comparable level with other OECD and PISA participating countries. However, there i s reason to remain optimistic that the situation in Mexico can improve. For more than a decade, the education system has been experiencing a dynamic process of change. Recent initiatives at all levels of government have been undertaken to address the challenge of education equity and quality (Box 2.1). Changes in legislation during the current administration aim at contributing to the construction of a more egalitarian society, improving and increasing education opportunities for indigenous peoples, and ensuring the continuity of educational policies. Box2.1 BasicEducationReformsand Improvement inthe 1990s The current reforms in basic education began in 1992 with the decentralization of educational services from the federal to the state level. With decentralization came a number of reforms and initiatives at the central and state levels, with each state experimenting with ways to improve its educational systems. Reform measures included: A far-reaching curricular reform that wholly reorganized the content and materials for primary education. A rigorous federal government effort to provide excellent and diversified teaching and learning materials to primary school teachers and students. The efforts resulted in many initiatives, including free textbooks in each subject area with special texts for teachers, large classroom libraries in most schools, and textbooks in indigenous students' native language. Introduction of information and communication technology in both primary and secondary schools through the use of satellite systems, technology-based teacher-in service training and new computers in schools. Development and establishment of innovative supply- and demand-side interventions to promote rural education. For example, the Secretariat of Social Development (SEDESOL) created Oportunidades,formerly known as PROGRESA, a major conditional cash transfer program in Mexico aimed at developing the human capital of poor households. The program provides monetary transfers to families contingent upon their children's regular school attendance. In recent years, the creation of the Quality Schools Program (Programa de Escuelas de Calidad, PEC) allows the government to focus on disadvantaged urban and rural schools through an innovative school-based management initiative. 26 2.25 However, as Mexico moves forward with education reform, there i s a need to explore the strengths and weaknesses of the current system. National and international assessments could be used to inform the secondary school reform process currently in progress. Additionally, the fact that over half of Mexican students barely attained minimal levels of math and reading lends itself to support for stronger accountability mechanisms. Furthermore, given the variance in performance between states, as well as between indigenous and non-indigenous students, it i s probable that there are many unobserved factors that contribute to achievement levels. Increased autonomy at the school and community level would allow schools to determine the policies customized to the local context. 27 CHAPTER 3. MEXICO'S PERFORMANCE INNATIONAL AND INTERNATIONAL ASSESSMENTS Analysis Usingthe NationalAssessments 3.1 With the exception of Chile, Mexico has the longest history of system-run, nationwide student assessment in Latin America. The history o f assessment in Mexico goes back to the beginningof the 1970s. Since then, but mainly during the last decade, significant amounts of data have been collected and the technical capacity for measuring learning achievement has been developed (Box 3.1). Box 3.1 NationalEvaluationSystemfor Education The national evaluation system for education is composed of several institutions: the General Directorate for Evaluation (DGE); the State Offices for the Evaluation of Education; the National Institute for the Evaluation of Education (INEE); and NGOs and academic institutions. DGE and INEE are especially important divisions because they are supported by the federal government and these divisions assume responsibility for collecting, and analyzing, and disseminating information about the education system. The DGE is a division of the Secretariat of Public Education (SEP), which was formerly in charge of the evaluation system. With the creation of INEE, DGE has been primarily devoted to evaluation of SEP programs, collecting information for other system participants and informing the State Offices about their performance according to the evaluation results. Specific DGE responsibilities include: evaluation of the professional training and learning outcomes as result of The Teacher Career Program (Carrera Magisterial);testing for admission into lower secondary and teaching schools; children's knowledge contests; preschool and initial education assessment; compensatory program evaluation; evaluating eligibility for applying tests for hiring or promoting teachers; and Evaluation-Diagnostic Census for all children at the beginning of the school cycle. INEE was founded in August 2002 with the aim of providing adequate, rigorous and more transparent evaluations of the basic and secondary education system. A main objective of INEE is to consolidate the national evaluation system with the objective of increasing the quality of Mexican education. Other objectives include: (1) coordinating international evaluations, such as PISA; (2) developing an indicator system based on the information collected by SEP; and (3) developing models for the evaluation of schools as an organizational unit. In 2003 and 2004, INEE released two reports on the quality of education in Mexico and has recently published an analysis of the PISA 2003 evaluations. 3.2 This section presents an empirical analysis of Mexico's National Assessment exam (Estdndures Nucionules) in order to ascertain the determinants of students' academic achievement in basic education (Velez and Lopez-Acevedo 2002). These data were collected by SEP through the DGE. SEP conducted national assessments annually between 1998 and 2002, to different grade levels. There are five types o f primary schools that are considered in the test: (I)urban public schools; (2) rural public schools; (3) indigenous schools; (4) 28 community schools operated by CONAFE; and (5) urban private schools. Background questionnaires were obtained for students taking the exam, from a random sample of fifth grade teachers in the selected sample schools - not necessarily instructors of the students who took the exam - and from principals of the sample schools. 3.3 In addition to measuring educational outcomes in reading and math, the available data includes indicators on teaching practices, school environment, students' socioeconomic characteristics and education experience and community level variables. The analysis estimates the relationship between school outcomes (achievement) and education inputs (teacher characteristics, teaching methods, educational materials, physical facilities) controlling for student characteristics (socioeconomic background, gender, education experience) and state level characteristics. In order to analyze the factors determining the levels of achievement among children studying basic education, the DGE randomly selected a sample of 14,714 fifth grade students; 51 percent of this sample was female and only 6 percent of the students had access to the Early Childhood Education (ECD) modality supported by the Government. However, about 88 percent had some preschool experience. About 17 percent reported grade repetition in primary education. About 12 percent of boys reported some labor (not at home) activity, in addition to school-work. In contrast, only 4 percent of girls report working. The fact that girls help more often with household chores may underestimate the true average for labor among girls. On the other hand, for family background, almost every child reported that his or her family supported them in their studies and the average reported mothers' year of education was about 7.5 years. In the same context, dwelling services index averages are 0.14 for girls and 0.24 for boys. Finally, for school characteristics level, a high percentage of boys and girls reported positive pedagogical practices in their classroom. From this sample, almost 90 per cent of children were studying in public schools and 12 per cent at indigenous schools. The regional distribution (that splits the sample in four regions: North, Center, South and Mexico City) shows that the sample distribution for both girls and boys throughout the country i s quite similar. A more detailed description of the sample characteristics i s presented in Annex Table 3. 3.4 The analysis reveals that while educational investment in Mexico has been successful in achieving many quantitative objectives, the anticipated quality performance levels have yet to be achieved. Although there are excellent schools in Mexico and many students learn enough to progress successfully up the educational ladder, evidence shows that a significant proportion of students do not achieve minimum levels. This i s especially the case among students inthe southern states, in particular among indigenous students, (Figures 3.1 and 3.2). 29 Figure 3.1 Mean StandardizedTest Scoresby Region Mexico D.F. North iBCenter SouthEast 0SouthWest ONational 2nd grade 4th grade 6th grade Source: Velez and Lopez-Acevedo2002 Note: Testscore is an average calculated by SEPfor math, reading and text comprehension Figure 3.2 MeanStandardizedTest Scores by Region 50 EUrbanPrivate 40 UrbanPublic 30 ElRuralPublic 20 10 0Community 0 0Indigenous 1St 2nd 3rd 4th 5th TOTAL grade grade grade grade grade Source: Velez and Lopez-Acevedo2002 Note: Test score is an average calculated by SEPfor math, reading and text comprehension 3.5 Among the student's characteristics, it i s important to note that while preschool experience i s positively related with outcomes, this is not the case for having experience with the Early Childhood Development Center (Centros de Desarrollo Infuntil, CENDI) model, an initial education program for children from 1month to 6 years of age. The effect of repetition on academic performance, as elsewhere in Latin America, i s negative. Not surprisingly, work experience and health problems are also negatively associated with academic performance. Indeed, over half of Mexican fifth grade students work in (58 percent) or outside (11 percent) the home. Both types of work are associated with worse exam performance and working outside the home i s associated with a decrease in exam performance by about 0.4 standard 30 deviations. Annex Table 4 presents the results of the multivariate analysis, controlling for the many factors that could be measured and that theoretically may have a simultaneous affect on academic performance. 3.6 Several family characteristics that are associated with academic performance in the literature have been reconfirmed in the results of the analysis. A mothers' education is important for both reading and mathematics, especially for girls. Indeed, having a house with services such as electricity and water, a measure of family welfare, i s a strong predictor of academic performance. Having books and computers at home i s significantly associated with achievement, and it i s more important for girls than for boys. A family's expectation of further studies for the student is a strong predictor of academic performance. 3.7 Finally, when considering some characteristics of the school, it i s clear that: (i) rural school students, especially boys, perform unambiguously worse than students in urban locations; (ii)indigenous schools also have lower quality achievement; and (iii)although there are good and bad schools both in the private and public spheres, the students in the former obtain significantly higher educational achievement. Regions do not appear to have a strong capacity to predict student achievement, especially in the case o f boys; however, boys perform better in the Federal District than in any other region. Girls perform better in the North than in any other region. 3.8 A key finding of the analysis that the index of pedagogical practices has a significant effect on students' achievement in math and reading. Curricula that seek the introduction of activity-based, student-centered learning with a clear emphasis on student research and discovery methodologies and that encourage creativity, independent thinking and a questioning attitude, seem to be more effective in increasing student learning in Mexico. This analysis confirms that in order for the quality o f the Mexican basic education system to improve, teachers will need to be trained in active pedagogic techniques that expose students to certain teaching methods such as problem solving and critical thinking. Returns to Quality 3.9 The strong and positive link between years of schooling and earnings of students once they enter the labor market i s by far one of the best established facts in labor economics. Recently, researchers have examined if the substantial observed returns to additional years o f schooling depend on the level of funding that schools receive. Put differently, would an increase in resources per pupil make as attractive an investment as requiring students to attend one additional year of schooling? In the United States, school quality - measured by, among other factors, the student-teacher ratio - has a positive but small effect on the rate of return to education. A decrease in the student-teacher ratio from 30 to 25, for example, i s associated with a 0.4 percentage point increase inthe rate of return to education (Card and Krueger 1992). 3.10 In Mexico, preliminary findings show that those educated in states with higher quality schools, as measured, for instance, by student-teacher ratios, teachers' schooling and test scores have an impact on earnings. Controlling for indigenous and community schools shows a negative impact on earnings for these populations (Garcia, Knaul and Patrinos 2005). More specifically, a decrease in the student-teacher ratio from 40 to 30, for example, i s associated 31 with a 0.1 percentage point increase in the rate of return to education. There is also a small variation in the rate of return to education across individuals born in different states and different times, and this variation is associated with differences in the quality o f schooling (Card and Krueger 1992). Test scores are shown to have less impact, but an increase in 1point on the average score increases earnings by 0.04 percent. Teachers' schooling has a positive impact on earnings: if average teacher schooling were to increase by one whole year, then earnings would increase by 9 percent. The percentage of teachers that have only basic education has a negative impact on earnings, implying a 1 percent decrease in earnings. Thus, there is some evidence that the quality o f education increases the returns to education in Mexico, although the impact i s lower than that found in the United States (Garcia, Knaul and Patrinos 2005). Summary of NationalAssessments Results 3.11 Several family and school factors that are associated with academic performance in the literature have been reconfirmed. Family factors continue to be strong predictors of academic achievement. However, the findings discussed above also demonstrate that schools in poor regions are capable of improving learning outcomes through interventions to improve quality. The analyses demonstrate that factors that are associated with significant improvements in reading and math scores include having students with educated parents, having students that attended preschool, having a house with basic infrastructure services (for example, electricity), having books and computers at home and having parental involvement. The findings also confirm what many other researchers have found concerning active pedagogic instruction. Most notable, active pedagogical instruction is an effective way of increasing student achievement and points to the need for teachers to be trained in active pedagogic techniques that expose students to certain teaching behaviors such as problem solving and critical thinking for basic education quality to improve. AnalysisUsing International Assessments 3.12 The results of PISA (http://www.pisa.oecd.org) 2000 and 2003 (Abdul-Hamid 2005) are used to understand how well-prepared Mexican students are in math, science and reading. The research is based on the analysis of the results for 15-year-olds in all three disciplines. PISA provides a measure of reading, mathematics and science achievement for a nationally representative sample, comparable across countries. PISA's assessment focuses on young people's ability to apply their knowledge and skills to real-life problems and situations, rather than on how much curriculum-based knowledge they possess. 3.13 A mixed-method approach is used to analyze the determinants of school achievement of Mexico (Annex Table 5 contains the variables and descriptives). The approach includes a distributional analysis of performance, factors affecting achievement using generalized least squares and conditional relationships among subgroups using quantile regressions. In the distributional analysis, we use descriptive measures o f achievement, such as analysis of means and variances, within Mexico and with comparisons across countries. We also look at measures of dispersion between top and low achievers. The statistical significance of these measures is assessed and confirmed. Overall performance of Mexican students i s compared 32 using standardized scores against international averages. Performance of schools at different geographic locations was also assessed and compared. PerformanceandEquity 3.14 The analysis shows that overall variation in performance among Mexican students is small compared to the other participating countries. The gap between the top 5 percent and the bottom 5 percent of performers in Mexico is similar to that of top achievers, such as Canada and Finland. When comparing countries on performance while controlling for level of dispersion, Mexico's results are different from other L A C countries. While it is grouped with the under-performing countries in terms of test scores, it ranks high on equity: Mexico i s in the same quadrant as Portugal, Serbia and Thailand (Figure 3.3). However it i s important to note that secondary education enrollment i s substantially lower than that o f the other countries in the same quadrant. The same pattern was seen in PISA 2000, with Mexico in the opposite quadrant from Argentina, Chile, Peru and Brazil and in the same quadrant as Finland and Korea. Figure3.3 MathScores andDispersionacross Countries, PISA 2003 600 300- 1 240 260 280 300 320 340 360 380 Dispersion (difterence between top and bottom 5%) 3.15 The equitable performance of secondary school students in Mexico can be viewed in a different light. For example, socioeconomic background has less o f an impact on academic achievement in Mexico than in Germany (McGaw 2004). Thus, Mexico, in terms of academic achievement, displays high equity and low quality, putting it in a category with Spain, Italy, Poland and Greece. The high equity, high quality countries continue to be Korea and Finland, among others. Examples of high quality, low equity countries are the United Kingdom and the United States. Low quality, low equity countries include Germany, the Czech Republic and Hungary, among OECD countries (McGaw 2004; OECD 2001, 2004a). Among non-OECD countries, the high equity, low quality countries include Thailand and Indonesia. All other L A C participants in PISA 2000 and 2003 are low quality and low equity. 33 3.16 Secondary school enrollment in Mexico, however, is comparatively very low. The key question is what would happen to Mexico's performance if the enrollment rate were higher? There are two ways to answer this question. The first i s provided in the recently released PISA 2003. Mexico was one of only three L A C countries to participate in the second PISA test. Mexico did increase enrollment at the secondary level between 2000 and 2003, by almost 5 percentage points. Average achievement did decrease by less than by 4 to 5 percentage points in reading and science, and by much less in mathematics. Dispersion increased but Mexico remains in the quadrant occupied by countries with relatively low test score inequality and i s not significantly different than Portugal and Spain. 3.17 The impact of increased enrollments on average scores and dispersion was also simulated by merging the 2000 household survey (ENIGH) with the PISA data base. Household wealth and ownership of assets were used to merge the two data sets. From the household survey information on 15-year-olds, such as enrollment status, household characteristics and family wealth, each child was assigned a "score" based on the familyhousehold characteristics in each dataset. Then the enrollment status for those not currently enrolled was randomly assigned. The simulation began by randomly adding an additional 5 percent at a time and look at average score and dispersion. This approach has a few limitations: (1) there are few comparable contextual variables in common between the two datasets; (2) since PISA performance, where most of the calculations were conducted based on internationally standardized scores, i s being used, one should expect that the imputed performance score for the randomized enrollment scenarios will be within the range of the observed scores; and (3) the household survey i s not representative for the 15-year-old age group; the enrollment rate was 65 percent instead o f the 51.6 percent estimated in PISA 2000. Nevertheless, based on the simulations, scores and dispersion in mathematics are not going to change dramatically by increasing enrollment. Increasing enrollment by 1 percent will decrease scores by 7 percent of a standard deviation. Table 3.1 shows the actual and simulated values for 2000 and 2003. The simulations accurately predict changes in math scores, but underestimate the increase in dispersion. It could be that PISA 2003, given that 30,000 students were tested, many more than in 2000, more accurately reflects the true performance of students. Still, average math scores did not decrease very much, and dispersion i s still within the favorable quadrant of highequity. Table 3.1 Average Scores andDispersion Enrollment Average Math Dispersion Scores PISA 2000 (actual) 51.6 387 273 +5% -0.01% 1.01% +lo% -0.11% 0.4% Simulated +15% -0.26% 0.3% decreasehcrease +20% -0.04% 0.4% I +25% -0.12% 0.6% +30% -0.24% 0.3% +35% -0.30% 0.2% PISA 2003 (actual) 55.1 385 280 34 3.18 What accounts for high equity in Mexico? International experience might provide some evidence. In the case of Korea, a highperformer with very low dispersion in text scores, this could be the result of the secondary school equalization policy, which aims and achieves high performance and low inequality (Box 3.2). While Mexico does not have such an equalization policy at the secondary level, there i s evidence of specific policy interventions that tend to increase equity at the basic education level. SEP's compensatory education interventions target schools in disadvantaged rural areas and increase resource allocations for those schools to give students more equal opportunities. Compensatory education programs tend to decrease dispersion in scores. Perhaps the cumulative impact of compensatory education from basic education is carried over to lower secondary schools, and explains in part the highequity observed in PISA results. Box 3.2 Korea's equalizationpolicy South Korea adopted an equalization policy in response to growing demand for better schools and rampant private tutoring. Under this policy, the competitive entrance exams were substituted with random assignment of students for all secondary schools (both private and public). In addition, the government subsidized private schools so that their students' tuitions and teachers' salaries are now equal to those in public schools. Under the equalization policy, all schools, public or private, had to give up their rights to select new students and are required to take all students assigned by the Ministry of Education through district-wide lottery. Meanwhile, the government guaranteed any deficit in operating cost, but not in capital cost, of all private schools. Accordingly, private schools became almost public in terms of the accessibility to the students, contents of the learning, and the quality of teachers. The only meaningful difference between private and public school remained in the governance structure. While the policy has contributed to the remarkable expansion of secondary school enrollments, competition for better colleges and private tutoring has not decreased. The policy has raised both equity and average achievement level of Korean 15-year-olds. Meanwhile, the lack of competition and diversity among secondary schools created little incentives for schools to respond to the need of students and parents. 3.19 Comparing performance of each socioeconomic group (SES) in Mexico to a similar group in other countries provides another level of analysis (Figure 3.4). When comparing Mexico's standardized scores at the socioeconomic categorization to other countries for each socioeconomic group, the following is observed: 0 Students from all four SES groups scored below the international average in their group; 0 Students who come from the lower SES group in Mexico performed better than their peers from L A C countries but were worse than peers in most other countries; students in other SES groups also did the same, but performed relatively worse than Argentina among the L A C countries; 0 Relative to their SES groups, all groups (except the richest), on average, had comparable performance; and 35 0 Students from the richest families performed relatively better than students from other SES groups, and the other three SES groups had a comparable performance, on average. 1.5 1 0.5 0 -0.5 -1 -1.5 -2 -25 Figure 3.4 Performanceby Socioeconomic Level Using Standardized Scores by Level, PISA 2000 Source: Abdul-Hamid 2000 Note: 2000 was used in order to have meaningfin1Latin American country comparisons because PISA 2000 included more Latin American countries. Schools Comparison 3.20 In most PISA countries a considerable proportion of the variation in student performance lies between schools. In Mexico, there i s a significant difference in performance between private and public schools (Figure 3.5). Type and location of schools affect performance. On average, private schools achieve better scores and have lower dispersion than public schools; at the same time, some public schools' performance i s similar to, and sometimes better than, some private schools. There i s a significant difference in school average scores and school dispersion based on location o f the school; schools in large cities had higher average scores than the rest; schools in villages and small towns had higher dispersion than schools in cities; and top achieving schools were from large and medium-sized cities. The significance of school type and location has also been confirmed by multivariate and generalized least squares methods for the disciplines (Annex Table 6). In addition to the location, type, size, educational materials and system, compelling evidence shows that performance is significantly associated with the school climate and enthusiasm of teachers. 36 Figure3.5 Performanceof Public andPrivateSchools in Math, PISA 2003, across performance distribution 600 500 2 400 8 5 300 E" 200 100 0 1Gth 25th 50th 75th 90th Average Quantile Public Private 3.21 There is also variation in scores between the different types of secondary schools in Mexico. Overall, general and technical high schools perform better than other types of schools in mathematics, reading and science. Technical secondary schools and telesecundarias score much lower than other types of schools. This i s particularly troubling considering the telesecundarias are the fastest growing sector in Mexico's education system. Math scores for students in telesecundarias are, on average, 100 points less than those of students in general high schools, almost 150 points less in reading, and about 100 points less in science. Since PISA 2003 is representative at the level of school type, one is able to see that telesecundarias produce the lowest raw scores. But the question is: what would happen if one controlled for other factors and compared them to other school types? Using a GLS model that controls for sex, mother's education and work, home educational resources, city and state residence, private school attendance and school size, among other factors, it i s documented that all other types of schools have a significant learning advantage over telesecunduria students (Figure 3.6). In math, the advantage i s only statistically significant for upper secondary technical and general high schools, technical professional schools and training for workers programs (bachillerato te'cnico, buchillerato general, profesionul te'cnico and capacitacidn para el trabajo). That is, there i s no significant difference between types of lower secondary school (secundariu general and te'cnica, and secundaria para trubujadores) once you control for other factors. But for science and reading, telesecundarias have a significant disadvantage relative to all other types (except training for workers). This analysis does not control for costs. Analysis o f the PISA results undertaken by INEEalso showed that telesecundurias performed worse than other types of schools: 89.3 percent of telesecunduria students were not competent in mathematics, compared to 71 percent of students in general secondary schools (INEE 2005). 3.22 However, a 15-year-old, the subject of PISA tests, should be in the first year of upper secondary. Those students who are in the first year of upper secondary perform significantly better than students still in the lower secondary (basic) education level. Students may be in lower secondary despite their age because of late entry or grade repetition, both associated with low performance. The large intake of disadvantaged students in recent years, partly due to the expansion of Oportunidades, may have exposed these new students to poor quality schooling. Thus, there is a need to couple efforts of expansion with programs to improve the quality of 37 schooling, especially at the lower secondary level, and rapidly expanding modalities such as telesecundaria. 3.23 The problem, however, i s that more and more children are entering this type of school, thus lowering overall scores, and their achievement levels are so low. A 15-year-old who is in the first year of upper secondary does relatively well in PISA. Students who are still in lower secondary have very low scores. Low learning levels limit their chances for success at higher levels of schooling or later on in the labor market. However, since PISA was not designed as an evaluation of secondary school types, and the fact that only one point in time i s examined, it i s recommended that Mexico undertake undertake a rigorous assessment of the impact of treatment by type of secondary school, with particular focus on telesecundarias. More generally, the issue of increased enrollment into low quality schools must be a priority for further analysis. Figure3.6 Performanceadvantageof school type vs. telesecunduriu (controlling for other factors, significant only) 90 1 80 70 60 50 40 30 20 10 0 Upper Upper Technical Lower Lower Training for Secondary Secondary Professional Secondary Secondary workers General Technical General Technical i Reading Math 0 Science 3.24 The above analysis, aggregate levels of performance for Mexico. However, given the wealth of information contained in the PISA 2003 data, it i s possible, and appropriate, to examine state-level data as well (see also INEE2004b). Analysis of state-level data shows that there are actually wide variations in different indicators between states. For example, a closer look at four different states, plus the Federal District, found that there are differences in how various types of secondary schools perform in each state. While professional technical schools have higher scores than general secondary schools, in the state of Durango the opposite is true. On average, scores for technical and general high schools are equal. However, in Colima, scores for the technical high schools in all three subjects are about 75 points less than those for general high schools. The Federal District seems to show the least amount of variation in 38 scores between all types of secondary schools. In Veracruz, the professional technical schools outperform all other types of schools. In Veracruz and the state of Mexico, which both report telesecunduria scores, telesecundarius still perform worse than any other type of school. 3.25 Additionally, while girls in Mexico outperform boys inreading, the same trend does not hold true for all states. For example, controlling for all other variables, in Campeche there i s substantial gender inequality across all levels of performance, while girls in Veracruz and Aguascalientes far outpace boys (Figure 3.7). Variables explaining differences in reading scores also vary between states. For example, controlling for other variables, the effects of teacher morale at school on the scores of low achievers are similar in Colima, a high performing state, and Veracruz, a low performing state (Figure 3.8). Figure3.7 Effectsof BeingFemale on ReadingScores by AchievementLevel I 1 Note: Q l O is lowest achievement group, while Q90 is highest Figure 3.8 Effect of TeacherMorale at Schoolamonglow achievers (Reading) ___I_- 39 School Resources 3.26 Overall, the marginal effect of an increase in the quality of educational resources tends to be highest in countries where deficiencies reported by principals are particularly pronounced. This negative relationship may suggest diminishing returns to investment in educational resources. However, the value of coefficients varies widely across countries. In Mexico, Argentina and Peru, and also in Germany, a one unit change of the index i s associated with differences in scores of 25 points or more, corresponding to an improvement of more than a third of a proficiency level on the combined reading literacy scale. 3.27 School educational resources are carefully explored and the analysis finds that only those that have a direct connection to the curriculum are important. In science, only laboratory equipment i s found to be significantly associated with performance and the value increased by level of performance as shown by the quantile regression estimates (Figure 3.9, Annex Tables 7a-7c). Figure3.9 Effect of Equiplmentand Labson Science Scores, PISA 2003, across achievementlevels 14 12 10 2 0 10th 25th 50th 75th 90th Quantile Quality of Teachers 3.28 Although PISA is probably not the best instrument for analyzing teacher performance - since it i s a one point in time analysis - there are some characteristics of teachers that are at least associated with higher performance. These characteristics include teacher morale, behavior, attitude, and qualifications. High teacher morale, as perceived by the school principal, i s associated with better performance. The results confirm the substantial evidence that teacher quality is highly correlated with student test scores (Rockoff 2004; Murnane 1975; Armor and others 1976). For schools where teachers work with enthusiasm, math and reading scores seem to be higher. When teachers take pride, it i s observed that students in that school perform better. When teachers value academic achievement, students significantly perform better in math and reading. Moreover, teacher morale i s more important for low and middle achievers in reading and math, and for middle achievers in science (Figure 3.10). 40 Tipre 3.10 Effect of Teacher Morale on Performanceby Achievement Level, PISA 2003 Science Math 0Reading 10th 25th 50th 75th 90th 3.29 Teacher behavior and teacher-related factors affecting school climate are associated with performance. In schools where teachers have high expectations, students are observed to perform better. When principals feel that there i s a strong relationship between students and teachers, students perform better. Moreover, students attending schools with high levels of teacher absenteeism, and lack of encouragement to achieve their full potential, perform worse, especially in mathematics and reading. Technology: Quality, not Quantity 3.30 International experience on the impact of computers and technology on academic performance, based on randomized evaluations or natural experiments, shows mixed results (Krueger and Rouse 2004; Angrist and Lavy 2002; Boozer, Krueger and Wolkon 1992; Goolsbee and Guryan 2002; Kirkpatrick and Cuban 1998; Wenglisky 1998). The research of Krueger and Rouse (2004) in the United States suggests that while the use of computer programs may improve some aspects of students' language skills, it does not appear that these gains translate into a broader measure of language acquisition or into actual readings skills. In the Netherlands, Leuven and others (2003) show that extra funding for computers and for language materials does not improve test scores in reading, arithmetic or information processing. All point estimates are negative. There is more evidence of negative effects of the computer subsidy, especially in math. 3.31 Angrist and Lavy (2002) examined the effect of computer funding on performance in Israeli schools. The study examined the installation o f computers in many elementary and middle schools and provides an opportunity for estimating the impact of computerization on both the instructional use of computers and pupil achievement. Results from a survey of Israeli teachers show that the influx of new computers increased teachers' use of computer-aided instruction (CAI) in the 4th grade, with a smaller effect on C A I in 8th grade. Although many of the estimates are imprecise, on balance, C A I does not appear to have had educational benefits that translated into higher test scores. OLS estimates show no evidence of a relationship between C A I and test scores, except for a negative effect on 8th grade math scores 41 in models with town effects. Estimates for 4th graders show lower math scores in the group that was awarded computers, with smaller (insignificant) negative effects on reading scores. 3.32 In an analysis of 31 countries usingPISA 2000 data, including Mexico and Brazil but no other developing country, Fuchs and Woessmann (2004b) find that students who use computers extensively at school have worse mathematics and reading performance. Those usingcomputers several times a week performed "sizably, statistically and significantly worse" than those who used them less often, according to an analysis using PISA 2000 for 31 countries. Once controlling for home and school resources, they find no impact of computers on performance. That is, for students from homes and schools with more resources in general, computer availability was not related to student performance. Computer use at home, particularly internet access, email and educational software, i s associated with better test performance. However students who hardly ever used computers did a little worse than those who used them between a few times a year and several times a month. 3.33 Although not a random experiment, analysis using PISA data for both 2000 and 2003 for Mexico, shows that there are mixed results regarding computer use. In general, the existence of computers and the computer-to-student ratio at the school does not make a difference on performance. This was tested using the computer-to-student ratio and the number of computers available to students only or to teachers only, as reported by the school principal. However, in Mexican schools where the use of computers i s significant, student outcomes were much higherthan other students in reading, math and science (while controlling for other factors). Moreover, this is confirmed usingquantile regression analysis. The analysis highlights two major findings for Mexico. First, although using computers at school is associated with positive achievement, it is observed that in mathematics, using calculators at school, as reported by students, also played an important factor. Hence, use of low threshold technology can also be beneficial. When controlling for achievement levels as revealed by the estimates from the quantile regressions, it i s observed that calculators show a higher contribution to achievement than computers among low achieving students. However, for high achievers, computers have this positive effect (Figure 3.11). Second, providing an opportunity for students to use computers at school has contributed mostly to achievement in reading rather than to science and mathematics. It can be hypothesized that this may be due to availability of software and learning modules in reading and lack of such for science and mathematics. 42 Figure 3.11 Effect of Student Use of Computersat School by AchievementLevel, PISA 2003 I ~~ I Note: I refers to lowest uchievement group, while 5 refers to highest 3.34 The issue of cost-efficiency is raised in light of these findings. While the use of computers and calculators are found to be significantly associated with performance in Mexico, using PISA 2000, especially in math, calculators are more cost-effective than computers. The price of a computer i s many times higher than that of a calculator (10,947 Mexican pesos, or $952, using an exchange rate of 11.5 pesos to the dollar, compared to 10.2 Mexicanpesos, less than $1, for a calculator). StudentRelatedFactors 3.35 Instrumental motivation, as measured by students' perception and understanding of education as a means to improve job opportunities and ensure a financially secure future, i s seen to be significant in explaining outcomes in science and mathematics. Students tend to achieve better results in these areas when they are aware of the importance o f studying mathematics and science for the labor market and their future careers. The level of association also varies between the different achievement quantiles. This is more significant for higher achieving students in science and i s only significant for the top two achieving quantiles (75th and 90th)in mathematics (Figure 3.12). Figure 3.12 Effect of MotivatingStudents on Value of Subject inMarket by Achievement Level, PISA 2003 1 2 3 4 5 Note: I refers to lowest achievement group, while 5 refers to highest 43 3.36 Memorization as a method of studying negatively affects student performance in all three disciplines while controlling for other variables in the generalized least squares models. The same significant negative effect i s observed among all different achievement groups (Figure 3.13). Figure3.13 Effect of Memorizationon Performanceby Achievement Level, PISA 2003 0.0 -1.0 -2.0 u -3.0 loth 25th 50th 75th 90th Rote: lo"' refers to lowest achievement group, while YVth rejers to highest 3.37 The awareness and enjoyment levels of the subject matter are important in reading performance and among the different achievement groups (Figure 3.14). In addition, allocating time to work on class assignments and homework i s associated with better performance. The noticeable importance of this factor i s observed among low achievers in mathematics and top achievers in science (Figure 3.15). Figure3.14 Effect of EnjoyingReadingby Achievement Level, PISA2000 10th 25th 50th 75th 90th Quantile 44 1I 3 - i 10th 25th 50th 75th 90th Quantile Family Factors 3.38 The most important family factors contributing to student achievement are mother's education level and the availability of educational resources in the home of the child. A mother's education level has the most significant effect in science, especially among the top achievers. Mother's work negatively affects performance only in mathematics (Figure 3.16). Figure 3.16 Effect of Mother with Secondary Education by Math Achievement Level, PISA 2003 j I 20 I 10th 25th 50th 75th 90th Quantk 3.39 The availability of educational resources at home affects reading performance the most and the effect increases by achievement level. Its effect on the performance of top achievers was twice as much as that for low achievers. Educational resources include books and a computer inthe home (Figure 3.17). 45 Figure 3.17 Effect of Home Educational Resources by Math Achievement Level, PISA 2003 l2 i 10th 25th 50th 75th 90th Quantile I 3.40 Results are summarized in Table 3.2 in terms of school factors, teacher-related factors, student factors and general characteristics. Table 3.2 Summary o f Determinants of Leaming in Mexico, PISA 2003 School Factors Positive Factors Private school over public in all subjects More girls in school is associatedwith better performance Location of school (cities especially large ones) has positive influence (villages and sma II towns disadvantaged) Mixed Effects School educational resources: .... Students who used computers (effectively) at school achieved better in all SUbjeCtS Total number of computers available to teachers does not have significant positive impact Computer-Student ratio at school does not have a clear impact Availability of science equipments and laboratories Negative Factors High student-teacherratio associatedwith low score Teacher -Related Factors Positive Factors ... High level of relationship between teacher s and students (perceived by school principal) High teacher morale associatedwith higher scores (perceived by principal) Teacher behavior andrelated factors affecting school climate associatedwith high scores Student and Learnine Factors Positive Factors .. Time on homework associatedwith better performance Interest in subject has positive effect Student perception of relationship with teacher (get along, interest in student. listen. extra help, treat fairly) Insnumental motivation has positive effect on student performance in math and science: understanding that science and math are associated with betterjob opportunities and future financial security Negative Factors . Memorization as a way of studying (in all subjects) is not effective General Characteristics Positive Factors Boys achieved better than girls in math and science. but girls perform better (but big difference) inreading Mother's education (above secondary) associated with better performance Home educational resources associated with hi gh performance in math and reading Negative Factors .. Mother's employment associated with low performance Number of siblings 46 Conclusions 3.41 The analyses presented above reaffirm what i s known about the impacts of socioeconomic status on learning and the limited role of physical investments. They also shed new light on the importance of school climate, expectations, participation, pedagogic methods, autonomy, accountability and the needto use assessmentsto informpolicy decisions. 3.42 A few key findings emerge from a comparison of the results of Mexico's National Assessments and PISA. Both find that increased maternal education and having educational resources at home positively impact student achievement levels. Results from both exams also underscore the importance of institutional factors in determining educational achievement: students from urban and private schools often perform better than students from rural and public schools and school climate and teaching practices matter. In schools where students and teachers have strong relationships and engage in participatory learning practices, students achieve higher test scores. Analysis also shows that schools in larger cities and high teacher morale are associated with better learning outcomes. Suggested implications are presented in Table 3.3. Table 3.3 Imdications for School Oualitv Basedon Analvsis of PISA (1) School Factors What i s working Private education, schools in cities High teacher morale and level of relationship between students and teachers Implications 1, Leam from best practices at private schools and apply them in public schools 2. Provide scholarships to needy groups 3. More altemative schools in rural areas and small cities 4. Rotate teachers from different types of schools to public schools 5. Build environment with high level of satisfaction and interest in teaching What i s not clear Existence of computers and calculators, science laboratories Implications 1. Avoid focus on computer-student ratio or number o f computers or Intemet 2. Think of cost-efficient resources 3. Build curriculum to accommodate usage o f computers 4. Trainteachers and students to use computers effectively incourse work 5. Make sure that science students carry out experiments and use laboratories (2) Student Factors What i s working Homework; interest in subject matter, understanding the value of subject Implications 1. Encouraging and supporting students to spend time on homework 2. Provide incentives to faculty to work more with students 3. Encourage group assignments so that students can help and motivate each other Understand future benefit o f learning subject matter 1. Train faculty to make students aware of value of subject matter in labor force 2. Encourage teachers to relate theory to real life applications 3. Enhance curriculum to include case studies and applications What i s not working Memorization as a way of studying Implications 1. Memorization as a way of studying should be discouraged 2. Schools could implement activities on how to be a highly effective leamer 3.43 One can also conclude from the above analysis that Mexican students perform as well as their peers in other Latin American countries. However, there remains much to be done to 47 bring them to a comparable level with other OECD and PISA participating countries. Issues that emerge are mostly related to instructional practices and whether students are oriented to be effective learners. Teachers cannot be rigid and too strict with students and should be able to encourage students to achieve their full potential. More important than physical resources, Mexico may need to focus on improving the learning environment and school climate. These factors were robust and confirmed by the quantile regressions for all three subjects. 3.44 School climate has been found to be of measurable importance for the different achievement groups. In other words, the climate factors played a significant role in the performance of low and high achievers indiscriminately. For example, when controlling for all other variables, we find that between 9 to 15 math and reading points were associated with a unit increase in the index of teacher morale and a 10 point increase for one unit of student sense of belonging at school (Table 3.4). Changes and improvements in the school climate (relations between students and teachers for example, belief in students' ability to learn and support for that to happen) and current schooling practices are simulated to increase the overall score of the bottom achievers by about 20 percent. Table 3.4 SimulatingImprovements,PISA 2000 Improving school climate for the low achievers to the School climate includes: average availble at schools with high achievers Improving teacher morale 20% 18% Improving teacher behavior 16% and relation toward 14% students 12% Improve teacher to student 10% ratio and certification 8% 6% 4% 2% 0% Science Math Reading 3.45 While it would be difficult for a central authority to regulate family and student factors, school climate and other determinants of learning, such as time spent on homework and the use of technology in schools, policies at the school and community level stand a far greater chance of affecting positive changes in learning outcomes. Increased autonomy, improved accountability, and use of assessments may be needed in order for schools to determine locally- appropriate policies. For example, the analyses above show that teacher morale, teacher- student relations, teacher behavior, school climate, teacher expectations of student performance, student's awareness of value of school for future earnings and active learningheaching styles are all associated with better learning outcomes. Increased autonomy may give schools the flexibility they need to empower teachers, thus improving the school climate as well as the relationship between students and teachers. The analyses also showed 48 that parental involvement with schools and attitudes about their children's' schooling have an impact on educational achievement. Strengthened accountability mechanisms could create space for parents to actively participate in the education system and, if successful, raise aspirations for greater educational attainment levels for their children. Finally, the results from an important international assessment such as PISA offer substantial material to be considered in the development of policies and strategies to improve learning outcomes. The next chapter provides a preliminary agenda for action that Mexico may consider for improving learning outcomes. 49 CHAPTER 4. A N AGENDA FOR ACTION 4.1. The first three chapters of this report provided an international review of the factors that impact learning outcomes, a description of the state of the quality of education in Mexico, and analyses of new data on learning outcomes, using innovative techniques and applications. The findings of the analysis of the national assessments and PISA both build on the recommendations of previous World Bank reports and extend their implications. The findings in this report are also confirmed by international research and have clear implications for improving the quality of education in Mexico. Household factors, as well as institutional, school and teaching characteristics, are associated with better performance. Ln the national assessments, the poor performance of indigenous students - very low levels o f learning outcomes and lower returnsto schooling - are documented. 4.2. In general more Autonomy, Accountability and Assessment - the three A's - will help improve the quality of education in Mexico, through improvements in learning outcomes (see also Schmelkes 2001, who calls for decision-making within the school, but with support from the center in matters of evaluation and specific assistance for weaker schools). This will also allow other policy actions to be more effective. That is, more autonomous schools can implement appropriate language policy, thus fulfilling national guidelines for indigenous students. A more accountable system will in turn encourage more active participation by parents and others, which is key to improving learning outcomes. Finally, a system that i s based on constant assessment and participation in international benchmarking exercises will improve cost-effectiveness. Box 4.1 summarizes the recommendations that follow. Increase School Autonomy at Public Schools 4.3. To improve quality, Mexico needs to continue efforts to move decision-making to the school level, thus increasing school autonomy. Results from the analysis of the National Assessments and PISA 2003 show that teacher morale, teacher-student relations, teacher behavior, school climate, teacher expectations of student performance, student's awareness of value of school for future earnings and active learning/teaching styles are all associated with better learning outcomes. Increased autonomy from the state level education secretariats to the local level may give schools the flexibility they need to empower teachers and parents, thus improving the relationships between students and teachers. The results of PISA 2003 suggest that both students and schools perform best in "a climate characterized by high expectations that are supported through strong teacher-student relations, students who are ready to invest effort and who show interest and lower levels o f anxiety with mathematics, and a positive disciplinary climate" (OECD 2004b). In most of the countries that performed well, local authorities and schools also have substantial responsibility for educational content and/or the use of resources, and many set out to teach heterogeneous groups of learners. 4.4. Increasing school autonomy can compensate disadvantaged schools. Two current government programs designed to increase school autonomy are improving school climate and showing signs of positive impacts on learning outcomes. A compensatory education program implemented by CONAFE builds an environment in which a high level of satisfaction and learning at schools exists and the PISA results provide some evidence that the compensatory 50 Box 4.1 A n Agenda for Action The agenda for action proposed in this study i s based on three main principles: (i)increased autonomy from the state-level education secretariats to local schools; (ii)implementation of accountability mechanisms between the federal and state education authorities, local schools; and communities, parents and students; (iii)constant assessment of student performance to inform education policy decisions. 0 Autonomy o To improve quality, Mexico needs to continue efforts to move decision-making from the state level secretariats to the school level, thus increasing autonomy o Increasing school autonomy can compensate disadvantaged schools o Autonomy can help raise the schooling outcomes of indigenous peoples o Schools needthe autonomy to develop locally appropriate education policies o School autonomy reinforces the role that attention given to homework, instruction of effective learning styles and student's perception of the future value of education play in raising student achievement levels o With more autonomy, schools could determine the appropriate mix of resources and technology for their students 0 Accountability o Accountability mechanisms could improve learning outcomes, by involving parents and communities in setting clear goals and visions for the school system o Accountability mechanisms that put people at the center of service provision can go a long way in making services work and improving outcomes o Flexible and wide-ranging accountability mechanisms could encompass various types of services o Continue to increase incentives for school enrollment, while improving accountability 0 Assessment o Assessment testing can be used to inform policy decisions-at the local, state and national levels o Analysis of assessments can foster public and civil society involvement in education reform o State governments should be proactive in encouraging public debate usingassessment results o Mexico has made remarkable efforts to improve assessment of the education system and i s encouraged to continue participating in international achievement tests and expand coverage of national assessments 51 program has improved equity in outcomes. Evaluations o f the program show that the program reduced learning gaps between similarly disadvantaged groups. The Quality Schools Program (Programa Escuelas de Culidad, PEC) also aims to increase parental participation and the empowerment o f the school community (school directors, teachers, parents, students). The most effective elements of the compensatory education program -parental management, on which PEC was based and expanded - could be mainstreamed into the education system. Moreover, a graduation system, in which improved schools are phased out of the compensatory program, could be designed, preferably in coordination with PEC expansion. Compensatory programs are also needed to ensure access to secondary education. The findings from on-going evaluations of these two programs should shed light on the effectiveness o f the programs as well as the impact of increased autonomy on learning outcomes. 4.5. Autonomy can help raise the schooling outcomes of indigenouspeoples. Indigenous peoples exhibit poor education outcomes and low returns to schooling. Therefore, it may be necessary to go beyond just compensatory programs for indigenous communities. A model worth considering i s Escuela Nueva (Box 4.1). The Escuela Nueva model represents a highly innovative reform movement that integrates an active pedagogy, reflective teaching, democratic decision-making, student leadership, cooperative learning and empowerment of teachers and the local community. The model may be particularly relevant for indigenous schools because it gives schools a high degree of autonomy in order to adapt the program for local needs. Multigrade students advance through flexible, but not automatic, promotion. Individual student work, emphasized in traditional schools, i s combined with work in small groups, a feature that could reinforce the collective nature o f most indigenous communities. This model can help address the needs of indigenous children, given its positive record in rural areas of Colombia and expansion to other countries. 4.6. Schools need the autonomy to develop locally appropriate education policies. The analysis shows that there was wide variation o f results by state, socioeconomic status, indigenous groups and types of schools, highlightingthe need to move decision making to the school level. For example, although there i s some variation, overall private schools perform much better than public schools. This may be partly attributed to the high level of autonomy over school resources and educational content that private schools enjoy. In addition, there i s a differentiated effect of technology and school inputs. For example, while computers have a positive effect on math scores among high achievers, calculators have a larger effect on math scores among low achievers than do computers. These examples illustrate that there are many unobserved factors that contribute to learning outcomes and it is difficult for a centralized authority to determine appropriate school interventions. Increasing autonomy would allow schools to determine locally appropriate policies, particularly in the case of schools that serve indigenous students and other disadvantaged schools. 52 Box 4.2 EscuelaNueva The Escuela Nueva program was introduced in rural schools in Colombia in 1975 to address the challenge of providing a complete primary education cycle in rural schools. At the time, half of Colombia's rural schools did not offer a complete primary education cycle and more than half of rural children between the ages of 7 and 9 had never attended school. The program has expanded to include over 27,000 schools. The Escuela Nueva system represents a highly innovative reform movement that integrates an active pedagogy, reflective teaching, democratic decision-making, student leadership, cooperative learning and empowerment of teachers and the local community. These principles are reflected in the daily activities of typical Escuela Nueva schools. For example, multigrade students advance through flexible, but not automatic, promotion. Individual student work, emphasized in traditional schools, is combined with work in small groups. Student work is oriented by self-instructional learning guides in mathematics, reading, science and social studies. Units in text books include learning objectives, guided activities to be completed and free activities, which require application of the knowledge gained. Some involve creative exploration and application of region specific knowledge. I n addition, in-service teacher training is divided into three one-week courses conducted throughout the first school year, designed to provide teachers with the pedagogical skills needed to implement the multi-grade classroom. Analysis shows the cost per student i s higher in Escuela Nueva schools. However, evaluation results indicate that repetition and dropout rates in Escuela Nueva schools are lower than those in the traditional rural schools. Additionally, student academic achievement in Escuela Nueva is higher compared to traditional schools, although this positive difference diminishes in 5th grade. This could be due to the better retention rates in Escuela Nueva (which retains the low achievers) or because this kind of education declines in the uppergrades. Escuela Nueva's success has been attributed to many characteristics-development over time, a structured yet flexible and multi-faceted program adapted to the local context and opportunities for meaningful involvement of students, teachers and community members. The Escuela Nueva reform provides a lesson and model on how policy makers and teachers can better educate their most challenging and poorest communities through innovation, cooperation and a deep understanding of the local context. Sources: McEwan and Benveniste 2001; Psacharopoulos, Rojas and Velez 1993 4.7. School autonomy reinforces the role of homework, learning styles and future value of education. Other important findings concern the role of homework and academic learning styles. There i s a strong relationship between the use o f homework, as well as interest in the subject matter, and academic performance. In addition, an increased understanding of the value of the subject matter in determining future access to jobs and earnings attainment will improve academic performance. Memorization as a way of learning i s not effective. Students need to be shown better and more effective alternatives to study and learn. Schools could be empowered to implement activities on how to be a highly effective learner as well as highlight the importance of education for future employment. 4.8. With more autonomy, schools could determine the appropriate mix of resources and technology for their students. What i s not clear i s the impact of technology on academic outcomes. There is a need to rethink how resources are used and not focus on computer to student ratios or the number of computers or internet access at school as indicators of improved learning outcomes. Technology is useful when it i s integrated in the learning process and when it i s used to improve the pedagogical approach. For example, technology can facilitate the pedagogical processes that will improve outcomes by making, for example, learning by 53 doing and the use of computer simulation technology, feasible learning strategies (Box 4.2). Schools could be given information about how technology can be used to improve the learning process and be given the freedom to adapt available technology to their own circumstances. Mexico's new technology program, Enciclomedia (www.enciclomedia.edu.mx), provides an excellent opportunity to evaluate the impact of technology on academic outcomes, especially if it is linked to the learning process and takes advantage of programs that empower schools through increased school autonomy and introduce flexibility in the school, such as PEC. Technology, in the form of e-learning, may also be a way to close learning gaps for groups typically not served by the education system, such as migrant workers, or for whom access to education is difficult, such as disabled students (Santillan 2004). Box 4.3 EducationTechnology Education technology has evolved from using film projectors in schools to courses offered on-line. The rapid and continuous development of computer technology offers the field of education the possibility of expanding the boundaries of the classroom as well as revolutionizing current pedagogical methods. According to Roger Shank, "it has been understood for a very long time that people learn by doing and that other kinds of learning don't really work all that well." Learning by doing is hard to find in the current education system because it is difficult to implement. But recent research on the brain and learning supports the idea that technology can be used in better ways to help people learn. More specifically, computers and on-line courses have the potential to put experiential learningback at the center of education. The role of technology in experiential education is that of simulation. Academic and computer experts would develop the curriculum and create a computer program that simulates a real world situation and provides feedback to students about the ramifications of the decisions that they make while engaged with the program. In this sense, students would learn what works and what does not work through their own actions instead of reading a chapter out of a book. For example, a business student can either read a case study about a businessthat failed, or they can run a business through a simulation and learn how their decisions cause a business to fail or to thrive. The potential uses for this type of technology in education reaches far and wide and can be practically implemented. I n fact, computer simulated programs are used every day to train pilots how to fly different types of aircraft. Sources: Schank 2001, 2004; OECD 2002 ImproveAccountability 4.9. Accountability mechanisms could improve learning outcomes. The poor overall results in Mexico highlight the need to assign responsibility for improving learning outcomes. In order to improve learning outcomes it is necessary that school empowerment be accompanied by a strengthened accountability framework that enhances social and parental participation in schools. Accountability can strengthen school quality by involving parents and the community more, and by setting clear goals and visions for the school system. This is particularly true for indigenous schools, for which decisions regarding curriculum are made at the federal level with little accountability. If accountability were delegated to the school level with focus on community and parental participation, learning outcomes would probably improve. 4.10. The PISA results show that time spent on homework and student interest in the subject matter are positively associated with learning outcomes, whereas memorization as a way of learning negatively impacts achievement test scores. The effects o f computers and lab 54 equipment have mixed effects, with computers benefiting mostly high achievers as well as overall only improving reading scores and lab equipment only improving science scores. While these factors are not necessarily amenable to changes in policy at the federal level, they would likely be influenced by stronger accountability mechanisms at the school and community level. If schools (teachers and administrators) and their communities (parents and students) were held responsible for results, they would be more likely to make sure that students are responsible for doing their homework, that teachers are more enthusiastic about subject matters and use active pedagogic methods, and that administrators are acquiring the appropriate mix of technology and equipment for their schools. It i s important to note that holding schools and communities accountable for improved results necessitates devolution of decision-making to the school level so that they can initiate and carry-out schoolkommunity appropriate policies. 4.1 1. Accountability mechanisms thatput people at the center of serviceprovision can go a long way in making services work and improving outcomes. More specifically, focusing on people enables them to monitor and discipline service providers and amplifies their voice in policymaking, and strengthens the incentives for providers to serve them. There are three key relationships in the service delivery chain that can be used to strengthen accountability: (1) between beneficiaries and providers; (2) between beneficiaries and policymakers; and (3) between policymakers and providers. There may be a need to look for strategies to strengthen the short route to accountability, the direct influence of beneficiaries on service providers. This could include enhancing client power or the leveraging of parents through choice or voice directly at the school level. Increased parental participation, choice of provider and demand- side financing are all examples of mechanisms that may increase the short route to accountability. Interventions that include choice (funding follows student) increase competition in the market and may improve quality as perceived by parents. Also, when parents control or manage the payments that go to providers, then their likelihood to play a monitoring role could increase; that is, voice would be expected to lead to better quality (through enhanced involvement). 4.12. Flexible and wide-ranging accountability mechanisms could encompass various types of services. For example, some services - such as schooling (measured as enrollment) - could be contracted out. The PISA results show that, in general, private schools achieve higher learning outcomes and this could be due, in part, to their self-management and greater accountability to their clients, via incentives in the form o f tuition payments. Contracting models, whereby poor students at the secondary level are given places in successful private schools, as in the case of targeted scholarships in Colombia and C6te d'Ivoire (Angrist and others 2002; Sakellariou and Patrinos 2004), can be used in the short term to increase school enrollment cost-effectively, while maintaining school quality as long as the schools selected to take in students perform well in standardized assessments. These options have the potential to both improve quality and increase access, especially at the post-compulsory secondary school level. Public finance of private providers i s used by many OECD countries, including Holland and Denmark (OECD 1994; Patrinos 2002), as well as Korea, where it i s combined with their secondary school equalization policy. Such models require adequate information flows to policymakers, providers and parents, and in the case o f Mexico could require extending the sample-based assessment system to cover more schools. However, generalized school choice 55 models are controversial and difficult to implement (Gauri and Vawda 2004; Hsieh and Urquiola 2003; Ladd 2002; McEwan 2000). The suggestions made here are merely short-term, targeted scholarship options. 4.13. Continue to increase incentives for school enrollment, while improving accountability. Other enrollment incentive programs could also benefit from strengthened accountability mechanisms. Mexico has had tremendous success in enrolling and retaining children in primary school through its conditional cash transfer program, Oportunidudes. Rigorous impact evaluations of the program indicate that it has significantly increased the enrollment of children, particularly girls, especially at the secondary school level. The results imply that children will have an average of 0.7 years of extra schooling because of Oportunidudes, although this effect may increase if children are more likely to go on to upper secondary school as a result of the program. Using panel data for Mexico for 1997 to 1999 (Behrman and others 2001; Skoufias and Parker 2001; Schultz 2004) it i s shown that Oportunidudes resulted in higher school attainment among indigenous children, and a significant reduction in the gap between indigenous and non-indigenous children (Bando and others 2005). Results show a significant reduction in the probability that indigenous children work after participation in the program. Similarly, indigenous children had lower school attainment compared to children that either only speak Spanish or are bilingual. The program has been expanded to urban areas and i s supposed to expand to the secondary level, an action that could be encouraged in order to help increase enrollment rates. Implementation of the Youth with Opportunities (Jdvenes con Oportunidudes) initiative, which provides additional resources to poor youth attending and completing upper secondary education, could also motivate youth to stay in school. Similar programs in the United States have been shown to be highly cost-effective (Greenwood and others 1998). 4.14. However, many of the new students entering the system are not prepared and are entering poorly performing schools. Rapid expansion may be showing up in poor results at the lower secondary school level, especially telesecunduriu. Oportunidudes has contributed to great gains in expansion of school enrollment. Nevertheless, telesecunduriu students are not performing satisfactorily in academic achievement tests, especially in writing and mathematics. This points to the need to address quality issues while expanding access through scholarships. Expansion of these programs to higher levels of schooling and to urban areas needs to be accompanied by greater accountability to ensure quality schooling for new students. Continue Learning from Assessments 4.15. Ifincreased autonomy and improved accountability are to leadto policies for achieving higher learning outcomes, then national and international learning assessments can help operationalize accountability and autonomy. In order to improve learning outcomes, countries must first have the capacity to measure levels o f achievement. 4.16. Assessment testing can be used to inform policy decisions. Mexico i s congratulated for its continuingparticipation in international student assessments, especially for the expanded and representative sample at the state level. It is important for Mexico to continue to participate in international assessments such as TIMSS and Luborutorio - but especially PISA 56 -andusethe results to benchmark itsperformance against other countries. As other OECD countries have shown, PISA results can be used to justify education reform initiatives. PISA provides an excellent opportunity for countries to evaluate their education systems, establish benchmarks for future tracking and inform policy responses, as recognized by Mexico in its own analysis of PISA results (INEE 2004b). This has been the case in Singapore and Jordan, where the results are continuously used to reform curriculum, train teachers and conduct research on the determinants of learning. In both cases, the use of the TIMSS results has provided significant returns (Box 4.3). The cost of participation in international assessments i s relatively low and the benefits significant. Box 4.4 UsingInternationalAssessment to Reform andImproveEducation: The Case of Jordan Jordan has benchmarked its education system against other countries in the areas o f education indicators and international achievement tests. This provides Jordan with comparable information by which to analyze systemic progress towards educational advancement and quality. Jordan i s also one of the World Education Indicator Countries, which benchmark their systems to OECD countries. I n 1990, Jordan became the first Arab country to participate in the International Assessment of Educational Progress (IAEP 11). The IAEP I1 study was launched simultaneously with Jordan's effort to undertake a thorough review of its education system that could be used to design a comprehensive reform program. The IAEP process not only provided crucial data on Jordan's educational performance (at the 8th-grade level), but also allowed national educational specialists the opportunity to learn the techniques that such an exercise involves - including sample selection, administration of tests, and implementation monitoring. The IAEP I1study was to be instrumental in building national capacity for independently conducting national surveys of education achievement in the future. Jordan's students ranked near the bottom in IAEP 11. Then Jordan participated in TIMSS in 1999. The results of the study came as a shock. About 75 percent of students in mathematics and about 67 percent of students in science scored lower than the international average. Jordan stood third from the bottom in both subjects among the 20 participating countries. An expert committee was subsequently established to investigate the causes of this poor performance. After an item-by-item examination of the test and school curricula, as well as administration of practice tests, Jordan re-administered the entire TIMSS examination. The results were almost identical to those obtained during the first round of testing. However, the results served to inform efforts to reform educational quality. More specifically, it served to: (a) establish benchmarks of 13-year-olds' achievement relative to 19 countries; (b) identify the areas of strength and weakness in each subject; (c) compare the performance of students in schools run by different authorities, regions and areas; (d) identify cognitive processes and respond with a view to informing teacher training; (e) analyze family and home characteristics associated with student achievement; and (0target negative andpositive influences of classroom practices, out-of school activities, and attitudes. Jordan's example indicates the importance of government commitment to use the results of international assessments for evaluating education systems, establishing benchmarks for future tracking and informing policy responses. Most importantly, the efforts paid off. In 2003, Jordan again participated in TIMSS. The results were impressive. Jordan improved its scores in both math and sciences. In science, Jordan's performance was above average. 4.17. Analysis of assessments can foster public and civil society involvement in education reform. A review of the published media in PISA participating countries suggests that the assessment results had an impact on educational policy making. Media coverage was extensive, especially in countries where the public was not happy with the results, but also in countries where the results were much better (Koda 2004). The media in many countries 57 widely covered PISA, focusing on: the overall rankings, disparity between the 5'h and 95th percentile scores, the effect of socioeconomic background on student performance and gender differences. Countries that did not perform well in PISA used the results to launch education reforms. In the case of Germany, which was dismayed by its poor results, media coverage was extensive and the public sector initiated a major review of its education system (OECD 2004a). Every stakeholder became aware of the PISA results and the reports that were put out by the government calling for a national effort to improve the education system. The report initiated fundamental changes in education policy, transforming the process from input to outcome driven reform. Among the many recommendations, the report emphasized decision-making at the school level. Denmark was also dismayed with the PISA results and asked for a review by the OECD. 4.18. However, governments must be proactive in encouraging public debate using assessment results. Interestingly, there are some countries that continue to participate in international assessments with limited public outcry and little debate among policymakers of the poor results over time in PISA, TIMSS and other international achievement tests. One may conclude that without proper use of the results - media coverage, public debate, sector review and policy reform - it i s unlikely that outcomes will improve over time. But countries that acknowledge the results, engage in public debate, conduct serious analysis of the results and launch programs to improve outcomes, will see improvements over time (for example, Jordan). This finding includes perennial top achievers such as Japan, which analyzed tests results and found that students were relatively stronger in computational skills, over analysis and applications. In Japan, this led to a change in the curriculum to reduce the emphasis on computational skills. 4.19. Mexico has made remarkable efforts to improve assessment of the education system. Mexico i s encouraged to continue participating in international achievement tests, as well as improve and expand the national system (Esta'ndares Nacionales). The assessment results need to be analyzed continuously and the results used to inform policy decisions. Assessment will also strengthen accountability measures. More information flowing to policy makers, the general public, communities, parents and schools will contribute to improving overall quality and learning outcomes. It i s recommended that INEEundertake constant analysis of PISA and other assessments in order to inform policymakers and guide reforms. 4.20. Mexico could expand coverage of the National Assessments. To ensure that information i s widely available, it may be necessary to expand the reach of the national assessments so that more schools are covered. This would make it possible to implement innovative initiatives such as those outlined above, as well as to ensure that more information flows to parents, especially if some form of public dissemination of results i s envisioned. The expanded PISA 2003 sample in Mexico, which i s representative at the state level and by school type, i s an excellent tool for analyzing differences by region and school type. Thus, deeper analysis of such information can help decision makers design appropriate policies. States, too, can conduct state-specific analysis for future tailoring of local education policies. 4.21. National and international assessments could be used to inform the secondary school reform process currently in progress. The Mexican Government i s adequately focused on 58 access to basic education, and has much to show for its efforts over the last ten years. Upper secondary education, in contrast, needs to become a key policy challenge for the coming years because Mexico's net secondary enrollment rate ranks among the lowest of upper-middle income Latin American countries and falls below the average for Latin America (62 percent compared to 65 percent). The government needs to address the challenge of improving access to upper secondary, while improving its quality and relevance. Mexico is currently undertaking a process of secondary education reform (Box 4.4). Expansion of coverage will help improve quality - or the yield - as more people complete their education in an improving system. Specific actions to improve access, enrollment, attendance and completion are needed. Secondary education, however, presents specific structural problems that point to the need for profound reforms of the curriculum and organization of secondary education, and assessments can be used to identify strengths and weaknesses of the current system. Moreover, given the poor results of certain modalities of secondary education - particularly telesecundurius, but also all types o f lower secondary, especially for 15-year-olds - there i s a need to investigate the reasons for poor performance. According to SEP, Mexico will not be able to improve their results in PISA without secondary school reform. In a speech given at the OECD Forum on "Mexico: Policies to Promote Growth and Economic Development," held in 2004, the Secretary of Public Education emphasized that the current "study plan and pedagogical structure that we have at the secondary school level is not adequate." Box 4.5 Secondary School Reform In 1993,secondary school attendancebecame mandatory. Butby the end of the decade, it was clear that, while progress was made in terms of coverage and completion, there were still significant gaps. For example, in 2000, 20 percent of the students of secondary school age were not enrolled in school and 20 percent of students enrolled in secondary school repeated a grade. Infact, according to Dr. Reyes Tamez Guerra, Mexico's Secretary of Public Education, at least 98 percent of the students who repeat a grade never finish secondary school. The secondary education curriculum has been characterized as follows: (a) it contributes to repetition because of the breadth of its subject matter relative to available time; (b) its content is encyclopedic and tends to include highly specialized material; (c) it i s fragmented (12 subject areas at the same time); and (d) it is not designed with the adolescent in mind. To address the problems with the secondary education system, SEP's 2001-2006 Education Program proposed the design and implementation of secondary education reforms. The Integral Secondary Education Reforms aim to guarantee the right to a quality education for all Mexicans. Among the objectives of the reforms are: (a) achieve universal secondary school enrollment rate; (b) reduce drop-out and failure rates to reasonable levels; (c) improve schooling achievement outcomes; (d) facilitate sharing of lessons learned in pedagogical methods and strategies for implementing policy; and (e) design curriculum modules that meet needs of students of different backgrounds. It is in this last area that major changes have been proposed. Studentscurrently study 11 to 12 different courses every year, placing incredible strain on both teachers and students. The reforms propose to replace the curriculum with one that offers eight courses each year. Up to three-quarters of the time would be spent in courses emphasizing four main areas: reading and writing, mathematics, sciences and technology, and history/geography/civics. The remaining class time would be spent on second language acquisition, physical education, fine and performing arts and various regional, state, or local requirements. This reduced course load has clear benefits. It takes pressure off teachers to be prepared for so many subjects, and enables them to concentrate on what is most important and spend more time with students. Students benefit from having more Contact with fewer teachers, creating an environment of stability in which to learn. Yource: www.sep.gob.mx 59 Additional Actions for Improving School Quality inMexico 4.22. Stronger efforts to increase capacity in Mexico to analyze learning outcomes data and evaluated programs are needed. Not only should the National Institute for Education Evaluation (Instituto Nacional para la Evaluaci6n de la Educacidn, INEE) continue its important task o f disseminating and reporting outcome data, but more rigorous analysis of the relationships and determinants could be undertaken. Mexico's Secretariat for Public Education (SEP) is managing many important programs. These need to be evaluated on a continuous basis. To do this, it is recommended that permanent capacity within SEP be built for analytical work, including impact evaluations on a periodic basis. Areas for Further Research 4.21 As Mexico continues to use incentives for school enrollment - through Oportunidudes - rigorous evaluation of the impact on learning is needed. In addition, since many Oportunidudesbeneficiaries are students in schools received compensatory education support, then the interactionbetween demand-side financing programs (Oportunidudes) and supply-side quality improvement programs (SEP's compensatory education program implemented by CONAFE) may also be evaluated. In addition, the main program that attempts to empower schools through enhanced autonomy and parental participation - Quality Schools Program (Progrumu Escuelus de Culidud, PEC), could be evaluated, including an assessment of how to improving accountability measures in this and the compensatory program. In addition, further researchi s needed on: 0 Impact of health and nutrition programs, especially initial, pre-school and early childhood development (ECD) programs on learning outcomes. Mexico i s undertaking an expansion of compulsory pre-school education. At the same time, there are a number of ECD programs already in operation. It would be worthwhile to evaluate the impact of such programs, as well as health and nutrition interventions that are part of Oportunidudes' early interventions, and their likely effect on basic education outcomes. 0 Low learning achievements associated with telesecundarias and the impact of using technology in the classroom. Many of the new students entering the system are enrolled in telesecundarias. These students are often ill-prepared for secondary education and the schools they enter are producing poor learning outcomes. It i s important to establish the reasons for such poor performance and take appropriate actions to improve outcomes. Also, technology in the classroom, combined with effective teaching practices, has the potential to improve learning outcomes. However, the international literature does not find a strong effect of technology on learning outcomes. Mexico's new programs offer the potential to evaluate the impact of new technologies on learning outcomes. e Impact of increased enrollments on learning outcomes. In general, given the need to increase enrollments, more work i s needed on the impact o f new students on learning 60 outcomes. This i s important because there may be a need to differentiate the type of schooling and support (for example, compensatory) that such students may need. e Zntersectoral links and education quality. While this report highlightedthe main labor market outcomes of education, and estimated the returns to quality, more work i s needed on the relevance of education in urban and rural areas. Also, the importance of education for the private sector could also be investigated, and the links between school quality and labor market productivity and international competitiveness could be researched. e Barriers to reform. Among the top priorities are institutional factors that prevent further improvements in learning and system reform, including the role of teacher unions and teacher training practices. e Zdentihing best practices. This i s especially related to understanding how and why some schools have better learning environments than others, and how to use educational and instructional material effectively at schools. 61 REFERENCES Abdul-Hamid, H. 2005. "Mexico's Performance in PISA." Background report prepared for the Mexico Quality of Education Study. World Bank (processed). Abdul-Hamid, H. 2003. "What Jordan needs to do to prepare for the knowledge economy: Lessons learned from TIMMS-R." University of Maryland (processed). Anderson, J.B. 2002. "The Effectiveness of Special Interventions in Latin American Public Primary Schools." The Dante B. Fascell North-South Center Working Paper Series, No. 5. University of Miami. Anderson, J.B. 2000. "Factors Affecting Learning of Mexican Primary Schools Children." Estudios Econdmicos 29(1): 117-152. Angrist, J., E. Bettinger, E. Bloom, E. King and M. Kremer. 2002. "Vouchers for Private Schooling in Colombia: Evidence from a RandomizedNatural Experiment." American Economic Review 92(5): 1535-1558. Angrist, J. and V. Lavy. 2002. "New Evidence on Classroom Computers and PupilLearning." Economic Journal 112(482): 735-765. Armor, D., P. Conry-Osequera, M. Cox, N. King, L. McDonnell, A. Pascal, E. Pauly and G. Zellman. 1976. "Analysis of the School Preferred Reading Program in Selected Los Angeles Minority Schools." Rand Corporation Report No.R-2007-LAUSD7 Santa Monica, CA. Bando, R., L.F. Lopez-Calva and H.A. Patrinos. 2005. "Child Labor, School Attendance, and Indigenous Households: Evidence from Mexico." World Bank Policy Research Working PaperNo. 3487, Washington, D.C. Barro, R.J. 2001. "Human Capital and Growth." American Economic Review, Papers and Proceedings91(2): 12-17. Behrman, J., P. Sengupta and S. Parker. 2005. "Progressing through PROGRESA: An Lmpact Assessment of a School Subsidy Experiment in Mexico." Economic Development and Cultural Change(forthcoming). Boozer, M.A., A.B. Krueger and S. Wolkon. 1992. "Race and School Quality since Brown v. Board of Education." NBERWorking Paper No. 4109, Cambridge, MA. Buchinsky, M. 1998. "The Dynamic of Changes in the Female Wage Distribution in the U.S.: A Quantile Regression Approach." Journal of Applied Econometrics 13(1):1-30. 62 Card, D. and A. Krueger. 1992. "Does School Quality Matter? Returns to Education and the Characteristics of Public Schools in the United States." Journal of Political Economy lOO(1): 1-40. Coleman, J. 1966. Equality of Educational Opportunity. Washington, D.C.: U.S. Department of Education. Fernandez, T. 2003. "Detemzinantes sociales y organizacionales del aprendizaje en la educacidn primaria de Mkxico en Tres estudios sabre determinantes del rendimiento escolar." Instituto Nacional para la Evaluacih de la Educacibn, MCxico. de Ferranti, D., G.E. Perry, I.Gill, J.L. Guasch, W. Maloney, C. SBnchez-PBramo and N. Schady. 2003. Closing the Gap in Education and Technology. The World Bank, Washington, D.C. Fertig, M. 2003. "Who's to Blame? The Determinants of German Students' Achievement in the PISA 2000 Study." IZA Discussion No. 739, Bonn. Fertig, M. and C.M. Schmidt. 2002. "The Role of Background Factors for Reading Literacy: Straight National Scores inthe PISA 2000 Study." IZA Discussion No. 545, Bonn. Fuchs, T. and L. Woessmann. 2004a. "What Accounts for the International Differences in Student Performance? A Re-examination Using PISA Data." CESifo Working Paper No. 1235, Center for Economic Studies and If0 Institute for Economic Research, Munich. Fuchs, T. and L. Woessmann. 2004b. "Computers and Student Learning: Bivariate and Multivariate Evidence on the Availability and Use o f Computers at Home and at School." CESifo Working Paper No. 1321, Institute for Economic Research, Munich. Garcia, V., F. Knaul and H.A. Patrinos. 2005. "Returns to Education and Quality of Education in Mexico." World Bank (processed). Garcia, V. and H.A. Patrinos. 2005. "Returns to Schooling across the Earnings Distribution: Indigenous and non-Indigenous People inMexico." World Bank (processed). Gauri, V. and A. Vawda. 2004. "Vouchers for Basic Education in Developing Economies: An Accountability Perspective." World Bank Research Observer 19(2): 259-280. Glewwe, P. 2002. "Schools and Skills in Developing Countries." Journal of Economic Literature (40) 2: 436-482. Glewwe, P. 1999. "Why Does Mother's Schooling Raise Child Health in Developing Countries? Evidence from Morocco." Journal of Human Resources 34(1): 124-159. 63 Goolsbee, A. and J. Guryan. 2002. "The Impact of Internet Subsidies in Public Schools.'' NBERWorkingPaper Series No. 9090, NationalBureau of Economic Research. Greenberg, E. 2004. "Climates for Learning." Paper presented at the American Educational Research Association 2004 Annual Meeting, April 12-16, San Diego, CA. Greenwood, P.W., K.E.Model, C.P. Rydell and J.R. Chiesa. 1998. Diverting Childrenfrom a Life of Crime: Measuring Costs and Benefits. SantaMonica: RAND. Hanushek, E.A. 2004a. "United States Lessons About School Accountability." Journal for Institutional Comparisons. Center for Economic Studies and the If0 Institute for Economic Research, Munich. Hanushek, E.A. 2004b. "What ifThere Are N o Best Practices?" Scottish Journal of Political Economy 51(2): 156-172. Hanushek, E.A. 1986. "The Economics of Schooling: Production and Efficiency in Public Schools." Joumal of Economic Literature 24(3): 1141-78. Hanushek, E.A. and J.A. Luque. 2003. "Efficiency and Equity in Schools around the World." Economics of Education Review 22(5): 481-502. Hanushek, E.A. and D.D. Kimko. 2000. "Schooling, Labor Force Quality, and the Growth of Nations." American Economic Review 90(5): 1184-1208. Heneveld, W. and H.Craig. 1996. Schools Count. World Bank Technical Paper Number 303, Africa Technical Department Series, Washington, D.C. Hsieh, C.-T. and M. Urquiola. 2003. "When Schools Compete, How Do They Compete? An Assessment of Chile's Nationwide School Voucher Program." NBER Working Paper 10008, National Bureau of Economic Research, Cambridge, MA. Instituto Nacional para la Evaluacion de la Educacion. 2004a. Panorama Educativo de Mexico: 2003 Indicadores del Sistema Educativo Nacional. Mexico: Instituto Nacional para la Evaluacidn de la Educacidn. Instituto Nacional para la Evaluacion de la Educacion. 2004b. Resultados de las Pruebas PISA 2000 y 2003 en Mkxico: Habilidades para la Vida en Estudiantes de 15 Aiios. Mexico: Instituto Nacional para la Evaluacidn de la Educaci6n. Instituto Nacional para la Evaluacion de la Educacion. 2005. La Telesecundaria Mexicana: Desarrollo y Problema`ticaActual. Mexico: Instituto Nacional para la Evaluacidn de la Educacidn. 64 Kirkpatrick, H. and L.Cuban. 1998. "Should we be worried? What the Research says about Gender Differences in Access, Use, Attitudes, and Achievement with Computers. Educational Technology 38(4): 56-61. Koda, Y. 2004. "The Media Coverage on the Programme for International Student Assessment (PISA)." World Bank (processed.) Krueger, A.B. and C.E. Rouse. 2004. "Putting Computerized Instruction to the Test: A Randomized Evaluation of a Scientifically Based Reading Program." NBER Working Paper No. ~ 1 0 3 1 5 . Ladd, H.F. 2002. "School Vouchers: A Critical View." Journal of Economic Perspectives 16(4): 3-24. Lee, J.-W. and R.J. Barro. 2001. "Schooling Quality in a Cross-Section of Countries." Economica 68(272): 465-88. Leuven, E., M. Lindahl, H. Oosterbeek and D.Webbink. 2003. "The Effect of Extra Funding for Disadvantaged Students on Achievement." SCHOLAR working paper 39/03. LeVine, R.A., S.E. LeVine and B. Schnell. 2001. "Improve the Women: Mass Schooling, Female Literacy, and Worldwide Social Change." Haward Educational Review (Spring): 1-50. Lockheed, M. and A. Verspoor. 1991. Improving Primary Education in Developing Countries. New York: Oxford University Press. Lopez-Acevedo, G. 2001. "Evolution of Earnings and Rates of Returns to Education in Mexico." World Bank Policy Research Working Paper No. 2691. Washington, DC. Lopez-Acevedo, G. ndl. "Public Expenditure on Education, Marginal Willingness to Pay for Education and the Determinants of Enrollment inMexico." World Bank (processed). McEwan, P.J. 2000. "The Potential Impact of Large-scale Voucher Programs." Review of Educational Research 70(2): 103-149. McEwan, P.J. and L.Benveniste. 2001. "The politics o f rural school reform: Escuela Nueva in Colombia." Journal of Education Policy 16(6): 547-559. McEwan, P. and L. SantibaAez. 2004. "Teacher and Principal Incentives in Mexico" (Processed). The World Bank, Washington, D.C. McGaw, B. 2004. "Education Strategies to Give Students Knowledge and Skills for Labour Market and Further Education." Presentation at the International Seminar on Education and Competitiveness, November 4-5, 2004, Mexico City, Mexico. 65 Murnane, R.J. 1975. Impact of School Resources on the Learning of Inner City Children. Cambridge, MA: Ballinger. Mwabu, G. and T.P. Schultz. 1996. "Education Returns across Quantiles of the Wage Function: Alternative Explanationsfor Returns to Education by Race in South Africa." American Economic Review 86(2): 335-39. OECD. 2004a. What makes School Systems Pe@orm? Seeing School Systems Through the Prism of PISA. Paris: OECD. OECD. 2004b. "Top-performer Finland Improves Further in PISA Survey as Gap between Countries Widens." Paris: OECD. OECD. 2002. Understanding the Brain: TowardsA New Learning Science. Paris: OECD. OECD. 2001a. Knowledge and Skills for Life: First Resultsfrom the OECD Programme for International StudentAssessment. Paris: OECD. OECD. 1994. School: A Matter of Choice. Paris: OECD. Ontiveros, M. 1998. "Eficiencia del Gasto Educativo. Una Evaluacidn utilizando la funci6n de producci6n Educativa." El Trimestre Econo'mico 64(4) 535- 555. OREALC. 1998. "Primer Estudio Internacional Comparativo sobre lenguaje, matemiticas y factores asociados en tercer0 y cuarto grado." Laboratorio Latinoamericano de Evaluaci6n de la Calidad de la Educaci6n. Santiago: OREALC. Paqueo, V. and G. Lopez-Acevedo. 2003. "Supply-side School Improvement and the Learning Achievement of the Poorest Children in Indigenous and Rural Schools: the Case of PARE." World Bank Policy Research Working Paper No. 3172. Washington, DC. Patrinos, H.A. 2002. "Private Education Provision and Public Finance: The Netherlands as a Possible Model." Occasional Paper No. 59, National Center for the Study of Privatization in Education, Teachers College, Columbia University. Patrinos, H.A. and S. Metzger. 2004. "Returns to Education in Mexico: An Update." World BanWniversidad de las AmCricas, Mexico (processed). Patrinos H.A. and G. Psacharopoulos. 1995. "Educational Performance and Child Labor in Paraguay." International Journal of Educational Development 15(1): 47-60. Patrinos, H.A. and C.N. Sakellariou. 2004. "Schooling and Labor Market Impacts of a Natural Policy Experiment." World Bank Policy ResearchWorking PaperNo. 3460. Psacharopoulos, G. and H.A. Patrinos. 2004. "Returns to investment in education: a further update." Education Economics 12(2): 111-134. 66 Psacharopoulos, G., C. Rojas and E. Velez. 1993. "Achievement evaluation of Colombia's Escuela Nueva: I s multigrade the answer?" Comparative Education Review 37(3): 263- 276. Ramirez, A. 2005. "Mexico," in G. Hall and H.A. Patrinos, eds., Indigenous People, Poverty and Human Development in Latin America: 1994-2004. World Bank, forthcoming. Raymond, M. and E. Sadoulet. 2003. "Educational Grants Closing the Gap in Schooling Attainment between Poor and Non-Poor." CUDARE Working Papers, No. 986. Department of Agricultural and Resource Economics, University of California, Berkeley. Rockoff, J.E. 2004. "The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data." American Economic Review 94(2): 247-252. Sakellariou, C.N. and H.A. Patrinos. 2004. "Incidence Analysis of Public Support to the Private Education Sector in C6te d'Ivoire." World Bank Policy Research Working Paper No. 3231, Washington, D.C. Santillan, M. 2004. "Communities at Disadvantage: Reinventing ICT Based Leaming Opportunities," in Promoting Equity Through ICT in Education: Projects, Problems, Prospects,inA. Karpati, (ed.), Budapest: OECD. Schank, R.C. 2001. "Educational Technology: The Promise and the Myth." Camegie Mellon University. (processed). Schank, R.C. 2004. "El Papel de la Tecnologia en la Educaci6n." Presentation at the Primera Reuni6n Nacional Plenaria Ordinaria del Consejo Nacional de Autoridades Educativas, June 24, 2004, Aguascalientes, Mexico. Schmelkes, S. 2001. "School Autonomy and Assessment in Mexico." Prospects 31(4):575- 586. Schmelkes, S. 2000. "Education and Indian Peoples in Mexico: An Example of Policy Failure,`` In Unequal School, Unequal Chances. The Challenges to Equal Opportunity in theAmericas. F.Reimers (ed.). Boston: Harvard University Press. Schmelkes, S. 1997. La Calidad de la Educacidn Primaria. Un Estudio de Caso. Mkxico: SEPECE. Schultz, T.P. 2004. "School subsidies for the poor: evaluating the Mexican Progresa poverty program." Journal of DevelopmentEconomics 74(1): 199-250. Secretaria de Educaci6n Ptiblica. 2001. Informe de Labores: 1999-2000. Mexico: SEP. 67 Sen, A. 1999. Developmentas Freedom. New York: Alfred Knopf. Shapiro, J. and J. Moreno Trevino. 2004. "Compensatory Education for Disadvantaged Mexican Students: An Impact Evaluation Using Propensity Score Matching." World Bank Policy ResearchWorking Paper No. 3334, Washington, D.C. Skoufias, E. and S. Parker. 2001. "Conditional Cash Transfers and their Impact on Child Work and School Enrollment: Evidence from the PROGRESA Program in Mexico." Economia 2(1): 45-96. Torres, R. and E. Tenti. 2000. Politicas Educativas y Equidad en Mexico: La Experiencia de la Educacidn Comunitaria, la Telesecundaria y 10s Programas Compensatorios. Buenos Aires: UNESCO. Valijarvi, J., P. Linnakyla, P. Kupari, P. Reinikainen and I.Arffman. 2002. The Finish Success in PISA - and Some Reasons behind It. Jyvaskyla: Institute for Educational Research. Velez, E. and G.Lopez-Acevedo. 2002. "Quality of Basic Education in Mexico." World Bank (processed). Walker, I. and Y. Zhu. 2003. "Education, Earnings and Productivity-Recent UKEvidence." Labour Market Trends (March). Wenglisky, H. 1998. Does it Compute: The Relationship Between Educational Technology and StudentAchievement in Mathematics. Princeton, NJ: Educational Testing Service. Woessmann, L. 2004. "The Effect Heterogeneity o f Central Exams: Evidence from TIMSS, TIMSS-Repeat and PISA." CESIFO Working Paper No. 1330. Center for Economic Studies and IF0Institutefor Economic Research, Munich. Woessmann, L. 2003. "Schooling Resources, Educational Institutions, and Student Performance: The International Evidence." Oxford Bulletin of Economics and Statistics 65(2): 117-170. Wolfe, B. and R. Haveman. 2001. "Accounting for the Social and Non-market Benefits of Education," in The Contribution of Human and Social Capital to Sustained Economic Growth and Well-being, J. Helliwell (ed.), OECDMuman Resources Development Canada. Vancouver: University of BritishColumbia Press. Wolff, E.N. and M. Gittleman. 1993. "The role o f education in productivity convergence: does higher education matter?" in A. Szirmai, B. van Ark and D. Pilat (eds.), Explaining Economic Growth. Amsterdam: North-Holland. World Bank. 2004. World Development Report: Making Services for Poor People. The World Bank, Washington, D.C. 68 World Bank. 2003. Lifelong Learning in the Global Knowledge Economy: Challenges for Developing Countries. The WorldBank, Washington,D.C. World Bank. 2000a. Mexico: Transforming School into Effective and Efficient Learning Centers (Report No. 20593-ME), Latin America and the Caribbean, Human Development. The World Bank,Washington, D.C. Yonker, M.M. 2004. "Acercamiento a1 Impact0 de la Polftica de Educacidn Intercultural Bilingiie en MCxico." Banco Mundial y Coordinacidn General de Educaci6n InterculturalBilingue,SEP. Zamudio, A. 2001. "Educacih y la distribucidn condicional del ingreso: una aplicacidn de regresi6ncuantil." El Trimestre Econdmico 68(269): 39-70. 69 Annex Table la. Levels of Reading Competencies RetrievingInformation Interpretingtext Reflectionand Evaluation Locate and possibly sequence or Either construethe meaning of Critically evaluate or - combine multiple pieces of nuancedlanguageor hypothesize, drawing on deeply embedded information, demonstrate afull and detailed specialized knowledge. Dea some of which may be outside understanding of a text. with concepts that are E the main body of the text. Infer contrary to expectations anc c;l which information inthe text in draw on a deep relevant to the task. Deal with understanding of long or highly plausible andor extensive complex texts. (competing information. Locate and possibly sequence or b s e ahigh level of text-based britically evaluate or erence to understand and thesize, drawing on or form. Infer which with ambiguities, ideas re contrary to expectation s that are negatively hrase. Compare, contrast or e a companson or rmation is not prominent erience and attitudes. low-level inferences are May be able to read, but have not acquired the skills to use reading for learning 70 Annex Table lb. Levels of Math Competencies Level 6 Students can conceptualize, generalize, and utilize information based on their investigations and modeling of comp-lex problem situations. They can link different information sources and representations and flexibly translate among them. Students at this level are capable of advanced mathematical thinking and reasoning. These students can apply this insight and understanding, along with a mastery of symbolic and formal mathematical operations and relationships, to develop new approaches and strategies for attacking novel situations. Students at this level can formulate and precisely communicate their actions and reflections regarding their findings, interpretations, arguments, and the appropriateness of these to the original situations. Level 5 Students can develop and work with models for complex situations, identifying constraints and specifying assumptions. They can select, compare, and evaluate appropriate problem-solving strategies for dealing with complex problems related to these models. Students at this level can work strategically using broad, well-developed thinking and reasoning skills, appropriately linked representations, symbolic and formal characterizations, and insight pertaining to these situations. They can reflect on their actions and can formulate and communicate their interpretations and reasoning. Level 4 Students can work effectively with explicit models for complex concrete situations that may involve constraints or call for making assumptions. They can select and integrate different representations, including symbolic ones, linking them directly to aspects of real world situations. Students at this level can utilize well-developed skills and reason flexibly, with some insight, in these contexts. They can construct and communicate explanations and arguments based on their interpretations, arguments and actions. Level 3 Students can execute clearly described procedures, including those that require sequential decisions. They can select and apply simple problem-solving strategies. Students at this level can interpret and use representations based on different information sources and reason directly from them. They can develop short communications reporting their interpretations, results and reasoning. Level 2 Students can interpret and recognize situations in contexts that require no more than direct inference. They can extract relevant information from a single source and make use of a single representational mode. Students at this level can employ basic algorithms, formulae, procedures or conventions. They are capable of direct reasoning and making literal interpretations of the results. Level 1 Students can answer questions involving familiar contexts where all relevant information is present and the questions are clearly defined. They are able to identify information and to carry out routine procedures according to direct instructions in explicit situations. They can perform actions that are obvious and follow immediately from the given stimuli. 71 Zountry for means groups) for variance betweenall Mexico 108.6a groups 1.1 between all Argentina 99.0a groups 1.5 not between Peru 58.0a 2&3 22.7a not between Chile 116.0a 2&1 0.5 between all Brazil 214.1a groups 23.6a not between Italy 17.7a 2&3,3&4 1.4 only between Korea 32.2a 1&others 3.0 not between Spain 33.7a 2&3 2.3 between all U.S.A 1O8.la groups 4.8a between all kortugal 95.3a groups 1.5 Source: PISA 2000 a statistically significant at the 0.001 level statistically significant at the 0.05 level 72 Annex Table 3. Variables used in the analvsis. basic statistics Girls Boys Variable (N=7552) (N=7 162) Mean Std. Dev. Mean Std. Dev Student Experience Measure of skill in reading 493.48 82.15 482.55 83.64 Measure of skill on mathematics 483.74 64.29 489.04 65.78 Health Problems Index -0.17 1.14 -0.09 1.20 CENDIpre-school education 0.05 0.21 0.06 0.25 Pre-school education 0.88 0.32 0.88 0.32 Change of school on last year 0.30 0.46 0.34 0.47 Student has repeatedat least one primary grade 0.14 0.35 0.19 0.40 Student like reading 0.95 0.22 0.90 0.30 Student has books in her or his home 0.68 0.47 0.66 0.47 Student has computer in his or her home 0.23 0.42 0.26 0.44 Student works 0.04 0.20 0.12 0.33 Family Background Mother's scholar years of education 7.61 3.57 7.75 3.63 House has potable water 0.97 0.16 0.98 0.15 House has sewerage 0.7 1 0.45 0.73 0.45 House has electricity 0.94 0.23 0.94 0.23 House has gas fire 0.87 0.34 0.86 0.35 House has refrigerator 0.68 0.47 0.71 0.45 House has washing machine 0.68 0.47 0.71 0.45 House has telephone 0.51 0.50 0.54 0.50 Dwellings services factor index 0.14 1.74 0.24 1.75 Family support to continue studding 0.98 0.10 0.98 0.09 School Characteristics Teacher works at the classroom 0.95 0.23 0.93 0.26 Teacher helps when student does not understand 0.97 0.17 0.95 0.21 Teacher was concern about student learning 0.98 0.13 0.97 0.17 Teacher check homework and correct mistakes 0.94 0.23 0.92 0.27 Homework was interesting for the student 0.96 0.19 0.94 0.23 Additional auxiliary texts 0.74 0.44 0.75 0.43 Factor index of pedagogical practices of teacher 0.19 1.21 0.05 1.39 School security index 2.33 0.70 2.37 0.69 Rural School 0.40 0.49 0.39 0.49 Private School 0.09 0.28 0.10 0.30 Indigenous School 0.12 0.32 0.12 0.33 North Region 0.33 0.47 0.32 0.47 Center Region 0.41 0.49 0.41 0.49 South Region 0.23 0.42 0.24 0.42 IMexico City School 0.02 0.15 0.03 0.18 Source: StandardizedAchievement Test Scores, SEP 73 Annex Table 4. Explaining Educational Achievement (Standard Regression Coefficients) Variables Girls Boys Reading Mathematics Reading Mathematics Student Experience Repetition -0.104" -0.119* -0.109" -0.119" Health problems -0.134* -0.110* -0.160" -0.014" CENDI -0.069" -0.073 -0.079" -0.083" Preschool 0.065" 0.053" 0.071* 0.059" Work -0.065" -0.053 -0.073" -0.072* Family Background Mother Education 0.097" 0.089" 0.077" 0.071* Services at home 0.086" 0.074* 0.047" 0.042" Parental support 0.025" 0.021" 0.029" 0.041* Books in home 0.030* 0.030" 0.010 0.020" Computer at home 0.069" 0.058" 0.047" 0.028" School Characteristics Pedagogy in classroom 0.033" 0.036* 0.055* 0.042" School security 0.079" 0.075" 0.089" 0.087" Rural school -0.033* -0.007 -0.070" -0.040" Private school 0.128" 0.071* 0.131" 0.083" tndigenous school -0.089" -0.098" -0.082* -0.109 Northern Region 0.038" 0.045* -0.040"" -0.018 Central Region 0.035 0.051 ** -0.040*" 0.006 Southem Region 0.031 0.032 -0.020 -0.007 Sample size 13,665 13,665 13,142 13,142 R-Square 0.24 0.18 0.24 0.19 R-Square adjusted 0.25 0.18 0.24 0.19 Source: Academic Achievement Survey, So Grade, 2001. SEP. Note:* Coefficientsignificant at the .5 level;** coefficientsignificant at the .10 level. 74 Annex Table 5. Variables usedin the analyses and descriptive statistics Label Mean Min. Max. Mother's education above Secondary 30% 0.00 1.00 Mother i s working 37% 0.00 1.00 Home educationalresources (index), basedon availability in home of dictionary, quiet placeto study, desk for study, books, calculators -0.88 -4.3 0.7 School is in a largecity 21% 0.00 1.00 School is in a mediumcity 17% 0.00 1.00 765 33 6378 Percentageof girls at the school 51% 0.00 1.00 Time spent on homework (index), derivedform amount student devotes to homework per week 5.85 0 30 Sense of belonging (index), derived from students' reportson their level of agreement with following statements: Ifeel like an outsider (or left out of things); Imake friends easily; Ifeel like Ibelong; Ifeel awkward and out of 0.08 -3.40 2.2 place; other students seemto like me; Ifeel lonely Memorization (index), derivedfrom frequency with which students usedthe following strategieswhen studying: Itry to memorizeeverything that might be covered; Imemorize as much as possible; Imemorize all new material so 0.56 -3.5 3.3 that Ican recite it; Ipracticeby saying the materialto myself over and over Teacher behaviors (index), derived from principals' reportson extent to which leaming by 15-year-oldswas hinderedby: low expectations of teachers; poor student-teacher relations; teachers not meetingindividual students' needs; teacher absenteeism; staff resistingchange; teachers being 0.67 -2.41 3.58 too strict with students; students not being encouraged to achievetheir full potential Teacher morale (index), derived from the extent to which school principals agreed with the following statements: the moraleof the teachers in this schoolis high; teachers work with enthusiasm; teachers take pride in this -0.02 -2.8 1.7 school; teachers value academic achievement Studentuses computersat school (dummy that measureswhether a student uses computer at school several times a week or several times a month) 30% 0.00 1.00 Studentuses the Intemet at school (dummy that measureswhether a student uses the Internetseveral time a week or severaltimes a month) 22% 0.00 1.00 Instrumental motivation (index), derived from the frequency with which students study for the following reasons: to increasemyjob opportunities; to insurethat my future will be financially secure; to get a goodjob 0.07 -2.44 1.48 Schooleducationalmaterial -0.4 -3.2 2.2 I Source: PISA 2003 75 innex Table 6. EducationProduction Function (based on GLS with School Fixed Effects) Math Science Reading Student is female -21.4 9.3 -22.0 (16.3) (6.7) (16.5) Mother education above secondary 1.7 2.1 8.1 (2.2) (2.4) (5.4) Mother is working 2.0 1.2 1.3 (1.4) (0.9) (0.9) Home educational resources 6.8 7.2 6.2 (11.1) (11.5) (9.9) School is Private 28.1 27.1 23.2 (5.6) (5.2) (5.0) Percentage o f girls at school 38.8 32.9 25.0 (2.4) (2.0) (1.7) School in medium city 17.9 20.9 12.9 (4.7) (5.3) (3.7) School in large city 22.8 24.2 16.8 (4.9) (5.0) (4.0) School size 0.01 0.01 0.01 (6.5) (6.6) (6.9) School educational material 0.5 1.o 1.5 (0.4) (2.7) (1.1) Time on homework 2.1 2.2 2.4 (17.6) (16.9) (18.7) Instrumental motivation (index) 2.8 -3.5 3.7 (3.2) (4.0) (4.0) Memorization (index) 0.5 -1.6 -0.1 (0.7) (2.0) (0.1) Teacher morale (index) 4.2 3.9 2.4 (3.0) (2.7) (2.3) Student uses computer often at school -5.7 2.9 -6.6 (3.5) (1.8) (4.2) Student uses Internet often at school -5.7 -3.5 -5.6 (4.1) (2.4) (4.0) Student self-belonging (index) 1.4 5.1 1.2 (2.0) (7.4) (1.6) Constant 375.6 380.0 398.7 (43.9) (43.0) (50.4) N 13,565 13,565 13,565 Yource: PlSA 2003 Vote: all are significant at the 0.05 level except these in bold 76 Annex Table 7a. Quantile Analysis Model (Science) Science Q l O Q25 Q50 475 Q90 Student is female -16.14 -17.95 -21.71 -25.34 -26.31 (7.34) (11.30) (16.19) (16.27) (14.97) Mother education above secondary 13.59 14.22 17.14 20.15 23.24 (6.42) (8.20) (12.28) (13.47) (13.12) Mother i s working 2.64 -0.39 1.63 0.76 0.98 (1.35) (0.26) (1.17) (0.47) (0.42) Home educational resources 8.90 8.71 9.14 8.85 9.16 (12.74) (14.81) (16.95) (14.56) (10.98) School i s Private 13.69 18.99 20.22 18.60 20.60 (4.12) (8.38) (8.97) (6.66) (7.62) Percentage of girls at school 32.11 16.12 23.02 34.93 28.04 (2.80) (1.94) (2.66) (4.07) (2.88) School in medium city 13.38 12.93 13.65 13.71 11.18 (6.18) (6.45) (8.09) (7.75) (5.44) School in large city 12.67 13.47 14.22 16.75 18.98 (5.07) (6.39) (6.97) (7.48) (7.36) School size 0.01 0.01 0.01 0.01 0.01 (7.25) (7.49) (8.97) (9.78) (8.44) School educational material (labs) 1.51 0.13 -0.38 0.82 1.63 (1.23) (0.17) (0.52) (0.92) (1.83) Time on homework 2.32 2.98 3.08 3.30 3.12 (14.16) (25.30) (28.47) (26.58) (18.32) Instrumental motivation (index) 1.10 3.24 4.35 4.36 4.86 (0.79) (2.91) (4.50) (4.00) (4.28) Memorization (index) 0.43 -0.50 -0.45 -0.32 -0.45 (0.39) (0.56) (0.63) (0.36) (0.44) Teacher behavior (index) 0.71 1.88 2.33 1.49 0.70 (0.56) (2.12) (3.02) (1.67) (0.72) Teacher morale(index) 1.64 1.15 1.56 1.87 1.60 (1.30) (1.39) (1.96) (2.06) (1.81) Student uses computer often at school 2.06 2.81 0.60 -1.68 -1.56 (0.95) (1.88) (0.39) (1.Ol) (0.82) Student uses Internet often at school -2.06 -1.58 -1.30 0.11 4.23 (0.63) (0.85) (0.79) (0.06) (1.93) Student self-belonging (index) 2.39 1.57 1.87 1.11 -0.66 (2.32) (2.16) (2.75) (1.39) (0.80) Constant 319.08 362.63 399.00 433.72 469.98 (43.10) (71.07) (71.27) (83.55) (78.02) Pseudo R-Square 0.12 0.13 0.13 0.13 0.12 Source: PISA 2003 77 Annex Table 7b. Quantile Analysis Model (Math) Math Q l O Q25 Q50 Q75 Q90 Student i s female -16.27 -17.63 -21.55 -26.43 -28.93 (8.64) (10.46) (15.79) (17.67) (14.11) Mother education above secondary 7.08 10.84 13.32 15.51 16.73 (3.34) (6.06) (9.76) (8.88) (6.78) Mother i s working 2.52 2.14 2.09 2.29 0.10 (1.45) (1.23) (1.35) (1.45) (0.04) Home educational resources 10.67 10.47 9.52 9.89 10.29 (13.77) (16.25) (17.86) (15.78) (11.42) School i s Private 25.17 21.32 21.01 23.22 25.O1 (9.59) (8.98) (11.59) (9.77) (7.93) Percentage o f girls at school 39.58 26.57 31.56 48.36 51.74 (4.29) (4.02) (4.24) (5.93) (4.95) School in large city 1.59 5.61 8.90 11.68 12.58 (0.72) (2.80) (4.66) (5.67) (4.30) School educational material -0.45 0.01 0.80 0.74 0.86 (0.37) (0.01) (1.21) (0.94) (0.80) School size 0.01 0.01 0.01 0.01 0.01 (9.05) (12.54) (11.98) (12.30) (9.62) Time on homework 2.57 2.67 3.03 2.97 2.88 (16.95) (20.66) (21.40) (21.34) (13.80) Instrumental motivation (index) -1.13 0.76 0.36 3.00 5.39 (1.01) (0.70) (0.37) (2.92) (3.82) Memorization (index) -0.23 -0.69 -0.05 -0.03 0.42 (0.21) (0.72) (0.06) (0.03) (0.29) Teacher behavior (index) 3.24 1.33 0.34 0.59 -0.29 (2.64) (1.63) (0.48) (0.68) (0.25) Teacher morale(index) 0.52 2.10 2.50 1.74 2.94 (0.52) (2.69) (3.65) (2.58) (2.92) Student self-belonging (index) 3.49 3.25 2.49 2.15 0.45 (3.57) (4.10) (3.61) (3.23) (0.44) Student uses computer often at school 4.73 3.93 2.77 1.92 1.25 (2.12) (2.13) (2.02) (1.37) (0.57) Student uses Internet often at school 5.31 0.46 2.25 3.37 3.70 (2.03) (0.20) (1.21) (1.74) (1.33) Constant 296.43 341.55 378.89 411.97 450.94 (48.24) (77.39) (80.10) (86.76) (73.23) xuce: PISA 2003 78 Annex Table 7c. Quantile Analysis Model (Reading) Reading Q l O Q25 Q50 Q75 Q90 Student is female 15.70 15.18 10.16 6.50 2.62 (7.12) (9.73) (7.47) (4.89) (1.52) Mother education above secondary 10.20 10.65 11.53 12.06 12.16 (4.58) (6.72) (7.67) (7.47) (5.72) Mother i s working 0.42 -0.41 0.00 1.03 3.64 (0.18) (0.25) (0.00) (0.79) (2.09) Home educational resources 11.35 10.73 10.94 10.40 10.09 (12.08) (16.62) (16.74) (17.55) (13.36) School i s Private 24.68 23.00 23.33 19.52 17.01 (5.79) (9.46) (10.19) (9.46) (6.34) Percentage of girls at school 37.03 32.29 28.75 38.8 1 47.82 (3.32) (3.95) (3.66) (5.54) (5.02) School in medium city 17.72 18.76 15.91 16.07 14.38 (6.92) (11.32) (10.70) (10.33) (6.66) School in large city 9.72 13.68 15.91 20.89 18.37 (2.28) (5.95) (7.3 1) (8.76) (6.46) School size 0.01 0.01 0.01 0.01 0.01 (8.13) (9.06) (13.48) (9.12) (7.38) School educational material (labs) 1.23 0.38 0.89 1.69 2.10 (1.04) (0.49) (1.35) (2.04) (2.46) Time on homework 2.66 2.69 2.71 2.84 2.77 (16.99) (21.18) (23.19) (20.44) (19.58) Instrumental motivation (index) -5.70 -5.26 -3.97 -3.68 -3.41 (3.77) (5.12) (3.97) (3.77) (2.36) Memorization (index) -1.79 -2.49 -1.50 -1.20 -1.83 (1.59) (3.26) (1.95) (1.77) (2.01) Teacher behavior (index) 1.28 2.59 1.09 0.21 0.09 (0.97) (2.89) (1.59) (0.31) (0.09) Teacher morale (index) 2.37 1.39 2.70 2.62 1.79 (1.74) (1.62) (3.80) (3.35) (1.86) Student uses computer often at school 3.30 4.2 1 2.64 2.23 -0.90 (1.26) (2.53) (1.83) (1.52) (0.49) Student uses Internet often at school 3.24 4.56 2.73 5.77 5.56 (1.13) (2.21) (1.75) (3.67) (2.09) Student self-belonging (index) 5.85 5.86 5.44 4.13 2.71 (4.86) (7.05) (7.66) (6.21) (2.95) Constant 295.79 338.62 387.33) 421.12 457.04 (41.60) (74.07) (88.22) (103.80) (75.82) Pseudo R-Square 0.12 0.13 0.13 0.13 0.12 ource: PISA 2003 79 App idix l-a: InternationalReview of Studies of the Determinants of Learning Data I Method I Results 1 v Advances Abdul- TIMSS 1999 for GLS Family and school related factors (including Hamid Jordan teaching methodology) play a role in 2003 achievement of students Fertig PISA 2000 for OLS and Negative factors include: poor school Associations 2003 Germany quantile conditions, non-native students, shortage of across regressions teachers, schools without regular exams achievement scores Fertig and PISA 2000 for Quantile Family background and school characteristics Schmidt Germany Regressions (including teacher provision) play a role in 2002 student achievement Fuchs and PISA 2000 cross- WLS I1School autonomy. improves student . Woessman country achievement in schools that have central exit 2004a 1I exams; institutional factors explain one- quarter of variation Greenberg NAEP 2000 math I Principal IIbehavior values had higher-mean math scores Studentsin schools with highest student 2004 component analysis than students in schools in middle or bottom of student behavior distribution Hanushek US National OLS; linear Accountability systems introduced in the 2004a Assessment of growth 1990s have clear positive impact on student Educational model achievement, but need to be tied to incentives Progress (NAEP) math 1992-1996; 1996-2000; reading 1994-1998; 1998- 2002 Hanushek US NAEP Log linear Relationship between school resources and Simulations 2004b model improved educational outcomes inconclusive with unobserved teacher variables Hanushek TIMSS 1995 cross- OLS Family background closely related to and Luque country schooling outcomes regardless of income 2003 level of country Lee and IEA and I M Pdata Cross- Family and school resources closely related to Barro 2001 1964-1998 for 64 country school outcomes countries Woessman TIMSS 1995cross- 2003 country personnel and process decisions, competition from private schools have positive student outcomes in 39 countries; negative factors - include influence of teacher unions Woessman TIMSS 1995, WLS; Positive role of central exams and school 2004 1999; PISA 2000 Quantile autonomy over teacher salaries and course regressions content on student performance 80 Appe dix lb: Review of Previous Studies of Determinants of Learning in Mexico Anderson 4 schools in Leon, 1OLS and IIncreased teaching hours, increased student- I I 2000 Guanajuato Logit teacher interactions, improved facilities improve student reading scores; direct evidence of parental participation impact on lower repetition rates Fernandez Estandares HLM Family and school characteristics have impact 2003 Nacionales 2000 on student achievement McEwan Camera Magisterial Teacher incentives have no impact on student Regression and 2002 achievement discontinuities Santibanez I IIto 2004 counterfactual Ontiveros PARE I OLS IImproved school facilities and textbooks haveI Quasi- 1998 positive impact on student achievement; experimental increasing teacher salaries has no impact design with control group Paqueo and PARE Experimenta Supply-side intervention has positive impact Quasi- Lopez- 1 Design on learning achievement of rural and experimental Acevedo indigenous schools design with 2002 control group Shapiro and Estandares Propensity Compensatory program improved student Propensity Moreno Nacionales score achievement score matching 2004 matching to create control group 81