33205 INTERNATIONALBANK FOR WORLD BANK R E T C N O E N STRUCTION PM AND DEVELO April 2005 No. 68 A regular series of notes highlighting recent lessons emerging from the operational and analytical program of the World Bank`s Latin America and Caribbean Region COMPENSATORY EDUCATION FOR DISADVANTAGED STUDENTS: EVIDENCE FROM AN IMPACT EVALUATION STUDY IN MEXICO Harry Anthony Patrinos, Joseph Shapiro and Jorge Moreno Trevino Overview A recent evaluation of the impact of CONAFE's compensa- The President of Mexico, Vicente Fox, proclaimed in 2001 tory programs finds that CONAFE is most effective in that his top education priority was the provision of equal and improving primary school math learning and secondary high quality education to all Mexicans--a sentiment that school Spanish learning. Telesecundaria education and reaffirmed the commitments of earlier Mexican administra- bilingual education for indigenous students are both shown tions. But effectively educating all citizens is difficult in a to improve student achievement. CONAFE is also shown to geographically disperse and culturally hetero- lower primary school repetition and failure geneous country such as Mexico. How should rates. Mexico educate the type of students who speak no Spanish, live in villages inaccessible by CONAFE and the World Bank roads, or come from families that cannot afford school uniforms? The Basic Education Development Program (APL) is in the second phase of a three-phase A similar question should concern education program that supports implementation of the policymakers in most poor- and middle-in- Government of Mexico's compensatory educa- come countries. High-quality basic educa- tion program. The Program supports the tion is necessary to end the transmission of Government's efforts to raise the level and qual- poverty from one disadvantaged generation ity of schooling in Mexico, bringing the to the next. Health and infrastructure im- country's education indicators substantially provements can build a framework for every closer to the other OECD countries.1 The pro- person to live a life free of poverty. But if a gram has two main project components. The country's most disadvantaged students do not receive a first improves quality in initial and basic educa- good quality education, those students will be largely tion with financing for (1) expanding and rehabilitating unable to escape the intractable and abject poverty that education infrastructure in targeted communities; (2) provid- characterizes too many disadvantaged communities. ing educational materials and equipment for students, teach- ers and schools; (3) training teachers, supervisors, other Mexico began to address this challenge as early as 1971 administrative staff and other education promoters, including by creating the National Council of Education Promotion technical assistance and performance incentives for primacy (CONAFE), a division of Mexico's Secretariat of Public school teachers; (4) support for community or parents asso- Education (SEP). CONAFE provides extra resources to ciations for school-based management activities, and (5) schools that enroll disadvantaged students. CONAFE's improving school supervision. compensatory education (see Box 1) programs now sup- port more than three million students in pre-primary and The second component strengthens institutional capacity at primary education, and about one million students in the federal and state levels. This second component (1) telesecundaria education, or secondary education deliv- coordinates national- and state-level evaluation of education ered via satellite television to remote schools. outcomes; (2) consolidates the national school mapping system and its use in planning at the state level for basic 1 education; (3) supports educational re- search; (4) strengthens the role of the Box 1 - Compensatory Education state-level Secretariats of Education to better provide basic education services Among the largest compensatory education programs is Chapter 1 (formerly by reinforcing the capacity of the Sec- Title 1) in the United States, which in 1991 allocated $4.3 billion to schools retariats to plan, program, budget, enrolling low-income students. A variety of evaluations in the 1980s found that monitor and evaluate service delivery, Chapter 1 funds effectively increased test scores over a 1-year period, but that and (5) encourages education innova- scores fell in the summer or year following the investment of Chapter 1 (Slavin tion (World Bank 2002).The Program's 1989). purpose is to ensure that children ages 0 to 14 in Mexico's most disadvan- More recent research has focused on the Head Start program in the United taged communities will begin and com- States, which provides extra resources to disadvantaged preschool students. plete basic education. Phase II of the Head Start began in 1965 by giving $1,000 (in 1999 prices) per student to about Program supports the most disadvan- 500,00 children aged three and four; today it supports about 800,000 students taged basic education schools in every with about $5,400 per student (Garces et al. 2002). Analyses of the program state of Mexico. Phase II also seeks to show that Head Start increases test scores, lowers dropout and failure rates, and consolidate and expand quality im- shrinks test score inequality between ethnic groups (Currie and Thomas 1999; provements and coverage of initial and Barnett 1995; Karly and others 1998). By third grade, however, the effects of basic education; to strengthen manage- Head Start seem to disappear (Currie and Thomas 2000; Aughinbaugh 2001). ment of the education systems; to put More recent analysis has shown that that adults who participated in Head Start in operation a competitive fund to sup- as children were not more likely to complete high school, attend college, or have port education innovations proposed higher earnings than students who did not participate in Head Start. One by the states, and to continue strength- suggested conclusion of this research is that in order achieve long-term ening states' institutional capacity. effectiveness, compensatory education must be sustained over long time periods (Aughinbaugh 2001). Methodology Chile operates a program called P-900 that more resembles CONAFE. Chile's Prior evaluations imperfectly measured Ministry of Education provides teacher training, textbooks and didactic materi- CONAFE's effect because student als, and infrastructure improvements to the schools with the worst performance socio-economic backgrounds differ on a national exam of student ability. Some recent analyses (MINEDUC 2000; markedly between CONAFE and non- Tokman 2002) compare performance of students in P-900 schools against CONAFE schools. For example, practi- students outside such schools. Those analyses find that P-900 increases test cally all indigenous students attend score performance over several years, but no analyses examine the effectiveness CONAFE schools, so one cannot deter- of P-900 in increasing school participation through the university level or in mine the effect of CONAFE on indig- increasing wages through adulthood. enous students by looking at only indig- enous students--one would have no appropriate group against which to illiteracy, community poverty, isolation from other learning compare those students. Construction of a control group of institutions and other sources of disadvantage ­ on student non-indigenous students, however, is difficult--indigenous achievement. Such a model was developed by estimating the students are more disadvantaged than their non-indigenous probability that the school of a particular student received peers, and indigenous students may lack Spanish as a mother CONAFE support--a probability that depends on back- tongue. ground characteristics of a student's community such as availability of public services, average literacy and average Therefore, a model is needed that can use available data on income. Although CONAFE supports a school and not a student test scores, CONAFE support, and student back- single student, one can model CONAFE support as a func- ground to distinguish the effect of CONAFE on student tion of both student and community background. The effect achievement from the effect of student background ­ parent of CONAFE, then, is the difference in performance improve- ment over time between students with similar backgrounds whose schools do and do not receive CONAFE support. To identify students with similar backgrounds, a propensity score matching algorithm was used that identified compa- rable CONAFE and non-CONAFE students. Matching is particularly appropriate to evaluate programs where (a) some individuals received an experimental treatment, (b) selection of participants for treatment was non-random and based on background features of each individual, (c) few individuals in the non-experimental group had similar characteristics to 2 observations in the experimental group, and (d) selecting comparable experimental and non-experimental observa- tions is difficult due to the high number of background features needed to determine comparability (Dehejia and Wahba 2002). CONAFE's support is nonrandom and based on complex background characteristics of students and schools, so evaluation of CONAFE is a good case for Figure 1 matching evaluation. An unbiased propensity score match would use sufficient background information of students to ensure that the assign- ment of CONAFE support among students with equal pro- pensity scores is purely random. Unfortunately, limited background data available to do this match suggests that negative score bias towards CONAFE students remains in this methodology, meaning that CONAFE students must overcome extra obstacles to show the same achievement as non-CONAFE students. A propensity score is the probabil- ity, given a school's background, that the school receives CONAFE support. A score could more simply be interpreted as the correspondence of a school's background with the profile of a typical CONAFE school. Figure 2 General Findings CONAFE's compensatory programs seek to support the most disadvantaged schools in all Mexican states. CONAFE ap- pears to be highly effective in this targeting, as CONAFE includes the most disadvantaged schools and nearly all indigenous schools. CONAFE support is relatively static, as 70% of schools in the five-year sample received support for with the most disadvantaged backgrounds. In all three all five years of the sample. But background indicators show groups, however, CONAFE was slower to close the Spanish that CONAFE has given the most years of support to the most test score gap than to close the math test score gap. Though disadvantaged communities. The communities in which CONAFE's Spanish and math programs both improve CONAFE schools are located have significantly lower levels student learning, it appears that CONAFE's math program of literacy, access to public services, and industrial develop- may be more effective than CONAFE's Spanish program is. ment than do the communities of non-CONAFE schools. This difference in math and Spanish performance exists even Furthermore, CONAFE's coverage of every Mexican state when controlling for the presence of indigenous students. has not diluted CONAFE's targeting mechanism. The portion of schools that receive CONAFE support varies significantly CONAFE Improves Indigenous Student Primary School by state and correlates strongly with poverty and presence of Performance indigenous communities in each state. Indigenous student exam performance rapidly increased over CONAFE Increased Primary School Test Scores and De- the sample period by an annual average of 27 points on math creased Inequality exams and 12 points on Spanish exams. CONAFE's instruction decreased the gap in math scores between comparable Composite Spanish and math scores of CONAFE students CONAFE and non-CONAFE students by 5 points annually. increased significantly over the sample period for all three That annual 5 point increase is the minimum effect attributable disadvantaged groups (See Figures 1 and 2). In each group, to CONAFE. Since nearly every indigenous student receives CONAFE students gained on non-CONAFE students by 2.4 CONAFE support, construction of an appropriate control group to 4.3 points per year in a math-Spanish composite score. against which to compare indigenous students is difficult. Bias of methodology used in this analysis understates Perhaps because the control group against which indigenous CONAFE's positive effect, so these numbers are the students were compared had a very different background than minimum positive effect attributable to CONAFE. CONAFE the typical indigenous student has, this analysis found no also decreased test score inequality between CONAFE and significant effect of CONAFE on indigenous Spanish scores. non-CONAFE students by 9% for the less-disadvantaged group and by 30% for the disadvantaged group. These results CONAFE Improves Secondary School Math and Span- are particularly encouraging, as they show that CONAFE is ish Learning most effective in eliminating learning inequality for students Spanish and math scores of telesecundaria students rapidly 3 Help Hispanic Children?" Journal of Public Econom- ics 74(2): 235-62. Currie, Janet and Duncan Thomas. 2000. "School Quality and the Longer-term Effects of Head Start." Journal of Human Resources 35(4): 755-74. Figure 3 Dehejia, Rajeev H. and Sadek Wahba. 2002. "Propensity Score Matching Methods for Non-experimental Causal Studies." The Review of Economics and Statistics 84(1): 151-161. Garces, Eliana, Duncan Thomas and Janet Currie. 2002. "Longer-Term Effects of Head Start." The American Economic Review 92(4): 999-1012. increased over the sample period from a composite average of Karly, Lynn A., Peter A. Greenwood, Susan S. Everingham, 499 points in 2000 to a composite average of 540 points in Jill Hoube, M. Rebecca Kilburn, Peter C. Rydell, Mat- 2002 (See Figure 3). Furthermore, telesecundaria education thew Sanders and James Chiesa. 1998. "Investing in caused much of this increase. Over the sample period for the our children: What we know and don't know about the less-disadvantaged group, telesecundaria education elimi- costs and benefits of early childhood interventions." nated 24% of math score inequality and 38% of math score Santa Monica: RAND MR-898. inequality between telesecundaria and non-telesecundaria students, controlling for variation in student background. MINEDUC. 2000. "Evaluación del Programa de las 900 This analysis defined inequality as the average test score Escuelas." División de Educación General, Chilean difference between the telesecundaria experimental group Ministry of Education. Cited in Tokman. and the non-telesecundaria control group. It appears that telesecundaria education is more effective for Spanish in- Shapiro, Joseph and Jorge Moreno Trevino. 2003. "Com- struction than for math instruction, which is particularly pensatory Education for Disadvantaged Mexican Stu- interesting given that CONAFE's primary school programs dents: An Impact Evaluation Using Propensity Score appeared to be more effective for math than for Spanish Matching." World Bank (Processed). instruction. Slavin, Robert E. 1989. "Students at risk of school failure: the problem and its dimensions." In Robert E. Slavin, Conclusions Nancy L. Karweit, and Nancy A. Madden, eds. Effec- Overall, CONAFE's compensatory programs are effective tive Programs for Students at Risk. Boston: Allyn and and well-targeted. At the primary and secondary levels, Bacon. CONAFE significantly improved student exam performance and decreased inequality between CONAFE and non- Tokman, Andrea. 2002. "Evaluation of the P-900 Program: CONAFE students. These results were robust even when A Targeted Education Program for Underperforming controlling for relevant background variables. CONAFE Schools." Working Paper 170, Central Bank of Chile. appears to be more effective in math instruction at the primary and in Spanish instruction at the telesecundaria World Bank. 2002. Project Appraisal Document: Mexico level. Generally, this analysis shows that the World Bank is Basic Education Development Phase II Project. World achieving its goal of improving and expanding educational Bank, Human Development Unit, Latin America and quality in Mexico through its support of CONAFE's the Caribbean Regional Office, Washington, D.C. compensatory programs. Notes Sources 1Mexico, the world's ninth largest economy, is a member of Aughinbaugh, Alison. 2001. "Does Head Start Yield Long- the Organisation for Economic Cooperation and Development Term Benefits?" The Journal of Human Resources (OECD). 36(4): 641-65. About the Authors Barnett, Steven. 1995. "Long-Term Effects of Early Child- Harry Anthony Patrinos is a Senior Education Economist, hood Programs on Cognitive and School Outcomes." Joseph Shapiro and Jorge Moreno Trevino are Junior Pro- The Future of Children 5(3): 25-50. fessional Associates all with the Human Development De- partment of the LatinAmerica and the Caribbean Region of Currie, Janet and Duncan Thomas. 1999. "Does Head Start the World Bank 4 INTERNATIONALBANK FOR WORLD BANK R E T C N O E N STRUCTION PM AND DEVELO 5