WPS4879 P olicy R eseaRch W oRking P aPeR 4879 The Performance of Decentralized School Systems Evidence from Fe y Alegría in Venezuela Hunt Allcott Daniel E. Ortega The World Bank Human Development Network Development Dialogue on Values and Ethics Unit March 2009 Policy ReseaRch WoRking PaPeR 4879 Abstract This program evaluation estimates the effects on students. These effects are significantly larger for standardized test scores of graduating from the Fe y households at the bottom of the distribution, and smaller Alegría private school system in Venezuela. The authors for those at the top. The authors posit that the better find an Average Treatment Effect on the order of 0.1 performance of the Fe y Alegría system stems from their standard deviations (approximately 16 percent of the labor contract flexibility and decentralized administrative average score), using a control group of public school structure. This paper--a product of the Development Dialogue on Values and Ethics Unit, Human Development Network--is part of a larger effort of the unit to document the role of faith-based organizations in service delivery. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The corresponding author may be contacted at Daniel. Ortega@iesa.edu.ve. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team The Performance of Decentralized School Systems: Evidence from Fe y Alegría in Venezuela1 Hunt Allcott Daniel E. Ortega Harvard University, Cambridge, MA Instituto de Estudios Superiores de allcott@fas.harvard.edu Administración (IESA) and Corporación Andina de Fomento, Caracas, Venezuela. daniel.ortega@iesa.edu.ve JEL Classification: I2, H4 Keywords: Schooling Quality, Private Education, Fe y Alegría. 1 The authors thank Rosa Amelia Gonzalez of IESA and administrators at Fe y Alegría, especially Noelbis Aguilar, for their invaluable support during this project. We also thank Pablo Acosta, Guido Imbens, Elizabeth King, Harry Patrinos, Miguel Urquiola, participants in the 2006 meeting of LACEA in Mexico City and participants in the PPP in Education Conference at the World Bank for valuable comments and suggestions. The paper also benefited from comments from Quentin Wodon at the World Bank's Development Dialogue on Values and Ethics. Hunt Allcott acknowledges financial support from the Ochoa Brillembourg Fellowship and from the Andean Development Corporation (CAF). Any remaining errors are because neither of the authors attended a Fe y Alegría school, and remain our own. 1. Introduction Public education in Venezuela has deteriorated steadily over the past 25 years. While the average educational attainment of the labor force increased from 6.1 years to 8.2 years (Ortega and Pritchett, 2006) and the literacy rate for people 15 and older went from 85 percent to 93 percent between 1981 and 2001 (Ortega and Rodriguez, 2008), the government's expenditures on education dropped 36 percent in real terms between 1980 and 2003 (Ortega, 2005). Average aptitude test scores for high school seniors have dropped from 21 to 6 in verbal ability and from 11 to 3 in math reasoning between 1987 and 2003. Possibly as a result of the deterioration in the quality of education, and of other changes in the labor market, the Mincerian returns to education have dropped from 15 percent in 1975 to under 10 percent in 2003. Working amid this disconcerting evidence is Fe y Alegría, a confederation of Jesuit schools targeting disadvantaged youth. The program's first primary school was established in Catia, a disadvantaged area of Caracas, in a home donated by a local bricklayer. Since then, it has expanded to serve 1.2 million students in 15 Latin American countries (Gonzalez and Arevalo, 2005). The organization has a number of initiatives, including job training, teacher training, adult and radio education, and support for microbusinesses, but the bulk of its efforts are spent in primary and secondary education. Most observers, from community members to academic researchers, consider Fe y Alegría to be quite successful, but no econometrically satisfying program evaluation has been undertaken. Through an econometric estimation of Average Treatment Effect, we compare Fe y Alegría graduates to a control group of Venezuelan public school students using the results of the Prueba de Aptitud Académica (PAA), a math and verbal test similar to the American SAT. Our results show that Fe y Alegría students perform slightly but significantly better on both parts of the PAA. The difference between two of our estimators suggests a heterogeneous treatment effect, which we then estimate, showing that the program is especially beneficial for the disadvantaged portion of the student body. Finally, we posit that this effect is due to the institution's organizational behavior: Fe y Alegría does not spend more money per pupil, but it does have evidently different management and cultural characteristics. Specifically, Fe y Alegría's management structure is much more decentralized, giving school principals budgetary authority and the ability to hire and fire teachers. Partially as a result of this decision making process, the organization has succeeded in instilling a "family feeling" in teachers, staff, and students, that we believe contributes to the treatment effect. Because Fe y Alegría is both private and decentralized, this research is related to both the literature on decentralization of public services and the literature on school privatization. Fe y Alegría represents a scalable alternative to these policy options, as evidenced by its rapid expansion within Venezuela and to other countries in the region. As we show, Fe y Alegría merits imitation and greater scale. 2. Related Literature Although we do not focus directly on the issue of decentralization of public services, this paper is related to that literature insofar as it touches upon the benefits and pitfalls of having decision rights closer to the individual. Galiani, Gertler and Schargrodsky (2005) argue that decentralization of public schooling in Argentina in the early 90's helped improve the quality of education, as measured by standardized test scores, in well-off regions, and had a negative effect in regions that were disadvantaged to begin with. Pães de Barros and Mendonça (1998) suggest that neither school financial autonomy nor local school boards in Brazil play a significant role in primary school performance, but that the principal's appointment power does have a positive and significant effect. Eskeland and Filmer (2002) find a positive correlation between performance and the autonomy of primary schools in Argentina and King and Ozler (2000) also suggest a positive effect of decentralization on parent participation in school decision making in Nicaragua. Aedo (1998) presents evidence that Chilean schools that have significant decision rights also perform better than centralized schools. More recently, Sawada and Ragatz (2005), di Gropello and Marshall (2005) and Parker (2005), as part of a larger investigation on teacher incentives in Latin America (Vegas, 2005), document educational reforms in El Salvador, Honduras and Nicaragua respectively, where either through spontaneous community organization or the government's initiative, some autonomy was transferred to local schools. The reported results are mixed in part because central authorities still retained significant decision rights, although key indicators such as teacher absenteeism and number of teacher strikes did seem to improve as a result of the reforms. Other alternative school systems have been extensively studied. Private versus public schooling in general has been one topic of interest both in the US (see for example Manski, 1992, Hanushek, 1994, Hoenack, 1994 among others) and in developing countries. Private school performance has been compared to that of public schools and has generally shown to outperform the public system according to several measures. Cox and Jimenez (1990), after controlling for selection issues, show that private schools perform better on standardized tests than public schools in Colombia and in Tanzania; Saavedra (1996) estimates a differential effect of private versus public schooling on the wages of Peruvian workers, and Contreras (2002) estimates a positive effect of the voucher system relative to public schools on test scores in Chile. Also related is the more narrow focus on Catholic schools, which has mostly been undertaken using data for the US; in particular, Evans and Schwab (1995) show that being Catholic per se does not affect educational outcomes, and then use Catholicism as an instrument for student participation in Catholic schools. They use the binary outcome of high school completion, claiming that it is a much more important predictor of future outcomes, and show that Catholic schools outperform public schools. There is a basic agency problem in the provision of public schooling. Principals (society and parents) contract implicitly with centralized government administrators to provide quality education. The school administrators may have different incentives, and the effects of their actions on school quality are difficult to observe. By making the agents informationally closer to the parents, decentralization and privatization might help to ameliorate the information problem. Empirically, these effects are difficult to tease out because both privatization and decentralization are bundles of policies that combine solutions to some incentive problems but at the same time may cause several others. It is not easy to find instances in which policies are undertaken in a way that allows identification of the impact of each of its components, i.e. to disentangle the effect of increased school principal authority from decreased central curriculum design. The articles in Savedoff (1998), although hampered by the natural limitations in the data, provide suggestive evidence as to the importance of these agency problems that may be resolved by means other than decentralization or privatization. Despite the high regard for Fe y Alegría and the availability of extensive data, no econometric evaluation has been done of the system's effectiveness. Navarro and De La Cruz (1998) evaluate test scores and use student-level demographic controls, but their analysis is restricted to two Fe y Alegría schools in one Venezuelan state. Our analysis also does not provide direct evidence as to the importance of decentralization, but suggests based on anecdotal evidence that this factor plays an important role in explaining the relative success of the Fe y Alegría system. This paper thus contributes to the discussion on decentralization and private schooling as ways of addressing incentive problems that arise in public administration. 3. Data In Venezuela, every graduating high school student takes the Prueba de Aptitud Académica, which is similar in spirit to the North American SAT. Extensive background data on each student is also gathered, ranging from the basics of age and gender to the profession of the father and what means of transportation the student uses to get to school. In total, there are 413,607 observations of graduating Venezuelan high school students who took the test in 2003. We include only those who are between 14 and 22 years old, are not night-school students, and actually graduated that year instead of earlier. We then drop the 4,662 students from public schools that are not included in a separate school registry that allows us to identify municipalities. Controlling for other observables, this group scores 0.15 standard deviations lower on the verbal section and statistically the same in math, relative to other public school students. Because there is substantial intrastate variation at the municipality level, however, we choose to omit these observations in order to include municipality dummy variables. Because these 4,662 public school students perform poorly relative to other control group students with similar observables, it is likely (depending on which municipality the schools are actually in) that their omission biases downward the estimated treatment effect of Fe y Alegría, since it raises the average performance of the control group. Our final dataset includes 46,460 public school students and 2,237 Fe y Alegría students. Table 1 shows the mean of each variable for the treated and non-treated cohorts. Test scores are normalized to mean 0, standard deviation 1. Family income, mother's education, house quality, and social class are reported in five classes, with 1 being the "highest." Although we could parameterize these variables, we instead use dummy variables for each bucket to retain the maximum flexibility. As we discuss in the results section, this non-parametric form is important because test scores will be non-linear --and even non-monotonic-- in some of these variables. Ideally, our program evaluation would compare students who were either selected into Fe y Alegría randomly or selected purely on observables to a control group of students who applied and were not selected, and there would be zero or random attrition through dropouts. Such application records are not available, however, and dropouts during primary and secondary school clearly are not random. Our econometric strategy, discussed in the following section, depends on the assumption that there is no unobservable factor correlated with both graduation from Fe y Alegría and test scores. This is often improbable, but we argue that several factors conspire to form a plausible natural experiment. The key factor behind the natural experiment is that Fe y Alegría schools are oversubscribed. Applications to Fe y Alegría at the primary and secondary school levels vastly outnumber the available spots: central administrators estimate that admit rates are around 35 percent. Each school then admits the poorest children from local neighborhoods in a non-standardized process. This process involves an application by the family, a series of interviews at the school and sometimes house visits by a social worker to more accurately determine the socioeconomic status of the family. As a result, conditional on having the motivation to apply for Fe y Alegría, which many students do, the selection of students into schools is on wealth and geographic location. Our observed variables capturing income and house quality proxy very well for the wealth aspect of schools' admission decisions. However, if the unobservable characteristics causing a student to apply for Fe y Alegría are both not widespread among public school students and positively correlated with test scores, our estimated treatment effect will be biased upward. Ideally, we would also observe the second implicit selection factor in admissions, proximity of each student's residence to each school. We argue, however, that any differences are orthogonal or weakly correlated with test scores. As part of the program's mission to serve underprivileged children in poor neighborhoods, Fe y Alegría schools were indeed often placed in the poorest neighborhoods. Over the life of the program, however, some of these neighborhoods have changed and experienced relative economic growth. In addition, many of the program's schools were once public schools that were transferred to Fe y Alegría at the community's request, and it's not obvious whether these schools would tend to be in "better" or "worse" neighborhoods. We thus assume that the areas near Fe y Alegría schools are econometrically identical to public school districts within the same municipality. If this assumption fails and Fe y Alegría districts are actually "worse," it will bias our treatment effect downward. Performing factor analysis on the subset of these variables related to socio- economic status generates a one-dimensional variable called SES, which we include in Table 1. Regressing SES on Fe y Alegría participation and a dummy variable for each municipality indicates that Fe y Alegría students are statistically of the same socio- economic status as the public school students within their municipality, which suggests that even though the program intends to target the poorest households, on average, its population is not very different from that attending the public school system. On the whole, the similarity on observables and the reality of the Venezuelan natural experiment suggest that it is reasonable to assume that unobservables do not substantially affect both Fe y Alegría enrollment and test scores. 4. Econometric Framework Our fundamental goal will be to calculate the Average Treatment Effect (ATE), typical of the program evaluation literature. The ATE measures the difference between the test score of each unit in both a treated and untreated state, i.e. how a student would have performed in Fe y Alegría vs. how she would have performed in public school: (1) Where: = an individual's test score if treated = the individual's score if not treated. We had initially planned to use program intensity at the municipal level as an instrument for participation. This identification is comparable to other program evaluation papers such as Duflo's (2001) evaluation of a school construction program or the previous literature on Catholic schools. This strategy requires that the placement of schools not be correlated with unobservables that affect test scores, which we claim above. Unfortunately, there is not enough variation in the instrument to obtain meaningful estimates in the first stage. The highest program intensity is under 25 percent, and even limiting the sample to the 31 municipalities where there is at least one Fe y Alegría high school, the average is under 5 percent. This makes the estimated ATEs unstable and implausibly high. As described above, however, this dataset and the natural experiment that created the data lend themselves to estimation of the Average Treatment Effect (ATE) through Ordinary Least Squares (OLS) and propensity score matching. OLS provides a consistent estimate of the ATE if there is no omitted variables bias and the treatment effect is homogeneous. We estimate (2) Where: =The outcome variables, which are the Math and Verbal PAA scores. = The treatment dummy, which takes the value of 1 if the student graduated from a Fe y Alegría school. = A vector of categorical dummy variables for {Venezuelan, Male, Married, Age, Student Works, Father's Profession, Mother's Education, House Quality, Income, Number of Siblings, How School Fees Are Paid, Transportation to School, Social Class} After calculating the OLS benchmark, we use matching estimators to estimate the Average Treatment Effect. Our first matching estimator matches treatment and control observations based on their propensity score. We estimate the propensity score using a standard probit model: where is the standard normal CDF and is a vector of parameters. The fitted values from this regression are the propensity score, which we denote . The empirical counterpart of (1) is: Where Y1 | Wi = 1 is simply the observed test score for a Fe y Alegría student, and Y0 | Wi = 0 is the observed test score for a control group student. The matching estimator simulates what the counterfactual outcome would have been based on the scores of the "nearest" observations: (Y0 | Wi = 1) = 1 (Y1 | WI = 1) J I J i (Y1 | Wi = 0) = 1 (Y0 | WI = 0) J I J i In these equations, is the number of neighbors used for the match; we use for consistency with Abadie, Drukker, Herr, and Imbens (2001). is the neighborhood around treatment (control) group observation i such that there are four observations from the control (treatment) group. For our matching estimators, we calculate robust standard errors and use the bias-corrected matching estimator à la Abadie, Drukker, Herr, and Imbens (2001). Additionally, most applications drop outlying propensity scores, and Crump, Hotz, Imbens, and Mitnik (2006) show that the heuristic of using only observations with closely approximates the optimal trimming rule. Although we do drop control group observations with propensity scores outside the support of the distribution in the treatment group, we do not otherwise censor the distribution because there are a substantial number of observations with in our sample. Chart 1, which shows the distribution of propensity scores in our data, illustrates this issue. As discussed in Heckman, Ichimura, and Todd (1997), our dataset lends itself to low bias in propensity score matching for two reasons. First, only a few members of the public school control group are not on the support of the distribution of the treatment group's observed characteristics. Indeed, as the above factor analysis and descriptive statistics show, Fe y Alegría participation appears similar to a natural experiment in that the distributions of many of the observed characteristics are similar, although not statistically identical. Previous studies using propensity score matching with job training programs often struggled with this, specifically that the observed employment rate or wages of the treated were lower than any controls in the pre-treatment period. We have eliminated private school students from consideration here precisely because their distribution of observables (and unobservables) is so different in Venezuela. Public school students, however, form an excellent control group. Second, the same questionnaire is administered to both treatment and control, and both groups are in a "common economic environment." These issues, of course, relate primarily to problems encountered with evaluation of job training programs. All of our data come from the same administration of the same test, with the same demographic questions asked of each student. Furthermore, unlike the American SAT, all graduating Venezuelan high school students take the PAA. Therefore, although the interpretation of the ATE is limited to those students who have not dropped out of school beforehand, there is no selection bias into the test itself. Although the Fe y Alegría natural experiment described above forms the basis of our assumption of selection on observables, we cannot fully rule out bias due to unobservables. However, the Heckman, Ichimura, and Todd (1997) job training data show that this bias can actually be less important than lack of overlap and differing economic environments, problems from which we do not suffer. The consistency of the matching estimator requires two assumptions, which together are called "strong ignorability of treatment" (Rosenbaum and Rubin, 1983): conditional mean independence and propensity scores strictly between zero and one. · Assumption I: for i={0,1} · Assumption II: Assumption I is the crucial assumption underlying any application of matching estimators. In the application to school choice, the concern would be that unobservable attributes of the student or her family such as motivation, proactivity, or valuation of education would cause the same types of students who select into Fe y Alegría to also do better in the counterfactual. If these decisions were made in a statistically random way, or through an observable nationally-uniform admissions process, this would lend itself to a different estimation strategy. As we discussed in the Data section, this is a decentralized admission process that in an unobservable way uses primarily observable variables. As a result, the conditional mean independence assumption is reasonable, and we can consider the propensity score matching results to be unbiased. Although all graduating students take the test, the ATE is conditional upon students actually graduating from high school. Although the support of the observables of Fe y Alegría and public school students is effectively the same, there is substantial selection through the years of schooling. Specifically, Fe y Alegría as a policy tries to maintain low dropout rates, and their average promotion rate is 10 percent higher than that in the public sector (González and Arévalo, 2005). Thus it is possible that some students in the treatment group have unobservables that would have caused them to drop out of public schools; these unobservables would cause the test scores of the treatment group to be lower. This effect will bias the ATE downward, but it is difficult to bound the effect in any reasonable way. In our application to Fe y Alegría, Assumption II requires that we drop observations from municipalities or states where there are no Fe y Alegría schools, and thus students have effectively zero probability of enrollment in Fe y Alegría. If we believed that there were no state- or municipality-level effects on test scores, we would omit the geographical area dummies from the probit estimation, and observations in non-program municipalities would have a non-zero propensity score. Since there quite plausibly are geographical-level fixed effects, however, we must include the geographical area dummies to satisfy Assumption I. This substantially reduces sample size but still leaves nearly 50,000 observations. 5. Results Using the above data and econometric technique, we estimate the ATE in test scores of being a Fe y Alegría student versus being in the public schools. Before beginning, we re- emphasize that Fe y Alegría is essentially a technical high school, not a college prep, and many of its effects on students are of course not measurable in test scores for college admission. The OLS results, shown in Table II, show that Fe y Alegría students perform 0.05 and 0.06 standard deviations higher in Verbal Score and Math Score respectively, after correcting for observables. Especially interesting in these regressions are the coefficients on several of the control variables. As might be expected, younger students tend to do better, as do students with fewer siblings. Instead of linear influences, however, the effects of family income and house quality seem to be in an inverted-U shape. Wealthier students living in "luxurious" houses actually tend to do worse on the exams than poor students. This may be because they have secured university admission through other university- specific tests and thus do not take the PAA seriously. Propensity score matching gives qualitatively similar results. The probit regression used to generate the propensity scores is shown in Table III, confirming anecdotal evidence that poorer students tend to be selected into (graduation from) Fe y Alegría. Table IV shows the Average Treatment Effect for Verbal Score and Math Score are 0.11 and 0.08 standard deviations (although the effect on verbal scores is not significant). Differences between parameter estimates in OLS and matching can be attributed to a heterogeneous treatment effect, as discussed in depth in Angrist (1998). Regression and matching estimators weight the underlying treatment effects differently: the weights applied in matching estimators are proportional to the probability of treatment, whereas the weights applied in OLS are proportional to the variance of treatment. The most basic test of a heterogeneous effect is to test the joint insignificance of the OLS parameters on the interaction between the treatment effect and the observables: here is a vector of coefficients on the interaction terms. A ² test rejects the joint insignificance of these coefficients; this is sufficient to explain the statistical difference between the matching estimator, which is consistent under our assumptions, and the OLS results. To explore the economic significance of the heterogeneous treatment effect, we use the same propensity scores and separately estimate the ATEs for relatively advantaged and for relatively disadvantaged students. Table V shows these results for both Math and Verbal scores when we break the sample into groups with high (1-3) and low (4, 5) Social Class and Mother's Education (4, 5 denotes primary of less; 1-3 denotes at least some high school). Relatively disadvantaged children benefit substantially more from participation in Fe y Alegría; the treatment effects are statistically different except for the Verbal scores for the different Mother's Education groups. Translating these scores back to the Venezuelan context, the average original scores across treatment and control groups were 6.9 on the verbal section and 3.0 in math, with standard deviations of 5.6 and 4.5. Thus, the above treatment effects of just under 1/10 of a standard deviation correspond approximately to a one-half point improvement in the raw results of the PAA. 6. Reasons for Fe y Alegría's Improved Performance We have shown, as cleanly as possible with the existing data, that Fe y Alegría offers better education than the public schools, as measured by test scores. We now suggest potential reasons for the effect. As González and Arévalo (2005) calculate, Fe y Alegría does not spend more money per pupil than public schools. Indeed, teachers do not receive retirement benefits and are thus often forced to view work at Fe y Alegría as a "side job." Thus, differences in financial inputs are not the cause of the improved performance of the program. Based on our conversations with school officials and researchers, we suggest key reasons for the program's success. As a result of its institutional history, Fe y Alegría's structure is different from that of the public schools on several dimensions, as discussed in Navarro and De La Cruz (1998) and González and Arévalo (2004). From the outset, the public school system was not viewed as an effective organizational model, and the initial spirit of volunteerism has morphed into a more established structure. Although religiosity was initially important, individual schools now vary substantially on that measure, with some schools run by nuns and others exhibiting little sign of Catholic influence. The initial growth in the number of schools was mainly financed by local community involvement and private donations, a process which led to significant autonomy at the school level underneath a national umbrella organization led by Father Velaz. This organically-developed structure was eventually formally adopted, with the principal and the school council at the center of local decision-making and the national leadership dealing with strategic issues such as growth plans and fundraising. Three specific organizational and cultural factors factors stand out: decentralized decision making, labor flexibility, and the potentially resultant feeling of a "family environment." · School-Level Autonomy. Although there exists a central authority at the national level as in the public system that determines general guidelines and principles for the organization as a whole, each Fe y Alegría school retains substantial administrative autonomy. Each principal can hire and fire teachers, purchase supplies, and sign maintenance contracts, among other things. Each school has the autonomy to plan, budget, procure funding for, and execute infrastructure investments. Although most fundraising activities for large projects are centrally coordinated, the initiative almost always comes from school-level administrators, whose ideas tend to be encouraged and well-received by the national administration. Furthermore, the schools, through the regional offices, play an active role in the national-level budgetary decision making. This contrasts with the public school administration, which is much more highly centralized. · Labor Flexibility. Fe y Alegría teachers are not unionized, and their labor contracts are much more flexible than those of the public school system. Teachers in the public school system are appointed by state-level committees that are often controlled by politically-motivated labor unions. In Fe y Alegría, they are hired by the school principal directly and given a one-year trial period before being offered more permanent positions. During this trial period, teachers are not only evaluated on formalities such as meeting the school's schedule of activities (e.g. showing up on time to class, grading exams and papers in a timely fashion, attending faculty meetings, etc.), they are monitored in the classroom every quarter and are coached by their more experienced peers. This flexibility relative to the public schools likely results in a selection process that produces higher teacher quality. Any differences in teacher quality, however, are not the result of higher pay: although its wages for teachers and staff are comparable to outside wages, Fe y Alegría does not offer a retirement plan. As a result, many Fe y Alegría teachers also work in the public schools simply to gain retirement benefits. It seems that Fe y Alegría's compensating differentials are principally the improved teacher training and the esprit de corps, which we describe below. · "Family Feeling." In visits to two Fe y Alegría schools in Catia and to the central administration offices in Caracas, we were struck by what teachers, students and administrative personnel termed a "family feeling": a sense of belonging to the organization of Fe y Alegría and agreement with the organization's objectives. As described above, this feeling reduces input costs by inducing teachers to work or volunteer longer hours for lower wages. It also likely increases efficiency of school input use, potentially by inducing students to respect school property more and pay better attention in class. As suggested by the literature in sociological economics such as Akerlof and Kranton (2005), it is possible that Fe y Alegría has succeeded in modifying students' utility functions to value education or discipline more highly. Even without this sort of "indoctrination effect," Fe y Alegría may have simply arrived at a high-performance equilibrium that attracts better teachers and induces continual good performance. Our impression is that this "family feeling" has been instilled in the organization's culture as a matter of policy and is substantially aided by the empowerment associated with school-level autonomy. 7. Conclusions Using a large, rich dataset, we have shown that graduation from Fe y Alegría increases scores on the Venezuelan college entrance examination relative to counterfactual graduation from public school. The effects are statistically significant for Math scores and robust to different estimators and on the order of one-tenth of a standard deviation, which represents approximately 16% of the average score in Math and 9% of the average Verbal score. These are economically significant effects. We also find that the most disadvantaged groups benefit much more from Fe y Alegría than the better off, with estimated effects in the order of 1/4 of a standard deviation for the bottom 2 quintiles as opposed to 1/20 of a standard deviation for the top three. This suggests that improvements in the quality of educational systems, in addition to having important productivity and poverty reduction effects, are likely to be equality enhancing, as the educational achievement of the most disadvantaged generally depends more on non-familial factors (e.g. quality of schools) than that of the less disadvantaged (see Ortega, 2006 for evidence on this for Venezuela). Because Fe y Alegría schools are oversubscribed and admit students based on observable poverty, and also because the dataset is rich and with uniform outcomes, matching estimators are plausibly consistent. These results suggest several further lines of research and policy recommendations. To strengthen the evaluation of Fe y Alegría, it would certainly be most convincing to randomly select or encourage a cohort to enter the program, creating a true field experiment. This would be the most satisfying way to deal with questions regarding strong ignorability and the exogeneity of participation in the program. Testing cohorts of students in years before graduation would allow the econometrician to better account for biased attrition, i.e. dropouts. In addition, a richer set of outcome variables characterizing the family and economic lives of Fe y Alegría graduates would likely give a full perspective on the effects of the program. This is not, however, the most interesting line of future research. If we believe that Fe y Alegría offers a better education, it is important to know the reasons for that, and also whether the program can be expanded or if its successes can be translated to public schools. If decentralized decision making is indeed a factor in the organization's improved performance, it would suggest that the program of decentralization pursued in Venezuelan schools in the 1990s should be continued more aggressively. On a more basic level, the fact that there is variance in school system quality suggests that policymakers should encourage school variety and choice. References Abadie, Alberto, David Drukker, Jane Herr, and Guido Imbens (2001). "Implementing Matching Estimators for Average Treatment Effects in Stata." The Stata Journal, Vol. 1, No. 1, 1-18. Aedo, Cristián (1998). "Diferencias entre escuelas y rendimiento estudiantil en Chile," in Savedoff, William, ed, La Organización Marca la Diferencia: Educación y Salud en América Latina. Washington DC: Inter-American Development Bank. Akerlof, George, and Rachel Kranton (2005). "Identity and the Economics of Organizations." Journal of Economic Perspectives, Vol. 19, No. 1 (Winter), 9-32. Angrist, Josh (1998). "Estimating the Labor Market Impact of Voluntary Military Service Using Social Security Data on Military Applicants." Econometrica, Vol. 66, No. 2 (March), 249-288. Contreras, Dante (2002). "Vouchers, School Choice and Access to Higher Education" Mimeo, Universidad de Chile. Cox, Donald and Emmanuel Jimenez (1990). "The Relative Effectiveness of Private and Public Schools: Evidence from two Developing Countries" Journal of Development Economics, Vol. 34, No. 1-2 (November), 99-121. Crump, Richard, Joseph Hotz, Guido Imbens, and Oscar Mitnik (2006). "Moving the Goalposts: Addressing Limited Overlap in Estimation of Average Treatment Effects by Changing the Estimand." Mimeo, Harvard University. di Gropello, Emanuela, and Jeffrey Marshall (2005). "Teacher Effort and Schooling Outcomes in Rural Honduras" in Vegas E. (ed.) Incentives to Improve Teaching. Lessons from Latin America. The World Bank, Washington, DC Duflo, Esther (2001). "Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment." American Economic Review, Vol. 91, No. 4 (September), 795-813. Eskeland, Gunnar, and Deon Filmer (2002). "Autonomy, Participation and Learning in Argentine Schools: Findings and their implications for Decentralization" World Bank Policy Research Paper Series No. 2766. Evans, William, and Robert Schwab (1995). "Finishing High School and Starting College: Do Catholic Schools Make a Difference?" Quarterly Journal of Economics, Vol. 110, No. 4 (November), 941-974. "Fe y Alegría: Informe Analitico." Mimeo, Instituto de Estudios Superiores de Administración, June 2005. Galiani, Sebastian, Paul Gertler and Ernesto Schargrodsky (2005). "School Decentralization: Helping the Good Get Better, but Leaving the Rest Behind." Mimeo, Universidad de San Andrés, UC Berkeley and Universidad Torcuato Di Tella. González, Rosa Amelia, and Gregorio Arévalo (2005). "Subsidized Catholic Schools in Venezuela." In Lawrence Wolff, Juan Carlos Navarro, and Pablo González, Eds, Private Education and Public Policy in Latin America. Washington, DC: Project For Educational Revitalization in the Americas. Hanushek, Eric (1994). "Making Schools Work: Improving Performance and Controlling Costs." Brookings Institution Press, Washington DC. Heckman, James, Hidehiko Ichimura, and Petra Todd (1997). "Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Program." The Review of Economic Studies, Vol. 64, No. 4 (October), 605-654. Hoenack, Stephen (1994). "Economics, Organizations and Learning: Research Directions for the Economics of Education." Economics of Education Review, Vol. 13, No. 2, 147-162. King, Elizabeth and Berk Ozler (2000). "What's Decentralization Got To Do With Learning? Endogenous School Quality and Student Performance in Nicaragua." World Bank Development Research Group, Washington DC Manski, Charles (1992). "Educational Choice (Vouchers) and Social Mobility." Economics of Education Review, Vol. 11, No. 4, 351-369. Navarro, Juan Carlos, and Rafael De la Cruz (1998). "Escuelas Federales, Estatales y Sin Fines de Lucro en Venezuela." In William Savedoff, Ed., La Organización Marca la Diferencia: Educación y Salud en América Latina. Washington, DC: Inter-American Development Bank. Ortega, D. (2006) "Intergenerational Mobility in Latin America: Evidence from Sibling Schooling Gaps in Venezuela" Mimeo, IESA, Caracas. _________ and Pritchett (2006) "Much Higher Schooling, Much Lower Wages: Human Capital and Economic Collapse in Venezuela" joint with Lant Pritchett. In Hausmann, R., and F. Rodriguez, eds., Venezuela's Economic Collapse, under editorial revision. _________ and Rodriguez (2008) "Freed from Illiteracy? A Closer Look at Venezuela's Robinson Literacy Campaign", Economic Development and Cultural Change, October Vol. 57, No. 1: pp. 1-30. Pães de Barros, Ricardo, and Rosane Mendonça (1998). "El Impacto de Tres Inovaciones Institucionales en la Educacion Brasileña" In William Savedoff, Ed., La Organización Marca la Diferencia: Educación y Salud en América Latina. Washington, DC: Inter-American Development Bank. Parker, Caroline (2005). "Teacher Incentives and Student Achievement in Nicaraguan Autonomous Schools." In Emiliana Vegas, Ed., Incentives to Improve Teaching: Lessons from Latin America. Washington, DC: The World Bank. Rosenbaum, Paul and Donald Rubin (1983). "The Central Role of the Propensity Score in Observational Studies for Causal Effects." Biometrika, Vol. 70, No. 1 (April), 41-55. Saavedra, Jaime (1996). "Public and Private Education: Their Relative Impact on Earnings. Evidence from Peruvian Survey Data." Mimeo, GRADE, Lima, Peru. Savedoff, William (1998). La Organización Marca la Diferencia: Educación y Salud en América Latina. Washington, DC: Inter-American Development Bank. Sawada, Yasuyuki and Andrew Ragatz (2005). "Decentralization of Education, Teacher Behavior, and Outcomes: The Case of El Salvador's EDUCO Program." In Vegas, Emiliana, Ed., Incentives to Improve Teaching: Lessons from Latin America. Washington, DC: The World Bank. Vegas, Emiliana (2005). Incentives to Improve Teaching: Lessons from Latin America. Washington, DC: The World Bank. Tables and Graphs Chart 1: Distribution of Propensity Scores Source: Authors' computations. Table 1. Variable means for Treated and Non-Treated Variable Fe y Alegria Public School Verbal Score 0.09 0.00 Math Score 0.20 -0.01 Male Dummy 0.47 0.40 Married Dummy 0.00 0.00 Age 16.96 16.73 Student W orks 0.03 0.03 Father's Prof: Professor or Exec 0.06 0.07 Father's Prof: Technician 0.09 0.09 Father's Prof: Employee 0.29 0.30 Father's Prof: Skilled Worker 0.37 0.39 Father's Prof: Unskilled Worker 0.17 0.14 Mother's Ed: University 0.08 0.09 Mother's Ed: High School 0.21 0.20 Mother's Ed: Some High School 0.27 0.28 Mother's Ed: Primary 0.39 0.37 Mother's Ed: Illiterate 0.04 0.04 House: Luxurious 0.01 0.01 House: Spacious 0.19 0.20 House: Normal 0.51 0.47 House: Deficient 0.24 0.25 House: Very Deficient 0.03 0.03 Highest Income Bracket 0.01 0.01 2nd-Highest Income Bracket 0.01 0.02 3rd-Highest Income Bracket 0.04 0.05 4th-Highest Income Bracket 0.18 0.19 5th-Highest Income Bracket 0.74 0.70 <3 Siblings 0.29 0.26 3 Siblings 0.24 0.24 4 Siblings 0.19 0.18 5 Siblings 0.12 0.13 6 Siblings 0.15 0.17 Pay: Parents 0.90 0.90 Pay: Family 0.00 0.01 Pay: Scholarship 0.01 0.01 Pay: Education Credit 0.03 0.03 Pay: Student's W ork 0.05 0.03 Trans: Own Auto 0.35 0.32 Trans: Parents' Auto 0.24 0.27 Trans: Friends 0.17 0.19 Trans: School Bus 0.10 0.10 Trans: Public Transit 0.10 0.08 Social Class: Highest 0.01 0.01 Social Class: 2nd 0.18 0.17 Social Class: 3rd 0.50 0.51 Social Class: 4th 0.29 0.28 Social Class: Lowest 0.02 0.01 Socio-Economic Status (Calculated) 0.01 0.00 Source: Authors' computations using the PEA (Prueba de Aptitud Académica) 2003. Table 2. OLS Regression Results Outcome Variable: Verbal Score Math Score Explanatory Variables Fe y Alegria Student 0.05** (0.03) 0.06** (0.01) Venezuelan Citizen -0.26** (0.01) -0.4** (0) Male Dummy 0.09** (0) 0.17** (0) Married Dummy -0.03 (0.77) -0.18** (0.03) 14 Years Old Dropped Dropped 15 Years Old -0.05 (0.59) -0.11 (0.17) 16 Years Old -0.14 (0.11) -0.21** (0.01) 17 Years Old -0.3** (0) -0.34** (0) 18 Years Old -0.46** (0) -0.47** (0) 19 Years Old -0.55** (0) -0.55** (0) 20 Years Old -0.62** (0) -0.62** (0) 21 Years Old -0.6** (0) -0.54** (0) 22 Years Old -0.48** (0) -0.61** (0) Student W orks -0.06** (0.02) -0.03 (0.29) Father's Prof: Professor or Exec 0.12** (0) 0.1** (0.01) Father's Prof: Technician 0.06 (0.1) 0.03 (0.47) Father's Prof: Employee 0.07** (0.04) 0.07** (0.05) Father's Prof: Skilled Worker 0.03 (0.38) 0.02 (0.57) Father's Prof: Unskilled Worker 0.04 (0.29) 0.03 (0.39) Mother's Ed: University 0.22** (0) 0.18** (0) Mother's Ed: High School 0.16** (0) 0.13** (0) Mother's Ed: Some High School 0.04 (0.34) 0.06 (0.16) Mother's Ed: Primary 0.04 (0.39) 0.06 (0.16) Mother's Ed: Illiterate 0.05 (0.3) 0.06 (0.19) House: Luxurious -0.14** (0.01) -0.13** (0.02) House: Spacious 0.14** (0) 0.08** (0.04) House: Normal 0.22** (0) 0.1** (0.01) House: Deficient 0.12** (0) 0.05 (0.21) House: Very Deficient 0.06 (0.19) 0.03 (0.52) Highest Income Bracket 0.03 (0.57) -0.05 (0.38) 2nd-Highest Income Bracket 0.06 (0.19) -0.08* (0.08) 3rd-Highest Income Bracket 0.13** (0) 0.02 (0.61) 4th-Highest Income Bracket 0.21** (0) 0.08** (0.01) 5th-Highest Income Bracket 0.19** (0) 0.06* (0.08) <3 Siblings 0.2** (0) 0.14** (0) 3 Siblings 0.15** (0) 0.13** (0) 4 Siblings 0.09** (0.04) 0.12** (0) 5 Siblings 0.09* (0.06) 0.07* (0.08) 6 Siblings 0.03 (0.45) 0.05 (0.27) Pay: Parents 0.03 (0.43) 0.05 (0.2) Pay: Family -0.13* (0.07) -0.04 (0.53) Pay: Scholarship 0.07 (0.26) 0.01 (0.87) Pay: Education Credit 0.14** (0.01) 0.11** (0.02) Pay: Student's W ork 0.16** (0) 0.15** (0) Trans: Own Auto -0.06** (0.03) 0 (0.89) Trans: Parents' Auto 0.05** (0.05) 0.09** (0) Trans: Friends 0.03 (0.21) 0.05** (0.04) Trans: School Bus 0.04 (0.17) 0.05* (0.06) Trans: Public Transit 0.03 (0.31) 0.04 (0.13) Social Class: Highest -0.07 (0.45) 0.02 (0.79) Social Class: 2nd -0.32** (0) -0.15* (0.06) Social Class: 3rd -0.43** (0) -0.24** (0) Social Class: 4th -0.46** (0) -0.25** (0) Social Class: Lowest -0.5** (0) -0.21** (0.02) Source: Authors' estimations. p-values in parenthesis. * significant at 10%; ** significant at 5%. Table 3. Propensity Score Estimation Equation (Probit) Coefficient P-Value Venezuelan Citizen 0.67 0.1 Male Dummy 0.12 0 Married Dummy 0.29 0.12 15 Years Old 0.04 0.87 16 Years Old 0.15 0.55 17 Years Old 0.37 0.14 18 Years Old 0.34 0.17 19 Years Old 0.22 0.38 20 Years Old 0.51 0.05 21 Years Old 0.47 0.11 22 Years Old 0.94 0.01 Student W orks -0.25 0 Father's Prof: Professor or Exec 0.04 0.74 Father's Prof: Technician 0.16 0.13 Father's Prof: Employee 0.12 0.23 Father's Prof: Skilled Worker 0.13 0.18 Father's Prof: Unskilled Worker 0.27 0.01 Mother's Ed: University 0.23 0.09 Mother's Ed: High School 0.26 0.05 Mother's Ed: Some High School 0.22 0.09 Mother's Ed: Primary 0.25 0.05 Mother's Ed: Illiterate 0.09 0.52 House: Luxurious -0.24 0.08 House: Spacious -0.21 0.03 House: Normal -0.15 0.11 House: Deficient -0.2 0.04 House: Very Deficient -0.23 0.05 Highest Income Bracket 0.06 0.73 2nd-Highest Income Bracket 0.01 0.93 3rd-Highest Income Bracket 0.07 0.5 4th-Highest Income Bracket 0.18 0.09 5th-Highest Income Bracket 0.28 0.01 <3 Siblings 0.17 0.18 3 Siblings 0.13 0.31 4 Siblings 0.15 0.25 5 Siblings 0.06 0.66 6 Siblings 0.03 0.81 Pay: Parents -0.05 0.65 Pay: Family -0.26 0.2 Pay: Scholarship -0.12 0.49 Pay: Education Credit -0.04 0.73 Pay: Student's W ork 0.1 0.41 Trans: Own Auto 0.32 0 Trans: Parents' Auto 0.22 0 Trans: Friends 0.24 0 Trans: School Bus 0.28 0 Trans: Public Transit 0.3 0 Social Class: Highest 0.2 0.47 Social Class: 2nd 0.01 0.97 Social Class: 3rd -0.1 0.69 Social Class: 4th -0.11 0.67 Social Class: Lowest 0.16 0.57 Source: Authors' estimations. Table 4. Results of Propensity Score Matching Verbal Score Math Score ATE 0.11 0.08 Standard error 0.07 0.03 Obs 46,287 46,287 Source: Authors' estimations. Table 5. Heterogeneous Treatment Effect Verbal Math Obs Class 4, 5 ATE 0.242 0.249 13,768 SE 0.101 0.095 Class 13 ATE 0.056 0.079 32,519 SE 0.059 0.032 Mother's Ed 4, 5 ATE 0.071 0.186 19,105 SE 0.042 0.049 Mother's Ed 13 ATE 0.052 0.074 27,182 SE 0.062 0.033 Source: Authors' estimations.