WPS4121 IMPACT EVALUATION SERIES NO. 10 The Impact of Private Provision of Public Education: Empirical Evidence from Bogotá's Concession Schools Felipe Barrera-Osorio1 Abstract. In 1999, the city of Bogotá, Colombia launched the Concession School program designed to broaden the coverage and quality of basic education. It consists of a contract between a group of private schools and the public educational system such that private agents provide education for low-income students. This paper tests three main hypotheses concerning the impact of concessions on the quality of education: first, dropout rates are lower in concession schools than in similar public schools; second, other public schools nearby the concession schools have lower dropout rates in comparison with other public schools outside the area of influence; third, test scores from concession schools are higher than scores in similar public schools. The paper presents evidence in favor of the three hypotheses, using propensity score and matching estimators. World Bank Policy Research Working Paper 4121, February 2006 The Impact Evaluation Series has been established in recognition of the importance of impact evaluation studies for World Bank operations and for development in general. The series serves as a vehicle for the dissemination of findings of those studies. Papers in this series are part of the Bank's Policy Research Working Paper Series. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. 1 Formerly Deputy Director of Fedesarrollo (Colombia); now Senior Education Economist, HDNED, World Bank. The World Bank provided financial support for this project. The Secretaria de Educación del Distrito provided generous help and data. I would like to thank the National Planning Department (Departamento Nacional de Planeaion, DNP) for providing some of the data used in this article. The first version of the paper was presented at the Harvard University and World Bank seminar "Mobilizing the Private Sector for Public Education." I am indebted to Raji Chakrabarti, Ronald Fryer, Brian Jacob, Harry Patrinos, Katja Vinha, and one anonymous referee for very useful comments. 1 1. Introduction: The Concession Schools Program In 1999, the city of Bogotá, Colombia launched an educational program designed to broaden the coverage and quality of basic education. The program, called Concession Schools, is a partnership between the public and private education sectors, with private schools providing public education in 25 schools for a period of 15 years. The state provides the infrastructure, selects the students and pays a pre-agreed sum per full-time student per year (approximately $1,200,000 Colombian pesos (US $520), according to Villa and Duarte, 2002), which is higher than what most regular public schools receive (approximately Col $1,000,000, or US $430). The concession schools must provide education to the population assigned to them by the state, but they are allowed relative flexibility to contract administrative and teaching staff and can freely implement their pedagogic model. The concession schools must also meet performance standards (on quality and quantity) set by the Secretary of Education. For instance, each school commits to surpass the mean score of standardized tests in similar schools. Over 25,000 students participate in the concessions program, representing close to 3 percent of the total public enrollment in the city (World Bank 2005). The schools were built following two main criteria. First, they were located in extremely poor areas of the city. Second, they were built in areas where the demand for primary and secondary education was higher than the number of places supplied by city public schools. Any student from the affected neighborhood can apply for enrollment in the concession schools. The Secretary of Education of the District uses means testing to allocate places in the concession schools among the applicants. The students not accepted into the concession schools are placed into nearby public schools. During 2004 and 2005, a team from the World Bank visited several schools in the public system, including two regular public schools, one private school that receives public students and three concession schools. During the visits and conversations with headmasters, teachers and students, it became clear that the differences across the three types of schools were very sharp. In contrast to regular public schools, concession schools had very good infrastructure, similar to good private schools. Also, the relationship between teachers and students in the concession schools 2 was open and very constructive. Finally, the concession school students were very proud of their schools. Concession schools are located in areas ranking at the lowest tail of the income distribution. Children in these zones in general face several problems, including lack of affection and other family problems, as well as poor nutrition. According to interviews with different headmasters, concessions, in their objective to provide high quality education, are providing psychological counseling and an environment of affection to students. Daily psychological counseling is offered to several students. Children that are suspected of having family problems are subject to home visits by social workers and psychologists from the school. Children that miss classes regularly are subject to home visits as well. To counter poor nutrition, several concession schools have their own food program, which complements the public program. The regular public school food program consists mainly of fruit, a piece of cake and a yogurt or a juice. Some concessions add a complete lunch, which generally includes a protein. Another clear objective of the concession schools is to work with the communities in which they operate. Several of them have an open door policy during weekends, and they encourage teacher-parent meetings on a regular basis. They work with the community through programs to resolve family problems and provide adult education. Finally, for many of the headmasters, the academic impact of the concession schools was considered a byproduct of the other measures: they need to provide psychological and nutritional balance to students before they can address academic achievement. Clearly, family problems and poor nutrition are also plague students in regular public schools. However, those schools do not have the resources to implement the strategies that concession schools use. In fact, non-concession public schools do not often have the resources to maintain what minimal infrastructure they have. For example, in both of the public schools that the World Bank team visited the bathroom facilities were limited and they were out of service on some days. 3 Despite the apparently good reputation of concession schools, in 2005 the program was under debate. Some members of Bogotá's City Council claimed that the program did not yield the expected results because the average standardized test scores were lower than the average scores for regular public schools. However, as discussed below, though the test scores for concession schools were lower according to a general comparison of means, in fact once confounding variables are controlled for, it appears that the concession schools do have a positive impact. The qualitative evidence compiled during the visits to the concession schools and the public debate in the Council induced three main hypotheses that are the subject of this paper: a. Dropout rates are lower in concession schools than in similar public schools. The dropout rate is another measure of schooling and, presumably, it responds quicker to interventions than do test scores. Through the various interventions described above, concession schools attempt to keep children in school. b. Regular public schools near the concession schools have lower dropout rates than public schools outside the influence of concessions. Concession schools have had positive externalities, through their community work, on nearby public schools. In addition, increased competition in the educational market due to the existence of concession schools has caused improvements in nearby schools. Previous literature on school choice (Hoxby 2002) supports this hypothesis. c. Concession school test scores are at least equal to or higher than test scores for regular public schools. Measuring the impact of any program on test scores can be complicated and theoretically we should observe changes in the schools after long- term exposure to the program. Given the recent implementation of the program, the most any student has attended a concession school is six years, and their early years were spent in regular public schools. Section 2 of this paper discusses methodological issues, underlining the reasons that concession schools may produce better educational outcomes than regular public schools. Section 3 outlines the empirical strategy, which is based on propensity score matching estimators. Section 4 4 presents the results. Section 5 finishes with conclusions and a discussion on the relationship between dropout rates and test scores. 2. Methodological issues and theoretical framework a. General framework The basic equation for estimation is based on the idea that educational outcomes depend on a production function based on certain inputs (Hanushek 1986, 1996). Specifically, the literature considers that a measure of educational attainment, Yi of individual i in school j in a given . j,t moment t, depends on the inputs of the school, Z j and on the characteristics of the individual, ,t X , such that: j,t Yi (1) , j,t = f (Xi ,Z ,i ) ,t j,t ,t Included among the variables of school inputs, Z j , are the student/teacher ratio, the educational ,t attainment of the teachers, school infrastructure and school size. The characteristics of the individuals, Xi , control for household income, the educational level of the parents, the number ,t of siblings, etcetera. In Equation (1), i captures the unobservable characteristics of the ,t individual, such as ability, skills and motivation that may influence test scores but cannot be measured. The dependent variables, Yi , are the standardized test scores and the dropout rate. , j,t Equation (1) depends on several premises: · The quantity and quality of the school inputs are fundamental in the production of better education, be they measured by higher standardized test scores or lower dropout rates. · The characteristics of the individuals are also critical elements in the production of better education. Individuals with better skills, better nutrition and with an adequate home environment should also perform better in school. · These two groups of characteristics (school and individual/family) interact to yield the third fundamental component in the education production function. 5 In a meta-analysis of Equation (1) Hanushek (1996) reaches several conclusions. First, there is no clear relationship between several of the school inputs and the quality of education. The pupil/teacher ratio, for example, may have a concave relationship with the outcome measures. A small number of pupils in the classroom may inhibit participation and it is possible that the positive externalities of participation do not occur. A large class size may also inhibit participation and may produce negative externalities such as noise. Second, the quality of the teacher is a key element in the production of education. And third, the role of the family is equally important as the quality of the teacher. The impact of the concession program on the quality of education can be explained through several channels (for a general discussion, see World Bank 2005). First, private participation assumes the application of an already proven pedagogic model since the private schools have already been operating for some time. Concession schools were, in fact, handed over to private schools with the highest standardized test scores. Likewise, concession schools are financially stable since the State pays a fixed sum per student, ensuring the stability of the pedagogic model. Second, the freedom to choose the teaching and administrative staff may lead to a better quality of education than in the public schools where the teachers' union makes it difficult to implement staff changes (for instance, see Borgas and Acosta 2005; Duarte 1996). Third, concession schools have contracts that establish goals in terms of standardized test. Fourth, the infrastructure of these schools is superior to that of public schools providing the students better learning opportunities. Concession schools were, in fact, built on better lots than were the average public school, with better equipment and a complete set of supplies for classrooms, laboratories and libraries. Finally, concession schools in general work actively with the pupils' parents and the surrounding communities. The community work has been one of the more discussed factors in visits to several concession schools: school officials, pupils, parents and other community members alike identify this community work as one the most important features of the initiative. b. Some empirical evidence 6 In the case of Colombia, Gaviria and Barrientos (2001), Barrera (2003) and Sarmiento and others (2004) have estimated Equation (1). These studies reach similar conclusions as Hanushek (1996). They show that school quality variables, such as the educational level of the teachers, tend to have a positive impact on standardized test scores. Also, they corroborate the importance of individual characteristics. Early articles about concessions focused on the description of the experience. Corpoeducación (2004) and Villa and Duarte (2002) describe the concession school initiative in great detail. Rodriguez (2005) presents an overview of the concession schools program, explaining the details of the program's design and potential effects of the program. Sarmiento and others (2005) present an evaluation of concession schools. They follow a different methodology than the one used here. In short, they applied a very detailed questionnaire to 22 concession schools and 10 public ones with similar characteristics. The questionnaire focused on internal processes and the educational production. Variables derived from the questionnaire were correlated with standardized test scores. They found that there are clear differences between the concession and pure public schools in terms of administration, autonomy of decisions, the capacity to adjust, and the impact on the community. Overall, they found that concession schools are performing better than the traditional public schools. Charter schools in the United States offer additional insight. Despite the differences across states in the implementation of charter school programs in the United States, they are similar to the concession schools in that public education is delivered by independent entities, with some degree of autonomy and are generally targeted to low-income students (www.uscharterschools.org). Solmon and Goldschmidt (2004) found positive impacts of charter schools on standardized test scores and other characteristics, such as the retention rate.2 In contrast, Hanushek and others (2005) present evidence of a negative impact of this type of schools. Hoxby and Rockoff (2004) present strong evidence in favor of charter schools, using the randomized character of Chicago's charter school program to determine the impact of charter 2Nelson and Hollenbeck (2001) present a critical view of Solmon and Goldschmidt. 7 schools on standardized test scores.3 The randomized nature of the allocation process allowed them to create a treatment and a control group that have the same observable and unobservable characteristics. They found that students who attended an average of two years in an elementary level charter school in Chicago scored higher on both math and reading tests. 3. Empirical strategy The basic methodology used to evaluate the impact of the program is to compare the group of individuals that were admitted to a concession school (treatment group) to the group of students who attend public school (comparison group) (see Heckman and others, 1999). The correct evaluation of a program requires the establishment of a counterfactual for the treatment group; that is, what would have happened had the treated individual not been treated. Of course, it is not possible to observe the same individual in the two states and it is therefore necessary to choose another individual, to serve as a comparison, who is as similar as possible, both in observable and unobservable characteristics, to the treated individual. Simple comparisons of mean test scores (or dropout rates) between students in concession schools with students in other types of schools may be biased measures of the true impact because the two groups may be very different in their observable and unobservable characteristics. In mathematical terms,4 let Y1 denote the standardized test score (or the dropout rate) for individual i who received the treatment (attended concession school) and Y0 if the individual went to another school. LetT =1 denote the "treatment condition" for those who received treatment, and T = 0 otherwise. The impact of the program (IP) will be given by IP = E(Y1 /T =1) - E(Y0 /T =1) , e.g., the mean test score of the same individual, with and without attending concession school. It is not possible, however, to observe the same individual in both states. What is observable are two different individuals, one attending a concession ~ school and the other not, IP = E(Y1 /T =1) - E(Y0 /T = 0). 3In fact, when there exists excess demand for school places, the majority of States allocate students to the charter schools based on lotteries. 4The basic reference for the discussion of the problem is Heckman and others (1999). This section is based on Chapter 2 in Vinha (2005), which presents a concise explanation of the problem. 8 This estimator can suffer from bias because of self-selection into the program. In short, the two types of individuals may differ systematically in observable and unobservable characteristics such that a typical control observation is not a good proxy of what would have happened to a treated individual had he not received the treatment. In the case of the concession school, the two groups, for example, are not comparable in terms of income, thus leading to bias. That is, E(Y0 /T =1) E(Y0 /T = 0) . Ideally, a program like concession schools can be evaluated using a simple randomization strategy: given that there exists excess demand for concession schools, the government can perform a lottery to allocate individuals between concessions and other public schools. The randomization would ensure that the group of students that attends a concession school (treatment group) and the one that attends other public schools (control group) are very similar in observables and unobservables characteristics. However, as discussed above, the school placements were not done by a lottery. A second approach5 would be to take as a control group students who applied to a concession school but were assigned to another public school. Still, evaluations based on the comparison of these two groups may be prone to bias, mainly because the selection of students into concession and other public schools is not random. 6 In contrast, this paper uses the whole sample of individuals who attend public schools and uses matching estimators as the strategy to "reduce" selection bias. In short, the estimator will try to match each treated individual with an individual who does not attend a concession school, based on the observable characteristics of the individuals (a vector X). In this way, the estimator can be ~ modified by IP = E(Y1 / X ,T =1) - E(Y0 / X ,T = 0) . Moreover, instead of using the vector of characteristics x, it is possible to determine the probability of participation in the program (where T is equal to 1 or 0), such that 5This approach was suggested by an anonymous referee. 6Moreover, the information on who applies to a concession school is not available to the public. 9 P(X ) = Pr(T / X ) = f (X ) . This probability is called the propensity score. In other words, the propensity score captures in a synthetic form the intention to participate in a program, based on a broad vector of observable characteristics thought to influence the participation decision and the outcome measures. For each individual (both in the treatment and the control group) a probability of participating in the program is estimated using the observable characteristics. Therefore, the following estimation is used to calculate the impact of the program: ~ ~ ~ IP = E(Y1 / P(X ),T =1) - E(Y0 / P(X ),T = 0) (2) Clearly, the endogeneity problem can arise due to either observable or unobservable characteristics. Equation (2) controls for the observable characteristics, but not for the unobservable ones. Heckman and others (1998, 1999) show that, in fact, the bias of estimation may come from three margins, and the most important one is the difference in the observable characteristics among individuals. As stated previously, it is essential for rigorous impact evaluation that the control and treatment groups have, on average, similar characteristics. Besides differences in the average treatment and non-treated individual, there may be differences in the support of the populations. For example, noneligible individuals cannot be part of the control group since there are no treatment observations with the characteristics of the noneligible individuals. The third potential difference between the groups arises from potential differences in the unobservable characteristics. According to the estimates of Heckman and others (1998), once the observable characteristics are similar between groups, and the observations fall within the same support the bias due to unobservable characteristics is relatively small. The evaluation of the concessions program in Bogota may suffer from self-selection into the program from two sources. First, the location of the schools was not randomly chosen. Authorities built the concessions in areas where there was a high concentration of low-income individuals and a scarce supply of public education. Second, an unknown proportion of the students are not randomly assigned to the schools. Individuals, when applying to public education, can state their preferred school in their neighborhood. As discussed above, the paper copes with the problem of endogeneity by using propensity scores and matching estimators. In 10 this way, I "minimize" the endogeneity problem and can obtain estimates that approximate the impact of the program. In any case, the predicted bias of the estimation is downwards since the pool of individuals that attend concessions comes from extremely poor areas. The paper presents direct and indirect impacts of the concession schools. In order to estimates these impacts, the paper separates the public schools into three categories: (i) concession schools, (ii) regular public schools close to the concession schools (that is, within the influence of a concession school) and (iii) public schools outside of the influence of the concession schools. The difference between students in concession schools and the matched students who go to public schools outside the influence of concession schools is the measure of direct impact. In contrast, the indirect impact of concession schools is measures as the difference between the students in public schools nearby the concession school and the students in similar schools outside the area of influence of the concessions. The area of influence is defined as being within one kilometer of a concession school. Given that some concessions are going to have more than one regular public school within this definition of proximity, we also rank the proximity among the nearby schools. 7 The indirect impact is governed conceptually by the idea that nearby schools may be forced to raise the quality of education to compete with concession schools, as well as respond to community pressure. This idea is similar to the argument that vouchers will increase the quality of education due to market and choice forces, as outlined by Friedman (1955). Recent literature finds evidence in favor of this type of mechanism. For instance, Hoxby (2002) shows that competition and choice (in the form of vouchers and charter schools) increase productivity of public schools significantly. In the case of Bogota, not only do the concession schools impact the nearby public schools through community outreach but they also encourage better performance by regular pubic schools due to the competition for resources. Since 2002, the central government has transferred educational resources to the localities based on the number of students enrolled in school. Presumably, concessions attract students away from the regular public school system, and the great the difference in quality between regular and concession 7One area of future research will be to test for robustness of the results using another radius for the area of influence. 11 schools, the greater the effect. In order to guarantee greater resources, the nearby public schools need to match the performance of the concession schools. To estimate the indirect impact of the program, I use a two-step process. I first identify the baseline. Given the identification of nearby schools as discussed above, I match them based on their common characteristics with similar schools outside the influence area in 1999. As stated before, concession schools started in 1999 were fully operational by 2000. These control schools are determined using propensity scores. Second, using data for 2003, I estimate the indirect impact using those students who attended schools close to a concession school as treated individuals, and those students who attended the matched schools found in the previous step as the control group. The baseline data can be used to control for differences in initial characteristics. However, another problem that may persist is the difference in pre-existing trends between the treated schools and the matched schools. Unfortunately, data prior to 1999 does not exist to test for this possibility. However, a priori, there is no theoretical ground to believe that there exist systematic differences in trends between schools under the influence of concessions and schools outside the area of influence of concessions. In short, the estimation that I present below is based on matching estimators using propensity scores to determine the treatment and control groups. The estimation will present evidence of "direct impact" (dropout rates and test scores for concession versus matched non-concession public schools outside the area of influence) and "indirect impact" (dropout rates for schools near the concession against matched schools outside the area of influence of concessions). 4. Data and Results a. Description of the data The paper uses data from two sources the Ministry of Education (surveys C600 and C100) and Institute for the Development of Higher Education (Instituto Colombiano para el Fomento de la 12 Educación Superior ICFES), which administers standardized tests in Colombia. The C600 and the C100 surveys provide general data on an important array of school characteristics and have an identification code that allows researchers to link these datasets with the ICFES test scores. The Ministry of Education collects the data directly via a questionnaire to all the schools in the country. Data are at the school and "schedule" level. The school level data present information on the number of administrative personnel, the number and level of education of teachers, the number of teachers by subject areas, the number of physicians in the school, the total number of students in the school by grade and by age, the number of students who failed a grade and the number of students who dropped out. These data are available for both 1999 and 2003. For 1999, there is also information on the physical characteristics of schools. The main variables are: furniture in the school (chairs and desks), support materials such as computers, the number of computers exclusively for teaching purposes, the total number of laboratories, the number of laboratories for specific subjects (physics, chemistry, biology, construction and farming), the total number of rooms, the number of classrooms, the number of libraries, the number of food facilities and dorms, the number of other types of rooms and the number of sports facilities (soccer fields and basketball courts). The surveys also breakdown the data at the schedule level, the time at which the school offers services. There are four main schedules during which students can attend school: in the morning (usually from 7:00 am to 12:00 pm), in the afternoon (from 12:00 pm to 4:00 pm), in the evening (from 2:00 pm to 7:00 pm); or they can attend a school with a "complete" schedule (from 8:00 am to 3:00 pm). The schedule mechanism was implemented throughout the country in the 1960s to maximize the use of school infrastructure. In this way, some schools that previously operated only during the morning were subsequently open for two schedules (e.g., morning and afternoon), thus allowing them to enroll twice the number of students. It is important to note that the data discriminate between the two schedules as if they were two different schools. 13 Second, data from ICFES provide test scores for individuals and some student characteristics. The test scores used here are those for a general test that is administered to all students who are finishing their secondary education. Although the ICFES data provides test scores for various subjects, the analysis here focuses on mathematics and reading scores. Student characteristics, such as gender and current enrollment status are also provided. Other variables linked to the individual students include: a school code, the city where the test was taken, the semester (either A, which is from February to November, or B, which runs from September to June), the type of secondary school that the individual attended (mainly academic or technical), if the school is public or private and the location of the school within the city.8 All ICFES data used in this analysis are for 2003. Unfortunately, ICFES data on individual characteristics are limited. Presumably, fixed effects by location can capture some of the socio-economic characteristics of the individuals living in the area. In fact, the data identified nine localities in the city that are quite homogeneous in terms of income. In contrast, the information on the schools from the Ministry of Education is very rich, and I will exploit this as the source of variation in order to estimate the impacts. In conclusion, in order to estimate the direct effect of the concession schools, this analysis uses the 2003 school and test score data. The panel data at the school level for 1999 and 2003 are used to estimate the indirect effect. b. Descriptive statistics Table 1 presents data of the full sample of basic characteristics of public schools, divided into concession (close to 1,050 individuals in 17 schools) and non-concession schools (36,000 individuals in 2,790 schools). Eligibility requirements preclude the use of data on private schools, which may downplay the competition argument, though only slightly since there is still competition for resources among public schools. 8The data identifies nine main geographical areas (subdivisions of location of the school in the north, south, west and east parts of the city). 14 The distribution of locations is quite similar between the two groups. The majority of the schools are co-ed. In terms of the type of secondary school, the majority of concession schools provide a classic academic education, whereas 30 percent of the public schools provide technical education. The distribution of schedules across public and concession schools is quite different: concession schools use complete schedules, whereas public schools are divided between morning, afternoon and evening schedules. Even though there are differences in the two groups, the estimators presented below balance the two samples of students in terms of observable characteristics. This is very important since the estimators have to isolate the effect of concessions from differences in other variables. For example, new literature on the length of school day and the quality of education shows a positive relationship between the two (Cerdan and Vermeersch, 1996). On average, public schools tend to be bigger than concession schools. The number of students in grades lower than their age group ("overage") is lower in concession schools than in the rest of public schools. The teacher-student ratio is quite similar (close to 28 students per teacher), as is the average years of education of teachers (16 years). The table also shows data on the two main impact variables, dropout rates and test scores. Dropout rates, measured by grade for grades 1 through 11, differ between the two types of schools. The dropout rate is 18 percent in regular public schools (with a standard deviation of 0.14), and 15 percent in concession schools (with a standard deviation of 0.09). 15 Table 1. Basic statistics (public schools) Non-concession Concession Mean Std. Dev. Mean Std. Dev. A. CHARACTERISTICS Localization 1 0.10 0.30 0.12 0.33 2 0.01 0.10 0.00 0.00 3 0.05 0.22 0.00 0.00 4 0.06 0.24 0.00 0.00 5 0.15 0.36 0.12 0.32 6 0.04 0.19 0.00 0.00 7 0.30 0.46 0.44 0.50 8 0.12 0.32 0.13 0.34 9 0.17 0.38 0.19 0.40 Type of school Male 0.07 0.26 0.00 0.00 Female 0.03 0.16 0.00 0.00 Coed 0.90 0.30 1.00 0.00 Type of secondary Academic 0.63 0.48 0.93 0.26 Technical 0.29 0.45 0.00 0.00 Both 0.07 0.26 0.07 0.26 Schedule Complete 0.03 0.18 1.00 0.00 Morning 0.40 0.49 0.00 0.00 Afternoon 0.14 0.35 0.00 0.00 Evening 0.43 0.49 0.00 0.00 Number of students Total school 2954.60 1376.76 987.98 90.17 Taking exam 122.15 79.16 65.33 9.98 In overage 47.28 45.15 12.11 7.03 Repeating grade 103.30 73.72 47.56 35.93 Ratio teacher/stud. 28.44 6.20 27.20 3.48 Years of education, 16.07 2.28 16.02 0.99 teachers B. OUTCOMES Dropout rates Grade 1 0.040 0.05 0.020 0.02 Grade 2 0.031 0.04 0.023 0.02 Grade 3 0.026 0.04 0.019 0.02 Grade 4 0.026 0.03 0.015 0.02 Grade 5 0.020 0.33 0.018 0.02 Grade 6 0.071 0.09 0.021 0.02 Grade 7 0.040 0.04 0,025 0.02 Grade 8 0.038 0.04 0.032 0.03 Grade 9 0.039 0.05 0.036 0.03 Grade 10 0.033 0.04 0.038 0.04 Grade 11 0.012 0.02 0.021 0.05 Test scores Mathematics 42.08 5.13 41.68 4.77 Reading 50.99 7.13 51.04 6.79 Number of observations Students Max 36244 1056 Min 34218 1013 Schools 2790 17 16 Graph 1 shows the dropout rate per grade. The dropout rates in concession schools are lower and exhibit smooth behavior across grades. In contrast, public schools mimic the behavior of the dropout rates found in others datasets using national statistics (for example, Barrera and Dominguez 2005), in which there is a significant increase in the dropout rate in the transition between grades five and six (last year of primary and first year of secondary). Graph 1. Drop-out rates, concessions versus non-concessions 0,08 0,07 0,06 0,05 0,04 0,03 0,02 0,01 Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10 Grade 11 Non-concession Concession Interestingly, dropout rates for concession schools increase significantly in grade 8. This may indicate a change in the composition of students in secondary education in the concession schools. Since attending a concession school reduces the dropout rates for grade 6, more students, including those who would have dropped out in the regular public system, are reaching higher grades. The population of students in secondary education in the concession schools includes those in the lower part of the income distribution, who have a higher probability of dropping out of school. 17 Graph 2 presents mean standardized test scores for private, regular public and concession schools. It shows that there are no clear differences between regular public schools and concession schools, whereas there exist substantial differences between all public and privates schools. However, concession schools deliver higher mean test scores than similar public ones after controlling by observable characteristics. Graph 2. Mean test scores 54 52 50 48 46 44 42 40 Private Public non-concession Concession Mathematics Physics Biology Reading Graph 3 presents mean test scores against the standard deviation of the scores by school type. There is evidence of a positive relationship between mean test score and dispersion. The graph also shows that public schools are more homogeneous (as measured by dispersion) than private schools. 18 Graph 3. Mean test scores against standard deviation Private Public non-conces. Concession 55 50 Mean 45 40 0 5 10 0 5 10 0 5 10 Std. Dev Table 2 presents basic statistics for the public schools within a kilometer of a concession school, as well as public schools outside the influence of concessions. 19 Table 2. Basic statistics, schools near concession versus far away schools A. CHARACTERISTICS OF SCHOOLS Schools near a concession Schools far away from 1. "Before concessions" (1999) Mean Std.dev Mean Std.dev Infrastructure Number of computers 17.82 16.75 24.96 22.69 Number of classrooms 21.40 10.58 22.24 12.93 Number of bathrooms 1.45 0.78 1.67 0.87 Number of sport facilities 1.78 0.54 2.59 1.52 Teachers Number of teachers 30.27 16.34 34.46 24.06 Number of admin. pers. 6.97 4.45 9.69 11.96 Number of physicians 0.06 0.24 0.45 1.10 Number of psychologists, etc 1.52 1.29 2.19 1.85 % teachers with second. educ 0.07 0.18 0.06 0.12 % teachers with college educ 0.60 0.23 0.60 0.62 % teachers with graduat. educ 0.25 0.17 0.34 0.34 Ration students / teachers 24.92 3.24 26.16 28.78 Drop-out rates 0.09 0.07 0.07 0.06 2. "After concessions" (2003) Localization 1 0.00 0.05 0.10 0.30 2 0.00 0.00 0.01 0.10 3 0.00 0.00 0.05 0.22 4 0.04 0.19 0.06 0.24 5 0.20 0.40 0.15 0.36 6 0.00 0.00 0.04 0.19 7 0.15 0.36 0.30 0.46 8 0.45 0.50 0.12 0.32 9 0.16 0.37 0.17 0.38 Type of school Male 0.00 0.00 0.07 0.26 Female 0.00 0.00 0.03 0.16 Coed 1.00 0.00 0.90 0.30 Type of secondary Academic 0.80 0.40 0.63 0.48 Technical 0.06 0.24 0.29 0.45 Both 0.14 0.35 0.07 0.26 Schedule Complete 0.04 0.20 0.03 0.18 Morning 0.28 0.45 0.40 0.49 Afternoon 0.07 0.25 0.14 0.35 Evening 0.61 0.49 0.43 0.49 Number of students Total school 2544.18 786.25 2954.60 1376.76 Taking exam 136.97 69.02 122.15 79.16 In overage 35.52 35.30 47.28 45.15 Repeating grade 114.93 88.69 103.30 73.72 Ratio teacher/stud. 31.42 2.56 28.44 6.20 Year of education, teachers 16.49 0.73 16.07 2.28 Drop-out rates Grade 1 0.035 0.04 0.040 0.05 Grade 2 0.030 0.03 0.031 0.04 Grade 3 0.030 0.03 0.026 0.04 Grade 4 0.018 0.02 0.026 0.03 Grade 5 0.021 0.02 0.020 0.33 Grade 6 0.054 0.06 0.071 0.09 Grade 7 0.029 0.03 0.040 0.04 Grade 8 0.051 0.04 0.038 0.04 Grade 9 0.046 0.03 0.039 0.05 Grade 10 0.040 0.03 0.033 0.04 Grade 11 0.012 0.02 0.012 0.02 Number of schools 23 416 20 In 1999, regular public schools near concession schools had, on average, fewer resources than other public schools. For instance, the number of computers in schools near concession schools was only 17, whereas it was 25 in schools in other parts of the city. Also, the absolute numbers of teachers, administrative personnel, physicians and psychologists were lower in regular public schools near concession schools. The educational attainment of teachers in 1999, measured as the proportion of teachers with secondary studies to those with college or graduate studies, was very similar across the two school types. Finally, regular public schools near concession schools had, on average, 24 students per teacher, compared to 26 students per teacher in the rest of the public system. In sum, there is evidence that regular public schools near concession schools were smaller and poorer than the rest of the public system, although the quality of the teachers was similar. Given that concession schools are located in low-income areas, these differences are not surprising and it is likely that any result based on an unmatched sample will be biased. In 2003, in general, the regular public schools near concession schools were similar to the rest of the public schools, with some exceptions. The proportion of secondary schools with an academic focus was higher in nearby schools than in public schools further away. Also, a higher percentage of the nearby schools used the evening schedule (0.61 versus 0.43, respectively). Furthermore, the nearby schools have lower overage ratios. Regular public schools near concession schools tended to have lower rates than other public schools in 2003. The dropout rate increases from 2.1 percent in grade 5 to 5.1 percent in grade 6, whereas the dropout rate for other public schools rises from 2 percent in grade 5 to 7.1 percent in grade 6. However, the dropout rates in regular public schools near concession schools tend to be higher than the ones in the rest of public schools at higher grade levels. Again, these results are based on the unmatched samples and thus the controls may not be a good counterfactual. c. Results Estimates of Equation (2), the average treatment impact on the treated, are presented below. There are several different estimators, ranging from the nearest neighbor to kernel estimators, that can be used to determine the counterfactual. In this paper the estimation is based on the 10 21 nearest neighbors matching estimator, with a caliper of 0.01; that is, the counterfactual is made of only those observations with a propensity score within 0.01 of the propensity score of the treated observation. All variables outside the common support were excluded (see Vinha 2005). This analysis also checks the robustness of the estimation. As a first approximation, results for the dropout rate are quite stable and independent of the type of estimator; on the contrary, the results for the impact on test scores vary with the type of estimation. The results for the direct and indirect effects are presented in Table 3. The first two columns present the results from the estimation of Equation (3) for concession schools, whereas columns three and four present the results for the indirect impact on nearby regular public schools. The estimation uses data at the school and grade level. In general, the coefficients of the probit estimation to determine the propensity scores are the move in the expected direction for both types of impacts. I test for the balance of characteristics across treated and control schools. The groups are balanced in all variables except two (number of students and proportion of students in grades lower than their age). In order to improve the balance in the two samples, I omit observations with the extreme propensity scores from the analysis. In any case, one variable remains unbalanced. 22 Table 3. Direct and indirect impact of concessions: effects on dropout rates Direct impact on dropout rates Indirect impact on dropout rates Probit estimates Dependent variable Concession Schools near concession Coefficient Std. Err. Coefficient Std. Err. Total number of students -0.008 0.001 * -0.015 0.003 * Dispersion in the test -0.366 0.080 * -0.708 0.178 * Localization 4 13.472 8.162 * Localization 5 0.024 0.168 9.129 8.556 Localization 7 0.212 0.138 13.929 8.089 * Localization 8 -0.052 0.163 12.411 8.234 Localization 9 0.048 0.155 12.455 8.311 Schedule Night -0.243 0.173 1.966 0.430 * Ration teacher / stud. -0.284 0.046 * 0.514 0.094 * Average year of educ. teac. -0.035 0.025 0.080 0.079 % of students repeating -1.723 0.351 * 1.407 0.502 * % of students in overage 3.327 0.719 * 5.040 1.709 * Number of computers, 1999 0.027 0.020 Number of class rooms, 1999 0.053 0.038 Number of restrooms, 1999 -4.686 0.719 * Number of sport facilities, 1999 -1.723 0.260 * Number of teachers, 1999 -0.476 0.278 * Number of administrative p. 1999 0.202 0.050 * Number of psychologist, 1999 -0.069 0.187 Ration teacher / stud., 1999 0.059 0.038 % teachers with second., 1999 35.714 10.066 * % teachers with college, 1999 33.968 9.841 * % teachers with grad. st., 1999 25.912 8.898 * Log likelihood -529 -231 PseudoR2 0.1391 0.5714 Number of obs. 2334 1217 Matching Impact variable: Dropout rate Impact variable: Dropout rate Variable: Dropout rate Difference Treatment-Control Unmatched -0.0130 -0.0069 ATT -0.0173 -0.0082 Bootstrap statistics Repetitions 1000 1000 Standard error 0.0039 0.0096317 Bias-corrected conf. Inter. -0.0244 -0.0112 -0.0341 0.0004 23 The lower part of Table 3 presents the impact of the program on dropout rates. The results show that concession schools have lower dropout rates (with a difference of 1.7 points) than similar public schools. Given the range of dropout rates (between 1.2 percent and 7.1 percent), this is an important and large effect. In order to find the standard error for the estimator, a bootstrap procedure was performed with 1000 repetitions. The effect is statistically different from zero at the 90 percent level of significance. Columns three and four present evidence of the indirect impact. That is, the probit is run with a dependent variable that equals one for regular public schools near concession schools and zero for public schools outside the influence of concessions. Again, in general, the probit estimates move in the expected direction and several of them are statistically significant. The estimated indirect impact of concession schools on nearby public school dropout rates is a reduction of 0.008 points. However, the standard error for the estimator (using a bootstrap procedure with 1000 repetitions) indicates that the 90 percent confidence interval is between -0.03 and 0.0004. Thus, I cannot rule out that there was no impact. Table 4 presents estimations of the direct impact of concession schools on the standardized test scores in mathematics and reading. The data are at the individual level. As in the previous regressions, the majority of the coefficients of the independent variables in the probit estimation move in the expected direction, and several of them are statistically significant, with one caveat: in contrast to the previous estimation, there are several variables that are not balanced in the treatment and control individuals (results are not show). To address this, I reduce the range of propensity scores in which the matching is performed, with some improvements on the balance of the samples. The implication is, again, that the treatment impact is for those individuals within this range of propensity scores and not for the whole population. The results hold for both estimations (math and reading). The calculated impact is positive and statistically significant for both reading and mathematics test scores. Mathematics scores for concession schools are almost one point higher than similar regular public schools and the effect is significantly different from zero. The impact on reading scores for concession schools is higher, with an estimated impact of almost 2 points. Given that 24 the average scores are 42.08 and 50.99, the results imply improvements of 2.4 percent and 4 percent, respectively. Table 4. Direct impact of concessions: effects on test scores (mathematics and reading) Direct impact on math tests Direct impact on reading tests Probit estimates Dependent variable Coeff. Std. Err. Coeff. Std. Err. Total number of students -0.018 0.001 * -0.018 0.001 * Dispersion in the test -0.351 0.033 * -0.197 0.029 * Localization 5 -0.441 0.079 * -0.542 0.080 * Localization 7 0.149 0.063 * 0.140 0.064 * Localization 8 0.073 0.075 0.079 0.076 Localization 9 -0.150 0.071 * -0.181 0.072 * Evening sche. 0.102 0.072 0.068 0.073 Ration teacher / stud. -0.071 0.005 * -0.072 0.005 * Average year of educ. teac. -0.089 0.012 * -0.095 0.011 * % of students repeating -2.508 0.183 * -2.575 0.179 * % of students in overage 19.165 0.779 * 18.789 0.774 * Constant 4.683 0.300 * 4.474 0.330 * Log likelihood 2152.87 -2846.556 Pseudo R2 0.2768 0.268 Number of obs. 18629 18630 Matching Impact variable: Mathematics Impact variable: reading Variable: Test scores Difference Treatment-Control Unmatched -0.2099 0.6234 ATT 0.9732 1.9364 Bootstrap statistics Repetitions 100 100 Standard error 0.522 0.748 Bias-corrected conf. interv. 1.2684 1.4655 1.5501 2.8472 25 It is important to note that a simple comparison between concession schools and other public schools shows that concession school have, on average, lower math tests. Indeed, the unmatched impact is -0.2099. However, once the estimation controls for the observable characteristics, the impact of concession on the score is positive and significant. Lastly, Graph 4 presents the relationship between test scores and dropout rates. In theory, better quality of education induces, in the short run, reduced desertion. Students who stay in school are more motivated and study more, which is likely to lead to higher test scores in the future. The graph presents long-term evidence of this assumption. Presumably, concession schools will have a larger impact on test scores the longer that students stay in the program. Graph 4. Relationship between mean test scores and dropout rates .4 .3 Drop out rate .2 .1 0 40 42 44 46 48 50 Mean, math test 26 5. Conclusions In conclusion, there is strong evidence of a direct impact of concession schools in reducing dropout rates. There is also some evidence of an indirect impact of the concession schools on the dropout rates in nearby regular public schools. Furthermore, there is evidence of a positive impact on test scores of students in concession schools when compared with students in other public schools. Moreover, as discussed previously, there is some evidence of downward bias in the estimations. Indeed, OLS estimators are lower than the matching estimators. This finding strengthens one important idea of the paper: given that propensity and matching estimators correct only partially the sources of bias, presumably the effects of concessions are larger than the ones presented here. The results for dropout rates are especially important in light of the current situation in the country. The enrollment rate in secondary education in the cities is reaching levels of 85 percent but the dropout rates are higher in the transition from primary to secondary education. Concession schools seem to be a promising intervention for reducing desertion from schools at this critical juncture. However, as public investment in each concession school is around $2.5 million, it is important to perform a cost-benefit analysis. The test score results are also promising, especially considering that they are just the initial impacts. The concession schools program started in 1999, four years before the test score data were collected. Individuals who took the exam were in grade 11, and therefore, they most likely transferred into the concession schools in grade 8. Presumably, the impact on individuals who start in a concession school from grade 1 onward will be higher. Instruments for improving the quality of education are limited. In fact, the educational sector in Colombia, and most likely in several other countries, is subject to inflexible policies adopted in the past under different conditions. Concession schools are an option that may be able to generate consensus as a positive measure. However, the potential scale of any such program may be limited. The program relies on private, high quality schools to manage public schools. 27 Clearly, there are only a limited number of such schools and, of those, even fewer may participate in the program. 28 References Barrera, F and C. Dominguez. 2005. "Educación Básica: Opciones futuras de política", work in process for the Misión para el Diseño de una Estrategia de Reducción de la Pobreza y la Desigualdad, Fedesarrollo. Barrera, F. 2003. Decentralization and Education: An Empirical Evaluation. Ph.D. dissertation, Department of Economics, University of Maryland College Park. Borgas, G. and O. L. Acosta. 2005. "Educational Reform in Colombia," in Institutional Reforms, A. Alesina (ed), Cambridge, MA: The MIT Press. Cerdan, P. and C. Vermeersch. 2006. "More Time is Better: An Evaluation of the Full Time School Program in Uruguay." Impact Evaluation Working Paper Series, World Bank. Corpoeducación. 2004. "Evaluation of Concessions Program." Unpublished document, Bogotá. Duarte, J. 1996. "La debilidad del Ministerio de Educación y la politización de la educación: dos problemas a enfrentar en el plan decenal." Coyuntura Social 14: 145-167. Friedman, M. 1955. "Role of Government in Education," in Economics and the Public Interest, ed., Robert Solo. New Brunswick, NJ: Rutgers University Press. Gaviria, A. and J.H. Barrientos. 2001. "Características del plantel y calidad de la educación en Bogotá." Coyuntura Social 25: 81-98. Hanushek, E.A. 1986. "The Economics of Schooling: Production and Efficiency in Public Schools," Journal of Economic Literature 24(3): 1141-1177. Hanushek, E.A. 1996. "Measuring Investment in Education." Journal of Economic Perspectives 10(4): 9-30. Hanushek, E.A., J. F. Kain, S. G. Rivkin and Gregory F. Branch. 2005. "Charter School Quality and Parental Decision Making With School Choice," NBER Working Papers 11252, National Bureau of Economic Research, Inc. Heckman, J., H. Ichimura, J. Smith and P. Todd. 1998. "Characterizing Selection Bias Using Experimental Data." Econometrica 66(5): 1017-98. Heckman, J., R. LaLonde and J. Smith. 1999. "The Economics and Econometrics of Active Labor Market Programs," in O. Ashenfelter and D. Card, eds., Handbook of Labor Economics, North Holland. Hoxby, C. 2002. "School choice and school productivity (or, Could School Choice be a Tide that Lifts all Boats?)." NBER Working Paper No 8873. 29 Hoxby, C. and J. Rockoff. 2004. "The Impact of Charter Schools on Student Achievement," HIER Working Paper. Nelson, C. and K. Hollenbeck. 2001. "Does Charter School Attendance Improve Test Scores?: Comments and Reactions on the Arizona Achievement Study," Staff Working Papers 01- 70, W.E. Upjohn Institute for Employment Research. Rodríguez, A. 2005. "CASE STUDY: Public School Concession model of Bogotá, Colombia," unpublished document, the World Bank. Sarmiento A, L Becerra y J González. 2000. "La incidencia del plantel en el logro educativo del alumno y su relación con el nivel socioeconómico". Mayo de 2004, borrador, DNP. Sarmiento, A., C.E. Alonso, G. Duncan y C.A. Garzon (2005) "Evaluación de la Gestión de los Colegios en Concesión en Bogotá 2000-2003". Documento de trabajo, DNP. Solmon, L. and P. Goldschmidt. 2004. "Comparison of Traditional Public schools and Charter Schools on Retention, School Switching and Achievement Growth" Policy Report, No. 192, Goldwater Institute. Villa L. and J. Duarte. 2002. "Los colegios en concesión de Bogotá, Colombia: una experiencia innovadora de gestión escolar". Red de Educación, Documento de trabajo, Banco Interamericano para el Desarrollo. Vinha, K. 2005. The impact of the Washington Metro on development patters, Ph.D. dissertation, Department of Agricultural and Natural Resource Economics, University of Maryland, College Park. World Bank. 2005. Colombia: Contracting Education Services (Report No. 31841­CO). Colombia and Mexico Country Management Unit, Education Unit, Human Development Department, Latin America and the Caribbean Region. 30