WPS5705 Policy Research Working Paper 5705 How Do Ex Ante Simulations Compare with Ex Post Evaluations? Evidence from the Impact of Conditional Cash Transfer Programs Phillippe Leite Ambar Narayan Emmanuel Skoufias The World Bank Poverty Reduction and Economic Management Network Poverty Reduction and Equity Unit June 2011 Policy Research Working Paper 5705 Abstract This paper compares the ex ante simulation of the poverty. The estimates reveal that ex ante predictions of impacts of conditional cash transfer programs against the certain impacts of conditional cash transfer programs ex post estimates of impacts obtained from experimental match up well against the benchmark estimates of ex post evaluations. Using data on program-eligible households experimental studies. The findings seem to support the in treatment areas from the same baseline surveys that use of this model to assess the potential impact and cost are used for experimental evaluations of conditional cash efficiency of a conditional cash transfer program ex ante, transfer programs in Mexico and Ecuador, the authors in order to inform decisions about how the program use a micro-simulation model to derive ex ante estimates would be designed. of the impact of the programs on enrollment rates and This paper is a product of the Poverty Reduction and Equity Unit, Poverty Reduction and Economic Management Network. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http:// econ.worldbank.org. The author may be contacted at eskoufias@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team How Do Ex Ante Simulations Compare with Ex Post Evaluations? Evidence from the Impact of Conditional Cash Transfer Programs Phillippe Leite Ambar Narayan Emmanuel Skoufias1 Keywords: Experiments, Conditional Cash Transfers, Mexico, Microsimulation, Ecuador. JEL Classification: C21, C52, I38, I2, I1 1 Phillippe Leite is an Economist in the Human Development Network (HDNSP) of the World Bank. Ambar Narayan and Emmanuel Skoufias are a Senior Economist and a Lead Economist, respectively, in the Poverty Reduction and Equity unit (PRMPR) in the PREM network of the World Bank. The authors are grateful to Francisco Ferreira for his input in the early stages of this work, to Tarhat Shahid for excellent research assistance, and to the Multi-Donor Trust Fund for Poverty & Social Impact Analysis for partial support. The authors share equal responsibility for the contents of this paper. The findings, interpretations, and conclusions are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent. I. Introduction Conditional Cash Transfer programs (CCTs) have been used by many governments in their attempts to reduce poverty and inequality and achieve longer term human development. CCTs are typically intended to generate incentives among the poor to improve human capital so that they can break free of poverty traps across generations. While CCT programs in different countries may vary in their design and purpose, they tend to share a few basic features. All of them involve transferring resources (usually in the form of cash) to poor households, which are usually provided directly to the mother with reciprocal conditions imposed on the household, to encourage changes in behavioral norms. These conditions commonly include requiring, for example, that children attend school and/or individuals have regular health facility visits. Different methods of evaluation have been developed to analyze the likely impacts of a monetary transfer on human capital accumulation and poverty alleviation. The array of literature on program evaluation presents two broad types of analysis, ex post and ex ante. Ex post evaluations can in turn be divided into two categories: experimental and quasi- experimental. Experimental analyses are those where households or geographic units are randomly assigned to both treatment and control groups prior to the implementation of the program. Quasi-experimental evaluations are those where treatment and control groups are not selected through random assignment (of households or geographic units), with the result that statistical techniques are necessary to correct and control for observable and unobservable differences between the two groups that might affect outcomes of interest. Ex post evaluation methods have major advantages in terms of generating statistically valid results about the impact of a monetary transfer through a CCT program, against a counterfactual of not having the program. However, such methods can also have a few important disadvantages, depending on the type of techniques used. Experimental ex post evaluations may provide more statistically valid and robust results relative to quasi- experimental methods, but they are often costly and time-consuming to implement, particularly on a large scale. Quasi-experimental techniques, while typically less costly and easier to implement than experimental analyses, have the disadvantage of requiring a high level of statistical complexity to account for selection bias and the effects of unobservable characteristics. 2 Regardless of the method used, an exclusive reliance on ex post evaluations is also likely to provide little evidence prior to implementation about the potential impact of programs– a question that policymakers are often most interested in, while designing large programs or reforms. Often, the design and implementation of large income transfer programs cannot be delayed till a series of pilots “testing” various design elements have been completed and evaluated using ex post techniques. In other cases, even when the pilot(s) have been tested and evaluated, concerns about “external validity” imply that the policymaker cannot be sure whether a program implemented on a much larger scale than the pilot would have similar impacts or lack thereof. Moreover, policymakers are often interested in questions like the sensitivity of outcomes to amounts of transfers, the differences between the impacts of conditional and unconditional transfers, or the efficiency and effectiveness of alternative program designs. While testing alternative program designs is possible in theory using ex post methods, the complications that arise in implementing and evaluating the relative impacts of multiple “treatments” make such evaluations extremely difficult in practice. Ex ante evaluations involve simulating the effects of a program on the basis of a household model, in most cases using a data set representative of all program beneficiaries. Such techniques, while having the disadvantage of not measuring the actual impacts of a program, can be particularly useful in addressing some of the questions mentioned above. At the same time these techniques, which rely on structural models of economic behavior, are often criticized about the strong underlying assumptions that are necessary. Bourguignon, Ferreira and Leite (2003) (henceforth BFL) propose such a behavioral model that allows policymakers to evaluate the potential effect of a CCT on poverty, inequality and school enrollment rates. The model combines arithmetic methods with models of demand for schooling, generating predictions about the potential occupational choice (one of which is the decision of going to school) of a child living in a region where a CCT is implemented. The BFL model generates estimates of likely effects of a monetary transfer program, comparing the simulated occupational choice of children with the status quo (absence of the CCT), keeping everything else constant. The evaluation literature in most part considers ex ante and ex post methods as competing or substitute methods. Ravallion (2008) underscores that the two approaches can instead be complementary – combining an ex post evaluation with an ex ante structural model of schooling 3 choices would allow policymakers to expand the existing set of policy alternatives in considering the optimal design of a given program. While this is true in principle, in order for the two approaches to complement each other in the context of a specific program, the consistency of the results produced by the two approaches would be an important consideration. The concerns about the validity of assumptions that underlie the ex ante behavioral models make it all the more important that their predictions are checked for consistency with the results of ex post experimental evaluations. Any evidence to suggest that ex ante approaches yield results that are at least broadly consistent with those from methodologically sound experimental evaluations would strengthen the case for using such approaches to complement ex post evaluations of similar programs in future, where the relative ease of employing ex ante approaches can be particularly useful in evaluating alternative design options before a program is implemented. Our paper represents an attempt to generate evidence on the consistency between the two approaches in the context of CCT programs by comparing ex ante predictions with results of ex post experimental evaluations, following the suggestion of Bourguignon and Ferreira (2003). Specifically, we compare the school enrollment, poverty and inequality impacts of a CCT program generated ex ante, using the behavioral model suggested by BFL, with the corresponding impacts estimated ex post from a randomized experiment involving the same program. The exercise is conducted for a number of CCT programs, of which the results of two exercises are presented in detail. The comparison presented in this paper is a validation test of the strongest hypothesis of the BFL model: that the cross-sectional income effects estimated ex ante with a representative household sample would coincide with the ex post estimates of income effects generated by the program. The ex ante simulation of the impact of CCTs on enrolment rate and poverty is conducted using the same baseline surveys that are used for the experimental evaluations of PROGRESA in Mexico and Bono de Desarrollo Humano (henceforth BDH) in Ecuador. The full set of ex ante predictions are then compared with the highly credible and accepted ex post results presented in Skoufias and Parker (2001) and Schultz (2000) for PROGRESA and Schady and Araújo (2008) for BDH. The choice of these programs for the exercise is partly to do with the fact that they represent large and well-known CCT programs that contain many of the typical features of such programs. Equally important for our purposes is that for both these programs, the randomization of program implementation allowed researchers to conduct 4 methodologically sound experimental evaluations. As a result, the results of these evaluations provide an appropriate benchmark against which ex ante predictions using the BFL model can be compared. The use of the baseline data from these evaluations to generate the ex ante predictions ensures that the ex ante simulations are based on the very same household sample on which the program impacts are estimated ex post. Section II below reviews the literature most relevant to our analysis. Section III provides a synopsis of the BFL model, including its underlying assumptions and how it works. Section IV discusses the two CCT programs on which the BFL model will be applied (PROGRESA and BDH), the data to be used for the analysis and the ex post evaluation studies that provide benchmark results. Section V presents the results of the comparison between the predictions of the BFL model and results of ex post evaluations. Section VI concludes the paper. II. Review of the Literature Our review of relevant literature consists of two related parts. Given the topic of our paper, it is important to discuss earlier work done by a number of researchers comparing results from ex ante models and ex post evaluations. Since we use results from experimental ex post evaluations as the yardstick to test the validity of ex ante predictions, it is also important to look back at some of the seminal literature, going back to a few decades, concerned with the predictive power of non-experimental estimators against the benchmark of experimental evaluation results. The act of forecasting outcomes is frequently subject to criticism, as it relies on estimation of parameters that are sensitive to model specification and to non-observable errors. Forecasts about the impacts of social programs are typically validated by comparing results with the estimated treatment effects from randomized experiments when they are available. Examples of such analyses include Todd and Wolpin (2004, 2006), Attanasio et al (2002) and de Janvry and Sadoulet (2006) for the Mexican case and Lise et al (2003) for the Canadian Self-Sufficiency Project (SSP).2 2 The SSP is a social program providing time-limited earnings supplements to Income Assistance recipients who are able to obtain full time employment within a 12 month period. 5 Todd and Wolpin (2006) use an ex ante model to generate child schooling estimates that compare reasonably well with experimental results. For children aged 12-15 years, the model predicts the effects of PROGRESA on school enrollments to be in the same direction as experimental results, but underestimates them by 21-25 percent. However, when the sample is disaggregated into age groups of 12-13 years and 14-15 years, the results compare less well, particularly among boys. The ex ante model does not predict any change in outcomes for boys aged 12-13 years and significantly overestimates the effects of the program for the 14-15 years age group. Todd and Wolpin also use a particular sample that is not comparable with the one used by Skoufias and Parker (2001) and Schultz (2000) to generate ex post results. They restrict their sample of analysis from the original 24,077 households surveyed under PROGRESA to only landless households where the spouse of the household head is a woman under 50 years of age, reducing their data sample to 3,401 households (and then by 209 more due to data problems). Attanasio et al (2005) present a complex model to simulate ex ante the likely effects of PROGRESA, as well as that of a hypothetical program that differs from PROGRESA in the way the transfers vary by the grade attended by the child. They do not compare the results of the first exercise with ex post results and instead only present the ex ante program impacts for different age groups. According to our computations using their estimates, the change in enrollment due to PROGRESA predicted by their model translates to enrollment increases of approximately 2 and 5.4 percent among children of age 10-13 years and 14-17 years respectively.3 In comparison, the enrollment increases attributed to PROGRESA – as estimated by the ex post evaluations of Skoufias and Parker (2001) and Schultz (2000) – are 2.4 and 7.5 percent for children of age 10-13 years and 14-17 years, respectively. The Attanasio et al model is thus able to generate predictions that match up quite well with ex post evaluation results, even as its complexity and challenges of implementation are considerable.4 Leite (2007) applies the BFL model to data from PNAD (Pesquisa Nacional por Amostra de Domicílios) 1999 in Brazil to forecast an increase of 3.9 percent in the enrollment rate of poor 3 Our computations involve extrapolating from the enrollment difference presented in Figure 1 of Attanasio et al (2005), by assigning weights to each age group in Figure 1 and using the baseline survey. 4 These authors also propose the use of a structural model to analyze issues that remain unexplored by standard Difference-in-Difference (henceforth DID) estimators. For example, they estimate that the performance of PROGRESA could have been improved by offering more resources to older age cohorts of children such as secondary school students and less to younger age cohorts or primary school students. 6 children of age 10-15 years. This result is in the ballpark of the actual 3 percent increase in enrolment for the same cohort of boys between 1999 and 2003 estimated from PNAD 2003, which is not a result from an impact evaluation. The forecast from the BFL model is also close to the results from a quasi-experimental evaluation of the Bolsa Escola program by Cardoso and Souza (2004), in which a matching estimator suggests a 3.1 percent increase in the enrollment rate in boys and 3.0 percent increase for girls aged 10-15 years associated with the Bolsa Escola. De Janvry and Sadoulet (2006) also use PROGRESA data to develop a predictive model. They analyze the potential for increase in school attendance in order to help increase the efficiency of such transfers, concentrating on secondary school attendance and the extent to which it can be affected by a conditional transfer. Lise et al (2003) work with data from the Canadian Self-Sufficiency Project (SSP) and construct a dynamic, partial equilibrium model to simulate the effects of the program in terms of labor market behavior and compare these to observed treatment effects on individuals. The SSP is a social program providing time-limited earnings supplements to Income Assistance recipients who are able to obtain full time employment within a 12 month period. Recipients (i.e. the treatment group) of SSP were picked at random from among those eligible for these benefits. Lise et al restrict their sample to single mothers only, with around 2,300 women in the control and treatment groups alike.5 Their ex ante model is calibrated to control group data and the experiment is simulated within this model to imitate the “welfare-to-work transition of the treatment group.”6 The results are then compared with those of an ex post evaluation of the SSP, which produces similar results. An earlier body of literature, including LaLonde (1986), Heckman and Hotz (1989) and others, use experimental results as benchmarks for evaluating the predictive power of partial equilibrium non-experimental estimators, mainly in the context of labor market programs. These papers concluded that non-experimental methods were not effective in evaluating program impacts. In more recent literature, propensity score matching was extended with 5 Lise et al’s calibrated search-matching model incorporates three segments of the market: employed individuals, unemployed individuals receiving unemployment benefits, and individuals receiving income supplements through IA. The model expands on Davidson and Woodbury (1993)’s equilibrium search model by assuming that expected lifetime income is maximized by individuals when they choose their employment state and the intensity with which they seek work if unemployed. 6 For further details, please see Lise, Seitz and Smith (2003) 7 kernel and local linear matching estimators, which use multiple nonparticipants to estimate the outcomes of the control group as opposed to pair-wise estimation. This method, used by Heckman et al. (1997) and Heckman et al. (1998), was implemented with longitudinal and cross- sectional data from the National Supported Work Demonstration, another labor market program.7 The results suggest that such estimators are able to replicate experimental results only when the data examined are very similar – from the same source(s), with treatment and control groups from the same geographic labor markets, and incorporating a range of variables that influence both participation and labor market outcomes.8 McKenzie et al (2006) build on the finding of previous authors: the best non-experimental estimates are obtained when the treatment and control groups are drawn from the same labor markets, data sources used are the same, and groups’ characteristics are similar in that the “likelihood of receiving the treatment is similar in both groups.” They compare experimental and non-experimental methods in analyzing the income gains from migration, using a survey on the quota of Tongans allowed to immigrate to New Zealand each year on the basis of a lottery from an excess number of applicants. They generate experimental estimates of income gain from migration by comparing gains of applicants whose names were not drawn in the lottery with those who immigrated through the lottery, after taking into account those who did not immigrate even after they “won” the lottery. They also survey a non-applicant group and generate non-experimental estimates for comparison with the experimental estimates. Because migrants are more likely to have certain observable and unobservable characteristics (such as ability and drive), the non-experimental methods are shown to overstate income gains from migration by between 9 and 82 percent. The ex ante labor market models described here suggest the difficulties in predicting the impacts of a program using a model representing standard economic incentives incorporated through wages. In our paper, however, the effects of programs are simulated using a model incorporating household income, where the incentives operate through channels that are quite different. 7 NSW was a subsidized work experience program providing recipients with training and assistance in finding regular jobs. 8 Summarized from Smith and Todd (2006). 8 Against the backdrop of the existing literature discussed above, the main contribution of our paper is in terms of testing the validity of an ex ante tool for evaluating the impacts of CCT programs (the BFL model) that is operationally useful, both in terms of ease of implementation and data requirements. Compared to the dynamic ex ante models suggested by Attanasio et al and Todd and Wolpin (see above), the BFL model makes fewer demands on data requirements and do not rely on the availability of panel data, as it employs a reduced-form approach that imposes simplifying assumptions on household behavior. If the predictions of the BFL model with all its simplicity were to compare well with the impacts measured by rigorous ex post evaluations, it would be a practical and credible ex ante tool for policymakers to employ during the design and planning phase of CCT programs. III. Ex Ante Simulation with the BFL Model9 This section presents a summary description of the BFL model, used as the ex ante evaluation tool in this paper (a more detailed discussion of the model can be found in the original BFL paper). Rather than constructing a complete structural model of demand for schooling and intra-household labor allocation, the BFL model aims to obtain reasonable orders of magnitude for the likely effects of cash transfers that are conditional on school attendance. The structural aspects of modeling are thus kept to the minimum necessary to capture the main effects of the program. There are four key underlying assumptions. Firstly, occupational choice is assumed to reflect (in reduced form) the end result of whatever decision-making process about a child’s time allocation unfolds in a household, regardless of how the decision is made. Secondly, the decision to send a child to school is assumed to be made after all occupational decisions by adults within the household have been made, so that the schooling decision does not affect those earlier decisions. Thirdly, the model does not allow for decisions about the occupational choice of multiple siblings in a household to be made simultaneously or jointly. Fourthly, the composition of the household is taken as exogenous. While each of these assumptions is an 9 A toolkit for running the BFL model was prepared by the World Bank. The toolkit provides a how-to guide in the application of the BFL micro-simulation model that enables users to analyze and compare several alternative scenarios on the basis on representative household survey data. It can thus be used in selecting the most cost-effective design or it can be used for sensitivity and cost-benefit analysis. The toolkit can be downloaded at www.worldbank.org/safetynets under toolkit or by clicking on the following hyperlink: BFL model toolkit 9 abstraction from reality (at least for some households), they make for a simple, reduced-form approach that is tractable with the baseline data available for most programs. Let j =0 be the occupational category of “not attending school,” j =1 be that of “attending school and working,” and j =2 be that of “attending school only.” Following the approach adopted in BFL, the utility function of child i corresponding to each occupational category j is given by the following:10 Where Zi are the characteristics of both the child and the household, Y-i is the household income without the child’s earnings, yij is the child’s income earned in alternative j; and vij is the random variable representing idiosyncratic preferences. γj and αj are parameters specific to occupational category j. The model is parsimonious in its representation of occupational choice of children and allows for all possible trade-offs between current income of the household and the schooling of the child.11 A key variable is the child’s earning yij since transfers depend on household income and the schooling/occupational status of a child, which in itself affects the child’s earning. The model implicitly treats the child's number of hours of work as a discrete choice between 3 alternatives, corresponding to the 3 occupational categories given by j. It seems reasonable to define child i’s earning in occupational category 0 as equal to her observed market earnings denoted by wi, since no time is spent on schooling. Assuming that earnings can be characterized by the standard Becker-Mincer human capital model, wi is given by: Where Xi is a set of individual characteristics (including age and schooling achieved) and ui is a random term that stands for unobserved determinants of earnings. The second term on the right hand side is a dummy variable representing the fact that a child who attends school and 10 See equation (2) in the BFL paper, which in turn is the linearized version of a more general specification given by equation (1) in their paper. 11 Note that the model can also implicitly represent a trade-off between current income of the household and future income of the child (and perhaps the household as well), since schooling would be expected to raise his/her future income 10 works for wages has less time available and is therefore likely to earn less. The child's contribution to the household income (yij,) in the various alternative categories for j is given by: Where yij covers income from both market-based and domestic work done by child i in occupational category j. Being in category 1, namely attending school while working outside the household, leads to a reduction in total income (relative to income in category 0) by the proportion (1-M). Similarly, going to school without working in the market leads to a reduction in total income by the proportion (1-D).12 While D is not observed, M is assumed to be the same for domestic and market work and can be estimated from observed earnings represented by equation (2) above. Replacing (3) in equation (1): Child i (or the household of child i) will choose the occupational category j that yields the highest utility among the 3 alternatives. Assuming that all parameters of equation (2) are known, along with actual or potential earnings (wi) and the residual terms (vij), household’s selection of a child’s occupation can then be defined by: If a CCT program is implemented where all children going to school receive a transfer T, equation (4) becomes: Equation (6) adds a transfer amount T to the part of household income that is independent of the child’s income, conditional on the child going to school. Thus in maximizing the utility function given by (6), the household must take into account the fact that it will receive T only in states (j=1) or (j=2). Under the assumptions made earlier, equation (6) is the full reduced-form model of the occupational choice of children, which would allow for simulations of the impact of 12 Note that total income in category 1 can be a combination of incomes from market and domestic work of the child, while total income in category 2 is from domestic child work only. 11 CCT transfers on those choices, once the estimates of the parameters of the model as well those of wi and vij's are obtained. Estimation of discrete choice model An assumption that the vij are independently and identically distributed across sample observations with a double exponential distribution leads to the well-known multinomial logit model, which can be used to estimate equation (6), where j=0, 1, 2 are the three alternative categories of occupation the household can choose between. The estimation of parameters is complicated by the nature of the model, which only allows the coefficients corresponding to a given category to be estimated as a deviation from those of a reference category. To see this, we start by noting that the probability a child/household i will select occupational choice k is given by: Taking (j = 0) as the reference state or occupation category, (7) can be written as: Multinomial logit estimation permits the estimation of only the differences (αj-α0), (βj-β0), and (γj-γ0) for j = 1, 2. Since the transfer is conditional on the child being in state 1 or 2, the income variable is asymmetric across alternatives. Thus in order to find the utility maximizing alternative (k*), it is necessary to find estimates of the coefficients α0, α1 and α2, for which the structural assumptions listed in equation (4) are useful. Let and be the estimated coefficients of the multinomial logit model corresponding to the income and the child earning variables for alternatives (j = 1) and (j= 2), the alternative (j=0) being taken as the reference. Using equation (4) we get the following system of equations:13 13 To obtain (9), recall from (4) that: and that: for j=1, 2. 12 Which is equivalent to Using (9a), we can derive estimates of α0, α1, α2 and D from the estimated coefficients of the multinomial logit and M; where M is obtained from the estimation of equation (2). In estimating the residual terms (vij-vi0), it is important to recall that these cannot be observed in a discrete choice model, and can instead only be known to belong to certain intervals. Thus the residual term for each child is drawn from the relevant interval, which is to say, in a way consistent with that child’s observed choice of occupational category.14 Finally, we note that equation (6) is not easily estimated without variable wi, which is unobservable for children who are not in the labor market. The most rigorous approach would be to estimate the discrete choice model and the earning equation simultaneously by maximum likelihood techniques – a cumbersome procedure.15 Correcting the estimation of the earning function for a selection bias turns out to be problematic as well (see discussion in BFL).16 Instead, the BFL model uses a simple approach, which has the advantage of transparency and robustness. This consists of estimating equation (2) by OLS, which is then used to predict the potential wage wi for children who are not in the labor market. A random term ui is added to the predicted earning of each non-working child to account for unobserved heterogeneity, by drawing from the distribution generated by the residuals of the OLS. Simulating the impacts of a CCT program Using the estimation steps described above, the outcomes of a given CCT program can be simulated with equations (5) and (6), but with the additional step of introducing a “means test” to identify eligible beneficiaries, which is a feature in CCT programs like PROGRESA and BDH. 14 For instance if child i has chosen state or category 1, the terms (vij-vi0) must be drawn so as to satisfy the inequality: . 15 To handle simultaneously the random terms of the discrete choice model and that of the earning equation, a multinomial probit would then be preferable to a multinomial logit. This would then however pose the difficult challenge of integrating tri-variate normal distributions. 16 Proper identification using the inverse Mill’s ratio to correct the earnings equation for election bias requires variables/instruments which influence earnings but not the occupational/schooling choice of children.) Such variables are not readily available from the data. Moreover, the standard correction using a two stage procedure is even weaker in the case of more than two choices. 13 The means test is represented by assuming that the transfer T is provided only if household income is less than or equal to a pre-determined threshold Y0. Taking into account both the means-test and the conditionality of child attending school, child (household) i would choose the state or occupational category that yields the maximum utility among the following alternatives: As explained earlier, the estimation of the multinomial logit regression, combined with the relationships shown in (9a), allows us to simulate the utility maximizing decision for household i to choose among the alternatives specified in (10). It is easy to see that the introduction of a transfer can induce households to move from occupational category 0 (no schooling) to category 1 or 2, and also from 1 to 2 if it were the case that the household qualifies for the transfer only when the child went to school and stopped working. The framework in (10) can be used to simulate the impacts of a variety of CCT programs that are conditional upon schooling enrollment, allowing for both the means test and transfer to be dependent on individual or household characteristics (e.g. transfer amount varying according to age or gender of the child). That said, important caveats or limitations apply to this framework, which are closely related to the assumptions set out at the beginning of this section. Firstly, the model cannot account for the effects of any upper limit on transfers going to a single household, which is a direct result of the model ignoring the possibility that decisions affecting multiple children in a household may be made simultaneously. Secondly, household income excluding the earnings of the child is treated as exogenous. This does not take into account for the possibility of the means test affecting adult labor supply – for example, when an adult decides to not participate in the labor market if the extra income makes the household ineligible for the CCT program. While this can be a serious issue when eligibility is defined by actual (or observed) 14 income, it is less so when eligibility is defined by a score-based “proxy” means test that attempts to reflect permanent income.17 IV. Programs and Data Description PROGRESA was launched in Mexico at a time when economic growth was found to be insufficiently pro-poor and existing safety net programs were seen as ineffective (Coady, 2004). PROGRESA was designed to alleviate poverty in both the short and long run, by making monetary transfers to poor families while requiring that they invest in the human capital of their children. Renamed Opportunidades in 2003, PROGRESA provides monetary grants to selected families (and usually to the mothers) conditional on children being enrolled in and regularly attending school, and on all family members attending scheduled visits to health care centers.18 The educational grant is provided for each poor child under the age of 18 years and enrolled in school between the 3rd grade of primary and 3rd grade of secondary level. The grant amount is adjusted every six months for inflation and varies by school grade and gender, increasing as children progress to higher grades (see Appendix, Table A-1). The grant is intended to reverse two observed tendencies among poor Mexican communities: older children are more likely to work, which implies that the opportunity cost of going to school increases with the grade level, and girls have higher dropout rates than boys at the secondary level. The health and nutrition grants components are intended to improve health indicators through regular visits to health care centers for all family members and enhance food consumption through nutritional supplements, especially for children under the age of 2 years and pregnant and breastfeeding women.19 17 Typically the proxy means test score is determined by factors such as ownership of durable goods and assets like land, housing conditions, education and occupational status of household members, and demographic characteristics like number of children and dependency ratio – none of which are directly influenced by the decision of an adult member to participate in the labor market. 18 For a more detailed description of PROGRESA see Skoufias and Parker (2001) or Parker and Skoufias (2000). 19 Skoufias and Parker (2001) present this component as two components: (i) health, providing basic healthcare for all members of the family without including any monetary transfer, and (ii) nutrition, including a fixed monetary transfer. 15 PROGRESA was first implemented during the first half of 1998 in 320 randomly assigned rural villages out of 506 villages pre-qualified for immediate participation.20 The remaining 186 villages were assigned as control villages and received their first monetary transfers in December 2000. Implementation of the program in a phased manner and random selection of program villages for the first phase of implementation were extremely important factors aiding rigorous evaluations of the program.21 The baseline survey (Encuesta Évaluation de los Hogares or ENCEL) for PROGRESA was collected in 1997 in all 506 pre-qualified villages, covering 24,077 households in seven states.22 PROGRESA administrators ran a follow-up survey over the same set of households once every six months, generating a rich household panel dataset. Many evaluations of PROGRESA were conducted on the basis of these surveys, including the previously mentioned studies by Skoufias and Parker (2001), Parker and Skoufias (2000), Schultz (2004), Todd and Wolpin (2006), Attanasio et al. (2005), and Coady (2004). Overall results show an increase in the enrollment rates of both boys and girls at the primary and secondary levels, a reduction in the incidence of illness among children of age 0-5 years, an increase in the annual growth rate of children of age 12-36 months and a reduction in all poverty indices. PROGRESA (Opportunidades) is now covering around 5 million households (18 percent of the country’s total population) with a budget equivalent to 0.4 percent of GDP, contributing around 20 percent of the income of participating families. In 2003, Ecuador updated Bono Solidario, its existing social program focusing on poverty reduction that had been created in 1999. The new program, Bono de Desarrollo Humano (BDH), is part of the Social Protection Plan of the Ministerio de Bienestar Social and re-targeted the transfers under the erstwhile Bono Solidario program. While the main purpose of BDH is to transfer cash to poor households, the transfers are also intended to be conditional on child enrollment and health care center visits. In its early days, compliance with the conditions was not monitored and consequently, non-complying households were not penalized. In order to 20 These villages were randomly selected out of 6,396 villages (4,546 considered to be in the treatment group and 1,850 in the control group) using probability proportional to size. Villages were identified on the basis of a community score based on information available from national census data regarding characteristics such as educational levels, occupational composition, and housing conditions. 21 Further details are available in Skoufias and Parker (2001). 22 The 7 states were: Guerrero, Hidalgo, Michoacan, Puebla, Queretaro, San Luis Potosi, and Veracruz. 16 correct this, program administrators launched a large-scale campaign to stress the importance of compliance. Under BDH, beneficiary households (again the mothers) receive grants of US$15 per month on the condition that children of ages 6-16 years are regularly enrolled, with at least a school attendance rate of at least 80 percent per month. Poor households with children of ages 0-5 years, additionally, have to fulfill the condition that they make scheduled visits to health centers for growth and development checkups and immunizations.23 According to Schady and Araújo (2006), the monthly US$15 transfers account for only 7 percent of household expenditure. BDH coverage reached approximately one million households (5 million people), covering 40 percent of the Ecuadorian population with an estimated cost of 0.6 percent of GDP in 2005. The baseline survey, collected between June and August 2003 for the evaluation of the BDH program, was drawn from four of 22 Ecuadorian provinces around the country.24 Within all four provinces, “paroquias” (parishes) were randomly assigned. For each selected paroquia, a random household sample was selected, generating a total sample size of 1,488 households. Around 90.5 percent of the household sample had at least one child aged 6 to 17 years at the time of the follow-up survey. None of the households in the sample were enrolled in BDH or Bono Solidario prior to the sample selection. Thereafter, some of the sample households were selected for immediate participation in the program while others were not eligible to receive any transfer in the first two years. The follow-up survey took place between January and March 2005 and collected information on households after program implementation, generating a panel sample of 1,306 households (2,875 children between six and seventeen years of age). According to Schady and Araújo (2008), around 27 percent of households in both treatment and control groups had stated that school attendance was a prerequisite for receiving BDH grants, despite the fact that the program administrator had never enforced any conditions. BDH was found to have a positive impact on school enrollment and a negative impact on child work; the 23 The transfers can be collected at any office of the Banred (the largest network of private banks in Ecuador) or from the National Agricultural Bank. 24 The 4 provinces were Carchi, Imbabura, Cotopaxi, and Tungurahua. 17 fact that households believed school attendance to be mandatory may explain the scale of these program effects.25 V. Comparison between Ex Ante and Ex Post Estimates of Program Impact26 The BFL model as described in Section III can be used to simulate school enrolment (and labor participation of children) in the presence of the CCT program, which can then be compared with the outcomes for the counterfactual – namely, the observed scenario in the absence of CCT. When the simulation is conducted with the treatment group – a sub-sample of the full baseline survey sample – the impact obtained may be considered as an estimate of the “Average Intent to Treat” (henceforth AIT) because it simulates the impact of the CCT program on the enrollment rate among all eligible children, regardless of whether they accept the transfer or not. Even if a child is eligible to receive the transfer, if the amount of transfer is insufficient to move him/her from category 0 to 1 or 2, the household would not accept the transfer and consequently, the transfer is not allocated to the household in the simulation.27 The AIT estimator is then given by: Where P* is a dummy variable taking the value 1 if children would be enrolled in school after receiving transfer T and P is a dummy variable taking the value 1 if children are currently enrolled in school. AIT* provides an estimate of the average impact of the availability of the program to eligible households in treatment communities. Two aspects of the ex ante AIT estimator must be noted. Firstly, it assumes good implementation of program in treatment communities. While the simulation accounts for eligible households opting to not receive the transfer and send their children to school, it does not account for “exclusion errors” in targeting whereby households eligible and willing to 25 Interestingly, these results suggest that even if the conditionalities of a CCT program are not enforced, just announcing the conditions may be enough to have some impact on behavior of participants. 26 Similar analysis for Nicaraguan Red de Protección Social (RPS) program was performed by the authors and main findings presented in this section also hold for the RPS program. Authors omitted the RPS analysis due to the length of the article but results are available per request (See Appendix, Table A-12 for summary results). 27 In maximizing its utility, an eligible household will decide to not accept the transfer if U(0) > U(1) even when Y-I + M.wi ≤ Y0, and U(0) > U(2) even when Y-I ≤ Y0 18 receive the transfer (complying with its condition of sending children to school) are left out of the program. Neither does it take into account other types of implementation flaws, including “inclusion errors” of non-eligible households receiving the transfers, lack of compliance with the conditions of the transfer, or schools being unable to accommodate increase in enrollments. Secondly, due to the nature of the BFL model, the ex ante AIT estimator does not take into account any time trend effects on outcomes on which impacts are measured. Given this, it is best to compare the ex ante AIT estimate with ex post AIT estimates obtained using double- difference or difference in difference (DID) method whenever possible – recognizing that this ex post method removes the time trend effect from the estimated impact. The impact of PROGRESA We compare the simulated ex ante impact of PROGRESA, applying the BFL model on the baseline survey (from 1997 – see section IV for details), with ex post results from Skoufias and Parker (2001). To make this comparison possible, all income variables from the baseline survey are converted into November 1999 pesos, and transfer amounts corresponding to the second half of 1999 (see Appendix, Table A-1 for the amounts) are applied in the simulations. The selected sample comprises only children who were of age 8-17 years at the time of the baseline survey in 1997. Given how the BFL model is set up, the data from the follow-up survey on the same children, which is to say the panel data, is not utilized for the simulation. Out of the 33,609 children in the baseline sample the simulation was applied to, around 30 percent were not attending school, while only around 4 percent of children were combining work outside the household with school. Children not attending school were on the average older and with higher education level than those who were in school and not working. Households with higher dropout rates were not necessarily poorer, indicating the importance of child income for the household. Child earnings increased with age and girls were more likely to drop out than boys, but boys were more likely to work and attend school simultaneously (see Appendix, Table A-2 for all descriptive statistics). Average household size was similar (around 7 members) for the 3 groups of children and more educated parents were more likely to have children in school. Enrollment patterns across age groups are important to note as well, with implications for how program impacts should be estimated. Dropout rates increase sharply around the age when children are expected to complete primary schooling, rising from 7 percent at age 11 to nearly 19 17 percent at age 12 and increasing exponentially thereafter to reach nearly 82 percent at age of 17. Because of the vast differences in enrollment across age groups, Skoufias and Parker suggest that the sample of children aged 8 to 17 years must be divided into two groups in assessing the potential impact of the program: children of primary school age (8-11 years) and those of secondary school age (12-17 years). The experimental design of PROGRESA ensures in theory that all possible sources of bias were evenly distributed among participants and non-participants, allowing us to strictly attribute differences between treatment and control groups to program effect.28 However, different papers have analyzed the pre-program composition and characteristics of treatment and control groups in detail, suspecting that despite randomization the two groups were not fully comparable. Skoufias and Parker also find some evidence of systematic differences between the two groups and propose that a double-difference (DID) estimator, which takes into account any pre-existing differences, is preferable in evaluating the impacts of PROGRESA.29 To be consistent with the Skoufias and Parker’s methodology for ex post evaluation, the BFL model is estimated (see appendix for model specification and estimations) for all boys and girls aged 8-17 years in the baseline sample, and the results are presented separately for three age groups and gender categories. This generates results for six age-gender categories. Following the argument at the beginning of this section, the ex ante AIT estimator generated by the BFL model (see (11) above) can be compared directly with the ex post results from Skoufias and Parker (Tables 5 and 6), who use the sample of all eligible households and measure the direct effect of the “intent to treat”, regardless of whether they in fact received a transfer or not. The ex post AIT estimates should be seen as a lower bound of the impact on households that actually received treatment, since the observed impact also includes the effect of flaws in program targeting, which would dilute the impact on eligible households (for example, when mis-targeting results in an eligible household being left out of the program). These estimates are derived from a regression-based approach that yields the DID estimate of the program’s impact 28 Heckman, La Londe and Smith (1999). 29 Also, Behrman and Todd (1999) found the null hypothesis of mean equality with respect to household characteristics between treatment and control group to be rejected more frequently than expected. 20 in each round (of the follow-up surveys), which is net of any preprogram differences between treatment and control households and any time trends in the values of the outcome indicator. Skoufias and Parker (in Figures 4a and 4b of their paper) show that the mean school attendance rate of both boys and girls were nearly identical for control and treatment villages at the outset of the program. For treatment villages, the enrollment rate among boys was higher than that of girls in the baseline year, where the difference is entirely a result of the higher dropout rate among girls of age 12-17 years (Table 1). For the age group 8-11 years, enrollment was high at around 94 percent for both boys and girls. Table 1: Impact of Transfers on Children’s Occupational Choice (1997-1999) Actual and counterfactual school enrollment rates for PROGRESA target population 2 2 Baseline 1 Ex ante AIT Ex ante AIT Ex post AIT (pre-program) Conditional Unconditional ** 8-17 years-old 74.5% - 4.0% -0.1% (0.4%) (0.7%) ** 8-11 years-old 93.8% 1.8% 0.0% -0.1% Boys (0.7%) (0.4%) (0.5%) ** ** 12-17 years-old 57.5% 5.8% 5.9% 0.0% (2.1%) (0.8%) (0.8%) ** 8-17 years-old 69.4% - 4.3% -0.1% (0.7%) (0.4%) 93.9% -0.3% -0.2% -0.1% Girls 8-11 years-old (1.0%) (0.4%) (0.4%) ** ** 12-17 years-old 47.9% 9.5% 6.6% -0.1% (2.2%) (0.8%) (0.8%) Source: Baseline Survey 1997 and Rounds 1-4;authors' calculation. Note: 1: Results from Skoufias and Parker (2001) - Table 6 (only the results for November 1999 are reported). The coefficients reported are the marginal effects of the PROGRESA program on the probability of attending school; 2: Results from simulation on baseline data using BFL model ** * Significant at 5% level; Significant at 10% level. Standard errors and standard deviations in parentheses. For ex post AIT, robust standard errors are shown, which account for clustering of individuals within villages. For ex ante AIT, standard deviations computed by bootstrap method. Table 1 shows the AIT estimates of impact of PROGRESA on school enrollment rates by age and gender (Skoufias and Parker, henceforth SP) as well as the ex ante AIT estimates using the BFL simulation model. Ex ante estimates are also shown for a hypothetical case where the transfer is 21 unconditional, namely provided to all potential beneficiaries, regardless of whether they are enrolled in school or not. The ex ante AIT (conditional) estimates compare quite well with the ex post estimates. Both types of estimates show the impact of PROGRESA on enrollment to be much higher among children of age 12-17 years than children of age 8-11 years, and higher among girls than boys in the 12-17 age group. While the ex ante and ex post estimates are almost identical for boys of age 12-17 and girls of age 8-11, the ex ante estimates understate the impact for girls of age 12-17 and boys of age 8-11. Boys of age 8-11 years are the only group for which the two types of estimates disagree on whether there is any significant impact or not – the ex post estimate show a small positive impact while the ex ante estimate is not different from zero. The confidence intervals of ex ante and ex post estimates for the same group overlay each other to a large extent. One advantage of ex ante methods is that they allow for the simulation of hypothetical scenarios. In the hypothetical case where transfers are unconditional, the simulated effect on enrollment rate is found to be zero for all age/gender groups. Table 2: Impact of Transfers on Children’s Occupational Choice (1997-1999) Actual and counterfactual proportions of children working for PROGRESA target population 2 2 Baseline 1 Ex ante AIT Ex ante AIT Ex post AIT (pre-program) Conditional Unconditional ** 8-17 years-old 23.0% -2.6% 0.0% (0.3%) (0.5%) 8-11 years-old 6.7% -1.1% 0.3% 0.0% Boys (0.8%) (0.4%) (0.4%) ** ** 12-17 years-old 37.3% -4.7% -3.7% 0.0% (2.2%) (0.5%) (0.7%) * 8-17 years-old 9.7% -0.7% 0.0% (0.3%) (0.3%) Girls 8-11 years-old 4.2% 0.0% 0.4% 0.0% (0.0%) (0.3%) (0.4%) * * 12-17 years-old 14.6% -2.3% -1.3% 0.0% (1.3%) (0.5%) (0.5%) Source: Baseline Survey 1997 and Rounds 1-4 – authors' calculation. Note: 1: Results from Skoufias and Parker (2001) - Table 5 (only the results for November 1999 are reported). The coefficients reported are the marginal effects of the PROGRESA program on the probability of children working; 2: Results from simulation on baseline data using BFL model ** * Significant at 5% level; Significant at 10% level. Standard errors and standard deviations in parentheses. For ex post AIT, robust standard errors are shown, which account for clustering of individuals within villages. For ex ante AIT, standard deviations computed by bootstrap method. 22 Table 2 shows the results of a similar exercise comparing ex ante and ex post AIT estimates of the impact of transfers on the proportion of children who are working outside home. The ex ante estimates are again quite close to the ex post SP estimates of impact. Both methods indicate that PROGRESA has the impact of reducing the rate of child labor among 12-17 year olds, but not among 8-11 year olds – quite consistent with what was seen in terms of impact of enrollment using both methods. Among the 12-17 year olds, the program’s impact on child labor is larger for boys than for girls. For both boys and girls of the 12-17 year age group, the ex ante estimates understate the size of the impact relative to the ex post estimates. Consistent with what was seen for enrollment, the ex ante simulations with unconditional transfers show no impact on child labor, suggesting that the conditionality of the transfer was a crucial factor for the impacts. While it is encouraging to find that the BFL model has good predictive power when compared to results from ex post evaluations with experimental design, the similarity of results may be driven by the restricted and homogenous nature of the baseline sample (of ENCEL) on which the simulations are run. Moreover, from a practical point of view, a specialized baseline survey of program areas may not be available in all cases to conduct ex ante simulations. CCT programs are often set up quickly responding to political needs or to utilize a brief window of opportunity, which may leave little time for baseline data collection. In such cases, a nationally representative household survey, which has much more heterogeneity across households than any baseline survey, is often the only source of data available to conduct simulations on. Therefore, for methodological as well as practical reasons, it is useful to test the validity of ex ante predictions from the BFL model using the Mexican National Household Survey (ENIGH). This is done by estimating the key parameters of the BFL model (αj’s, M and D) for all children of age 12-17 years with ENIGH 1996 (the round that was the closest to the launch of PROGRESA), and comparing the estimated parameters with those estimated with a similar model specification for the same age group of children from the ENCEL survey. The estimated parameters from ENIGH do not differ significantly from those estimated with the baseline survey (Table 3). For all key parameters, the 95 percent confident interval generated by ENIGH and ENCEL overlap each other and can thus be considered statistically similar. However, 23 ENIGH parameters generate smaller program effects compared to the results in Table 1 because all the parameters are a little smaller in size than ENCEL-generated parameter estimates. These results confirm the findings of Leite (2007) for Brazil, which show that estimations based on a more heterogeneous sample such as the one surveyed in the national representative PNAD leads to smaller parameter estimates. By estimating the same set of parameters for a sub- sample of homogenous households, Leite (2007) found a much higher impact on the enrollment rate and reduction of child work.30 Table 3: Estimating Key Parameters from BFL model Using Baseline 1997 and Enigh 1996 Surveys Baseline - ENCEL Enigh Boys and Girls 8-17 y.o. 12-17 y.o. 12-17 y.o. ** ** ** α0 0.025 0.021 0.018 (0.008) (0.008) (0.009) ** ** ** α1 0.025 0.021 0.017 (0.009) (0.008) (0.009) ** ** ** α2 0.025 0.021 0.018 (0.009) (0.008) (0.009) ** ** ** M 53% 54% 61% (2.02%) (3.09%) (5.01%) ** ** ** D 65% 76% 76% (14.8%) (12.2%) (9.9%) Source: Baseline Survey 1997 and ENIGH 1996 ** * Notes: Significant at 5% level; Significant at 10% level. Standard Deviations (in parentheses) computed by bootstrapping. The BFL model also makes the estimation of program effects on poverty and inequality possible, which is useful to examine the importance of PROGRESA as an anti-poverty program in the short 30 Leite (2007) estimates the set of parameters of interest using a sub-sample of children around the means test threshold. This is done by defining two homogenous groups – potential beneficiaries who are below the means test threshold (treatment) and non-beneficiaries who are above the threshold (control) – who have made their choices subject to similar income constraints. He finds that 65 percent of the movement of children out of the labor market that were already in school can be explained as a response to the Bolsa Escola. These two groups around the means test threshold, according to the Regression Discontinuity Design, are more likely to have similar observable and unobservable characteristics which would make it a good approximation of a randomized experiment. 24 run. This is done by setting 750 pesos as a maximum transfer per household (consistent with the rules of PROGRESA), using the same poverty line as in Skoufias and Di Maro (2008)31 and taking into account individual level decisions through the BFL model. The proportion of poor households in the baseline treatment village sample is estimated to be 61 percent, which is the same as in Skoufias and Di Maro. Applying the DID estimator on poverty measures, Skoufias and Di Maro find that between November 1997 and November 1999, PROGRESA was responsible for a 16.5, 24 and 27 percent reduction in the poverty headcount rate, poverty gap and squared poverty gap, respectively, in treatment villages. In comparison to their results, our ex ante simulations suggest a similar impact of PROGRESA on poverty headcount rate (13 percent reduction), but a somewhat larger impact for poverty gap (27 percent reduction) and squared poverty gap (33 percent reduction).32 Table 4 below shows these results. Table 4: Poverty Index: Observed (ex post) and Simulated (ex ante) FGT(0) FGT(1) FGT(2) % σ % σ % σ Treatment Communities (baseline) 60.9% 1.1% 30.8% 1.1% 21.5% 1.1% Change in poverty ex post: Skoufias and Di 1 Maro (2008) -16.5% 1.6% -24.3% 1.5% -29.2% 1.4% 2 Change in poverty ex ante: Simulated -13.1% -26.6% -33.1% Source: Baseline Survey 1997 and Rounds 1-4; 1 : Skoufias and Di Maro (2008), Table 4; 2 : authors' calculation using BFL model Notes: Prices of November 1999 (pesos); poverty line: mean of Nov 98 consumption per capita (taken from Skoufias and Di Maro, 2008) The final step in our analysis is the simulation of the cost of the program based on the BFL model. Table 5 shows that 4,558 out of 7,110 (or 64 percent of) poor families in treatment villages would enroll in the program and accept the condition of school attendance. The average cash transfer for these families in treatment villages is estimated at 303 pesos/month, which is very close to the average transfer of 300 pesos in January 2002 for 1.6 million families observed from official program data. Based on an analytical exercise of scaling up the cost of the program 31 The poverty line is equivalent to 176.17 pesos per month (for details, see Skoufias and Di Maro, 2008) 32 The ex ante results presented in Table 4 remain the same when the simulations were done without incorporating the conditionality of schooling. This suggests that in terms of meeting the poverty reduction objective of safety net programs, conditionality does not necessarily imply better results. In other words, it suggests that if the main objective of a CCT program is poverty reduction, the transfers are important but the conditions are not – which makes perfect intuitive sense. 25 from the BFL model to the national figure using ENCEL 1997, the total cost of the program is estimated to be around 6.08 billion pesos, which is very close to the actual program cost of 6.06 billion pesos in January 2002 from the official program data (Table 5). Table 5: Estimated and Observed PROGRESA's Cost 1,2 1,3 BFL Simulation January 2002 Number of Families with Children Receiving Benefits 4,567 1,681,254 Average Monthly Transfer for Families with Children $301 $300 Estimated Total Annual Transfer $16,517,580 $6,061,221,243 Scaling Up Total Transfer: Simulated average times $6,080,632,364 n.a. families in 2002 Note: 1: November 1999 prices 2: Estimated by BFL model using Baseline survey of 1997, Rounds 1-4 3: National official numbers from: http://www.oportunidades.gob.mx/indicadores_gestion/ene_feb_02/indice.htm Thus the BFL model applied to the PROGRESA program in Mexico appears to have strong predictive power for the impact of the program on outcome indicators (i.e. enrollment rate, child labor and poverty) as well as the cost of the program, when compared against relevant benchmarks – namely AIT impacts estimated by ex post evaluations using experimental methods and actual or observed program cost. Since ex post AIT estimates represent a lower bound of the impact on households that actually received treatment (for reasons explained earlier), the ex ante estimates of program impact from BFL should also be seen as a lower bound for the actual impacts of the program on beneficiary households. The impact of Bono de Desarrollo Humano (BDH) As before, we compare the ex ante estimates of program impact from the BFL model with ex post findings from Schady and Araújo (2008), henceforth SA. For ex ante simulations, we use the baseline survey of BDH collected in Carchi, Imbabura, Cotopaxi and Tungurahaua and described in section IV above, which is also used by SA as the baseline data in analyzing impact. It is relevant to note that by the authors’ own admission, the results from SA should be treated with some caution as they are not based entirely on random assignment. 26 A summary of descriptive statistics from SA reveals no significant differences between the pre- defined treated and non-treated samples for a large number of variables (see Appendix, Table A-3 – first two columns), suggesting that the random design of the experiment in terms of the intended treatment and control groups was successful.33 However, the match between the intended groups with actual BDH recipients is imperfect – transfers went to about 78 and 42 percent of households in the so-called treatment and control groups, respectively. A comparison between recipients and non-recipients of BDH transfer shows significant differences between the two groups for variables like child enrollment, parental education and household size (Appendix, Table A-3, third and fourth columns). Thus selection into the BDH program appears to be non-random. In order to avoid misspecification due to bias from the contamination of both treatment and control groups, SA computed the “treatment-on-the-treated” or TT estimator using the Two Stage Least Square method. The TT estimator suggests that the BDH increased enrollment rates among compliers by 9.7 percent, with the statistic being only weakly significant (i.e. significant at the 10 percent level but not at 5 percent). Notably, around 25 percent of households in the sample believed that compliance with the condition of school attendance was mandatory for receiving the cash transfer (to be called “conditioned households” henceforth), despite the fact that this conditionality was not enforced by BDH administrators. Out of these conditioned households, 55 percent were from treatment and 45 percent from control groups. Following the re-weighting scheme of Hirano et al (2003), SA reported the Difference-in-Difference or DD estimator, adding a dummy variable for “conditioned” and an interaction term of the “conditioned” dummy variable with the treatment group dummy variable. As previously mentioned, our BFL model estimator is a type of “Intend-to-treat” estimator, ITT. In this case, it is easy to transform the ex post TT estimator from SA (results from Table 3 in SA) for comparison with our estimator, as follows: 33 Children from the intended treatment and control groups are indistinguishable in the baseline data in terms of their enrollment rate, grade attainment, labor participation, gender, per capita expenditure, assets, parental education and household size, 27 Another issue to take into account is the amount of transfer to be allocated to each child. Our BFL model is based on the assumption that the decision to enroll (or not) every child in school is made (by the child or the household) independently, taking into account the transfer amount for the individual child. But the BDH program transfers US$15 per household, independent of the number of enrolled children in the household, which makes the decision of sending or keeping every child in school essentially a household level decision. On the one hand, if we set $15 as the direct transfer amount, we suppose that each child will add to its utility an amount of $15.αj (for all j= 1, 2),34 which will lead to an overestimation of the effects of the program. This can then be considered as the upper-bound of the estimated impact of BDH from the BFL model. On the other hand, the BFL model does allow us to construct alternative scenarios about how the $15 is distributed within the household, namely the size of the de facto transfer to every child when there are multiple children in the household. We propose three alternative scenarios. The first scenario is one where the transfer received by each child is equal to $15 divided by the number of children aged 6-17 years in the household, with a minimum of $3 per child. In the second scenario, we set an individual transfer of US$5 because the average number of potential beneficiaries is close to three per household. In the third scenario, the individual transfer is set at $7.5. In order to account for the cost of the program and average size of transfer per household, we enforce the official household maximum of $15 as defined by the BDH program and also compute its impact without any household maximum. Table 6 shows all ex post estimators obtained by SA and the results of the different scenarios proposed using the BFL model. As expected, the ex ante or simulated ITT estimator under the assumption of US$15 transfer per child (IIT1), at 14.58 percent, is almost double the size of the ex post impact measured by SA at 10 percent. The annual cost of the program under scenario 1 (US$15 per household, despite individual decision) is US$166,000 if we set the household ceiling at US$15. Allowing for different amounts of transfer per household would increase the average transfer to US$32.33 and the total annual cost would more than double. However, the effect on enrolment rate generated by scenario 1 is unlikely, as a child living with its siblings will never make a decision on program participation based on a US$15 transfer. 34 Refer to Equation (4) earlier. 28 The next three proposed scenarios have lower estimated impacts ranging from 5.86 percent to 9 percent; and for all three cases, the ex post ITT estimator falls in the 95 percent confidence interval of the ex ante ITT estimator. For these scenarios, the average transfer per household ranges from US$9.8 to US$15.95, regardless of whether the US$15 ceiling is taken into consideration. As a consequence, the estimated cost of the program never extrapolates to the cost in scenario 1 of US$15 per household. For example, this result suggests that we would have observed higher program effects (without affecting the overall cost of the program) by making individual, smaller transfers of US$7.5 per child, so that the amount of transfer per household ranges from US$7.5 to US$45 depending on the size of the household. For the reasons described earlier, we cannot calculate ex post results for the BDH program in a way that would allow for a direct comparison with ex ante predictions as we have done for the case of PROGRESA. However, the prediction power of the BFL model is reasonable and consistent with our expectations conditional on the assumption made about the allocation of the household transfer among beneficiaries. As we would expect, considering each child in the sample to be a household under the BFL definition (and receiving US$15) overestimates the impact of the BDH. Keeping the average transfer per household close to aor above US$15 and using smaller amounts of transfer per child, the BFL model estimates program effects on enrollment rate that are not statistically different (at 5 percent level of significance) from ex post results reported by SA for all simulated ITT estimators. VI. Conclusion The results of our paper suggest that ex ante and ex post methods for estimating impacts can effectively complement each other to inform the design and evaluation of social programs. In principle, combining an ex ante simulation exercise using a structural model of schooling choices with an experimental evaluation design would help policymakers to assess the implications of alternative choices for program design, potential impacts and cost implications upfront, and rely on ex post evaluation results to validate and identify the actual impacts of the program. For ex ante models to be used this way, however, their results need to be credible in terms of accuracy or predictive power. Our findings make some progress in this direction. Our results suggest that ex ante predictions of certain impacts of CCT programs using the BFL model match up well with 29 ex post “gold standard” results from experiments, which should lend confidence to policymakers about the reliability of ex ante estimates from this model. The primary analysis of the paper consists of comparing the ex ante simulated impact on enrolment rates using the BFL model with the ex post impact estimates from a randomized experiment. For the case of the PROGRESA program in Mexico, we can also compare the ex ante and ex post estimates of program impact on poverty. Such a comparison of methods is also a test to validate the hypothesis central to such ex ante models: cross-sectional income effects, estimated on the basis of the representative household sample, coincide with the income effects generated by a program. For PROGRESA, we estimate that the predicted impact tends to understate but is not significantly different (at 5 percent level of statistical significance) from ex post “intent to treat” or ITT estimators of impact for all age groups of boys and girls. Moreover, the BFL model estimates similar results in terms of changes in poverty levels. For the Ecuadorian program Bono de Desarrollo Humano (BDH), the use of the BFL model, with reasonable assumptions about how the household level transfers would translate to de facto transfers for every child, yields simulated impacts that are again statistically similar to ex post ITT impact estimates. The fact that ex ante predictions from the BFL model are validated by ex post estimates also lends confidence to the use of the model for analyzing other types of questions related to program design. These are questions that are important for the program designer before designing the program, but too expensive or cumbersome to implement as a pilot to evaluate the actual impacts ex post. An example of such questions is: What would have been the likely impact of PROGRESA on enrollments if it were an unconditional cash transfer? Using the BFL model we find that transfers without conditions would not have produced the enrollment impacts seen from the conditional transfer. In case of BDH, the ex ante simulations suggest that program impact on enrollment rates could have been higher if the transfers were smaller and for each individual child (instead of a flat amount every household), for an unchanged overall cost of the program. Thus the ex ante method proposed by Bourguignon, Ferreira and Leite (2003) can be a powerful simulation method in designing or reforming CCT programs that aim to improve enrollments. The built-in advantages of an ex ante approach – in terms of cost, practicality and the stage of 30 the program cycle when it can be done – implies that a model that performs well against the benchmark of “gold-standard” ex post evaluations can be extremely helpful to inform policymakers about program design. One major advantage is that in many cases, such a model can allow for comparison between the likely impacts of alternative ways of designing a program – without the time, cost and logistics involved in setting up multiple pilot experiments to obtain ex post evaluation results. There are important caveats as well to the use of the BFL model, the most important among which is that ex ante analysis cannot incorporate any unanticipated changes that may occur during the process of program implementation, be that in the implementation process or in any other extraneous events (including large shocks) that can affect the behavior of program participants. The ex ante exercise would then need to be followed up with an ex post impact evaluation that is able to capture the actual effect of the program on the outcomes of interest, taking into account all the possible changes in the program and the environment it operates in that could have occurred as it was being implemented. 31 Table 6: Children’s Occupational Choices in Ecuador 2003-2005 Actual and Counterfactual Enrollment Rate for Target Population Ex post Ex ante Probability of TT ITT receiving benefits if 1 2 3 4 d1 eligible Simulated ITT Simulated ITT Simulated ITT Simulated ITT (a) (c)=(a)(b) (b) Table 4 - Results 2SLS 10.00% 10.00% 77% 7.7% 14.58% 7.63% 5.86% 9.00% 4.72% 4.72% 0.53% 0.82% 1.10% 0.60% Average Transfer per Household with Ceiling of $15 per household 15.00 11.91 9.80 12.31 Number of households 645 631 596 603 Annual Cost 116,100.00 90,192.00 70,080.00 89,100.00 Average Transfer per Household without Household Ceiling 32.33 11.92 10.35 15.95 Number of households 645 631 599 605 Annual Cost 250,200.00 90,264.00 74,400.00 115,830.00 Source: Schady and Araújo (2008) table 4 column (3). Note: 1 - R$15 per child assuming unitary child household. 2 - $15 divided by number of children age 6-17 in the household. Minimum transfer $3 per child. 3 - $5 per child. 4 - $7.5 per child. ** * Significant at 5% level; Significant at 10% level. Standard Deviation computed by bootstrap method. 32 References Attanasio, O., C. Meghir and A. Santiago (2005). “Education Choices in Mexico: Using a Structural Model and a Randomized Experiment to Evaluate PROGRESA.” University College London IFS Working Paper EWP0501. Bourguignon, F., F. Ferreira and P. Leite (2003). “Ex-ante Evaluation of CCT Programs: the Case of Bolsa Escola.” Policy Research Working Paper No. 2916. The World Bank. Washington DC Cardoso, E. and A. Souza (2004). “The Impact of Cash Transfers on Child Labor and School Attendance in Brazil”. Department of Economics, Vanderbilt University; Working Paper No. 04- W07. Nashville, TN. Coady, David. 2004. “Alleviating Structural Poverty in Developing Countries: The Approach of PROGRESA in Mexico.” Background Paper for the 2004 World Development Report. Available at: http://econ.worldbank.org/files/27999_Coady.pdf Davidson, C. and S. Woodbury (1993). "The Displacement Effect of Reemployment Bonus Programs," Journal of Labor Economics, Vol. 11(4): 575-605. De Janvry, A. and E. Sadoulet (2006). "Making Conditional Cash Transfers More Efficient: Designing for Maximum Effect of the Conditionality." World Bank Economic Review; Vol. 20: 1- 29. Heckman, J. and J. Hotz (1989). “Alternative Methods for Evaluating the Impact of Training Programs”. Journal of the American Statistical Association; Vol. 84 (408): 862-880. Heckman, J., H. Ichimura and P. Todd (1997). “Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme”. Review of Economic Studies; Vol. 64 (4): 605-654. Heckman, J., H. Ichimura and P. Todd (1998). “Matching As An Econometric Evaluation Estimator”. Review of Economic Studies; Vol. 65 (2): 261-294. LaLonde, R. (1986). “Evaluating the Econometric Evaluations of Training Programs with Experimental Data”. The American Economic Review; Vol. 76(4): 604-620. Leite, P. (2007) “Les politiques d'allocations scolaires et l’inégalité des chances au Brésil” mimeo 33 Lise, J. S. Seitz, and J. Smith (2009) “Evaluating Search and matching Models Using Experimental Data,” Available at: https://www2.bc.edu/~seitzsh/ssp_red01.pdf Maluccio, J., and Flores, R. (2004). "Impact Evaluation of a Conditional Cash Transfer program: The Nicaraguan Red de Protección Social," IFPRI – FCND Discussion Papers no 184, International Food Policy Research Institute (IFPRI). McKenzie, D., J. Gibson and S. Stillman (2006). "How Important Is Selection? Experimental vs. Non-Experimental Measures of the Income Gains from Migration," IZA Discussion Papers 2007, Institute for the Study of Labor (IZA). Parker, S. and E. Skoufias (2000). The Impact of PROGRESA on Work, Leisure, and Time Allocation. Report submitted to PROGRESA. International Food Policy Research Institute, Washington, D.C. Ravallion, M. (2008). “Evaluating Anti-Poverty Programs” in Handbook of Development Economics (eds. T. Schultz and J. Strauss). North-Holland Schady, N. and M. Araujo (2006). “Cash transfers, conditions, school enrollment, and child work: Evidence from a randomized experiment in Ecuador. Policy Research Working Paper No. 3930. Impact Evaluation Series No. 3. The World Bank. Washington DC. Schady, N. and M. Araujo (2008). “Cash Transfers, Conditions, and School Enrollment in Ecuador”. Economia; Vol. 8(2): 43-70. Brookings Institution Press. Schultz, T. (2000). “School Subsidies for the Poor: Evaluating the Mexican Progresa Poverty Program”, Journal of Development Economics, 74(1):199-250. Skoufias, E. and V. Di Maro (2008). "Conditional Cash Transfers, Adult Work Incentives, and Poverty." Journal of Development Studies; Vol. 44 (7): 935 - 960. Skoufias, E. and S. Parker (2001). "Conditional Cash Transfers and Their Impact on Child Work and Schooling: Evidence from the PROGRESA Program in Mexico." Economía; Vol. 2 (1): 45-86. Smith, J. and P. Todd (2005). “Does matching overcome LaLonde’s critique of nonexperimental estimators?” Journal of Econometrics, 125: 305–353. 34 Todd, P. and K. Wolpin (2006). "Assessing the Impact of a School Subsidy Program in Mexico: Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility." American Economic Review, 96(5): 1384–1417. 35 Appendix Descriptive Tables Table A-1: PROGRESA Monthly Cash Transfer (Pesos - July to December 1999) Educational grant per child Boys Girls Primary 3rd 80 80 4th 95 95 5th 125 125 6th 165 165 Secondary 1st 240 250 2nd 250 280 3rd 265 305 Cash grant for food per household 125 Maximum transfer per household 750 Source: Skoufias and Parker (2001). 36 Table A-2: Descriptive Statistics from PROGRESA Baseline Survey (Children aged 8-17 in 1997) Working Not and Attending Attending Total Attending School School School Age 14.8 12.6 11.2 12.3 Years of Schooling 5.5 5.3 4.6 4.9 Household Per Capita Income 301.7 238.5 241.3 259.3 Observed Children's earnings 8 536.5 282.1 0.0 381.0 9 484.2 217.3 0.0 301.6 10 593.2 317.1 0.0 424.8 11 680.4 389.4 0.0 537.9 12 706.1 323.1 0.0 518.8 13 618.3 332.9 0.0 541.9 14 702.8 415.1 0.0 653.4 15 793.8 453.2 0.0 760.8 16 813.5 549.5 0.0 792.6 17 838.7 690.5 0.0 830.4 Years of schooling of more educated 2.7 3.3 3.7 3.4 parent Age of Older Parent (years) 47.5 45.3 45.2 45.9 Number of Household members 7.3 7.3 7.2 7.2 Male (%) 48.2 72.8 51.5 51.4 46.6 34.7 51.3 49.2 Father works in agriculture sector (%) Proportion of universe (%) 30.0 4.3 65.7 100 Population 10,081 1,435 22,093 33,609 8 (%) 4.9 2.5 92.6 100 9 (%) 4.2 3.5 92.3 100 10 (%) 5.6 3.5 90.9 100 11 (%) 7.0 4.5 88.6 100 12 (%) 16.6 6.1 77.3 100 13 (%) 28.1 5.8 66.1 100 14 (%) 42.0 5.4 52.6 100 15 (%) 57.7 4.1 38.2 100 16 (%) 71.3 4.3 24.5 100 17 (%) 81.6 3.1 15.2 100 Source: Baseline Survey 1997; and Authors' calculation. 37 Table A-3: Differences between Treatment and Control Groups – BDH Baseline Survey Differences between Differences between recipients and Treatment and Control Groups non-recipients of the BDH transfers Mean: Difference Mean: non- Difference Control recipients Probability that child is 0.770 -0.003 0.736 0.052*** enrolled (0.421) (0.017) (0.441) (0.018) Mean years of schooling 4.59 0.031 4.65 -0.071 completed (2.47) (0.086) (2.46) (0.090) Probability that a child is 0.463 0.012 0.478 -0.014 working (0.499) (0.027) (0.500) (0.027) Child age 11.67 0.047 11.81 -0.174 (2.86) (0.094) (2.86) (0.097) Child is male 0.517 -0.031 0.506 -0.009 (0.500) (0.018) (0.500) (0.019) Log of per capita expenditures 3.44 0.013 3.48 -0.046 (0.543) (0.030) (0.564) (0.031) Asset index 0.007 -0.015 0.016 -0.028 (0.882) (0.049) (0.966) (0.052) Father’s education 4.72 0.087 4.44 0.523*** (2.58) (0.156) (2.76) (0.160) Mother’s education 3.52 0.283 3.30 0.598*** (2.99) (0.168) (2.99) (0.172) Household size 5.83 -0.116 5.64 0.208** (1.78) (0.102) (1.86) (0.105) F-statistic (p-value) 0.15 0.002 Note: All means refer to baseline values. Standard errors in estimated difference for child-specific variables adjust for within-sibling correlation. **Significant difference at the 5 percent level; *** at the 1 percent level. The sample is limited to households who had school-aged children in both the baseline and follow-up surveys. Sample size is 2876 for all child-specific variables, and 1309 for all household- specific variables except father’s education (n=1227) and mother’s education (n=1279). Source: Schady and Araújo (2006) – Table 2. 38 Regression results Estimations for Mexico (PROGRESA) Table A-4: OLS Regression using ENCEL 1997 Number of Obs = 3549; Adj R-squared = 0.2718; Root MSE = 0.65883 l_earn Coef. Std. t P>|t| ws -0.6354 0.037176 -17.09 0 _Iage_9 0.157519 0.219373 0.72 0.473 _Iage_10 0.460302 0.189121 2.43 0.015 _Iage_11 0.779767 0.184945 4.22 0 _Iage_12 0.758285 0.168635 4.5 0 _Iage_13 0.778888 0.164875 4.72 0 _Iage_14 0.910126 0.163087 5.58 0 _Iage_15 0.964133 0.162588 5.93 0 _Iage_16 1.016574 0.162404 6.26 0 _Iage_17 1.029843 0.162258 6.35 0 _Isex_1 0.229161 0.058058 3.95 0 school 0.022947 0.007705 2.98 0.003 _IsexXscho~1 -0.02151 0.008536 -2.52 0.012 lsalm 0.234924 0.033639 6.98 0 iml7954r -0.04904 0.023816 -2.06 0.04 pobreden -0.09103 0.03038 -3 0.003 _Istate_13 -0.1022 0.078887 -1.3 0.195 _Istate_16 -0.00615 0.096089 -0.06 0.949 _Istate_21 -0.08348 0.083244 -1 0.316 _Istate_22 -0.06409 0.097702 -0.66 0.512 _Istate_24 -0.12837 0.088878 -1.44 0.149 _Istate_30 -0.10423 0.086443 -1.21 0.228 MUNICIPIO DUMMIES _cons 2.435359 0.250553 9.72 0 39 Table A-5: MLOGIT Regression using ENCEL1997 Number of obs = 33383; LR chi2(284) = 18780.43; Prob > chi2 = 0; Log likelihood = -16133; Pseudo R2 = 0.3679 sit Coef. Std. z P>|z| 1 rendomo -0.00034 0.000169 -2.02 0.043 what -0.01211 0.004561 -2.66 0.008 sex 1.105229 0.146383 7.55 0 _Iage_9 0.370892 0.190677 1.95 0.052 _Iage_10 0.164794 0.191683 0.86 0.39 _Iage_11 0.21229 0.22004 0.96 0.335 _Iage_12 -0.5613 0.210174 -2.67 0.008 _Iage_13 -1.34199 0.218375 -6.15 0 _Iage_14 -1.94908 0.247149 -7.89 0 _Iage_15 -2.6782 0.267839 -10 0 _Iage_16 -2.95637 0.285206 -10.37 0 _Iage_17 -3.47639 0.296784 -11.71 0 _IsexXscho~1 0.035069 0.023569 1.49 0.137 _Istate_13 -1.5521 0.220932 -7.03 0 _Istate_16 -1.43423 0.223446 -6.42 0 _Istate_21 -0.99432 0.21386 -4.65 0 _Istate_22 -1.94792 0.261181 -7.46 0 _Istate_24 -2.11771 0.229534 -9.23 0 _Istate_30 -0.96735 0.227391 -4.25 0 MUNICIPIO DUMMY school 0.223272 0.0235 9.5 0 ranki -0.01069 0.037808 -0.28 0.777 ncri 0.080001 0.021681 3.69 0 ed_che 0.041625 0.012431 3.35 0.001 id_che -0.00107 0.003113 -0.34 0.73 pereagr -0.5809 0.070275 -8.27 0 ESC_p 0.820492 0.243902 3.36 0.001 ESC_s 1.533228 0.37282 4.11 0 distSEC -0.04933 0.019245 -2.56 0.01 _cons 0.061112 0.477692 0.13 0.898 2 rendomo -2.6E-05 5.78E-05 -0.46 0.649 what -0.00881 0.002527 -3.49 0 sex 0.418524 0.076557 5.47 0 _Iage_9 -0.05109 0.118538 -0.43 0.666 _Iage_10 -0.51105 0.113398 -4.51 0 _Iage_11 -0.84711 0.127128 -6.66 0 _Iage_12 -2.13563 0.117257 -18.21 0 _Iage_13 -3.17132 0.11892 -26.67 0 _Iage_14 -4.05299 0.13363 -30.33 0 _Iage_15 -5.00498 0.143825 -34.8 0 _Iage_16 -5.80871 0.156125 -37.21 0 _Iage_17 -6.54823 0.163246 -40.11 0 40 _IsexXscho~1 -0.01839 0.011115 -1.65 0.098 _Istate_13 0.008042 0.108736 0.07 0.941 _Istate_16 -0.54651 0.121999 -4.48 0 _Istate_21 -0.69317 0.115782 -5.99 0 _Istate_22 -0.41603 0.119283 -3.49 0 _Istate_24 -0.39484 0.117655 -3.36 0.001 _Istate_30 -0.24031 0.12064 -1.99 0.046 MUNICIPIO DUMMY school 0.341356 0.010447 32.67 0 ranki 0.094725 0.021523 4.4 0 ncri -0.03456 0.011984 -2.88 0.004 ed_che 0.064438 0.00701 9.19 0 id_che 0.004859 0.001671 2.91 0.004 pereagr 0.012498 0.036473 0.34 0.732 ESC_p 0.105365 0.130649 0.81 0.42 ESC_s 1.33154 0.263694 5.05 0 distSEC -0.10569 0.010067 -10.5 0 _cons 2.479397 0.255503 9.7 0 (sit = 0 is the base outcome) Table A-6: OLS Regression using ENCEL 1997 12-17 years old Number of obs = 3422; Adj R-squared = 0.2312; Root MSE = 0.63528 l_earn Coef. Std. t P>|t| ws -0.6075 0.037497 -16.2 0 _Iage_13 0.036609 0.069916 0.52 0.601 _Iage_14 0.173965 0.064705 2.69 0.007 _Iage_15 0.229693 0.062767 3.66 0 _Iage_16 0.2821 0.062238 4.53 0 _Iage_17 0.294646 0.06213 4.74 0 _Isex_1 0.204211 0.058664 3.48 0.001 school 0.019267 0.007614 2.53 0.011 _IsexXscho~1 -0.01878 0.0085 -2.21 0.027 lsalm 0.232743 0.032937 7.07 0 iml7954r -0.04817 0.023427 -2.06 0.04 pobreden -0.0924 0.02966 -3.12 0.002 _Istate_13 -0.10255 0.077337 -1.33 0.185 _Istate_16 0.014942 0.094584 0.16 0.874 _Istate_21 -0.07067 0.08192 -0.86 0.388 _Istate_22 -0.06368 0.096157 -0.66 0.508 _Istate_24 -0.12769 0.087166 -1.46 0.143 _Istate_30 -0.13361 0.084659 -1.58 0.115 MUNICIPIO DUMMY _cons 3.176226 0.201298 15.78 0 41 Table A-7: MLOGIT Regression using ENCEL 1997 12-17 years old Number of obs = 19282; LR chi2(276) = 8979.81; Prob > chi2 = 0; Log likelihood = -11890.6; Pseudo R2 = 0.2741 sit Coef. Std. z P>|z| 1 rendomo -0.00036 0.000204 -1.75 0.08 what -0.00974 0.005737 -1.7 0.089 sex 1.336258 0.216444 6.17 0 _Iage_13 -0.76115 0.121148 -6.28 0 _Iage_14 -1.37954 0.139038 -9.92 0 _Iage_15 -2.1014 0.160786 -13.07 0 _Iage_16 -2.3889 0.180381 -13.24 0 _Iage_17 -2.90052 0.196287 -14.78 0 _IsexXscho~1 0.004386 0.030551 0.14 0.886 _Istate_13 -1.59308 0.263266 -6.05 0 _Istate_16 -1.8076 0.273142 -6.62 0 _Istate_21 -1.11296 0.255425 -4.36 0 _Istate_22 -2.08594 0.313184 -6.66 0 _Istate_24 -2.32464 0.27933 -8.32 0 _Istate_30 -1.05794 0.269358 -3.93 0 MUNICIPIO DUMMY school 0.209312 0.02827 7.4 0 ranki -0.00671 0.057266 -0.12 0.907 ncri 0.033444 0.025891 1.29 0.196 ed_che 0.018113 0.015336 1.18 0.238 id_che 0.000436 0.003751 0.12 0.907 pereagr -0.40546 0.082632 -4.91 0 ESC_p 0.583097 0.287884 2.03 0.043 ESC_s 1.591755 0.381581 4.17 0 distSEC -0.05203 0.023577 -2.21 0.027 _cons -0.16313 0.640248 -0.25 0.799 2 rendomo -7.4E-05 6.17E-05 -1.2 0.229 what -0.00509 0.002922 -1.74 0.082 sex 0.667565 0.100408 6.65 0 _Iage_13 -1.00445 0.065313 -15.38 0 _Iage_14 -1.89471 0.072623 -26.09 0 _Iage_15 -2.82463 0.081792 -34.53 0 _Iage_16 -3.62027 0.095427 -37.94 0 _Iage_17 -4.35211 0.104709 -41.56 0 _IsexXscho~1 -0.04866 0.013445 -3.62 0 _Istate_13 -0.39296 0.123475 -3.18 0.001 _Istate_16 -1.09535 0.139274 -7.86 0 _Istate_21 -1.05066 0.132351 -7.94 0 _Istate_22 -0.90335 0.136086 -6.64 0 _Istate_24 -0.89142 0.132328 -6.74 0 _Istate_30 -0.51679 0.136568 -3.78 0 MUNICIPIO DUMMY 42 school 0.323843 0.011453 28.28 0 ranki 0.095179 0.028809 3.3 0.001 ncri -0.04391 0.013294 -3.3 0.001 ed_che 0.058434 0.007735 7.55 0 id_che 0.006369 0.001868 3.41 0.001 pereagr -0.00557 0.040552 -0.14 0.891 ESC_p 0.075362 0.144107 0.52 0.601 ESC_s 1.224399 0.272513 4.49 0 distSEC -0.12436 0.011692 -10.64 0 _cons 0.603831 0.313559 1.93 0.054 (sit==0 is the base outcome) Table A-8: OLS Regression using ENIGH 1999 12-17 years old Number of obs = 1229; R-squared = 0.3535; Root MSE = 0.70413 Robust l_earn Coef. Std. t P>|t| ws -0.49817 0.080114 -6.22 0 _Ischool_1 -0.83503 0.251169 -3.32 0.001 _Ischool_2 -0.0031 0.138087 -0.02 0.982 _Ischool_3 -0.81435 0.107551 -7.57 0 _Ischool_4 -0.80037 0.209365 -3.82 0 _Ischool_5 -0.4339 0.553829 -0.78 0.434 _Ischool_6 -1.13081 0.307738 -3.67 0 _Ischool_7 1.419835 0.154841 9.17 0 _Ischool_8 -0.24301 0.263424 -0.92 0.356 _Ischool_9 -0.57605 0.367697 -1.57 0.117 _Iage_13 -0.71879 0.134218 -5.36 0 _Iage_14 0.082265 0.207099 0.4 0.691 _Iage_15 -0.46776 0.083621 -5.59 0 _Iage_16 -0.46811 0.315204 -1.49 0.138 _Iage_17 0.041793 0.172272 0.24 0.808 SCHOOL x AGE interaction sex 0.026597 0.057631 0.46 0.645 lsalm 0.677629 0.078768 8.6 0 urbano 0.172929 0.064045 2.7 0.007 _cons 1.326435 0.304249 4.36 0 43 Table A-9: MLOGIT Regression using ENIGH 1999 12-17 years old Number of obs = 9183; Wald chi2(64) = 1401.81; Prob > chi2 = 0; Log pseudo-likelihood =-5138.31; Pseudo R2 = 0.34 Robust sit Coef. Std. z P>|z| 1 rendomo -0.00024 0.000273 -0.88 0.377 what -0.00708 0.00279 -2.54 0.011 hhsize -0.00035 0.036806 -0.01 0.992 _Ischool_1 21.25102 9.787838 2.17 0.03 _Ischool_2 13.79116 3.118599 4.42 0 _Ischool_3 20.71353 4.514764 4.59 0 _Ischool_4 15.23906 2.693013 5.66 0 _Ischool_5 17.30204 2.785797 6.21 0 _Ischool_6 6.892596 2.004884 3.44 0.001 _Ischool_7 16.8561 2.878271 5.86 0 _Ischool_8 23.75594 3.598322 6.6 0 _Ischool_9 5.006638 3.116995 1.61 0.108 age -2.38333 1.126347 -2.12 0.034 _IschXage_1 -1.27471 0.762994 -1.67 0.095 _IschXage_2 -0.73041 0.216395 -3.38 0.001 _IschXage_3 -1.23374 0.334568 -3.69 0 _IschXage_4 -0.74307 0.180915 -4.11 0 _IschXage_5 -0.84201 0.179096 -4.7 0 _IschXage_6 -0.23137 0.123083 -1.88 0.06 _IschXage_7 -0.70804 0.184502 -3.84 0 _IschXage_8 -1.1285 0.224 -5.04 0 _IschXage_9 0.015271 0.183439 0.08 0.934 age2 0.073948 0.039074 1.89 0.058 sex 0.924371 0.137466 6.72 0 ncri -0.06586 0.06578 -1 0.317 ed_che 0.089529 0.027089 3.3 0.001 id_che 0.010682 0.007018 1.52 0.128 _Ichef_ana~1 0.27459 0.234554 1.17 0.242 _Iurbano_1 -0.18893 0.187438 -1.01 0.313 _IcheXurb_~1 0.086548 0.340221 0.25 0.799 _Ipereagr_1 -0.89107 0.255841 -3.48 0 _IperXurb_~1 -0.05922 0.46076 -0.13 0.898 _cons 12.0094 8.08279 1.49 0.137 2 rendomo 0.000757 0.000151 5.03 0 what -0.00365 0.001875 -1.95 0.052 hhsize -0.03056 0.029843 -1.02 0.306 _Ischool_1 20.01243 6.287175 3.18 0.001 _Ischool_2 14.05995 4.56299 3.08 0.002 _Ischool_3 20.30527 4.525572 4.49 0 _Ischool_4 18.9947 4.291385 4.43 0 _Ischool_5 19.82598 4.33549 4.57 0 44 _Ischool_6 12.39132 3.53123 3.51 0 _Ischool_7 20.08583 3.948379 5.09 0 _Ischool_8 24.65777 4.47102 5.52 0 _Ischool_9 8.250779 3.943195 2.09 0.036 age -1.23758 0.864201 -1.43 0.152 _IschXage_1 -1.29412 0.47178 -2.74 0.006 _IschXage_2 -0.79471 0.326542 -2.43 0.015 _IschXage_3 -1.27234 0.325518 -3.91 0 _IschXage_4 -1.14464 0.30586 -3.74 0 _IschXage_5 -1.11763 0.302626 -3.69 0 _IschXage_6 -0.69827 0.247192 -2.82 0.005 _IschXage_7 -1.03201 0.273385 -3.77 0 _IschXage_8 -1.2717 0.300841 -4.23 0 _IschXage_9 -0.27782 0.267857 -1.04 0.3 age2 0.033179 0.02873 1.15 0.248 sex 0.254487 0.087417 2.91 0.004 ncri -0.10554 0.0526 -2.01 0.045 ed_che 0.144227 0.016025 9 0 id_che 0.021386 0.004875 4.39 0 _Ichef_ana~1 0.465103 0.178107 2.61 0.009 _Iurbano_1 0.771415 0.121484 6.35 0 _IcheXurb_~1 -0.62397 0.246272 -2.53 0.011 _Ipereagr_1 -0.14676 0.171696 -0.85 0.393 _IperXurb_~1 -0.15915 0.284332 -0.56 0.576 _cons 5.892521 6.822494 0.86 0.388 45 Estimations for Ecuador (BDH) Table A-10: OLS Regression using Baseline 2003 Number of obs = 267; R-squared = 0.4264; Root MSE = 0.67868 Robust l_earn Coef. Std. t P>|t| ws -0.49977 0.201318 -2.48 0.014 age -0.24467 0.356101 -0.69 0.493 age2 0.012886 0.012762 1.01 0.314 sex 0.180629 0.091735 1.97 0.05 lsalm 0.396124 0.174559 2.27 0.024 urban 0.126072 0.135503 0.93 0.353 _Ischool_1 -2.19578 0.530415 -4.14 0 _Ischool_2 -1.00641 0.568661 -1.77 0.078 _Ischool_3 -0.77323 0.407913 -1.9 0.059 _Ischool_4 -1.66233 0.449178 -3.7 0 _Ischool_5 -0.89796 0.448714 -2 0.047 _Ischool_6 -0.50477 0.324735 -1.55 0.121 _Ischool_7 0.040771 0.403345 0.1 0.92 _Ischool_8 -0.68354 0.444674 -1.54 0.126 _Ischool_9 -0.44851 0.483612 -0.93 0.355 CANTON DUMMY _cons 2.784753 2.51582 1.11 0.269 Table A-11: MLOGIT Regression using Baseline 2003 Number of obs = 2876; Wald chi2(94) = 1285.13; Prob > chi2 = 0; Log pseudo-likelihood = -1619.63 Pseudo R2 = 0.4679; Std. Err. Adjusted for 1306 clusters Robust sit Coef. Std. z P>|z| 1 rendomo 0.002349 0.001251 1.88 0.06 what -0.05672 0.015445 -3.67 0 hhsize -0.20205 0.06061 -3.33 0.001 age2 -0.02822 0.003554 -7.94 0 mae 0.101713 0.291116 0.35 0.727 domes 1.120512 0.201157 5.57 0 sex 0.819148 0.147756 5.54 0 school -0.06533 0.370655 -0.18 0.86 school2 0.087116 0.022173 3.93 0 ed_che 0.11097 0.034543 3.21 0.001 analf_che -0.06359 0.230309 -0.28 0.782 id_che 0.005117 0.009819 0.52 0.602 indig_che -0.04508 0.24461 -0.18 0.854 46 n05 0.077647 0.135983 0.57 0.568 nenrol 0.003889 0.100783 0.04 0.969 necoact 0.718026 0.124442 5.77 0 ranki -0.41967 0.155434 -2.7 0.007 water 0.010287 0.187873 0.05 0.956 toilet -0.59454 0.248398 -2.39 0.017 wealth1 0.074518 0.112082 0.66 0.506 urban 0.269404 0.226346 1.19 0.234 CANTON DUMMY _Igage_1 2.243918 2.029061 1.11 0.269 _Igage_2 1.332428 1.583292 0.84 0.4 _IgagXscho~1 0.035771 0.391392 0.09 0.927 _IgagXscho~2 -0.22326 0.214244 -1.04 0.297 _cons -0.21259 2.179989 -0.1 0.922 2 rendomo 0.00277 0.001241 2.23 0.026 what -0.05448 0.015159 -3.59 0 hhsize -0.06539 0.058908 -1.11 0.267 age2 -0.02977 0.003425 -8.69 0 mae 0.091806 0.294041 0.31 0.755 domes 0.576011 0.195201 2.95 0.003 sex 0.200328 0.151231 1.32 0.185 school 0.269437 0.440786 0.61 0.541 school2 0.070922 0.025259 2.81 0.005 ed_che 0.142764 0.035089 4.07 0 analf_che -0.42179 0.22192 -1.9 0.057 id_che -4.5E-05 0.009591 0 0.996 indig_che -0.48946 0.269788 -1.81 0.07 n05 -0.05303 0.130141 -0.41 0.684 nenrol 0.294744 0.104771 2.81 0.005 necoact -1.71552 0.179068 -9.58 0 ranki 1.03071 0.192825 5.35 0 water 0.340674 0.167277 2.04 0.042 toilet 0.006488 0.20741 0.03 0.975 wealth1 0.070699 0.105214 0.67 0.502 urban 0.088551 0.205876 0.43 0.667 CANTON DUMMY _Igage_1 4.479492 2.512193 1.78 0.075 _Igage_2 2.424846 2.01771 1.2 0.229 _IgagXscho~1 -0.46749 0.464386 -1.01 0.314 _IgagXscho~2 -0.32443 0.263182 -1.23 0.218 _cons -0.76923 2.578151 -0.3 0.765 (sit==0 is the base outcome) 47 Estimations for Nicaragua (RPS) Table A-12: Ex Ante simulations using the BFL for the Red de Protection Solidaria (RPS) program in Nicaragua: Statistics from RPS Baseline and follow-up Survey Actual and Counterfactual for Target Population Ex Post* AIT  Ex Ante AIT  Enrollment Rate 1 *** *** Children 7-to-13 years old 22.1% 4.0% 21.7% 2.2% Child Labor 2 ** Children 7-to-13 years old -2.5% 2.7% -2.1% 0.9% Source: RPS Baseline survey 2000 and Follow-up survey 2001. * 1 2 Notes: Results from Maluccio and Flores (2004); Table 10 - Maluccio and Flores (2004), Table 13 - Maluccio and Flores (2004) Table A-13: The Red de Protección Social program  The RPS program Started in 2000 and it was expanded in 2002  The RPS targeted poor households in rural areas using PMT to identify beneficiaries  The RPS program is a conditional cash transfer (CCT)  The RPS beneficiaries must keep children aged 7-13 years-old in school and participate in educational workshops.  The RPS beneficiaries must attend health center if pregnant or if have children 0-5 years old in the household;  The RPS transfer was fixed per household regardless number or children plus individual school supply transfer 48