77391 the world bank economic review, vol. 16, no. 2 275–295 Impact Evaluation of Social Funds The Impact and Targeting of Social Infrastructure Investments: Lessons from the Nicaraguan Social Fund Menno Pradhan and Laura B. Rawlings The benefit incidence and impact of projects financed by the Nicaraguan Emergency Social Investment Fund are investigated using a sample of beneficiaries, a national household survey, and two distinct comparison groups. The first group is constructed on the basis of geographic proximity between similar facilities and their corresponding communities; the second is drawn from the national Living Standards Measurement Study survey sample using propensity score matching techniques. The analysis finds that the social fund investments in latrines, schools, and health posts are targeted to poor communities and households, whereas those in sewerage are targeted to the bet- ter-off. Investments in water systems are poverty-neutral. Education investments have a positive, significant impact on school outcomes regardless of the comparison group used. The results of health investments are less clear. Using one comparison group, the analysis finds that use of health clinics increased as a result of the investments; using both, it finds higher use of clinics for children under age six with diarrhea. With nei- ther comparison group does it find improvements in health outcomes. Social fund in- vestments in water and sanitation improve access to services but have no effect on health outcomes. Social investment funds have quickly gained in popularity because of their ca- pacity to carry out community development projects rapidly and with broad participation. An alternative to strategies led by central governments, social funds allow communities control in determining investment priorities. This model, widely implemented in a short period, has been the basis for the World Bank’s first large-scale experience with small, community-led projects. The first social fund was created in Bolivia in 1987; today almost all countries in Latin America Menno Pradhan is with the Nutritional Science Department at Cornell University and the Economics Department at Free University in Amsterdam, and Laura B. Rawlings is a Senior Monitoring and Evalu- ation Specialist in the World Bank’s Latin America and the Caribbean Region. Their e-mail addresses are mpradhan@feweb.vu.nl and lrawlings@worldbank.org, respectively. The authors would like to thank the participants in the Nicaraguan Emergency Social Investment Fund (fise ) workshop held November 1999 at the World Bank for useful comments, particularly Florencia Castro-Leal, Carlos Lacayo, John Newman, Berk Ozler, Geert Ridder, and Steve Younger. They would also like to thank the team at the fise for guidance and support and the team at the National Statistical Institute in Nicaragua for excel- lent data collection and processing. Funding for the research was provided by the governments of Nica- ragua and Norway and the World Bank. © 2002 The International Bank for Reconstruction and Development / THE WORLD BANK 275 276 the world bank economic review, vol. 16, no. 2 and the Caribbean have social funds or development projects that embody many of their operational characteristics. Social funds have also been established in Africa, Asia, Europe, and the Middle East. Social funds finance small projects using a demand-driven process that allows the fund to appraise, finance, and supervise the implementation of social projects identified and executed by a range of local actors (Jorgensen and van Domelen 1999). In Latin America most social funds concentrate their investments in social infrastructure, particularly school construction projects (Goodman and others 1997). Although organizationally part of the central government, social funds generally operate outside the norms regulating public agencies, including those governing staff salaries. Some observers have praised social funds because as a result of these features, they can attract high-level professional staff and operate efficiently by separating the financing of investments from their provision; others have criticized them for providing a means of avoiding essential reforms in line ministries. Despite the widespread use of social funds, until recently they have not been subjected to rigorous impact evaluations. As a result there is little knowledge and much debate about whether these demand-driven mechanisms can reach poor communities and households and whether the infrastructure investments they finance affect welfare outcomes. Given the scope of social funds and the national and international resources they have quickly absorbed, the need for serious evaluation of social funds is clear. This evaluation provides empirical data shed- ding light on the extent to which social funds have realized their goals. This impact evaluation, one of the first on social investment funds, was moti- vated by the prominence of the Nicaraguan social fund and the interest in Nica- ragua and beyond in assessing the ability of social funds to reach the poor and contribute to changes in their welfare. The Nicaraguan Emergency Social Invest- ment Fund (Fondo de Inversión Social de Emergencia, or fise) is the primary financier of health and education infrastructure in Nicaragua, with total opera- tions accounting for more than 1 percent of the country’s gross domestic prod- uct (gdp). It has grown remarkably since its establishment in 1990 and played a key role in expanding public infrastructure. In 1991–98 the fise carried out 40 percent of the public investments in Nicaragua’s social sector infrastructure (Bermudez and others 1999). The fise is patterned on the general model for social funds. Its central func- tion is to finance infrastructure improvements in schools, health centers, water systems, and sanitation facilities at the request of local communities. It has also focused increasingly on combining its financial role with strengthening the plan- ning and implementation capacity of local government. This article examines whether fise investments—in primary schools, rural health posts, latrines, and water and sewerage systems—are targeted to poor communities and poor households, improve access to basic social services, and help improve health and education outcomes. In doing so, the article contrib- utes to the thin literature on the effects on household behavior and outcomes in Pradhan and Rawlings 277 developing economies of quality improvements in health facilities (Alderman and Lavy 1996, Hotchkiss 1998, Lavy and others 1996), education facilities (Glewwe 1999, Hanushek 1995, Kremer 1995), and water and sanitation fa- cilities (Brockerhoff and Derose 1996, Lee and others 1997). Existing studies are based on cross-sectional variations in quality and do not take account of the endogeneity of the placement of government investments. By dealing explicitly with the endogeneity of investments, this article makes a new contribution. The article also contributes to the large literature on targeting, providing informa- tion on the outcomes achieved through the novel strategy of combining explicit targeting to poor areas with a demand-driven approach. Because the locations of fise interventions are determined through a non- random selection process, a simple comparison of health and education outcomes between areas that benefited from fise investments and those that did not would not yield a valid estimate of the impact of the investments on beneficiaries. With only postintervention data available, the choice of evaluation techniques to ad- dress this selection issue is limited. This analysis applies a matched comparison technique in which each treatment subject is matched with a comparison subject that did not benefit from a fise investment. Two comparison groups are used. One was drawn from similar schools and health posts in the proximity of the treatment facilities. The other was constructed using propensity score matching, a technique building on recent advances in the evaluation literature that has been applied mainly in labor market evaluations (Dehejia and Wahba 1998, Heckman and others 1998). This household-level impact evaluation is part of a larger evaluation of the Nicaraguan social fund carried out by the World Bank in coordination with the fise. The larger evaluation also includes an analysis of the quality and sustainability of fise projects based on the results of a project-level survey, a review of the insti- tutional evolution of the fise, a comparison of the cost-effectiveness of fise in- vestments with that of similar projects carried out by another agency, and a contextual process evaluation of fise projects in a subsample of 18 fise com- munities selected from those surveyed for the impact evaluation. The results of all the studies are summarized in World Bank (2000). I. The fise and Data Sources for the Impact Evaluation The fise was created in November 1990 to fund small-scale projects designed to meet the basic needs of the poor and create temporary employment, thereby contributing to the poor’s economic and human capital and involving them in Nicaragua’s economic and social development (Bermudez and others 1999). In 1991–98 the fise invested US$191 million, making it the largest social invest- ment fund (as a percentage of gdp) in Latin America. On average, the fise in- vested $11.2 million a year in education, and the Ministry of Education invested $11.7 million. The social fund’s average yearly investment in health was $5.8 million, and the Ministry of Health’s was $17.2 million. The fise directed most 278 the world bank economic review, vol. 16, no. 2 of its investments to infrastructure and equipment for primary and secondary schools (57 percent), health posts and health centers (8 percent), infrastructure for water and sanitation (9 percent), latrine facilities (7 percent), and public works (16 percent). This article considers all but the last category. The fise uses a poverty map to target investments to the poor. The poverty map used to guide the projects reviewed in this evaluation is based on the 1993 Living Standards Measurement Study (lsms) survey—a nationwide household survey—and contains a poverty measure developed by the fise for each munici- pality.1 Estimates based on the ranges established by this poverty map show that in 1991–98, 23 percent of fise investments went to municipalities with “extreme� poverty, 53 percent to municipalities with “high� poverty, and 24 percent to municipalities with “medium and low� poverty. Both the municipalities with extreme poverty and those with high poverty received a larger share of fise in- vestments than their share of the population, pointing to a progressive geographic distribution of resources by the standards of the poverty map (table 1). Moreover, the fise’s allocation of resources to extremely poor municipalities has improved, rising from 11 percent of investments in 1991 to 34 percent of investments in 1998. The fise has recently updated its poverty map using results of the 1998 lsms survey and the 1995 census and applying new methodologies that allow the imputation of consumption-based poverty levels.2 The poverty targeting and impact analyses carried out in this study rely on three sources of data: the 1998 lsms survey, the fise household survey, and administrative data. The fise household survey applied the same questionnaire and was fielded at the same time as the 1998 lsms survey. Both surveys fol- lowed the established practices developed in the World Bank lsms initiative (Grosh and Glewwe 1995). The fise household survey sampled from house- holds in the area of influence of randomly chosen fise projects and matched comparison (non-fise ) projects (in health and education only). The area of influence was determined on the basis of service provision norms for schools and health centers and project records on fise construction for water, sewer- age, and latrine projects. At the sampling stage there was concern that random sampling would not yield sufficient observations of households that actually used the facilities tar- geted by fise investments. For this reason, choice-based sampling techniques 1. The poverty map is based on several weighted measurements used to construct a composite pov- erty score assigned to municipalities based on their basic needs, per capita income, and population size. First, the poverty map is based on three indicators of poverty, each with the following weights: infant malnutrition (40 percent), access to drinking water (40 percent), and the proportion of displaced indi- viduals (20 percent). The results are then weighted to favor the poorest municipalities using a relative poverty indicator (rpi ), which measures income levels relative to the cost of a basket of basic goods. Based on the rpi, municipalities are divided into three groups: extreme poverty, high poverty, and me- dium and lower poverty. Finally, the poverty map score is weighted by the size of municipal populations using estimates based on the 1971 census. 2. For more information on techniques combining census and survey data to estimate poverty rates see Elbers and others (forthcoming). Pradhan and Rawlings 279 Table 1. Poverty Targeting of fise Investments Across Municipalities, 1991–98 Municipal Share of Total investment, poverty Number of population 1991–98 Average annual per ranking municipalities (percent) (US$ millions)a capita investment(US$) Extreme 42 18.4 143.6 (22.8) 6.25 High 96 51.6 101.7 (53.2) 5.33 Medium 9 30.0 146.1 (24.1) 3.79 and low Total 147 100.0 191.34 (100.0) 4.98 aFiguresin parentheses are percentage shares of the total. Source: World Bank (2000). were applied. Within the randomly chosen set of census segments in the area of influence, all households were classified as either direct beneficiaries or potential but not direct beneficiaries.3 Two samples were drawn, one from the group of direct beneficiaries that were confirmed as users of the social fund investment and one from the group of potential beneficiaries. Weights were constructed to correct for the sampling in the analysis stage (Manski and Lerman 1977). Sample sizes for the fise survey (which includes households in the area of influence of both fise projects and non-fise schools and health posts) are shown in table 2. The sample size for the lsms survey, from which compari- son groups were constructed using propensity score matching methods explained later, was 4,040 households. The administrative data used in the analysis come from a data file contain- ing the census segments associated with the areas of influence of the universe of fise health and education projects by type of project. A census segment is included in the database if more than 50 percent of the segment is located within the area of influence of a selected project. This file makes it possible to sepa- rate the households in the 1998 lsms survey into two groups: potential benefi- ciaries and others.4 In addition, the analysis uses data from the poverty map employed by the fise in targeting its investments. This map contains the esti- mated poverty head count ratio (share of the population in poverty) for each municipality. 3. For education projects, direct beneficiaries are households that have at least one child in the fise school. For health projects they are households in which a member has visited the fise clinic in the past year. For sewerage projects they are households that have a flush toilet connected to the sewer. Water and latrine projects are public access facilities, allowing no distinction between direct and potential beneficiaries. 4. Those living outside the area of influence of a fise project could decide to benefit from the project. For instance, children living outside the area of influence of a fise school could enroll in the school. Thus, there is no guarantee that those living outside the area of influence of a fise project did not benefit from the intervention, potentially biasing the comparison group. This caveat holds for both types of com- parison groups. 280 the world bank economic review, vol. 16, no. 2 Table 2. Sample Size of fise Survey (number of households) Treatment or Direct Potential comparison group beneficiaries beneficiaries Total Education Treatment 161 79 240 Comparison 142 99 241 Health Treatment 165 34 199 Comparison 164 35 199 Water Treatment 95 0 95 Sewerage Treatment 74 30 104 Latrines Treatment 234 0 234 Total 1,312 Source: World Bank (2000). II. Targeting of fise Investments The analysis distinguishes between two levels of targeting. First, it explores com- munity-level targeting by examining the characteristics of households in the area of influence of fise projects—the potential beneficiaries. Second, it investigates household-level targeting by examining the characteristics of households using the fise investments—the direct beneficiaries. To evaluate the benefit incidence of social fund investments, the analysis applies a conventional benchmark by comparing an implicit transfer with a uniform transfer. The implicit transfer is obtained by assuming that everyone who uses a social fund facility obtains an equal benefit. A uniform transfer assumes an equal transfer to every individual in the population. When the social fund investments reach a larger proportion of the poor than a uniform transfer would, the social fund is considered progres- sive (on a per capita basis). Concentration coefficients are used to assess the targeting of fise investments to the poor. The analogue of Gini coefficients for Lorenz curves, concentration coefficients are derived from concentration curves, which show the cumulative percentage of benefits received by the population ranked according to a welfare measure, in this case per capita consumption. The coefficients range from –1 (all transfers go to the poorest) to 1 (all transfers go to the richest). The concentra- tion coefficient is defined as 1 – 2 ∫G(x)dx, where G(x) is the concentration curve.5 A major advantage of using concentration curves is that information on the av- erage probability of benefiting from an intervention is not needed. For any con- sumption level x, the concentration curve shows the fraction of the population with per capita consumption below x (derived from the lsms survey) against the fraction of beneficiaries with per capita consumption below x (derived from the fise beneficiaries survey). The curve can thus be computed using two indepen- dent surveys. 5. For information on the concentration curves constructed for this study see World Bank (2000). Pradhan and Rawlings 281 The analysis also examines the share of social fund benefits accruing to those below the poverty line and the extreme poverty line used in Nicaragua.6 In 1998, 48 percent of the population of Nicaragua lived below the poverty line, and 17 percent below the extreme poverty line. If the share of social fund benefits ac- cruing to these groups is larger than their population share, the investments are progressively targeted to these groups. The concentration coefficients for fise investments in education show that they are distributed with a slight propoor bias, although the incidence of bene- fits is close to neutral for the extreme poor (table 3). This is a common finding in analyses of the benefit incidence of education investments, and it arises mainly from the fact that poor households generally have more children.7 When the analysis includes only direct beneficiaries (households with at least one child enrolled in a fise school) rather than potential beneficiaries, the concentration curve falls slightly higher, indicating that fise schools have been relatively suc- cessful in reaching poor children within the communities where the schools are located. fise health interventions reveal a more propoor distribution than the educa- tion interventions. This outcome is explained in part by the fact that health posts are typically in rural areas, whereas primary schools are in both rural and urban areas. Whether potential or direct beneficiaries are used in the analysis makes little difference in the targeting results for health interventions, indicating that the likelihood of visiting an fise facility, conditional on living in an area where one is present, does not depend on income. The targeting outcomes for water and sanitation investments reveal a great deal of heterogeneity. Latrine investments are the most progressive of all those analyzed in the impact evaluation. Water investments are distributed quite evenly across the population, showing neither a strong prorich nor a strong propoor bias. Sewerage interventions are very poorly targeted, both at the community level (potential beneficiaries) and at the household level (direct beneficiaries). In considering the poverty targeting results, it should be kept in mind that the nature of projects can affect their potential to reach poor households. Water and sewerage projects need to reach a certain scale to be cost-effective and thus are typically concentrated in more populated areas, which tend to be wealthier. Latrines tend to be used only by the poor, so the success of latrine investments in reaching the poor and the extreme poor reflects the self-targeted nature of this 6. The poverty line is set at $344, considered to be the level of annual per capita consumption nec- essary for a person to attain the minimum caloric requirements. The measure takes into account non- food items. The extreme poverty line (also called the food poverty line) is set at $181, considered to be the level of annual per capita food expenditure necessary for a person to satisfy the minimum daily requirement of 2,226 calories. 7. The benefit incidence of education investments depends in part on the number of children enrolled from a household and the poverty ranking of the household. The choice of welfare measure here—per capita consumption—assumes that there are no economies of scale in household consumption; changing this assumption could lead to reversals in poverty rankings (Lanjouw and Ravallion 1995). 282 the world bank economic review, vol. 16, no. 2 Table 3. Targeting of fise Investments to the Poor and Extreme Poor, 1998 Share of benefits reaching Share of benefits extreme poor (percent)a reaching poor (percent)b Concentration coefficient Among Among Among Among Type of Potential Direct potential direct potential direct project beneficiaries beneficiaries beneficiaries beneficiaries beneficiaries beneficiaries Education –0.061 –0.111 18.0 18.1 53.9 59.2 Health –0.120 –0.115 17.0 12.3 64.1 65.2 Water –0.004 n.a. 12.3 n.a. 49.9 n.a. Sewerage n.a. 0.420c n.a. 5.1c n.a. n.a. 0.430 0.370d 4.0 8.3d 10.7 8.6 Latrines –0.301 n.a. 26.9 n.a. 73.3 n.a. n.a. = Not applicable. aThe 1998 lsms survey observed an extreme poverty rate of 17 percent. bIncludes extreme poor. The 1998 lsms survey observed a poverty rate of 48 percent. cBased on broad definition of direct beneficiaries (households with any access to sewerage system). dBased on narrow definition of direct beneficiaries (households with flush toilet connected to sewerage system). Source: Authors’ calculations based on 1998 lsms survey, fise survey, and fise administrative data. type of investment. The targeting outcomes for water and sanitation investments by fise are consistent with those from other countries (Rawlings and others 2002). III. Impact Evaluation The central question posed by the impact evaluation is this: If the fise had not existed, what would the condition of the beneficiaries have been? The analysis compares this counterfactual condition with the results from the survey of pro- gram beneficiaries to estimate the impact of fise investments in health posts, primary schools, water systems, and sanitation (sewerage systems and latrines) on beneficiaries’ access to and use of these basic services as well as their health and education status. Impact Evaluation Methodology Because the impact evaluation was designed without the benefit of baseline data, the counterfactual was constructed using a matched comparison technique.8 This method defines a comparison group of individuals who did not have the oppor- tunity to benefit from an fise project. If this group is similar to the treatment 8. For an overview of different methods of impact evaluation see Grossman (1994). An alternative approach is difference in differences, but this was not feasible because of the lack of a preintervention survey. Another alternative is to use instrumental variables. This technique was not applied because there were no good candidates for variables that influence the selection of a community into the program but not the outcome. Such variables typically measure the ability of a community to obtain a project. This information is usually collected through a community questionnaire, which was not included in the 1998 lsms survey. Pradhan and Rawlings 283 group in all relevant preintervention characteristics, a direct postintervention comparison of the comparison and treatment groups provides an estimate of the impact of the fise intervention. The two groups should be similar in both ob- servable and unobservable characteristics that influence outcomes and selection into the program. Constructing such a comparison group is a nontrivial matter. A simple comparison of health and education outcomes between areas that benefited from fise investments and those that did not would not yield a valid estimate of the impact of the investments because of the nonrandom selection process for fise interventions. These selection issues arise from the allocation process for social fund investments, which takes into account the preferences of both communities and the social fund. Communities take the initiative in apply- ing for a social fund project, including selecting the type of project, such as con- structing latrines or rehabilitating a school. Communities’ ability to prepare and execute project proposals also determines in part the likelihood that they will receive a project. The preferences of the social fund come into play during the promotion and review of projects. For instance, fise, using its poverty map, al- locates more resources to poorer areas. Two types of matched comparison methodologies were used to construct a comparison group for estimating the counterfactual. First, for health and edu- cation projects only, a fise comparison group was constructed during the sam- pling stage of the study, before the fise survey was implemented. Each fise facility included in the survey was matched to the nearest non-fise facility, with the match restricted to facilities of similar size and type.9 The fise survey col- lected information on households in the area of influence of the fise facilities as well as households in the area of influence of the non-fise comparator facilities; this second set of households made up the fise comparison group. Second, a propensity score comparison group was constructed separately for each of the interventions from the 1998 lsms sample using propensity score matching techniques. This score measures the probability that a subject receives an intervention as a function of observable characteristics. Rosenbaum and Rubin (1983) show that if it is valid to match using these characteristics, it is equally valid to match using only the propensity score. This matching method greatly simplifies the problem and allows the inclusion of many variables in the propen- sity score, thereby reducing the role of unobservables. One can say little a priori about which comparison group should be preferred for analyzing the impact of fise health and education investments. Both rely on presumptions about the method that is most suitable for creating a comparable comparison group. The fise comparison group is based on the notion that the 9. Characteristics used for matching fise and non-fise facilities include location (urban or rural) and the poverty category of the municipality. Number of classrooms was also used to match schools and type of facility (according to Ministry of Health norms) to match health posts. Based on these criteria, each fise facility was matched to the nearest non-fise facility that did not have an overlapping area of influence. 284 the world bank economic review, vol. 16, no. 2 nearest similar non-fise health post (or school) and the corresponding house- holds are equivalent to the fise health post (or school) and corresponding house- holds before the fise intervention. The propensity score match is valid under the assumption that the variables included in the propensity score functions are sufficient to eliminate the selection bias between the treatment and comparison groups. Propensity Score Matching Methodology The variables entering into the propensity score function are chosen with the knowl- edge that the preferences of both local communities and the fise determine the final allocation of projects. The variables measure the ability of a community to prepare project proposals, the preferences of the fise (poverty map data), and, where available, preprogram outcomes (outcome indicators are available only for the water and sanitation interventions). With only a postintervention survey avail- able, the analysis must rely on recall information for preprogram outcomes. All other explanatory variables are valid under the assumption that they have not changed as a result of the fise intervention. This limitation again emphasizes the need for comparable baseline data when evaluating social programs.10 Propensity score matching requires one to estimate the probability that an individual lives in the area of influence of a facility receiving an investment. To estimate this function precisely, one needs to know exactly which communities received a fise investment and which did not. The lsms survey might appear to be the most appropriate source for this information, since it asks households whether or not they benefited from an fise investment. But many households do not realize that their community received an fise investment—and worse, many households whose community did not receive an fise investment think that it did.11 The analysis therefore relies on fise administrative data, which provides the census segments associated with the areas of influence of all fise health and education projects by project type. This file, merged with the lsms survey results, separates households between those that are in the area of influ- ence of an fise project and those that are not. This allows an estimate of the propensity score associated with living in the area of influence of an fise project for individual i: (1) Pr(potential beneficiary) = F(Xib). 10. The 1993 lsms survey could not serve as a baseline because it covered different communities than the 1998 lsms survey, and because of sample size limitations relative to the population of fise beneficiaries. 11. The fise survey includes information on whether or not respondents are in a community that had received an fise investment. The results show, for instance, that only 90 percent of the households classified as direct beneficiaries of an fise education project claimed that they had benefited from an fise education investment, whereas 25 percent of those in the fise comparison group claimed that they had benefited from such an investment. Pradhan and Rawlings 285 The function F is estimated using a probit model. Xi are the observed character- istics of individual i. They include community characteristics and fise targeting instruments. Modeling the propensity score for a fise water, sewerage, or latrine invest- ment is easier than constructing the corresponding model for a health or educa- tion investment. Almost all the water and sanitation projects were included in the fise beneficiaries survey, and thus the likelihood that a household included in the lsms survey is in a community that received one of these projects is neg- ligible. Moreover, the area of influence of water and sanitation projects is geo- graphically defined. The propensity score function is therefore estimated using the combined data of the lsms survey and the fise beneficiaries survey, with the assumption that none of the households in the lsms survey benefited from a water and sanitation project. Sampling weights are used to correct for the choice-based sampling (Manski and Lerman 1977). Because there is no limit on the number of explanatory variables that can be included in the propensity score function, the analysis uses a fully interacted model for health and education investments. But because the coefficient estimates are difficult to interpret, the article presents the estimates of the probit models in which none of the variables is interacted. To ensure comparability between the results for the fise comparison group and those for the propensity score comparison group, the treatment population is always defined as those identified as (potential) fise beneficiaries in the fise household survey. The propensity score comparison groups are drawn from the lsms sample, restricted to the areas in which the fise has no projects of the type being investigated. The population from which the match is drawn depends on the impact variable used. If the focus is on children’s enrollment, for instance, the comparison group is restricted to school-age children. The population is also limited to the geographic region in which the treatment population lives, based on the assumption that households within a region share characteristics that are not fully captured by the regional dummy variable in the model. Limiting the selection of comparison group subjects to those living in the same region as the treatment group increases the likelihood that the two groups will be similar. The geographic restriction did not affect the ability to find a good match for every treatment group. Predicting Participation in fise Projects. Estimation results for the prob- ability of living in the area of influence of an fise health or education project as defined in equation 1 show that the geographic variables are highly significant (table 4). This finding reflects the tendency of the fise to invest in poorer areas, a preference confirmed by the benefit incidence analysis and reconfirmed by the significant positive effect of the poverty head count ratio of the municipality from the poverty map. Results for the access road variables, included as a proxy for the remoteness of the municipality, show that households with worse access roads have a higher chance of living in the area of influence of an fise project. 286 the world bank economic review, vol. 16, no. 2 Table 4. Probit Estimates of Geographic Location of fise Education and Health Projects Education projects Health projects Variable Coefficient t-statistic Coefficient t-statistic Managua 0.594 4.72 n.a. n.a. Pacific urban 0.757 8.08 –0.545 –11.19 Pacific rural 0.434 5.28 0.571 15.71 Central urban 0.132 1.43 –0.515 –11.06 Atlantic urban 0.193 2.09 –1.332 –21.87 Atlantic rural –0.287 –2.83 –0.034 –0.81 Log per capita consumption –0.040 –1.01 –0.049 –2.57 Paved road to house –0.030 –0.44 –0.133 –3.52 Dirt road to house 0.141 2.04 0.213 6.54 Non-fise projects from which household 0.040 1.13 0.045 2.35 benefited Non-fise projects from which household –0.202 –2.40 0.023 0.60 benefited and in which it participated Total membership of community –0.047 –1.32 –0.005 –0.31 organizations Distance to school or clinic –0.152 –5.74 –0.048 –15.11 Head count ratio in municipality based 0.014 4.89 0.001 0.84 on fise poverty map Gini coefficient in region 1.089 3.71 1.514 9.92 Constant –0.387 –0.50 –0.774 –2.17 Pseudo R2 0.052 0.111 n.a. = Not applicable. Note: The data in the table are not the estimates for the propensity score function. The propensity score function applies the same explanatory variables but is fully interacted. Dependent variable = 1 if household lives in the area of influence of a health or education project. Source: Authors’ calculations based on matched fise administrative data on geographic locations of projects and lsms sample. The number of non-fise projects from which a household has benefited and the number of such projects in which a household has participated are included as proxies for a community’s ability to develop projects and obtain project fi- nancing. The results reveal that the number of non-fise projects a community has received has no significant effect on the probability of its receiving a fise education project. By contrast, community participation has a negative effect, possibly because once a community has obtained a non-fise education project, it is less likely to seek a similar fise project. The number of non-fise projects has a positive effect on a community’s ability to obtain fise health projects. As expected, distance to a fise facility has a negative effect on the probability of living in its area of influence. Income inequality in the region, as measured by the estimated Gini coefficient, has a positive effect on the probability of obtain- ing a fise project. The R2 in education is 0.052 and increases to 0.1038 when the fully inter- acted model is used, whereas the R2 in health is 0.111 and increases to 0.1632. Pradhan and Rawlings 287 A low R2 does not necessarily mean that the propensity score function is not good. In the extreme case, when the allocation of projects has been de facto ran- dom, the R2 would be zero, but the resulting propensity score comparison group would be perfect. The estimated propensity score functions for latrine and sewerage projects in- clude a higher-order term for consumption than do those for water projects be- cause of the strong targeting bias found (propoor for latrine projects and prorich for sewerage investments). Interaction terms are not used for water and sanitation projects. Experiments with interaction terms for these models, which have fewer degrees of freedom, found that their use worsens the overlap of the propensity score functions. It was therefore decided to continue with a limited set of descriptive explanatory variables, which yielded a good overlap. (The results, omitted here because of space limitations, are available in Pradhan and Rawlings 2000.) Matching Process. Beneficiaries observed in the fise sample are matched to similar individuals from the 1998 lsms survey who did not live in the area of influence of a fise project. Individuals can be matched only once—that is, with- out replacement. To test the quality of the propensity score match, propensity scores were plotted for the treatment and comparison groups for each area of investment under evaluation (education, health, water, sewerage, and latrines). Except for a few observations in the health treatment group with high propen- sity scores, the curvatures of the functions observed for each treatment group come very close to overlapping with those of the comparison group. These results indicate strong similarities between the treatment and comparison groups and a high-quality match.12 The Impact of FISE Investments on Beneficiary Households An unbiased estimate of the average treatment effect of an fise intervention can be obtained by simply comparing mean outcomes in the comparison and treat- ment groups. For the treatment group, constructed using propensity score match- ing, the t-test for equal means has to take account of the uncertainty arising from the fact that the comparison group sample is based on an estimated coefficient vector in the propensity score function. The standard errors for this comparison are calculated using bootstrapping with 400 replications. In each iteration a new comparison group is constructed using a random draw from the estimated dis- tribution of the coefficient vector of the propensity score function. Following the usual bootstrap procedures, a random sample of equal size is drawn from the matched sample with replacement. The observed difference in means in the bootstrapped sample takes account of both the uncertainty arising from the fact that the comparison group sample is based on an estimate and the fact that the treatment group estimate is based on a sample of limited size. 12. The probit estimates for the treatment and comparison groups are compared in Pradhan and Rawlings (2000). 288 the world bank economic review, vol. 16, no. 2 Education. The average impact of living in the area of influence of a fise school is estimated for several indicators—enrollment, the education gap (the difference between the ideal educational attainment, given a child’s age, and the highest grade attended), age for grade, repetition, attendance, and age in first grade. Enrollment appears to have increased as a result of the fise investments (table 5). The net enrollment ratio for the treatment group is almost 10 percentage points higher than that for the propensity score comparison group, though the differ- ence is smaller—4.5 percentage points—and insignificant for the fise compari- son group.13 Results for both comparison groups confirm the impact of the fise in reducing the education gap from around 1.8 years to 1.5. The effect is signifi- cant for both comparison groups. No significant effect is found for the share of children in the correct grade for their age. However, the age of first grade children dropped sharply—from 8.6 to 7.9 years —as a result of fise education invest- ments, a finding confirmed by results for both comparison groups. Nonetheless, absenteeism is high in fise schools, averaging 6.8 days a month. Although this rate is slightly better than that observed in the fise comparison group, it is sig- nificantly worse than that observed in the propensity score comparison group, rendering the results inconclusive. Results based on the two comparison groups in education are fairly consis- tent and are also significant. This suggests that the significant, positive effects of fise investments in primary education on enrollment, the education gap, and age in first grade are robust. The effects of fise education investments are also estimated separately for different consumption quintiles and by gender (table 6). Results based on both comparison groups confirm that fise education investments have a greater ef- fect on girls’ enrollment than on boys’. They show that the investments have a greater effect in reducing the education gap and increasing the share of children in the correct grade for age for children in poorer quintiles. Health. The effects of fise interventions in health are less clear, rendering the results inconclusive. Beneficiary households had a higher contact rate (that is, were more likely to have visited a health post or health center in the past month) than the propensity score comparison group, but there was no significant differ- ence between the treatment group and the fise comparison group (table 7). Es- timation results based on the fise comparison group indicate an improvement in the contact rate for children with diarrhea, but those based on the propensity score comparison group show no significant effect. Although results based on the fise comparison group point to an improvement in indicators of acute mal- nutrition resulting from fise investments, those based on the propensity score comparison group do not confirm this finding. For most of the other outcome variables the differences between the treatment and comparison groups are in- 13. An evaluation of the Peruvian Social Investment Fund also found a positive effect of social fund investments on school enrollment (Paxson and Schady 2002). Table 5. Impact of fise Education Investments on School Outcomes t-test on Propensity score t-test on Treatment group fise comparison group means comparison group means Indicator No. observations Mean No. observations Mean (p-value) No. observations Mean (p-value)a Net primary enrollment 341 91.7 358 87.2 0.056 341 82.1 0.073 ratio (percent) Education gap (years)b 338 1.5 357 1.7 0.039 335 1.9 0.0279 289 Children in correct grade 341 26.0 358 25.5 0.889 341 21.8 0.5094 for age (percent) Days of school missed 302 6.8 313 7.3 0.394 259 1.7 0.000 in past month Age in first grade (years) 76 7.94 85 8.60 0.001 77 8.56 0.0875 aBased on bootstrapped estimates with 400 replications. bDifferencebetween ideal educational attainment, given a child’s age, and the highest grade attended. Source: Authors’ calculations based on 1998 lsms survey, fise survey, and fise administrative data. Pradhan and Rawlings 289 290 the world bank economic review, vol. 16, no. 2 Table 6. Impact of fise Education Investments by Consumption Quintile and Gender Quintilea Indicator and population group 1 (poorest) 2 3 4 5 (richest) Boys Girls Net primary enrollment ratio (percent) Treatment group 82.8 96.1 96.4 94.7 90.2 90.0 93.9 fise comparison group 85.9 86.9* 97.9 82.0* 84.6 87.1 87.4* Propensity score comparison group 69.2* 93.3 85.1* 73.9* 89.1 82.4* 81.7* Education gap (years)b Treatment group 1.8 1.4 1.7 1.3 0.6 1.6 1.3 fise comparison group 2.2* 2.0* 1.5 1.5 0.7 2.1* 1.3 Propensity score comparison group 2.6* 2.0* 1.8 1.7 0.6 2.1* 1.7* Children in correct grade for age (percent) Treatment group 16.8 23.6 25.4 24.8 55.3 21.8 31.5 fise comparison group 12.4 19.5 36.7 27.0 48.2 22.9 28.2 Propensity score comparison group 4.5* 9.1* 21.2 43.1 66.4 17.1 27.9 Days of school missed in past month Treatment group 6.0 9.2 6.5 6.9 4.5 6.9 6.6 fise comparison group 8.9* 8.1 7.6 7.4 3.5 8.6* 6.3 Propensity score comparison group 1.6* 2.0* 2.2* 0.6* 1.9 1.7* 1.9* *Difference between treatment and comparison groups significant at the 5 percent level. aBased on the national distribution of per capita consumption as observed in the 1998 lsms survey. bDifference between ideal educational attainment, given a child’s age, and the highest grade attended. Source: Authors’ calculations based on 1998 lsms survey, fise survey, and fise administrative data. significant. Estimation of the average treatment effect by gender and consump- tion quintile leads to similar inconclusive results. Water Systems. Impact estimates for fise water projects show that the invest- ments had a significant, positive effect on water supply (table 8). The variables for change in access to infrastructure, constructed using recall information from 1993, before the fise investments, are equivalent to difference-in-difference estimators. The results show that the share of households with access to piped water increased by about 21 percentage points more in areas where the fise invested than in areas where it did not. Variables for rates of malnutrition and diarrhea all indicate an improvement in health status, but the results are not significant. Sewerage Systems. The fise has had a significant, positive impact on access to sewerage systems in the areas where it has invested (see table 8). The treat- ment group is defined as direct beneficiaries—households with a flush toilet con- nected to the sewerage system. The propensity score comparison group is drawn from the eligible population—households not connected to a sewerage system in 1993, based on recall data on access to water and sanitation facilities in that year. Without a fise intervention, only 8.7 percent of households in the propen- Table 7. Impact of fise Health Investments on Health Outcomes Treatment group fise comparison group Propensity score comparison group No. Mean No. Mean p-value for No. Mean p-value for Indicator observations (percent) observations (percent) equal means observations (percent) equal meansa Contact rateb 1,169 10.3 1,196 11.1 0.523 1,169 5.6 0.029 Contact rate for children under age 6b 223 23.4 207 19.4 0.315 223 5.6 0.008 Contact rate for people over age 5b 946 7.2 948 9.6 0.053 946 5.7 0.425 Incidence of diarrhea in past month 220 27.0 207 22.6 0.286 220 16.9 0.153 in children under age 6 Contact rate for children with diarrheab 50 43.3 40 18.1 0.009 47 17.0 0.255 Incidence of cough or other respiratory 1,169 22.5 1,196 23.5 0.562 1,169 18.9 0.342 disease in past month Share of women giving birth in past 104 76.1 107 69.3 0.271 104 87.4 0.293 five years who had at least one prenatal checkup in that period 291 Share of institutional births 104 69.0 107 55.0 0.036 104 70.8 0.881 Share of births attended by 104 97.7 107 94.5 0.236 104 94.9 0.324 skilled health staffc DPT vaccination coveraged 36 86.7 25 94.2 0.320 36 96.3 0.518 Polio vaccination coverage 36 93.6 25 97.3 0.491 36 99.8 0.564 Prevalence of wasting 164 0.4 144 4.7 0.020 164 1.1 0.739 (low weight for height)e Prevalence of stunting 164 20.5 144 24.2 0.436 164 17.3 0.717 (low height for age)e Prevalence of underweight 164 10.1 144 19.5 0.021 164 11.4 0.739 (low weight for age)e aBased on bootstrapped estimates using 212 replications. bContact rate shows the percentage of individuals who visited an outpatient public health care provider in the past month. cGynecologist, nurse, nurse assistant, or midwife. Pradhan and Rawlings dDPT is diphtheria, pertussis (whooping cough), and tetanus. eModerate malnutrition with z-scores less than –2 for children under age 6. Source: Authors’ calculations based on 1998 lsms survey, fise survey, and fise administrative data. 291 Table 292 8. world of Impact the fise bank Water review, economic and Sanitation Investments vol. 16, no. 2 on Health and Infrastructure (percent, except where otherwise specified) Propensity score Treatment group comparison group p-value No. No. for equal Indicator observations Mean observations Mean meansa Water investments Incidence of diarrhea in past month 79 18.8 157 25.4 0.399 among children under age 6 Prevalence of wasting 102 3.4 108 3.6 0.946 (low weight for height)b Prevalence of stunting 102 13.6 108 24.0 0.204 (low height for age)b Prevalence of underweight 102 15.6 108 18.5 0.690 (low weight for age)b Distance to water source in 1997 (km) 95 0.0090 189 0.075 0.334 Change in distance to water source 95 –0.1298 189 –0.042 0.157 between 1993 and 1997 (km) Share of households with piped water 95 84.6 189 56.5 0.0000 in 1997 Change in share of households with 95 27.3 189 5.9 0.0000 piped water between 1993 and 1997 (percentage points) Sewerage investments Incidence of diarrhea in past month 23 9.4 45 21.9 0.237 among children under age 6 Prevalence of wasting 0 0 n.a. (low weight for height)b Prevalence of stunting 31 12.2 30 16.9 0.683 (low height for age)b Prevalence of underweight 31 16.0 30 6.9 0.414 (low weight for age)b Share of households with flush toilet 31 100.0 61 8.7 0.000 in 1997 Change in share of households with 31 100.0 61 8.7 0.000 flush toilet between 1993 and 1997 (percentage points) Latrine investments Incidence of diarrhea in past month 226 29.16 451 24.52 0.365 among children under age 6 Prevalence of wasting 313 5.8 312 4.7 0.694 (low weight for height)b Prevalence of stunting 313 23.7 312 22.4 0.817 (low height for age)b Prevalence of underweight 313 12.7 312 13.9 0.798 (low weight for age)b Share of households with no toilet 224 1.86 447 23.00 0.000 in 1997 Change in share of households with 224 –31.87 447 –13.19 0.000 no toilet between 1993 and 1997 (percentage points) aBased on bootstrapped estimates with 200 replications. bModerate malnutrition with z-scores less than –2 for children under age 6. Source: Authors’ calculations based on 1998 lsms survey, fise survey, and fise administrative data. Pradhan and Rawlings 293 sity score comparison group managed to obtain a flush toilet.14 None of the health-related impact variables is significant, but the results may reflect small sample sizes.15 Latrines. Again using recall data for 1993, the analysis finds that in areas re- ceiving fise investments in latrines, the share of households with access to sani- tation facilities increased by nearly 20 percentage points more than it did in areas without fise investments. No significant results are found for the impact on diarrhea or malnutrition. IV. Conclusions This article presented estimates of the impact and benefit incidence of the Nica- raguan Emergency Social Investment Fund. Impact estimates were derived using two comparison groups. One was constructed on the basis of geographic proxim- ity and similarities with the facilities (schools and rural health posts) receiving the social fund investments (the fise comparison group). The other was constructed using propensity score matching techniques and drawing from the household data collected by the 1998 Living Standards Measurement Study survey (the propen- sity score comparison group). The benefit incidence analysis indicates that fise investments in the health and education sectors, which together receive the largest share of fise financ- ing, have a pro-poor bias. Latrine investments also are strongly biased toward the poor. By contrast, sewerage investments generally benefit the better-off, while water investments are equally distributed, favoring neither the poor nor the rich. The impact evaluation shows that fise investments in education have had a positive impact on enrollment and the education gap, although the size and significance of the effect found depends on the comparison group used. As a result of fise investments, children enroll half a year earlier on average. En- rollment ratios improved more for girls than for boys, and the share of chil- dren in the correct grade for their age increased more among the poor than among the better-off. 14. When potential fise beneficiary households (all those that could have connected to the fise - financed sewerage system) are matched to similar households, the analysis reveals a 34.4-percentage- point increase in the share of households with a flush toilet from 1993 to 1998 in the treatment group, compared with a 2.5-percentage-point increase in the propensity comparison group. Thus the net in- crease in access to flush toilets resulting from fise investments is almost 32 percentage points. 15. When potential fise beneficiaries are matched to their corresponding propensity comparison group, estimation results show that fise-financed sewerage investments have a significant impact on the incidence of diarrhea in children under age six. This suggests that sewerage investments may have a community-level effect even in the absence of high rates of connection to the sewerage system. The larger sample size ob- tained when matching potential beneficiaries (rather than the smaller sample of direct beneficiaries with toilets) also underscores the importance of having sample sizes large enough to estimate specific impacts, especially for a particular population such as children under six. 294 the world bank economic review, vol. 16, no. 2 The results for fise investments in health are less clear. The strongest result points to a 5-percentage-point increase in the share of households using health clinics. But this effect is found only when the propensity score comparison group is used and is not confirmed by the fise comparison group. When the fise com- parison group is used, fise investments are found to have had a significant ef- fect on acute malnutrition, but this effect is not confirmed by the propensity score comparison group. There is similar inconsistency for other indicators of impact. This lack of consistency undermines confidence in the results for fise investments in health. Social fund investments in water and sanitation improved the physical infra- structure and the share of households with access to services. They also appear to have had a positive effect on health indicators, but the effects are generally insignificant, possibly as a result of the small samples. The results of the evaluation were discussed at length in two workshops held in Managua, Nicaragua, with the fise ’s management, representatives of its principal multilateral and bilateral donors and representatives of government agencies working closely with the fise , including the Ministries of Health and Education. The evaluation informed key policy changes. For instance, the fise suspended investments in sewerage for two years in response to the findings of poor poverty targeting and lack of measurable effects on health. In addition, the evaluation results helped generate World Bank support for a new pilot project aimed at increasing the development impact of fise investments for the extreme poor. The pilot project will provide subsidies to households that send their children to school and use health services for basic preventive care. Finally, the results helped inform policy debates within Nicaragua, particularly those relating to the development of the country’s Poverty Reduction Strategy, and helped foster an evaluation culture within the country. References Alderman, Harold, and Victor Lavy. 1996. “Household Responses to Public Health Ser- vices: Cost and Quality Tradeoffs.� World Bank Research Observer 11(1):3–22. Bermudez, Gustavo, Lifia Maria Castro Monge, and Luz Marina Gracias Fonseca. 1999. “Analisis Institucional del fise.� gb Consultants, Managua, Nicaragua. Brockerhoff, Martin, and Laurie F. Derose. 1996. “Child Survival in East Africa: The Impact of Preventive Health Care.� World Development 24(12):1841–57. Dehejia, Rajeev H., and Sadek Wahba. 1998. “Propensity Score Matching Methods for Nonexperimental Causal Studies.� nber Working Paper 6829. National Bureau of Economic Research, Cambridge, Mass. Elbers, Chris, Peter Lanjouw, and Jennifer Lanjouw. Forthcoming. “Welfare in Towns and Villages: Micro-Measurement of Poverty and Inequality.� Econometrica. Glewwe, Paul. 1999. The Economics of School Quality Investments in Developing Coun- tries: An Empirical Study of Ghana. Studies on the African Economies. New York: St. Martin’s. Pradhan and Rawlings 295 Goodman, Margaret, Samuel Morley, Gabriel Siri, and Elaine Zuckerman. 1997. “Social Investment Funds in Latin America: Past Performance and Future Role.� Inter- American Development Bank, Evaluation Office and Social Programs and Sustain- able Development Department, Washington, D.C. Grosh, Margaret E., and Paul Glewwe. 1995. “A Guide to Living Standards Measure- ment Study Surveys and Their Data Sets.� lsms Working Paper 120. World Bank, Washington, D.C. Grossman, Jean. 1994. “Evaluating Social Policies: Principles and U.S. Experience.� World Bank Research Observer 9(2):159–80. Hanushek, Eric A. 1995. “Interpreting Recent Research on Schooling in Developing Countries.� World Bank Research Observer 10(2):227–46. Heckman, James J., Hidehiko Ichimura, and Petra Todd. 1998. “Matching as an Econo- metric Evaluation Estimator.� Review of Economic Studies 65(2):261–94. Hotchkiss, D. R. 1998. “The Tradeoff between Price and Quality of Services in the Phil- ippines.� Social Science and Medicine 46(2):227–42. Jorgensen, Steen, and Julie van Domelen. 1999. “Helping the Poor Manage Risk Better: The Role of Social Funds.� World Bank, Human Development Network, Washing- ton, D.C. Kremer, Michael R. 1995. “Research on Schooling: What We Know and What We Don’t. A Comment on Hanushek.� World Bank Research Observer 10(2):247–54. Lanjouw, Peter, and Martin Ravallion. 1995. “Poverty and Household Size.� Economic Journal 105(433):1415–34. Lavy, Victor, John Strauss, Duncan Thomas, and Philippe de Vreyer. 1996. “Quality of Health Care, Survival and Health Outcomes in Ghana.� Journal of Health Econom- ics 15(3):333–57. Lee, Lung-fei, Mark R. Rosenzweig, and Mark M. Pitt. 1997. “The Effects of Improved Nutrition, Sanitation, and Water Quality on Child Health in High-Mortality Popula- tions.� Journal of Econometrics 77(1):209–35. Manski, Charles F., and Steven R. Lerman. 1977. “The Estimation of Choice Probabili- ties from Choice-Based Samples.� Econometrica 45(8):1977–88. Paxson, Christina, and Norbert Schady. 2002. “The Allocation and Impact of Social Funds: Spending on School Infrastructure in Peru.� World Bank Economic Review 16(2):xxx–xxx. Pradhan, Menno, and Laura B. Rawlings. 2000. “The Impact and Targeting of Nicaragua’s Social Investment Fund.� World Bank, Latin America and the Caribbean Region, Human Development Department, Washington, D.C. Rawlings, Laura B., Lynne Sherburne-Benz, and Julie van Domelen. 2002. “Evaluating Social Fund Performance: A Cross-Country Analysis of Community Investments.� World Bank, Social Protection Network, Washington, D.C. Rosenbaum, Paul R., and Donald B. Rubin. 1983. “The Central Role of the Propensity Score in Observational Studies for Causal Effects.� Biometrika 70(1):41–55. World Bank. 2000. “Nicaragua: Ex-Post Impact Evaluation of the Emergency Social Investment Fund (fise).� Report 20400-NI. Latin America and the Caribbean Region, Human Development Department, Washington, D.C.