WiS 2 8so POLICY RESEARCH WORKING PAPER 2800 Did Social Safety Net Preliminary evidence favors focusing safety net Scholarships Reduce scholarships-designed to Drop-Out Rates during the reduce dropout rates during an economic crisis-on lower Indonesian Economic Crisis? secondaryschools, continuing to target children (especially older students) Lisa A. Cameron from large families, scaling back scholarships to private schools at the lower secondary level, or targeting the households hurt most by the crisis. The World Bank Development Research Group Poverty Team March 2002 I POLICY RESEARCH WORKING PAPER 2800 Summary findings Cameron uses regression and matching techniques to design was in reaching the poor. Comrmittees that evaluate Indonesia's Social Safety Net Scholarships allocated the scholarships followed the criteria diligently, Program, which was developed to keep large numbers of but a significant percentage of scholarships did go to children from dropping out of school as a result of the students from households with high reported per capita Asian crisis. It was expected that many families would expenditures, if household expenditure data are reliable. find it difficult to keep their children in school and that It is unclear how targeting can be improved, giving the dropout rates would be high, as they were during a scarcity of accurate local household data in most recession in the 1980s. But dropouts did not increase countries. Using local monitoring could help but then markedly and enrollment rates remained relatively monitoring for accountability would be more difficult. steady. Cameron examines the role the scholarship Preliminary evidence favors focusing safety net program played in producing this result. scholarships-designed to reduce dropout rates during She found the scholarships to have been effective in an economic crisis-on lower secondary schools, reducing dropouts in the lower secondary school (where continuing to target children (especially older students) students are more susceptible to dropping out) by about from large families, scaling back scholar slhips to private 3 percentage points. They had no discernible impact in schools at the lower secondary leyel, or targeting the primary and upper secondary schools. households hurt most by the crisis. Cameron also examines how well the program adhered to its documented targeting design and how effective that This paper-a product of the Poverty Team, Development Research Group-is part of a larger effort in the group to study the welfare impact of the East Asian crisis. Copies of the paper are available free from the World Bank, 1818 H Street NW, Washington, DC 20433. Please contact Patricia Sader, mail stop MC3-306, telephone 202-473-3902, fax 202-522-1153, email address psader@worldbank.org. Policy Research Working Papers are also posted on the Web at http:// econ.worldbank.org. The author may be contacted at Icameron@unimelb.edu.au. March 2002. (46 pages) The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessanily represent the view of the World Bank, its Executive Directors, or the countries they represent. Produced by the Research Advisory Staff Did Social Safety Net Scholarships Reduce Drop-Out Rates during the Indonesian Economic Crisis? Lisa A. Cameron* Department of Economics University of Melbourne March 2002 I would like to thank Vivi Alatas, Stephen Baines, Deborah Cobb-Clark, Deon Filmer, Emanuela Galasso, Gavin Jones, Santosh Mehrotra, Menno Pradhan, Lant Pritchett, Jerry Strudwick, Sudarno Sumarto, Yusuf Suharso, Asep Suryahadi, Duncan Thomas, and participants at the RAND Labor and Population Program and World Bank seminars for helpful comments. The assistance of Asep Suryahadi and Yusuf Suharso with the matching of the different rounds of data was invaluable. Research support from the World bank, UNICEF, and SMERU is gratefully acknowledged. Addresses for correspondence: Department of Economics, University of Melbourne, Victoria, 3010, Australia. Email: Icameron@unimelb.edu.au I. Introduction Indonesia's gross domestic product dropped by 13% in 1998, the rupiah plummeted from a pre-crisis level of approximately Rp 2500 per US dollar to Rp 16,000 and inflation reached 77%. Accompanying the dramatic decline in the country's economic fortunes was an appropriate concern for the social impact of the crisis - its effect on poverty, health, fertility, child labor and school enrolment rates. Somewhat surprisingly the social impact of the crisis has been much more muted than what was expected given the magnitude of the financial decline. That's not to say that people didn't and aren't still suffering. Poverty increased from 11% to about 20% of the population and real wages fell dramatically. Health and education indicators were however remarkably, and somewhat inexplicably, robust to such a drastic change in the country's fortunes. This is in sharp contrast to the 1980's recession which saw enrolments decline substantially. Various explanations have since been advanced for the very small or non-existent decreases in enrolment rates to date (as documented in Jones, Hagul and Damayanti (2000) and Pradhan and Sparrow (2000)). These include decreases in the opportunity cost of children's time due to the excess supply of adults in the labor market; that unlike in the 1980's, children have not been forced out of school if school fees went unpaid; and a possible increase in the value Indonesian parents place on their children's education and hence added efforts to keep their offspring in school. The extensive Social Safety Net (Jaring Pengamanan Sosial, JPS) scholarship program that was put in place at the start of the 1998/99 school year is an additional contender. This program was an ambitious I undertaking. Funded by the World Bank, the Asian Development Bank and other bilateral donors to the tune of US$350 million over three years, it undertakes to reach approximately 6% of the country's enrolled primary school students, 17% of lower secondary students and 10% of the country's upper secondary students. Somewhere between 1.2 and 1.6 million scholarships were disbursed in the 1998/1999 school year, Jones et al., (2000). This paper has two aims. First, we use data from the 100 Villages Survey to examine how well the program adhered to its documented targeting design and whether the design resulted in those from poorer households receiving a greater proportion of the scholarships. Secondly, we use regression and matching techniques to estimate the impact of the program on school attendance. Detailed knowledge of the program's selection criteria and data reflecting these and other household characteristics enable us to test for selection into the program on the basis of unobservable characteristics. The results show that the project reduced drop-outs at the lower secondary level but had no discernible effect at the upper secondary and primary level. The data cover a period only 4 months after the start of the program, albeit a period in which the crisis accelerated unexpectedly to its peak in December 1998, hence, it is a period in which one might have expected to see significant school drop-outs in the absence of the program but it is also possible that the program will have longer term effects that cannot be measured here. In this sense the results should be considered preliminary until confirmed with data over a longer time period. The paper is structured as follows. Section 2 presents summary statistics of changes in Indonesian school enrolment rates during the crisis period and discusses the JPS 2 scholarships program in some detail. Section 3 discusses the methodology that will be used to evaluate the impact of the program. The 100 Villages data is discussed in Section 4. Section 5 examines the targeting performance of the program and section 6 presents the impact evaluation results. Section 7 concludes. II. Background and Details of the Scholarships Program Indonesian Education System Indonesia's school system consists of three levels - primary schools, lower secondary schools and upper secondary schools. Since the 1970's there has been a primary school in every Indonesian village and the government's stated goal of universal primary school education was attained in the mid-1980's. The major educational challenge since then has been to stem the large number of dropouts which occur at the lower secondary school level. High dropout rates at this level mean that by upper secondary school, students are largely from better-off families. Educational gender differentials in Indonesia are very low by international standards. Crisis Impact The large nationally representative sample of households in the Indonesian Statistical Agency's Susenas (Survei Sensus Nasional) survey provides the most comprehensive data on the impact of the crisis on enrolments to date. The Susenas enrolment rates show that primary school enrolments hardly changed during the crisis (Table 1). There was however a very small dip in enrolments at lower secondary schools from 77.5 per cent in 1997 to 77.2% in 1998 but this rebounded to above the pre-crisis level at 79% in 1999. Upper secondary enrolments did not decline at all (48.6% in 1997, 3 49.3% in 1998 and 51.2% in 1999). The Susenas also show that the tendency for enrolment rates to increase has come largely from poorer households, Jones et al. (2000).' While these figures are promising, it nevertheless remains possible that enrolment rates will decline more sharply in the longer term. It has been widely noted for instance that enrolments fell substantially in the four years following the economic decline of 1986/87. It is however also true that the enrolment declines that occurred during 1986/87 were larger than what has occurred to date during this crisis. There are a number of differences between this crisis and the 1980's recession which might explain this result. Government spending on education has been maintained, fees have not increased as dramatically as other costs of living, students who have been unable to pay their fees have not been forced out of school and requirements to wear uniforms have been relaxed, Jones et al. (2000). Another possibly important difference is the existence of the JPS program. JPS Scholarships Program The scholarship program discussed in this paper is just one program in Indonesia's newly constructed social safety net. This combination of programs was put together by the Indonesian government with the financial assistance and advice of international aid organizations in an attempt to lessen the social impact of the crisis.2 The 'This trend in national enrolment rates is confirmed by the Ministry of Education's enrolment figyures which show very small declines. The Indonesian Family Life Survey is another source of data on school enrolment and shows a larger decrease and significant variation in dropout rates across geographic areas and with socio-economic status, see Beegle et al. (1999). The 100 Villages data show a rise in enrolment at the primary school level. At the lower secondary level (children aged 13-15 years) enrolments iniitially dropped quite sharply from 0.69 in May 1997 to 0.65 in August 1998. This had rebounded to 0.7 in May 1999, Cameron (2000). Upper secondary enrolment rates were relatively stable. All of the sources however show a much smaller decline in enrolments than that which was initially forecast. 2 The other elements are block grants to schools, the employment creation programs, credit programs, health and nutrition programs and subsidized rice programs. 4 scholarships provide RplO,000, Rp. 20,000 and Rp. 30,000 per month for primary, lower secondary and upper secondary school students respectively. These amounts generally cover the cost of school fees and can be used for that purpose or to cover other expenses. Scholarships funds were first allocated to schools so that "poorer" schools received proportionally more scholarships. Details of the geographic allocation are given in the appendix. Scholarships were then allocated to individual students by school committees which consisted of the school head teacher, the chair of the parent's association, a teacher representative, a student representative and the village head. School students in all but the lowest three grades of primary school were officially eligible. Participant students were to be from the poorest backgrounds. Committees were to use household data from school records and pre-existing household classifications prepared by the National Family Planning Coordinating Agency (Badan Koordinasi Keluarga Berencana Nasional, BKKBN) to identify potential participants. The BKKBN classifies households as Pre-Prosperous, Prosperous I, Prosperous II, Prosperous III or Prosperous III+. The BKKBN data is collected by family planning volunteers and was originally designed to be used to target families for family planning programs. The BKKBN ranks households on the basis of a number of simple questions including whether all householders have different sets of clothes for when they are at home, working or going to school and going out, whether the house has a dirt floor, whether when a child is sick or if a householder wishes to use family planning methods, they are taken to a health center and receive modem medicines and whether the householders generally are able to perform their religious duties. The data have been criticized for their lack of reliability 5 and comparability across regions.3 However this is the only data available in Indonesia that attempts to cover every household in the archipelago. All other official data sets cover only small geographic areas comprehensively, or are random samples of the population and so are not able to identify which particular households (as opposed to geographic regions) are poor. Hence, when the crisis hit Indonesia, the BKKBN data was the only data that could be used for this purpose. Scholarships were to be allocated to children from households in the two lowest BKKBN rankings first. If there were a large number of eligible students such that not all of the poor students could receive a scholarship, then additional indicators were to be used to identify the neediest students. The additional indicators were those living far from school, those with physical handicaps and those coming from large or single parent families. Also, a minimum of 50% of scholarships, if at all possible, were to be allocated to girls. As will be discussed in more depth below, the 100 villages data provide information on many of these indicators. The program however also allows local knowledge to play a role in the allocation of scholarships. Hence, we will need to assess the role played by unobservable factors and the degree to which they are correlated with the probability of dropout below. The allocation of JPS funding (6% of primary students, 17% of lower secondary and 10% of upper secondary) coupled with the higher average socio-economic status of students in upper secondary school means that only a relatively small number of poor students will be able to be targeted at the primary school level, a much greater percentage of children from poor households will receive scholarships at the lower secondary level 3Suryahadi et al. (1999) for example find that the BKKBN rankings are not strongly correlated with 6 and a majority of poor students should be able to receive scholarships at the upper secondary level. Qualitative evidence on the impact of the program indicates that it has been well-received in the villages, has been reasonably well-targeted and has played a role in keeping at least some children in school, Hardjono (1999). III. Impact Evaluation Methodology An issue that has to be dealt with in every evaluation of a targeted program is the endogeneity of selection into the program. In the case of the JPS scholarships program, this means the selection of the most 'at risk' children. That is, children were chosen on the basis of characteristics, possibly both observed and unobserved, that ex-ante were expected to be correlated with low educational attainment. For example, children in households who were classified as BKKBN Pre-Prosperous or Prosperous-I were targeted, as were children in large households and households with a single parent. The targeted nature of the program can introduce a negative correlation between program participation and the probability of being in school, which it is necessary to correct for when estimating the program impact. If one simply calculates the difference between the dropout rate of participants and non-participants without controlling for these differences then the impact estimate will be biased downwards and may even show the program having a negative impact on enrolments. If selection into the program is based purely on observables (that is variables that we are able to control for) then we can remove this bias. Equations 1 and 2 illustrate this point. Equation 1 is the scholarship participation equation. It shows an underlying response variable, JPS;, which can be thought of as the individual's probability of reported per capita expenditures. 7 receiving a scholarship. This variable is determined by a vector of observed variaLbles, Xio ~and a random error term, e,o . However, we only observe whether an individual actually receives a scholarship or not, JPSj . JPSj = iAXio + 6io JPs = I if JPS;>O (1) =0 otherwise. Educational attainment at time t, Si,, as shown in equation 2, is similarly determined by a vector of observables (which may in practice differ from those in equation 1 but which we will denote with the same notation here for simplicity), a vector of unobservables 77,, participation in the program, and a random error term, u,t. Sit = fik, + yJPSi + (0, + u,) (2) In this case we can obtain an unbiased estimate of the program's impact, y, by estimating equation 2 using ordinary least squares. Now consider the possibility, as in the JPS scholarships case, that participation in the program may also be based on unobservables and these unobservables may be correlated with educational attainment at time t. In this case equation 1 becomes: JPS; =AX;o + (7,o + ,io0) JPS =I if JPs; >O (1') =0 otherwise. In this case, estimating equation 2 by ordinary least squares will no longer provide an unbiased estimate of the program's impact because, if the unobservables are correlated over time, program participation is correlated with the error term (ii,, + u, ) via q0 . The 8 problem is the same as if we just compared the dropout rates across participants and non- participants. In this case it is unobservable characteristics of participants which make them less likely to be enrolled than non-participants, even controlling for their observable characteristics. In an ideal world, one would have access to data from a randomized experiment which would guarantee that your control group had the same underlying distribution of observables and unobservables as the group that received the scholarship. So, when one calculated the difference between the average enrolment rate of the children who received the scholarship with the average for those who didn't, then the unobservables would cancel out. That is: E[(Sj, I JPSj = 1) -(Sj, I JPSj = 0)] =E[,fXi, + yJPS, + 7i, + uj, I JPSj = 1] - E[/Xi, +7ij, + u, I JPSj = 0] =y (3) However in our sample, there is at least the possibility ("hope" from the program designers view and "concern" from the econometricians view) that the unobserved variables differ across participants and non-participants, with participants having unobservable characteristics that make them less likely to be enrolled in school at time t. In this case, this simple difference estimator will not provide an unbiased estimate of r. We can however make some progress on eliminating the bias due to unobservables by examining differences in educational attainment across time. Consider two time periods, one just immediately prior to the start of the program (t=O) and one after the program is in operation (t=l ). Seeing that JPS1 must equal 0 at t=O, we can write: S, = fiXi + YJPSi + q,1 + u,1 (4) 9 Sjo fiX= o + 1ho + Uio (5) Note that for every individual: (Sii - Sio ) = riPsi + 68oxio - Alxii) + (17io - 770 + (uijo - uj,l) (6) Hence, if the unobservables are constant over time ( qjo = i7j, ), they difference out and so the JPSi variable is no longer correlated with the error term. The retrospective data in the December 1998 round, along with the data from the August 1998 data, allow us to estimate equation 6. Note however that there are still two possible forms of bias. First, bias thaLt arises from time-varying unobservables that determine participation and enrolment status. However, we know that JPSj is only a function of the unobservables at t=O. Given that our "baseline" data is collected almost exactly at t=O (or possibly even a little later) this is unlikely to be an important source of bias in this case. This is because there was very little chance for the unobservables to change in a way that would be correlated with scholarship participation. The second source of bias that may remain is potentially more serious. In some ways the whole differences-in-differences approach just introduces a new set of semantics to the problem of eliminating bias due to unobservables when one is examining a scholarships program. It is true that the bias due to unobservables that affect educational attainment at time t=O and t=1 in the same way are differenced out by the method proposed in equation (6), but it is also possible (although I will argue maybe not in this particular instance) that participants were chosen on the basis of unobservables that are thought to be correlated with "changes in educational attainment" rather than educational attainment at any point in time. That is, students that were thought to be most 10 likely to have dropped out between t=O and t=1 (and so have a smaller increase in educational attainment over the period) were chosen to participate. In most scholarship cases this is not an unlikely scenario. However, in the current Indonesian context and given the data used in this study, this may not be the case. As will be explained below, the data restrict our attention to the period August to December 1998. This was a period of massive social upheaval in Indonesia. Rice and other food prices went through the roof and real wages fell. Some households were however sheltered from this turmoil. Those that produced most of their own food for example may have actually benefited from the price increases. These dramatic changes in prices were largely unpredictable and even if the price increase was anticipated, it would still have been difficult for school committees to identify the winners and losers from an as yet unrealized change in prices. Hence, it may be that the brevity and unpredictable nature of the period of study works in our favor and that unobservables on which the committees based their allocation decisions were largely uncorrelated with the unobservables that determined dropout between August and December 1998.4 Fortunately, we don't have to leave the existence of such bias to chance. Below we are able to test whether the residuals in the participation equation are correlated with the residuals in the dropout equation. We find that they are not. Given that we are then dealing with a world of selection on observables, we are able to obtain unbiased estimates of j from estimating a version of equation 6 and are also able to construct estimates based on the method of matching to compare with the regression based estimate. The matching methods used will be described in detail in the results section. IV. Data The data used in this study are from the "100 Village Survey" (Survei Seratus Desa, SSD). This is a survey of 120 households in each of 100 villages across Indonesia which is conducted by the Indonesian Central Statistical Agency (BPS) and funded by UNICEF. The villages are located in 10 districts (kabupaten), spread across eight of Indonesia's 27 provinces.5 The villages were chosen to represent different types of villages in the rural economy. It was not designed to be a nationally representative sample and focuses disproportionately on rural and relatively poor areas. Hence, it may not be appropriate to generalize the specific estimates generated here to the country at large. However, the results are informative in that they indicate whether the program has been successful in these villages and hence provide some information on the likelihood of it having been successful elsewhere. The first round of the survey was conducted in 1994. It has since been conducted in May 1997, August 1998, December 1998, May 1999 and August 1999.6 The data can in theory be merged across time to form a panel because in each round the majority of households from the previous round are reinterviewed. In this paper we use the matched data from the August 1998 and December 1998 rounds. We limit the analysis to these two rounds because of the loss of sample size that results from the use of each subsequent round of data. Of the 12,000 households interviewed in December 1998, only 8751 were 4 Selection on unobservables is also likely to be less serious here than in the case of self-selection into programs. The provinces covered are Riau, Lampung, West Java, Central Java, Bali, Central Nusa Tenggara, East Kalimantan and South East Sulawesi. 6 The 1999 rounds were not available for analysis at the time of writing. 12 also interviewed in August 1998. If we extended the analysis to May 1997, the joint sample would decrease further to 6201 households.7 Merging the data was difficult and time-consuming. Households were matched manually using the village of residence and the name of the household head. Further checks were then made using demographic characteristics. The SSD provides both information on the household in which the child lives and information on the individual characteristics of the child.8 In the December 1998 round households were also asked whether they had received help in the form of JPS scholarship funds. Thus we can identify households who received funds. We cannot however identify the actual child that received the scholarship. Nevertheless, this household information is sufficient to allow us to examine the targeting of the program and to identify the program impact on dropout.9 Changes in the data's individual identification codes makes it virtually impossible to track children across school years. We are fortunate however in that the December 1998 questionnaire asks parents whether each of their children is in school at the time of the survey. Then it also asks if the child is not in school, whether the child dropped out in the current school year or in a previous year. Thus we can construct a variable that indicates whether the child was enrolled at the start of the school year which was in the 7 Those households who appear in both rounds in most respect don't differ substantially from those that leave the sample after one round. Their incomes and expenditures are however slightly lower and they may have been slightly less adversely affected by the crisis. s Information is gathered on the demographic attributes of the interviewees, on education, health and fertility behavior, migration, labor market activity, socio-economic status and crime. The post-crisis surveys focus to a greater extent on the living standard of the household and gather information on coping mechanisms. 9 We also know whether individual children receive a scholarship from the government but not whether it was a JPS scholarship or one of a range of other scholarships. Another problem with the child level scholarship question is that it was only asked of children who were in school at the time of the survey and so can't be used for an analysis of drop-outs. 13 last week of July. We can further construct a variable which indicates whether the child dropped out of school in the current school year. We restrict our sample to the 7686 children who were eligible for the scholarship because they were attending school at the start of the school year and who appear in both the August and December 1998 rounds. The August data is needed because it contains information in and before August that was used for targeting. The August 1998 data also provide us with some variables that are not available in the December data, such as the household's BKKBN classification. The JPS scholarship program commenced in the 1998/99 school year. The December 1998 data thus gives us the opportunity to examine the impact of the program in the preceding 4-5 months. This is a relatively short period of time in which to witness the impact of a program that could reasonably be expected to have predominantly long term impacts. The period August 1998-December 1998 was however the peak of the Indonesian crisis. Rice prices increased dramatically and reached their highest point in December. They have since dropped sharply and at the time of writing are below the government's floor price.'0 Hence, it was between July and December that households were under the greatest strain and during which the threat of student drop-out was high. As discussed briefly above, it is however possible that households would have been able to afford school fees through running down assets during this period and that the real pinch would only come several months later when asset stocks were seriously depleted. It is hence possible that even if the scholarship program shows no impact during this period, that it would have one at this latter stock-out point. It would also have been ° This is possible because farmers are not required to sell to the government logistics agency, EBULOG, and BULOG no longer has the funds to purchase the rice stocks that are required to keep the price at the floor price. 14 preferable to have data on drop-out at the end of the school year because this is when most drop-outs occur, rather than during it. The results presented here should thus be interpreted as an analysis of the short term impact of the program. V. Targeting In this section we examine how many households received the scholarships and how closely actual scholarship receipt followed the selection criteria - specifically that girls, single parent households, large households and households in the two lowest BKKBN rankings be targeted. We also examine to what extent the program reached the poor where the poor are defined by their level of per capita expenditure. As mentioned above the data only provide us with information on whether any child in the household received a scholarship, not on which child within the household received a scholarship. It is hence possible that one or more of the children were actually scholarship recipients. In this section and elsewhere references to "scholarship recipients" should be taken to mean "children in households that received one or more scholarships.""II Table 2 shows the incidence of scholarship receipt by school level. It shows that 9.5 1% of all children who were in school at the start of the 1998/99 school year were in households that received some scholarship funds. Disaggregating by school level, 8.44% of primary school students, 13.56% of lower secondary students and 9.57% of upper secondary school students are in a household that received JPS scholarship funds. It is tempting to compare these figures with the official targets of 6%, 17% and 10% for " If we knew that only one scholarship was allocated per household then we could weight observations inversely to the number of children in a household. There is no reason to suppose however that only one scholarship is awarded per household. The characteristics of the household that caused one child to be a recipient may well result in more than one child participating in the project. Hardjono (1999) finds multiple scholarships recipients in over a third of households that received scholarship funds. 15 primary school, lower secondary and upper secondary respectively but our definition of scholarship recipient will of course overestimate the actual percentage of children receiving a scholarship. 12 Nevertheless, a sizeable proportion of children are in households that have benefited from the program.'3 Table 2 also breaks the scholarship awards down further by gender and by EBKKBN rankings. It shows that consistent with the written criteria, girls were slightly favored over boys and this is the most marked at the upper secondary level. The figures by BKKBN ranking also show that the selection criteria were being put into practice, although not strictly. All scholarships were meant to be given to households in the two lowest BKKBN rankings (Pre-Prosperous and Prosperous I) and any additional scholarships were then to be allocated to households in the higher categories. This stipulation has not been followed to the letter because although coverage of the lowest two categories is less than 100% in many villages, some scholarships were awarded to students in households in the upper rankings. For some reason a large majority of the households (82%) report that they have never been classified by the BKKBN. One explanation is that households are required to have an identity card before they can be assessed. Obtaining such a card often involves paying the appropriate "unofficial fees" to the relevant government officials and so is difficult for the poor to obtain. That the 100 villages are relatively poor might thus explain why a large proportion aren't classified, although it seems unlikely that only 18% 12 A comparison of these figures with the official targets is also problematic because of the unrepresentativeness of the 100 villages data. 13 Jones et al. (2000) used the nationally representative 1999 Susenas data and found that the program reached less than its targeted number of children. For instance, only 8.4% of lower secondary students received a JPS scholarship. He suggests that this could at least in part be due to underreporting by households. There was also a delay in the disbursement of scholarships in the first year so that some students may only have received their scholarship for the 1998/99 school year after the Susena- was conducted in February 1999. 16 would be able to afford such a card. Figure Al in the appendix presents kernel density estimates of per capita expenditure for each of the BKKBN categories and for those without a classification. Those without a classification seem to lie somewhere between the two lowest classifications in terms of per capita income. 14 The low proportion of households reporting a BKKBN ranking is another reason to question the appropriateness of BKKBN data for targeting purposes. The number of scholarships and the targeting ratio drops as the BKKBN classifications rise. This is true overall and for each school level with the one exception being Prosperous II households with children at upper secondary school. The high targeting ratio for this group is however coming off only 3 households receiving JPS funds in this small category. Only 2 scholarships were awarded in the entire sample to children from Prosperous III or above households. In every case, except primary schools, those without a BKKBN classification receive somewhere between the number of scholarships allocated to Pre-Prosperous and Prosperous I households. Rather than present cross-tabulations of scholarship receipt by household size and the other specified criteria, we can examine their impact on scholarship receipt via estimation of a participation equation analogous to equation 1. That is, we estimate a probit of scholarship receipt controlling for these and additional variables. We will use these results below to test for selection bias in the estimates of program impact and as a means of constructing a matched comparison group but we discuss them here in the light of what they contribute to our understanding of the targeting performance of the project. Table 3 presents the results. 14 Median per capita expenditure for the households with no BKKBN ranking is RP 69,887 which lies between the medians for Pre-Prosperous and Prosperous I households (Rp 56534 and Rp 74290 respectively). 17 Probit Results Table 3 presents two sets of results - those with and without village level effects. Presenting both sets of results allows us to examine first, how well scholarships were allocated across the entire population (the specification without village dummies) and secondly, how well school committees allocated the scholarships within their geographic region (the specification with village dummies). 15 The probits control for all available variables that could be expected to influence the allocation of scholarships. That is we control for variables that feature in the specified list of criteria - the child's gender, the number of school aged children in the household, whether the household head is female and BKKBN status.'6 We first interacted every variable with school level variables and estimated separate coefficients for every school level. We then tested the coefficients to see if they were constant across the school levels. For those that were we then went back and estimated only one coefficient As mentioned above, a large number of households claim never to have been classified by the BKKBN. These households form the omitted category for the BKKBN indicator variables. However, we can do better than this because the December 1998 data also asked the questions on which the BKKBN rankings are supposedly based. Dummy variables reflecting the answers given to these questions are also used as explanators.'7 15 Note that when one includes village level dummy variables it is necessary to drop villages in which there is no variation in scholarship receipt from the sample. 16 We are not able to control for whether the child is handicapped nor the distance to school, although the latter will be captured to some extent by the village level dummies. There is at least one primary school in each village. There may however be only one lower and/or upper secondary school per sub-district (kecamatan). 17 The inclusion of the information in the questions underlying the BKKBN categories in both the scholarship and dropout equations in effect makes the BKKBN categories a valid instrument in the exogeneity tests below. This is so because the BKKBN categories themselves, once one controls for their 18 In addition we include variables that reflect other information that the committees may have used in the allocation process and/or proxies for these variables. These variables reflect the household's socio-economic status and the child's characteristics. Specifically we control for August 1998 per capita household expenditure, the education of the household head, whether the household head is unemployed, whether the household is a farn household and whether it produces most of its own food and whether August 1998 income measured in rupiah is lower than rupiah income 12 months earlier.18 These latter three variables might capture the impact of the crisis. Farmers for instance may have benefited from the increase in food prices. We also control for the child's level of schooling, whether the school attended at the start of the school year was public or private, the child's age and whether the household lives in a rural area. The results are consistent with the tabulations already presented. Even after one has controlled for all the other factors, households ranked Prosperous III and above are on average 6.9% less likely to receive a scholarship than households without a BKKBN ranking.'9 Girls were approximately 6 percentage points more likely to receive a scholarship at the upper secondary level (significant at the 10% level). The variable that reflects the number of school aged children in a household shows that each additional child in this age group increases the probability of the family receiving a scholarship by approximately 2 percentage points. This could however be picking up that households with more children have a higher likelihood of receiving a information content, will affect the selection into the program because they are written into the selection rules but should not affect the probability of dropout. 18 A household is defined as a farm household if more than a third of household income is derived from agriculture. Households are defined as producing their own food if they indicate that they don't buy food from the market. This category will include a small number of households who rely on gifts of food. '9 The coefficients on the indicators that reflect the answers to the questions underlying the BKKBN classifications are difficult to interpret because they are the effect of these variables once the BKKBN rankings have been controlled for. 19 scholarship rather than an increase in the probability of an individual child receiving a scholarship. Single parent households were also meant to get priority. The 100 villages survey does not allow us to identify every child's parents so instead we constructecl a "female headed household" variable to proxy for this variable. Female headed households were 6.1 percentage points more likely to receive the JPS scholarship funds. Hence, every variable that was meant to have been taken into account in the allocation of scholarships is having a positive and significant impact on the probability of scholarship receipt. There is also evidence that other information was taken into account. School records may have contained some information on household income and or expencliture or this information may have been available in other forms of "local knowledge". The results show that the probability of a household receiving a scholarship decreases as expenditure increases. Scholarship allocation is most sensitive to expenditure per capita at the upper secondary level and least sensitive in primary schools. An extra RplOO,000 in monthly per capita expenditure (mean monthly per capita expenditure is Rp8 1,427) decreases the probability of receiving a scholarship by about 4 percentage points at primary school level, 6 percentage points at lower secondary and 10 percentage points at upper secondary school. The education of the household head also reflects the socio-economic status of the household and is found to be negatively related to scholarship receipt, although it becomes insignificant once village level effects are introduced. We also control for whether the household's rupiah income had decreased in the 12 months prior to the survey. Those that responded that it had were 2.6 percentage points more likely to receive a scholarship, controlling for their current expenditure level. 20 The variable that indicates whether the household produces most of its own food is also significant. Those who produced their own food were 4.5 percentage points less likely to receive a scholarship than those who don't. We also included a variable that indicates whether anyone in the household lost their job in the preceding 12 months. This variable was insignificant. The significance of some of the variables that reflect crisis impact is interesting because crisis impact was specifically added to the allocation criteria for the 1999/2000 school year. The results from the equation without fixed effects suggest that crisis impact may have already implicitly been taken into account during the 1998/99 school year. Note however that the inclusion of village dummy variables makes these variables insignificant. This suggests that the geographic targeting of funds may have resulted in worse-affected regions being targeted but that within the village crisis impact was not a specific selection criteria.20 The school level and age variables show that a child is least likely to receive a scholarship in the lower years at primary school (these children are officially not eligible but were left in the sample due to reports of some schools nevertheless allocating scholarships at these levels). Having controlled for all of the other variables, upper secondary students are more likely to receive a scholarship than students at any other level. In addition to the above variables we also included two variables that attempt to capture the "political economy" of scholarship awards. The first is a variable that indicates the number of social activities and organizations in which the household was 20 Or alternatively that there is not much variation in crisis impact within villages. 21 involved in August 1998.21 If committees used their local knowledge in addition to the specified criteria to award scholarships then it may be that scholarships were allocated disproportionately to those households that were better known to the committee. (A less generous interpretation would be that committees preferred to direct funds to those they knew, for reasons not related to the probability of child drop-out). This variable is positive and significant at the 10% level in the regression with village effects but very small in magnitude. It becomes negative (also small and significant) when village level effects are introduced. Ideally we would like to also construct a variable that equals 1 if any households worked in the public sector. Although householders were asked if they worked in the public sector in the May 1997 round of the survey, this question was dropped in the later rounds. As a proxy we construct a variable that equals 1 if anyone in the household is an employee in the services sector. This would hence capture public servants and teachers who both might have some control over the allocation of funds. This variable was statistically insignificant. To further examine the relationship between per capita expenditure and scholarship allocation, Figure 1 plots the percentage of total scholarships received by each of the 100 Villages expenditure quintiles and so shows us how scholarship receipt varies with expenditure category, without controlling for the other variables. (Table 2 contains the actual rates on which the figure is based). It shows that although the probit results indicate that the probability of receiving a scholarship decreases as household per capita expenditure rises, only 25% of households that reported receiving funding were in the bottom quintile of the August 1998 per capita expenditure distribution. These figures 21 These include mother's groups, sports clubs, young people's organizations, funerals, religious groups, 22 vary by school level. Secondary schools targeted poorer households more accurately. 31% and 37% of scholarships were awarded to bottom quintile households in lower and upper secondary schools respectively compared to 22% at primary school. Part of what Figure 1 shows to be a relatively poor targeting performance may be due to measurement error in the per capita expenditure data. It also may reflect committees having difficulty differentiating the very poor from the poor. 22 This may be exacerbated in the 100 villages sample because the sample is poorer than the population at large. Figure 2 replots the distribution of scholarships using quintiles from the 1996 Susenas data. This gives a considerably more positive impression of the program's targeting. Now over 60% of scholarships went to households in the lowest quintile and 84% to the two lowest quintiles. 23 Notwithstanding this result, Figure 1 shows that to the extent we can trust the expenditure data, committees were not so successful in isolating the most needy students. These findings are consistent with Jones et al. (2000) who analysed the Susenas data on JPS scholarship receipt.24 In summary, the data support the view that the program allocation criteria were quite closely adhered to and we find scant evidence of undue influence of parties close to and savings associations. 22 Figure AI in the appendix shows for instance that there is substantial overlap in expenditure per capita across the BKKBN categories and hence any program that relied heavily on these categories is going to produce a distribution of funds that shows substantial leakage to groups with relatively high per capita expenditure. 23 Now also primary schools outperform secondary schools. This is mostly because there are a lot less poor students at secondary schools than at primary schools. Unlike in Figure 1 where the quintile cut-off points were calculated separately for each school level, due to limited access to the Susenas data, the quintiles are calculated for the whole population. The August 1998 per capita expenditure figures were deflated back to 1996 for this exercise. The deflation was conducted crudely. First August 1998 figures were deflated back to May 97. Because food price inflation was so much higher than other inflation over this period, the food share implicit in the price deflator has a large impact on the appropriate inflation rate. We used a price deflator that allows for a food-share of 68% of expenditure (that is equivalent to that of the lowest 30% of households in the Susenas). Prices were much more stable prior to this period. We used the official CPI to deflate the May 1997 figures back to February 1996 which is when the 1996 Susenas was collected. 24 Suryahadi et al. (1999) found great variation in the program's coverage of the poor across districts (kabupaten). 23 the allocation process on the distribution of scholarship. Nevertheless, a fair number of scholarships appear to have been received by households that appear in the upper quintiles of the per capita expenditure distribution. This finding may be due to measurement error in reported expenditure in the 100 villages survey. It may also reflect difficulty in differentiating between poor and poorer households due to the lack of appropriate household data. It is not clear how targeting can be improved given the paucity of data at the household level. Some of these households would however be ranked in the poorest quintile of the national distribution. Accurate geographic targeting of funds can hence mitigate some of the household level targeting problems. The extent to which the program was geographically targeted is something that could be assessed using nationally representative data such as the Susenas. VI. Impact Evaluation Results A. Regression-Based Estimates Table 4 presents the results corresponding to estimation of equation 6. The 100 Villages data do not provide us with the actual date of dropout so we use a dependent variable that is discrete and indicates whether the child dropped out of school in the current school year rather than the continuous variable shown in equation 6. The dependent variable, Di, equals 1 if the child dropped out during the school year and 0 otherwise. Again results are presented with and without village fixed effects. The rationale for including village level effects here is that they will further reduce any problematic unobservables that may affect the probability of scholarship receipt. In 24 addition they will control for any positive effect the scholarship program may have had on school finances and for the geographic placement of programs.25 Village level dummies also ensure that we are comparing like with like in terms of comparing individuals in close geographic proximity.26 Using village level dummies does however significantly reduce our sample size because attention is restricted to villages in which there is dropout variation. That is, at least one child dropped out. The probits are run separately for each school level. Test of Exogeneity We conduct two tests of whether the correlation between unobservables that determined selection into the program also determine the probability of dropout. The first test involves taking the residuals (which contain the unobservables) from equation (1') and testing whether they are significant in the dropout equation. That is we estimate: 25 It has been hypothesized that the scholarship programs may have contributed to reduced dropouts via its effect on school finances rather than or in addition to its impact on individual recipients. Unlike during the 1980's recession, children were not forced to leave school if their parents were unable to pay school fees. However, it has been reported that when awarding the scholarships, if the child's school fees were in arrears, the parents were often notified that the amount of the scholarship had been deducted from the school fees owed and so the scholarship funds in effect went straight to the school. Hence, they may have played a significant role (in conjunction with the parallel program of school grants) in maintaining school quality. If this is the case, then children in a village that received a lot of scholarships might be less likely to dropout regardless of whether they received a scholarship themselves or not. We did initially attempt to assess the funding effect by using the percentage of eligible children in each village who received the scholarship as an explanatory variable in the dropout equation. This variable was however clearly endogenous and we were unable to find suitable instruments for it. We hence do not attempt to test this effect here. 26 A series of recent papers have examined the magnitude of biases arising from the use of comparison groups for estimating program effects given the lack of randomized experiments. Heckman, Ichimura and Todd (1998) find that one of the largest sources of bias is due to using a comparison group from a different geographic region. Other sources arise from the comparison group having been administered a different survey instrument, using data from outside the region of common support when calculating estimates and differing distributions of the probability of being a program participant within the area of common support . Selection bias was a further source of bias but was found to be the smallest of these possible bias. Given that our information on participants and non-participants come from the same survey, the first two sources of bias are automatically eliminated. Estimates are obtained over the region of common support and the matching methods used below generate distributions of the predicted probabilities of receiving a scholarship that is the same for both participants and non-participants. 25 Di= YJPS, + (/1oXjO - ,/Xj,) + irRj, + vi, (8) where Ri, are the residuals from the first stage and we test the hypothesis that ir = 0. This is the Sargan-Wu-Hausman test suggested by Jalan and Ravallion (2000) in this context. A difference here is that the second stage equation (the dropout equation) has a limited dependent variable.27 If we use a probit in the first stage, the second stage estimates of equation (8) will be inconsistent unless the first stage is exactly correctly specified (Angrist, 2000). This is however not the case if both stages are estimated using linear probability models which is the method we proceed with here.28 The inclusion of the information in the questions underlying the BKKBN categories in both the scholarship and dropout equations makes the BKKBN categories a valid instrument for this test. This is so because the BKKBN categories themselves will affect the selection into the program even once their information content (the answers to the underlying questions) are controlled for because they are written into the selection rules. In contrast, the actual categories should not affect the probability of dropout given one has controlled for the information underlying the classification. 29 The questions underlying the BKKBN variables were included in both equations estimated for the tests. They are not included in the equations presented in Table 3 because they were insignificant. 27 Jalan and Ravallion (2000) evaluate a workfare program. They have a limited dependent variable in the first stage but income is the dependent variable in the second stage. 28 Angrist (2000) examines a model with a limited dependent variable and a potentially endogenous limited explanatory variable. He suggests three estimation techniques in the case that one finds the qualitative explanatory variable to be endogenous, one of which is to use the linear probability model and proceed using standard two stage least squares. He argues that it is safer to estimate both stages by linear probability than the first stage by probit or logit because the second stage estimates will only be consistent if the first stage is the exactly correct model. Seeing as the test used here arises from a two-stage estimation procedure, for the reasons Angrist states we estimate both stages of the test using the linear probability model. 29 The inclusion of the community groups variable and the public servant variable also contribute to the identification of the test but these are insignificant once village dummies are introduced. 26 To allay fears that the test results may be unreliable due to the use of this model we also jointly estimated the participation and dropout equations jointly using bivariate probits. We then conducted a Wald test of whether the estimated correlation between the residuals in the second equation was significantly different from zero. The results were consistent with those from the linear probability model. For each school level, with and without fixed effects, we were not able to reject the null of exogeneity.30 That is that the correlation between the unobservables in the two equations is insignificantly different from zero. Hence we proceed on the basis of selection on observables.3' Scholarship Impact The coefficient on the scholarships variable is our estimate of scholarship impact. In the equations without village dummies the coefficients are negative but insignificant at. all school levels. However, once we include village level dummies the scholarships variable becomes strongly significant at the lower secondary level. A lower secondary school student in a household that received scholarship funding had a 3.5 percentage point lower probability of dropping out than a similar student in a household that received 30 The bivariate probits were unable to estimate the large number of coefficients associated with the inclusion of the village fixed effects. The bivariate probit test was hence conducted only without fixed effects. The p-values for the model without fixed effects were 0.23, 0.18 and 0.69 for the linear probability model tests (primary, lower and upper secondary respectively) and 0.22, 0.81 and 0.21 for the bivariate probit test. For the fixed effects model the p-values were 0.24, 0.25 and 0.46. 3 In the early stages of the scholarship program there were delays in the disbursement of funds. Hence, it may be that a child actually had to be in school some date after the start of the school year to receive the scholarship and this might lead to correlation in the unobservables. That is a child may have had to be in school until September for example to have received the funds and being in school in September would increase the probability of being in school in December. Hence this endogeneity would positively bias the impact estimate. By testing for the endogeneity of JPSj we are also testing for the existence of this type of bias. 27 no funding. The scholarships variable remains insignificant at the other schooling levels.32 Other Variables Apart from the age variables which are significant in each equation, little else is consistently significant across the dropout equations. For primary and secondary school students, having more siblings has a positive but very small impact (less than 1 percentage point) on a child's probability of dropout. The variable that indicates attendance at a private school is negative in each equation although only significant at the 5% level and quantitatively important for lower secondary school students who are 3 percentage points less likely to dropout. Girls are 3 percentage points more likely to drop out at upper secondary school than boys. There is no gender difference at the lower school levels. Controlling for changes in income, primary school children in households with lower expenditure per capita in August 1998 have a higher chance of dropping out of school by December. The magnitude of this effect is however very small. The variables reflecting changes in income and expenditure are significant in some of the equations but inconsistently signed and also very small in magnitude. B. Matching Methods Given that we have concluded that scholarships are not endogenous in the: differenced equation, we can also use matching methods to construct estimates of the 32 As mentioned above, including village level effects reduces our sample size significantly. By focusing on villages in which there is at least one dropout, we are focusing on less well-off villages. It hence may be that we are picking up this effect not so much because we controlling for village effects but because these are poorer villages where children are more likely to drop out of school and where scholarships might be expected to be more successful. The matching methods used below however obtain estimates using the entire sample (in the region of common support) of participants. 28 impact of the scholarship program on school enrolments. The idea behind matching methods is that one can examine program impact by comparing the outcome of a participant in the program with the outcome of a similar individual who is not a participant. Regression based methods do this by controlling for characteristics of the participants and non-participants in the regressions. Matching methods actually pair participants with one or more non-participants with similar attributes. Here we match participants with non-participants using the predicted probability of scholarship receipt (propensity score) estimated from the probit above.33 We then calculate the average difference between the dropout of participants and matched non-participants. The impact estimator can be written as: P NP G = (Djp -ZW,D,NP)/ P (8) j=I 1=1 where DP equals 1 if the jth participant dropped out of school, 0 otherwise; D N7 is defined analogously for non-participants, Wij is a weight applied to non-participant i when paired with participantj, P is the number of participants and NP is the number of non-participants. There are a variety of ways in which participants can be matched. Here we use the "five nearest neighbors" method and a kernel-based method. The five nearest neighbors method involves matching each participant with the five non-participants who have propensity scores closest to that of the participant. That is, the five "closest" non- participants are given a weight of 1/5 in equation 8 and all other participants are given a weight of zero. The average dropout rate of these 5 non-participants is then deducted 3 Rosenbaum and Rubin (1983) show that matching on propensity scores is sufficient to eliminate bias in the case of selection on observables. 29 from the dropout rate of the participant and these "differences" are then averaged across all participants.34 The kernel based method calculates a weighted average dropout rate across all non-participants in the sample with the weights being a declining function of the difference between the participant's propensity score and the non-participants' propensity scores. We use a biweight kernel and use Silverman's optimal bandwidth.35 Details are given in the appendix. In both methods matches were constrained to occur within school level and within the same geographic district.36 Figure A2 in the appendix presents kernel density estimates of the propensity scores before and after matching. Table 5 presents the matching method results. The results are consistent with those obtained from the regression based method. The scholarships had no significant effect on dropout at both primary and upper secondary levels, at lower secondary level however the estimates are negative and significant. The 5 nearest neighbors method indicates that being in a household that received a scholarship reduces the probability of dropping out of school by 3.8 percentage points (p=0.022). The kernel based method indicates a marginal effect of scholarship receipt of 3.3 percentage points (p=0.05:3). These estimates are very similar in magnitude to the regression-based estimate. Interpreting the Magnitude of Scholar'ship Impact 34 We sample from the non-participants with replacement. 35 Heckman et al. (1997) found matching estimates are susceptible to bias arising from use of observations in the tails of the non-participant propensity score distribution. To deal with this we trimmed 2% from the top and bottom of the non-participant distribution of propensity scores. We also eliminated observations for which the non-participant kernel density estimate was zero so that the estimates were conducted over a region of common support. 36 Matches were made within sub-districts (kecamatan) rather than villages to ensure each participant could find a non-participant with a suitably close propensity score. Kecamatan is the next largest geographic region after village. 30 We can use the fixed effects probit results to examine the impact of scholarship receipt on dropout rates. The mean predicted dropout rate using the actual scholarship receipt as a predictor is 7.30 percent in the fixed effects sample. This is very close to the actual dropout rate in these villages of 7.52%.37 If we set JPSi equal to zero for all households in this sample then the predicted dropout rate becomes 9.65 percent. Hence, we estimate that at least in poorer villages such as those sampled in the 100 Villages data, the scholarship program reduced dropout by about 2.35 percentage points at the lower secondary school level. This is quite a large decrease, especially as we are talking about mid-school year dropouts which are normally much lower than end of year dropouts. This corresponds to a 24 percent decrease in dropouts. VII. Conclusions The analysis above supports the contention that the committees who allocated the scholarships were diligent in following the criteria laid out in the program implementation plan. Those groups that were slated to be targeted did by and large receive a greater than proportional share of the scholarships. Nevertheless, if the expenditure data in the 100 villages data can be trusted, then a significant percentage of scholarships were nevertheless allocated to students from the upper quintiles of the distribution. It is not clear how targeting can be improved in programs of this type in Indonesia and elsewhere, given the scarcity of accurate household data at the village level in most countries. Placing greater weight on local knowledge may help but it also makes monitoring more difficult and weakens accountability mechanisms. Furthermore, there is 37 Note that these are only the villages in the fixed effects sample. 31 already scope for local knowledge to be used in the selection of participants into this particular program. Accurate geographic targeting can to some extent offset targeting errors made at the local level. The impact evaluation finds that the scholarships significantly reduced the probability of dropout at the lower secondary level but did not affect dropout rates at primary and upper secondary schools, at least in the first few months of the program's operation. This result is consistent with lower secondary school students being most susceptible to dropout and so being in the position to benefit the most from a program of this kind. Prior to the crisis the majority of school dropouts occurred at the lower secondary level. The program impact is quite large. It was estimated to have reduced lower secondary school dropouts by 2.35 percentage points, or 24 percent. Nevertheless, it seems likely that other factors - like the decreased opportunity cost of children's work for example - have also played a role in keeping children at school at all three school levels. Other than scholarship receipt, the age of the child was a significant determinant of dropout. Within upper and lower secondary school, older children are a lot more likely to dropout. Having more siblings also increases the probability of dropout slightly. Private school students are less likely to dropout at lower secondary school. At some school levels children of farmers (who were less likely to have been adversely affected by the crisis) were slightly less likely to dropout. Hence, this is preliminary evidence in favor of focusing funding on lower secondary schools, continuing to target children from large families, possibly focusing funding on later year students, scaling back scholarships to private schools at the lower secondary level, and for targeting households that have been 32 most adversely affected by the crisis - as has already been written into the 1999/2000 implementation plan. This paper gives a timely evaluation of the JPS scholarships program. It is nevertheless limited by the data that was available at the time of writing. Further research on later data will prove valuable. It is possible that the scholarships may reduce dropout to a greater extent in the future if, say, the crisis does not abate and households' ability to draw down assets and borrow are severely constrained. Also, it may be that all students benefit from the scholarships through its impact on school funding. The scholarships enable parents to pay fees that then flow into the schools' coffers. This hypothesis was not tested here. If recipient students do not benefit more than other students though, as the evidence presented here suggests is the case at primary and upper secondary level, there is an argument for extending the program of block grants to schools rather than wasting resources on the costly allocation of individual scholarships - unless other benefits of allocating funds to individual recipients, such as the possible monitoring and accountability advantages, have been persuasively proven. 33 References Alatas, V. (1999), "Evaluating the Left Behind Villages Program in Indonesia: Exploiting Program Rules to Identify Effects on Expenditures and Employment", unpublished paper, Nov, ppl-17. Angrist, J. (2000), "Estimation of Limited-Dependent Variable Models with Dunmrny Endogenous Regressors: Simple Strategies for Empirical Practice", NBER Technical Working Paper 248. Beegle, K., Frankenberg, E. and D. Thomas (1999), "Measuring Change in Indonesia", RAND Working Paper 99-07, August, pp1-84. Bollen, K., Guilkey, D. and T. Mroz (1995), "Binary Outcomes and Endogenous Explanatory Variables: Tests and Solutions with an Application to the Demand for Contraceptive Use in Tunisia", Demography, 32(1), February, ppl 11 -131. Cameron, L. (2000), "The Impact of the Indonesian Financial Crisis on Children: An Analysis Using the 100 Villages Data", unpublished paper, University of Melbourne. Hardjono, J. (1999), "The Micro Data Picture: Results of a SMERU Social Impact Survey in the Purwakarta-Cirebon Corridor", SMERU Working Paper, July 9th, ppI- 42. Heckman, J., Ichimura, H. and P. Todd (1998), "Matching as an Econometric Evaluation Estimator", Review of Economic Studies, 65, 261-294. Heckman, J., Hidehiko, I. And P. Todd (1997), "Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme", Review of Economic Studies, 64, 605-654. Heckman, J., Ichimura, H., Smith, J. and P. Todd (1998), "Characterizing Selection Bias using Experimental Data" Econometrica, 66(5), Sept, 10 17-1098. Heckman, J. (1997), "Instrumental Variables: A Study of Implicit Behavioral Assumption Used in Making Program Evaluations", Journal of Human Resources", XXXII(3), pp441-462. Jalan, J. and M. Ravallion (1999), "Income Gains to the Poor from Workfare: Estimates for Argentina's Trabajar Program", World Bank Policy Research Working Paper No 2149, July, ppl-32. Jones, G., Hagul, P. and Damayanti (2000), "A Study of the Impact of the Scholarships and Grants Program", Central Independent Monitoring Unit of the Scholarships and Grants Program, Jakarta. Ministry of Education and Culture, Ministry of Religious Affairs, Ministry of Home Affairs (1999), Program Implementation Plan 1999-2000, Scholarships and Grants Program for Primary and Secondary Schools, June, Jakarta. Pitt, M., Rosenzweig, M. and D. Gibbons (1993), "The Determinants and Consequences of the Placement of Government Programs in Indonesia". Pradhan, M. and R. Sparrow (2000), "Basic Education Outcomes During Crisis: An Analysis using the 1995, 1997, 1998 and 1999 SUSENAS, unpublished paper, Feb th 15 Ravallion, M. (1999), "The Mystery of the Vanishing Benefits: Ms. Speedy Analyst's Introduction to Evaluation", World Bank Policy Research Working Paper No 2153, July, pp1-40. 34 Ravallion, M. and Q. Wodon (1999), "Does Child Labor Displace Schooling?: Evidence on Behavioral Responses to an Enrollment Subsidy", World Bank Policy Research Working Paper No 2116, May, ppl -21. Rivers, D. and Q. Vuong (1988), "Limited Information Estimators and Exogeneity Tests for Simultaneous Probit Models" Journal of Econometrics, 39, 347-366. Rosenbaum, P. and D. Rubin (1983), "The Central Role of the Propensity Score in Observational Studies for Causal Effects, Biometrika, 70:41-55. Skoufias, E., Suryahadi, A. and S. Sumarto (1999), "The Indonesian Crisis and Its Impact on Household Welfare, Poverty Transitions, and Inequality: Evidence from Matched Households in 100 Village Survey", SMERU Working Paper, September, pp1-47. Smith, R. and R. Blundell (1986), "An Exogeneity Test for a Simultaneous Equation Tobit Model with an Application to Labor Supply", Econometrica, 54(3), May, pp679-685. Sumarto, S., Suryahadi, A. and L. Pritchett (2000), "Safety Nets and Safety Ropes: Comparing the Dynamic Benefit Incidence of Two Indonesian "JPS" Programs", SMERU Working Paper, February, pp1-38. Suryahadi, A., Suharso, Y. and S. Sumarto (1999), "Coverage and Targeting in the Indonesian Social Safety Net Programs: Evidence from 100 Village Survey", SMERU Working Paper, August, ppl-39. 35 Appendix Allocation of Funds to Schools (i) Lower and Upper Secondary Schools For secondary schools this involved allocating funds to districts (kabupaten) on the basis of a poverty index calculated from random household samples which are representative at the district level.38 District committees then allocated the funds to schools. Wealthy schools were excluded and then schools were ranked according to (i) the percentage of the poor in the community served by the school (as indicated by the BKKBN rankings), (ii) the total school income per student head at the school and (iii) the percentage of "left behind villages" in the sub-district (kecamatan) in which lower and upper secondary schools are located.39 (ii) Primary Schools Unlike secondary schools which may be distributed with only one in each sub- district, every village in Indonesia has a primary school. The districts allocated the primary school scholarship funds to sub-districts using the same criteria as were used above for districts. Individual schools were then allocated scholarships on the basis of the percentage of poor households in its community, school income and the individual villages' "left-behind" status, and for these villages, their location in an urban, non-urban or remote area. 38 For the 1999/2000 school year this was changed so that districts were also targeted according to the severity of the crisis impact. 39 Villages in Indonesia are classified as left-behind on the basis of their infrastructure and village access to services. These so called IDT (Inpres Desa Tertinggal) villages have been the recipients of various 36 Matching Methods The biweight kernel weights observations using the following formula: k(z)=15/16(l-z2)2, -1l. where z = [P(X.jo) - P(X io)] / h. P(Xjo) is the propensity score of participantj and P(Xio) is the propensity score of non-participant i, and h is the bandwidth. Silverman's optimal bandwidth is used here and is: h = 2.42 * Min(or,0.75 * IQR) * N"5 where a is the standard deviation of the differences in the propensity scores, IQR is the interquartile range (difference between the 75th and 25'h percentile) and N is the sample size. government programs over the years. See Alatas (1999) for details of the classification system. The scholarships program, used the 1995 village classifications. 37 Table 1: Susenas Enrolment Ratios Susenas 1993 1994 1995 1996 1997 1998 1999 School Year School Level 1992/93 1993/94 1994/95 1995/96 1996/97 1997/98 1998/99 Primary School 92.8 94.1 93.9 94.4 95.4 95.1 95.2 Lower Secondary 68.9 72.4 73.2 75.8 77.5 77.2 79.1 Upper Secondary 42.6 45.3 44.6 47.6 48.6 49.3 51.2 Source: Jones et al. (2000) 38 Table 2: JPS Scholarships Targeting Performance All Gender BKKBN Rankings 100 Villages Per Capita Expend. Quintile Male Female None Pre-Pr. Pr. I Pr. II >Pr.III Ql Q2 Q3 Q4 Q5 %receiving 9.51 9.06 9.98 9.89 11.6 7.11 6.52 1.22 0.118 0.149 0.096 0.061 0.051 % of scholarships 0.49 0.51 0.85 0.08 0.04 0.03 0.00 0.25 0.31 0.2 0.13 0.11 Targeting Ratio 0.95 1.05 1.04 1.22 0.75 0.69 0.13 1.25 1.55 1 0.65 0.55 Primary % receiving 8.44 8.08 8.81 9.06 8.22 5.9 4.29 0 0.091 0.137 0.093 0.056 0.045 % of scholarships 0.49 0.51 0.87 0.06 0.04 0.02 0.00 0.22 0.32 0.22 0.13 0.11 Targeting Ratio 0.96 1.04 1.07 0.97 0.70 0.51 0.00 1.1 1.6 1.1 0.65 0.55 Lower Secondary % receiving 13.56 13.14 14.03 13.4 21.84 12.5 11.48 5.41 0.208 0.185 0.154 0.07 0.061 % of scholarships 0.50 0.50 0.81 0.09 0.05 0.03 0.01 0.31 0.27 0.23 0.1 0.09 Targeting Ratio 0.97 1.03 0.99 1.61 0.92 0.85 0.40 1.55 1.35 1.15 0.5 0.45 Upper Secondary % receiving 9.57 7.75 11.45 8.93 27.27 4.35 14.29 0 0.178 0.14 0.066 0.056 0.038 % of scholarships 0.41 0.59 0.63 0.10 0.02 0.06 0.00 0.37 0.29 0.14 0.12 0.08 Targeting Ratio 0.81 1.20 0.76 2.31 0.37 1.21 0.00 1.85 1.45 0.7 0.6 0.4 * The Targeting Ratio (TR) = % of total scholarships awarded to that category / % of population in that category. If TRj>1 (TRj<1) group j receives a greater (lesser) than proportional share. Table 3: Marginal Effects from Probit of Scholarship Receipt No Village Fixed Effects Village Fixed Ettects dF/dx Std. Err. dF/dx Std. Err Per Capita Expenditure ('00000 Rp) x primary -0.033 0.014 - 7 Per Capita Expenditure ('00000 Rp) x lower sec. -0.067 0.021 * -0.065 0.025 Per Capita Expenditure ('00000 Rp) x upper sec. -0.12 0.036 * -0.103 0.038 Rp. Income Less than 12 mths ago 0.026 0.009 * 0.014 0.011 Primary Educated Head -0.022 0.014 # -0.018 0.016 Lower Secondary Educated Head -0.046 0.013 * -0.032 0.017 Upper Secondary Educated Head -0.037 0.015 * -0.023 0.020 Membership of Community Groups 0.004 0.002 # -0.007 0.003 Householder(s) is an employee in services sector 0.003 0.014 -0.01( 0.015 Farm Household -0.014 0.009 -0.010 0.012 Female Headed Household 0.061 0.025 * 0.113 0.035 Household Produces Most of Their Own Food -0.045 0.015 * -0.021 0.028 Unemployed Household Head 0.019 0.037 -0.024 0.025 Rural Residence x primary 0.025 0.014 # -0.107 0.076 Rural Residence x lower secondary 0.064 0.027 * -0.054 0.040 Rural Residence x upper secondary -0.023 0.02 -0.076 0.014 BKKBN Status: Pre-Prosperous 0.004 0.019 0.003 0.024 BKKBN Status: Prosperous I -0.018 0.018 -0.018 0.023 BKKBN Status: Prosperous II -0.004 0.022 0.037 0.040 BKKBN Status: Prosperous III or IV -0.069 0.01 * -0.065 0.014 Eat at least twice a day. 0.042 0.016 * 0.021 0.028 Own a change of clothes. -0.024 0.024 -0.028 0.027 House's floor is of dirt. -0.054 0.011 * 0.015 0.045 Observe Religious Duties. 0.005 0.012 0.011 0.015 Buy medicine when needed. -0.023 0.016 0.015 0.014 Age of Child: 7 yrs 0.011 0.019 0.005 0.021 8 yrs -0.01 0.015 -0.015 0.018 9 yrs 0.032 0.019 0.024 0.021 10 yrs 0.013 0.018 0.009 0.020 11 yrs 0.046 0.022 * 0.037 0.024 12 yrs 0.052 0.022 * 0.061 0.026 13 yrs 0.042 0.023 * 0.049 0.027 14 yrs 0.028 0.024 0.044 0.030 15 yrs 0.009 0.025 0.013 0.029 16 yrs 0.029 0.031 0.034 0.036 17 yrs 0.023 0.035 0.034 0.042 18 yrs 0.017 0.037 0.013 0.038 1998/99 School Level: Lower Secondary 0.065 0.038 * 0.037 0.038 1998/99 School Level: Upper secondary 0.183 0.099 * 0.123 0.095 # Female x primary 0.008 0.007 0.008 0.008 Female x lower secondary 0.008 0.014 0.007 0.014 Female x upper secondary 0.05 0.034 # 0.060 0.040 # Private School x primary 0.078 0.024 * 0.011 0.021 Private School x lower secondary -0.039 0.011 * -0.031 0.013 # Private School x upper secondary 0.002 0.026 -0.001 0.029 No. of school aged children in h'hold 0.012 0.004 * 0.019 0.005 Pseudo-K2 0.079 0.221 N 7686 5915 Standard Errors allow tor clustering within househo ds. () denotes signiticance at t ee 5% (10%lvel. Omitted categories are: a head with no education, no BKKBN status. Table 4: Marginal Effects from Probit on Dropout No Village Fixed Effects Village Fixed Effects Primary Lower Sec. Upper Sec. Primary Lower Sec. Upper Sec Household Received a JPS Scholarship -0.00008 -0.008 -0.001 -0.0001 -0.035 * 0.042 (0.00007) (0.005) (0.009) (0.0002) (0.011) (0.082) Change in Exp per cap., Dec'98 -Aug'98 -0.0005 * -0.011 0.007 -0.0019 * -0.031 0.001 ('00000 Rp) (0.0002) (0.009) (0.007) (0.0012) (0.023) (0.002) August 1998 Expenditure per Capita -0.0004 * -0.014 -0.006 -0.0017 * -0.024 ('00000 Rp) (0.0001) (0.0106) (0.0087) (0.0010) (0.0233) Rupiah Income Decreased in Previous 12 mths 0.0002 -0.005 -0.004 0.0004 * -0.021 # 0.000 (0.0001) (0.0048) (0.0052) (0.0004) (0.0109) (0.0005) Primary Educated Head 0.0000 -0.007 -0.008 -0.0001 -0.011 -0.005 (0.0001) (0.0095) (0.0120) (0.0002) (0.0194) (0.0121) Lower Secondary Educated Head -0.0001 -0.006 -0.011 # -0.0003 0.001 -0.001 (0.0001) (0.0068) (0.0051) (0.0002) (0.0220) (0.0026) At Least Upper Secondary Educated Head -0.0001 -0.010 -0.013 -0.0004 # -0.022 -0.001 (0.0001) (0.0056) (0.0092) (0.0003) (0.0112) (0.0042) Age 7 0.4169 * 0.7212 * (0.1588) (0.2117) Age 9 0.5067 * 0.8192 * (0.1433) (0.1487) Age 10 0.4573 * 0.7356 * (0.1311) (0.1696) Age>=l 1 0.3265 * 0.5786 * (0.0691) (0.1281) Age 14 0.008 0.013 (0.0088) (0.0178) Age 15 0.032 * 0.082 * (0.0156) (0.0353) Age >=16 0.095 * 0.289 * (0.0300) (0.0718) Age 17 0.012 0.004 (0.0122) (0.0142) Age 18 0.095 * 0.206 (0.0335) (0.1410) Rural Abode 0.0000 0.008 0.001 -0.0031 -0.018 -0.006 (0.0001) (0.0048) (0.0054) (0.0055) (0.0517) (0.0161) Female 0.0000 -0.003 0.010 0.0001 -0.011 0.033 (0.0001) (0.0047) (0.0070) (0.0002) (0.0100) (0.0424) Private School -0.0001 -0.016 * -0.008 -0.0002 -0.030 * -0.003 (0.0001) (0.0043) (0.0061) (0.0002) (0.0097) (0.0072) Number of School Aged Children 0.0001 0.005 * 0.001 0.0002 * 0.009 * 0.000 (0.0000) (0.0020) (0.0020) (0.0001) (0.0041) (0.0005) Farm Household -0.0002 * 0.002 -0.008 -0.0005 * 0.011 -0.001 (0.0001) (0.0049) (0.0057) (0.0004) (0.0116) (0.0018) Female Headed Household 0.0000 * -0.001 -0.005 -0.0001 0.034 0.000 (0.0001) (0.0106) (0.0063) (0.0002) (0.0446) (0.0004) Produce Their Own Food -0.0001 -0.004 0.0006 -0.010 (0.0001) (0.0075) (0.0014) (0.0148) Pseudo-R2 0.1900 0.194 0.217 |0.2630 0.308 0.554 N 5496 1460 488 1901 585 105 Std errors are shown in parentheses and allow for clustering within households. * (#) denotes significance at the 5% (10%) level. Omitted categories are: a head with no education, no BKKBN status. The variables omitted from the fixed effects regression at the upper secondary level are omitted because they precdict perfectly in this sample. Table 5: Matching Results Primary Lower Secondary Upper Secondary 5 Nearest Neighbors Method 0.001 -0.0377 -0.002 (0.163) (-2.291) (-0.084) Biweight Kernel Method* 0.004 -0.0329 -0.0119 (-0.718) (-1.937) (-0.36) N 451 159 44 t-statistics shown in parentheses. Silverman's optimum bandwidth was used with a 2% trimming rule. Figure 1: Distribution of Scholarships 0.40 0.35 _ 0.30- 0.25 l~_ 0. <0 0.20_ l i I-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~[1 0 0 .15_ ** .. 1 2 3 4 5 Per Capita Expenditure Quintiles (100 Villages Data) Figure 2: Distribution of Scholarships Across Susenas Per Capita Expenditure Quintiles 0.70 . . . ....... ........... _ --______.___-.. 0.6 0.50 ____ uE 0.0 E_ 0 0.30 1 2 3 4 5 Susenas Per Capita Expenditure Quintiles Table Al: Summary Statistics N=7686 Mean Std. Dev. Min M[ax Dropout Rate 0.0151 0.12 0 1 Primary 0.0097 0.10 0 1 Lower Secondary 0.0320 0.17 0 1 Upper Secondary 0.0262 0.29 0 1 Dec '98 - Aug '98 Expenditure per capita (mthly) 2861 41294 -691733 601239 Aug 1998 Expenditure per capita (mthly) 81421 48252 13504 995375 Primary Educated Head 0.67 0.47 0 1 Lower Secondary Educated Head 0.09 0.29 0 1 At least Upper Secondary Educated Head 0.13 0.34 0 1 Age of Child: 7 years 0.10 0.29 0 1 8 years 0.11 0.31 0 1 9 years 0.10 0.30 0 1 10 years 0.11 0.32 0 1 11 years 0.11 0.31 0 1 12 years 0.11 0.31 0 1 13 years 0.10 0.30 0 1 14 years 0.07 0.25 0 1 15 years 0.05 0.22 0 1 16 years 0.04 0.20 0 1 17 years 0.03 0.17 0 1 18 years 0.02 0.14 0 1 Rural 0.79 0.40 0 1 Female 0.49 0.50 0 1 Private School 0.12 0.32 0 1 Number of School Aged Children 2.56 1.27 1 8 Newly Unemployed Household Head 0.03 0.17 0 1 Farm Household 0.48 0.50 0 1 Female Headed Household 0.06 0.23 0 1 Household Produces Most of its Own Food 0.04 0.20 0 1 Rupiah Income declined in previous 12 months 0.41 0.49 0 1 BKKBN Rankings: Pre-prosperous 0.06 0.24 0 1 Prosperousl 0.06 0.23 0 1 Prosperous II 0.04 0.20 0 1 Prosperous III 0.02 0.13 0 1 No. of Community Groups 2.34 1.99 0 7 Householder is an employee in the Services Sector 0.17 0.37 0 1 Unemployed Household Head in August 1998. 0.02 0.12 0 1 Eat at least twice a day. 0.96 0.19 0 1 Own a change of clothes. 0.96 0.21 0 1 Dirt floors. 0.07 0.25 0 1 Observe Religious Duties. 0.80 0.40 0 1 Buy medicines when needed. 0.89 0.31 0 1 * Omitted categories are: household head with no education, no BKKBN ranking. Figure Al: Kernel Density Estimates of Per Capita Expenditure by BKKBN Ranking. * no ranking a pre-prosperous o prosperous I prosperous 11 .000015- 1.0e-05 - Prosperous III & IlI-+ 5.oe-06- 0 100000 200000 300000 Per Capita Expenditure (Rp) Figure A2: Kernel Density Estimates of the Predicted Probability of Scholarship Receipt X0