76745 Evaluating the Impact of Conditional Cash Transfer Programs Laura B. Rawlings • Gloria M. Rubio Several developing economies have recently introduced conditional cash transfer programs, which provide money to poor families contingent on certain behavior, usually investments in human capital, such as sending children to school or bringing them to health centers. The approach is both an alternative to more traditional social assistance programs and a demand-side complement to the supply of health and education services. Unlike most development initiatives, conditional cash transfer programs have been subject to rigorous evaluations of their effectiveness using experimental or quasi-experimental methods. Evaluation results for programs launched in Colombia, Honduras, Jamaica, Mexico, Nicaragua, and Turkey reveal successes in addressing many of the failures in delivering social assistance, such as weak poverty targeting, disincentive effects, and limited welfare impacts. There is clear evidence of success from the first generation of programs in Colombia, Mexico, and Nicaragua in increasing enrollment rates, improving preventive health care, and raising household consumption. Many questions remain unanswered, however, including the potential of conditional cash transfer programs to function well under different conditions, to address a broader range of challenges among poor and vulnerable populations, and to prevent the intergenerational transmission of poverty. Conditional cash transfer programs are an innovative approach to the delivery of social services. They provide money to poor families conditional on investments in human capital, such as sending children to school or bringing them to health centers on a regular basis. That conditionality makes this new generation of social programs an instrument for longer-term human capital investments as well as short-term social assistance. Additionally, along with school voucher programs and certain subsidized health insurance schemes, conditional cash transfer programs are part of a growing policy emphasis on the use of market-oriented demand-side interventions to directly support the poor. They complement traditional supply-side mechanisms, such as © The Author 2005. Published by Oxford University Press on behalf of the International Bank for Reconstruction and Development / THE WORLD BANK. All rights reserved. For permissions, please e-mail: journals.permissions@oupjournals.org. doi:10.1093/wbro/lki001 20:29–55 general subsidies or investments in schools, health centers, and other providers of social services. Conditional cash transfer programs aimed at improving children’s human capital have been established in numerous countries in recent years. Six are reviewed here. The first large-scale program to incorporate both health and education components was Mexico’s Education, Health, and Nutrition Program (Progresa), launched in 1997.1 Following a similar model, Colombia has the Families in Action (FA) program, Honduras has the Family Assistance Program (PRAF), Jamaica has the Program of Advancement through Health and Education (PATH), Nicaragua has the Social Protection Network (RPS), and Turkey offers the Social Solidarity Fund (SSF). Each program promotes long-term human capital accumulation as a primary objective, recognizing its role in breaking the intergenerational transmission of poverty (table 1). The programs focus primarily on children as the recipients of the human capital investments promoted by the programs and closely monitor compliance with conditions as a prerequisite for receiving the transfers. Traditional social assis- tance strategies have focused on short-term poverty alleviation through redistribution during times of crisis. Implementation of conditional cash transfer programs has been accompanied by systematic efforts to measure their effectiveness and understand their broader impact on households’ behavior, a marked departure from the limited attention to rigorous impact evaluations in the past.2 This article reviews the experience to date of six countries in setting up and evaluating the impact of such programs. The programs were selected to include those that provide conditional cash transfer for both health and education because policy and evaluation experience exist for such programs, as well as for those that provide in-kind conditional transfers.3 This review draws from program documents provided by administrators and evaluation reports produced by research institutions. Evaluation results are analyzed to draw conclusions about the welfare impact of this type of program and about how the evaluations have been used to inform policy decisions. Expected insights from forthcoming evaluations are briefly considered, followed by some reflections on the future direction of evaluations of social sector programs. Conditional Cash Transfer Programs: A New Approach to Social Assistance Conditional cash transfers together with other social assistance programs constitute a country’s formal, publicly provided safety net system.4 Conditional cash transfer programs represent a new approach to social assistance that explicitly addresses several criticisms often levied at more traditional social programs, including weak poverty targeting; high administrative or component costs, such as materials in workfare 30 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) Table 1. Objectives, Components, and Target Population of Selected Conditional Cash Transfer Programs Components Target population Program Objectives Education Health and nutrition Education Health and nutrition Colombia FA Increase human capital Bimonthly school Nutrition subsidy; Poor households with Poor households with children investment in extremely subsidy health education children ages 7–17 ages 0–6 not participating in poor families in smaller enrolled in school other programs municipalities; serve as (grades 2–11) a safety net Honduras Increase the accumulation Demand incentives Demand incentives Poor households with Poor households with PRAF II of human capital among (educational voucher); (nutrition andhealth children ages 6–12 pregnant women or children children of the poorest supply incentives for voucher); supply who have not yet under 3 families and help break primary schools incentives for health completed grade 4 the circle of poverty care centers; nutrition training for mothers Jamaica Increase educational Education grant Health grant; health Poor households with Poor households with children PATH attainment, improve education children ages 6–17 ages 0–5, pregnant and health outcomes, and lactating women, elderly thus reduce poverty; over 65, people with reduce current poverty; disabilities, and destitute reduce child labor; serve adults under 65 as a safety net (Continued) Table 1. (Continued) Components Target population Program Objectives Education Health and nutrition Education Health and nutrition Mexico Improve the educational, Educational grants; Cash grant for food Poor households with Cash grants: poor households; Progresaa health, and nutritional support for school consumption; basic children 8–18 enrolled nutrition supplements: status of poor families, materials; supply health care services in primary (grade 3 and pregnant and lactating particularly children and quality of education package; nutrition higher) and secondary women, children ages and mothers services and health education; (grades 1–3) school; 4–24 months, and mal- improved supply of age limit raised to 20 in nourished children ages 2–5 health services; 2001 to include upper years nutrition supplements secondary students Nicaragua Promote human capital Education grant; Cash grant for food; Poor children ages Cash grants: poor households; RPS accumulation in house support for school nutrition and health 6–13 enrolled in health care services: children holds living in extreme materials; supply education; basic health primary school ages 0–5 poverty incentive care package for grades 1–4 children under 5; supply incentive Turkey SSF Increase the human Education grant Health grant Poor households with Poor households with capital investment in children ages 6 and children ages 0–5 extremely poor families older enrolled in nationally; serve as grades 1–11 a safety net Source: Authors’ compilation based on FISE (2001), IFPRI (2000a), Skoufias (2001), World Bank (2001c, d, 2002). a In March 2002 Progresa changed its name to Oportunidades and broadened its objectives. The renewed program aims to create income-generating opportunities for poor households through preferential access to microcredit, housing improvements, and adult education. programs; lack of integration of disparate projects with a multiplicity of overlap- ping or unrelated goals; accusations of paternalism and clientelism; and excessive focus on reducing current poverty with little attention to long-term, structural poverty. The traditional role of social assistance programs has been to redistribute income and resources to the needy, helping them overcome short-term poverty during periods of crisis. Social policies and programs are changing, however, and are beginning to encompass objectives of longer term economic growth and human capital development. As Ravallion (2003) explains, by making insurance available, helping credit- constrained poor people become productive workers, and providing incentives for long-term investments in human capital, safety nets are now seen to have a potentially important role in compensating for the market failures that help perpetuate poverty, particularly in high-inequality settings. With an emphasis on human capital accumulation and long-term poverty reduction, conditional cash transfers are perhaps the clearest policy manifestation of this new thinking on social assistance programs. Conditional cash transfer pro- grams address both future poverty, by fostering human capital accumulation among the young as a means of breaking the intergenerational cycle of poverty, and current poverty, by providing income support for smoothing consumption in the short run. Conditional cash transfer programs have also introduced other key design features that depart from traditional social assistance programs. First, they provide grants directly to poor households, thereby changing accountability relationships among the national government, service providers, and the poor. The conditions required by the grants provide an incentive for poor households to use available health and education services, strengthening the link between ser- vice providers and the poor. Conditional cash transfer grants also allow national governments to forge a direct relationship with poor families, seeking to foster coresponsibility by requiring families to assume responsibility for schooling, health care, and the appropriate use of the cash grants. The programs reviewed here designate mothers as recipients of the grants in recognition of the inter- national evidence that suggests that women often make more optimal household spending decisions affecting children’s welfare. Second, they seek to exploit com- plementarities between elements of human capital development through their inclusion of health, nutrition, and education components. Third, the use of cash is promoted as efficient and flexible. It gives households spending discretion and avoids the price distortions and creation of secondary markets that are often asso- ciated with in-kind transfers. Finally, many conditional cash transfer programs also incorporate good technical program design features, including explicit pov- erty targeting criteria, often based on proxy-means tests, and strong monitoring and evaluation systems. Laura B. Rawlings and Gloria M. Rubio 33 Education and Health Components The programs reviewed here have both an education component and a health and nutrition component. The education component consists of a cash grant conditioned on school enrollment and regular school attendance (usually 80–85 percent of school days). The size of the grants varies considerably across countries (table 2). In Honduras, Mexico, and Turkey, the education grant covers both direct costs (school fees, school supplies, transportation costs) and opportunity costs in lost income from sending children to school rather than work. In the other countries the grant generally covers only part of the opportunity cost. In Colombia and Mexico education grants are higher for secondary school than for primary school, to reflect the increasing opportunity cost of work as children grow older. In Mexico grants at the secondary level are higher for girls, to provide an added incentive for reversing a pattern of unequal gender participation in secondary education and to internalize the educa- tion externalities that accrue as they raise families of their own (Skoufias 2001). In Turkey the value of the grant decreases proportionally according to the number of children in the family. Health and nutrition grants are targeted to children up to the ages of 2 or 3 years and in some cases up to the time they enroll in primary school. In Honduras, Jamaica, and Mexico, pregnant and lactating women are designated as program beneficiaries, and their inclusion is being discussed in Turkey. This component consists of a cash transfer aimed at food consumption, as well as health care and nutrition education for mothers. In Mexico and Nicaragua this component explicitly stipu- lates the provision of a basic health care package for the target household members. Receipt of the cash transfer is conditional on compliance with a predetermined number of health center visits and health and nutrition workshops.5 Children’s health care visits are linked to growth monitoring and often to vaccination protocols. Health care visits for pregnant and lactating women seek to ensure appropriate prenatal, childbirth, and puerperal care. In Mexico and Jamaica adult household members other than pregnant and lactating women are also required to get a check-up once or twice a year (see table 2). The value of the monthly cash grant for the health and nutrition component var- ies across countries (see table 2). In Honduras, for example, the value of the nutri- tion and health voucher is equivalent to the value of the time invested by the mother in the trip to the health center and waiting for care. Jamaica set the health grant per beneficiary per month at US$9, the same level as the education transfer and twice average monthly spending per person on health care and medicine in 1999. Colombia set the grant to the mean income required to allow an average indigent family to reach the extreme poverty line and so to consume a nutritiously adequate diet. Jamaica and Turkey provide health and nutrition grants to individuals rather than family-based allocations. 34 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) Table 2. Conditionality and Transfer Size of Selected Conditional Cash Transfer Programs Conditionsa Transfer size Program Education Health and nutrition Education Health and nutrition Colombia At least 80% school Regular health care visits Primary school: Col$14,000 (US$6) per child Col$46,500 (US$20) per family FA attendance in a 2-month for child’s growth and per month; secondary school: Col$28,000 per month cycle development monitoring (US$12) per child per month Honduras School enrollment and Compliance with the required Voucher: L$828 (US$58) per child per year; Voucher: L$660 (US$46.3) per PRAF II maximum 7 days of frequency of health center visits average supply incentive of L$57,940 family per year; average supply school absence in (US$4,000) per school per year incentive L$87,315 a 3-month period (US$6,020) per facility per year Jamaica Minimum school Compliance with the required Grant: J$500 (US$9) per child per month after J$500 (US$9) per eligible PATH attendance of 85% number of health visits per second year (program began at J$300 per child household member per month (maximum 9 days of year, which varies by per month) school absence per term) beneficiary age / status Mexico School enrollment, with Compliance by all household Primary school: varies by grade, Mex$80–165 Mex$125 (US$13) per Progresaa minimum attendance of members with the required (US$8–17) per child per month plus Mex$100 household per month in 1999 85%, monthly and number of health center visits (US$11) per year per child for school materials; of Mex$750 per month for food annually and mother’s attendance at secondary school: varies by grade and gender, support and educational grants health and nutrition lectures Mex$240–265 (US$25–32) per child per month plus Mex$200 (US$20) per year per child for school materials Nicaragua School enrollment, fewer Regular health care visits for Grant: C$240 (US$17) every 2 months per C$480 (US$34) per household RPS than 6 days of unexcused growth monitoring, up-to-date household plus C$275 (US$20) per child per year every 2 months school absence in vaccinations, and attendance at for school materials; supply incentive: C$10 a two-month period, and health and nutrition talks (US$0.7) per student every 2 months school grade promotion Turkey SSF School enrollment, with Regular health care visits for Primary and secondary school grant: TL TL8.1 million (US$6) per child minimum attendance growth monitoring and vacci- 12.8 million (US$9.50) per month for first child, per month of 85% nations according to a schedule TL10.8 million (US$8) for second, and TL8.8 set by the Ministry of Health million (US$6.50) for each subsequent child Source: Authors’ compilation FISE (2001), IFPRI (2000a), Skoufias (2001), World Bank (2001c, d, 2002). a Programs have not always enforced all conditions. Supply-Side Support Conditional cash transfer programs can be interpreted as a response to the perceived failures of traditional supply-side interventions, such as schools and health clinics, which have been underutilized by the poor because of unmanageable out-of-pocket expenditures, high opportunity costs, difficult access, and a lack of incentives for investing in children’s human capital. However, conditional cash transfers are not a substitute for the provision of high-quality supply-side investments. Rather, they complement such investments by directly addressing the problem of insufficient demand for health and education services from the poor. This makes these pro- grams’ ultimate success dependent on access to high-quality health and education services. No program should be conditioned on the mandated use of poor-quality, ineffective services. Because of the critical role of good quality health and education inputs, some countries go beyond providing demand-side monetary incentives to families by strengthening the supply of these services. In Nicaragua teachers receive a modest bonus per child participating in the program, half of it intended to pay for school materials, and nongovernmental organizations (NGOs) are contracted to provide health services. Mexico sets aside resources to ensure an adequate supply of equipment, medicines, and material to meet the increase in health services demand arising from the program. Honduras provides grants directly to schools and health centers as part of an experiment to compare the effectiveness of three alternative interventions combining demand and supply incentives. Poverty Targeting Directing benefits to the poor or vulnerable is a critical feature of each reviewed conditional cash transfer program. Most rely on both geographic and household targeting, using targeting mechanisms appropriate to the type of data available (table 3). At the geographic level Jamaica uses annual consumption data to construct a scoring formula to identify poor households at the parish level for allocating PATH funds. In Mexico eligible communities in rural areas are selected using a marginality index based on census data, whereas in Honduras malnutrition data from the Height Census of First Grade School Children are used to select program municipalities. In most countries, the criteria applied to select communities to receive the conditional cash transfer program include consideration of the supply capacity to respond to the increased demand in health and education services. At the household level programs are experimenting with proxy-means tests that estimate household poverty levels as a criterion for program participation (table 3). In Nicaragua the results of household-level proxy means tests are being compared 36 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) Table 3. Targeting Criteria of Selected Conditional Cash Transfer Programs Program Geographic Household Colombia FA Municipalities other than department capitals Level 1 families in the SISBEN with fewer than 100,000 inhabitants; (an information system based municipalities not participating in other national on a proxy means test for programs with adequate supply of education and identifying poor households) health services and a bank; municipalities with up-to-date SISBEN database Honduras PRAF II Municipalities with the lowest average height None for age z-scores Jamaica PATH All parishes participate in the program; funds A scoring formula with a are distributed across parishes depending predetermined cutoff point on their poverty incidence Mexico Progresaa Rural communities with a high marginality Within eligible localities, index with more than 50 and fewer than 2,500 eligible households are inhabitants and access within a certain distance identified using discriminant to primary and secondary school and health care analysis of household income center; urban areas with a high marginality and other characteristics index have been included since 2001 Nicaragua RPS Departments and municipalities with high Pilot phase 1: all households extreme poverty incidence, good access to in selected census areas with schools and health care centers, good transport less than 14.1 ha and no and communication infrastructure and local vehicle; pilot phase 2: capacity; within eligible municipalities census eligibility is determined areas were classified in two groups according to by a scoring formula a marginality index based on family size, access to basic sanitation and safe water, and literacy rates, with group 1 eligible in the pilot phase 1 and group 2 in the second pilot phase Turkey SSF National coverage; no geographic targeting Proxy-means test based on a scoring formula Source: Authors’ compilation based on FISE (2001) IFPRI (2000a), Skoufias (2001), World Bank (2001c, d, 2002). with the results of geographic targeting alone. Turkey will use a proxy-means test being developed especially for the conditional cash transfer program to target the poorest of the poor at the national level. Other countries are taking advantage of economies of scale in the use of proxy-means tests. In Colombia household eligibility is based on an existing information system managed by municipalities, the System for Selecting Social Program Beneficiaries (SISBEN). The system classifies households according to an unmet basic needs index and other indicators, such as average house- hold schooling that serve as income proxies. Used primarily to identify eligibility for the subsidized health regime, SISBEN is now being expanded to other social sector Laura B. Rawlings and Gloria M. Rubio 37 initiatives, including the conditional cash transfer program. Jamaica is planning to expand the use of the PATH scoring formula to other safety net programs to avoid duplication of administrative systems and increase coordination across programs. Some countries periodically review beneficiaries’ eligibility. In Mexico and Jamaica household poverty status is reevaluated every three years. Nicaragua’s RPS is designed to last three years in a beneficiary community before the cash transfers are phased out. Only the supply interventions are retained for two more years without a reassessment of eligibility. An Expanding Role in Poverty Alleviation As reflected in budget allocations and the number of beneficiaries, conditional cash transfer programs are playing an increasingly important role in poverty reduction strategies. Mexico’s Progresa went from covering 300,000 households when it began operations in 1997 to reaching more than 4 million families in 2002, some 20 percent of the population. The program’s 2002 annual budget was around Mex$18 billion (US$1.8 billion). PATH is a key element of the Jamaican government’s initiative to transform the social safety net into a fiscally sound and more efficient system of social assistance for the poor and vulnerable. It aims to consolidate three major income transfer programs, strengthen targeting measures, improve the cost-effectiveness of delivering benefits, and adjust benefit levels to meet assessed needs. Turkey’s SSF was introduced as part of a handful of crisis-response mechanisms to ease the impact of the 2001 economic crisis on poor households. It has an annual operating budget of US$100 million. Colombia’s FA is the flagship program of the three safety net programs introduced in 2001 to provide relief from the effects of an economic recession. The program, designed to run through 2004, has a budget of US$455 million and is expected to reach more than a million children. In Brazil, Bolsa Familia is being introduced as an overarching welfare program that will consolidate numerous smaller programs to become a mainstay of Brazil’s poverty reduction approach. Evaluation of Design and Implementation The first generation of conditional cash transfer programs in Colombia, Honduras, Nicaragua, and Mexico prioritized the early use of robust evaluations as a key element for informing program design and expansion. All but Colombia’s program used ran- domized control designs as the primary evaluation methodology underpinning a large-scale social experiment, carefully planned well in advance with strong support from program staff and policymakers. 38 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) The first generation of conditional cash transfer evaluations aimed at assessing program impact and operational performance by examining the programs’ adminis- trative adequacy, the extent to which programs reached poor areas and poor house- holds, the presence and size of expected impacts, any unanticipated effects, stakeholders’ perceptions about the program, and the cost-effectiveness of delivery mechanisms. The impact evaluations focused on measuring changes in short- and medium-term indicators of human capital accumulation rather than on the income redistribution effects of the grants. In education the evaluations assessed changes in school enrollment and attendance rates, and some also analyzed changes in promotion and repetition rates. Evaluation of Honduras’s PRAF and Mexico’s Progresa went beyond outcome indicators to measure changes in impact indicators, such as average test scores. In addition, given PRAF’s evaluation objective of comparing the impact of supply- and demand-side interventions, evaluators are examining changes in the availability and quality of education inputs (percentage of teachers trained, percentage of schools with basic teaching materials). In health and nutrition, the evaluations included a wide range of health care utili- zation and quality indicators. Program variations in target population are reflected in the diverse selection of child, maternal, and adult health indicators. Child health indicators typically include vaccination coverage, malnutrition rates, incidence of diarrhea, and participation rates in child growth and development monitoring. Maternal health indicators include utilization rates and satisfaction with pre- and postnatal care. Honduras’s PRAF evaluation is measuring final program impacts by analyzing changes in maternal and infant mortality. Changes in consumption levels and patterns are also central to many evaluations. Total consumption per capita disaggregated by food and nonfood items, such as health and education spending, is frequently used as an indicator. Given the implicit objective of reducing current poverty, Mexico’s Progresa evaluation investigates the impact of cash transfers on the poverty headcount ratio, poverty gap, and poverty severity index. Development programs often have unplanned direct and indirect effects, both positive and negative. Some of the conditional cash transfer evaluations have ana- lyzed such impacts. For example, the distribution of cash grants directly to mothers may have an effect on resource allocations within households and on power relations. Cash transfers may crowd out remittances and other private transfers to households or affect household work incentives. Household-level targeting may also affect community relations when not all members of a community receive program benefits. Evaluation Design Program impacts are measured by assessing whether a program changes the mean value of an outcome variable among participants compared with what the outcome Laura B. Rawlings and Gloria M. Rubio 39 would have been had they not participated. The central evaluation problem then is that program participants cannot be simultaneously observed in the alternative state of no participation (the counterfactual). Evaluators typically simulate the counter- factual by comparing program participants (the treatment group) with a control or comparison group with similar characteristics. Construction of the counterfactual determines the evaluation design, which can be broadly classified as experimental or quasi-experimental. These evaluation designs vary in feasibility, cost, and the clarity and validity of results. Experimental or randomized control designs involve the random assignment of individuals (or another unit of analysis) into the treatment group or the control group. Because participants are selected randomly, any differences between the groups is due to chance, not selection. For this reason, experimental designs are usually regarded as the most reliable evaluation method and the one yielding the easiest results to interpret (Freeman and Rossi 1993; Grossman 1994). When randomization is not feasible, a quasi-experimental design is used to gener- ate a comparison group through alternative means. Statistical matching on the basis of observable characteristics is commonly used to select comparison group members who are comparable in essential characteristics to participants. Because unobservable characteristics of beneficiaries, such as motivation or organizational capacity, can strongly influence program impacts but are generally not addressed or addressed only with difficulty in evaluations using quasi-experimental designs, these approaches are often considered less methodologically robust. The first generation of conditional cash transfer evaluations took advantage of the gradual implementation of these programs (because of logistical complexities, fiscal constraints, and uncertainty about the magnitude of program impacts) to randomly add beneficiaries as the programs expanded.6 This approach reflected pragmatism and a desire to rigorously explore the impact of these new programs. Experimental designs are usually maintained for only a few years, however, thus limiting their ability to provide rigorous evidence on longer term program effects. Most first-generation conditional cash transfer evaluation designs rely on random allocation of program benefits by geographic area rather than by household (table 4). The broad geographic nature of some of the program components and the difficulties arising from having treatment and control groups in the same community made randomization at the household level impractical. In Mexico’s Progresa, evaluators randomly assigned localities to treatment and control groups. Treatment localities entered the program in November 1997, whereas control localities started receiving Progresa benefits in December 2000. Randomization was implemented at the municipal level in Honduras’s PRAF and at the census level in Nicaragua’s RPS. In Honduras the evaluation objectives required three different treatment groups to compare the impacts of different combinations of demand and supply incentives. Randomization by municipality was the preferred option because 40 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) Table 4. Completed and Ongoing Evaluations of Selected Conditional Cash Transfer Programs Evaluation activities Sample size Logical Operations Qualitative Impact Program framework evaluation studya evaluation Evaluation design Main indicators Data sources Control group Treatment group Colombia X X X X Quasi-experimental Targeting efficiency; Household survey; 50 munici- 50 municipalitiesb FA with 2 rounds: school enrollment; surveys of schools palities (10,660 house- participating child nutrition and and health centers; (8,347 holds) municipalities were health status; con- surveys of households) matched with sumption patterns community similar day care centers municipalities not part of the program Honduras X X Experimental with Education outcomes Census of G1 and G2 20 munici- G1 = 20 PRAF II panel data: random (test scores, repetition, municipalities; palities municipalities assignment of promotion, attendance); household surveys (1,600 (1,600 households); municipalities into availability and quality (baseline plus two households, G2 = 20 four groups: G1 of education inputs; follow-ups—1 and 2 80 in each municipalities (vouchers), G2 health outcomes years after program municipality) (1,600 households); (vouchers + supply (maternal and infant start); school and G3 = 10 incentives),G3 mortality); utilization health center municipalities (supply incentives and satisfaction with diagnostic surveys; (800 households) only), and G4 health care services; standardized test (control group) health care practices scores (Continued) Table 4. (Continued) Evaluation activities Sample size Logical Operations Qualitative Impact Program framework evaluation studya evaluation Evaluation design Main indicators Data sources Control group Treatment group Mexico X X X Experimental with School enrollment and Census of evaluation 186 localities 320 localities Progresaa panel data: random attendance; utilization localities; household (4,682 (7,887 eligible assignment of of health care services surveys (baseline eligible households) localities into and health status; child plus 5 follow-up households) treatment nutritional status; surveys collected and control household consumption about every group and caloric availability; 6 months)c; school poverty incidence; and clinic surveys; changes in fertility; community question- women’s status and naires; test scores; intra-household school and clinic relations; time allocation; administrative data private transfers Nicaragua X X Experimental with Targeting efficiency Census of program 21 census 21 census RPS panel data: random (leakage and coverage area; baseline areas (812 areas (773 assignment of rates); school enrollment household survey; households)d households) census areas into and attendance; follow-up treatment and consumption patterns; household survey; control group utilization and quality institutional of child health care assessment for services (including schools timely immunization) Source: Authors’ compilation based on Attanasio and others (2002), Behrman and Todd (1999), IFPRI (2000b, 2001a), and Skoufias (2001). a Including beneficiary assessments. b This is the designed sample. The preliminary results presented in this article are based on the early incorporation of 25 municipalities into the treatment group. c The evaluation also benefited from data on anthropometric measures and blood samples collected by the National Institute of Public Health. d Includes only the first baseline data collected. of their well-defined borders and the feasibility of linking each household, school, or health center with a particular municipality. Program municipalities were selected using data from the School Height Census. A subset of municipalities was randomly assigned to one of four evaluation groups: those receiving demand vouchers, those receiving vouchers plus improvements in service quality, those receiving improve- ments in service quality, and the control group. RPS in Nicaragua followed a similar process, randomly allocating census areas into treatment and control groups. In contrast to the other programs Colombia’s FA applied a quasi-experimental design. Program implementers ruled out randomization, instead targeting the program to medium-size municipalities able to provide adequate health and education services and with at least one bank to be able to set up family accounts. A comparison group consisted of municipalities similar to the treatment group in terms of population and infrastructure but not qualifying for the program, often because of lack of a bank. Although not originally planned, the program was launched in a few treatment municipalities before the baseline data were collected. This provided an opportunity for a preliminary evaluation of the unanticipated treatment group and a comparison group constructed through propensity score matching techniques using the baseline data (the comparison group is constructed from nonprogram households with a participation probability closest to program beneficiaries, as determined by the probability of program participation based on socioeconomic characteristics). This required adjustments to the sampling frame, the inclusion of retrospective questions in the survey questionnaires, and additional econometric techniques to control for possible nonrandom selection of early participating municipalities, but the exercise yielded valid (if less precise) impact estimates. Data Collection Early planning of most evaluations allowed for the collection of baseline data, thus permitting comparisons of households in the treatment and control groups before and after program implementation. In this way evaluations can account for charac- teristics that do not change over time within treatment and comparison households, as well as for those that do and that are common to both groups. Random assignment into treatment and control groups, combined with the collection of baseline and follow-up data, allows measuring program impact using difference-in-differences estimators. Except for Colombia, all first-generation conditional cash transfer evalu- ations have baseline data collected before program implementation.7 Household surveys are the main data collection instrument in each of the cases reviewed. Each questionnaire contains a core set of questions about the demographic composition of the household, household expenditures and remittances, and socio- economic status, education, health, migration, and labor market participation of household members. Some country questionnaires include additional modules, such Laura B. Rawlings and Gloria M. Rubio 43 as anthropometrics (height and weight), fertility, participation in other programs, and time allocation. Honduras also incorporates two modules on the quality of health services and schools to evaluate the supply-side component of PRAF.8 School and health center surveys and community questionnaires are also frequently used for evaluation. In Honduras and Mexico student achievement test scores were used to analyze program impact on academic performance. Beneficiaries and other stakeholders’ perceptions about the program are often captured through qualitative studies. As part of the operational evaluation of the program, Progresa conducted semistructured interviews with secondary school and health clinic staff and focus group discussions with beneficiaries, nonbeneficiaries, and community mothers who serve as local contacts for Progresa. Qualitative studies have also been used in Nicaragua to complement impact eval- uations. A study on beneficiaries’ perceptions of the program’s impact on welfare included a beneficiary survey; focus group discussions with beneficiaries and com- munity mothers; key informant interviews with representatives from the Ministries of Health and Education, the mayor’s office, health care providers, NGOs, and local program office staff; and six case studies of beneficiary families in different munici- palities. Another study assessed perceptions of the poverty targeting mechanism and included surveys, focus group discussions with beneficiaries and nonbeneficiaries, and key informant interviews. Implementation Issues Social experiments present challenges at each stage of implementation. Evaluations of conditional cash transfer programs reveal two particular issues: the difficulty of coordinating impact evaluations with the implementation schedule and the chal- lenge of winning the political support required for a successful impact evaluation. New, logistically complex programs, such as cash transfer programs, can run into implementation delays or, as in Colombia, may move ahead of the evaluation schedule. Likewise, political changes (such as an upcoming election or changes in program administration) may also affect implementation or even program design. Natural disasters, such as flooding in Jamaica, can also alter the implementation schedule. Such events can effect the evaluation in various ways. For example, in Nicaragua baseline data was collected during August–September 2000, and follow-up data collection was scheduled for the same time a year later. But when the health component was delayed until June 2001 because of difficulties coordinating the health care providers, evaluators had to postpone follow-up data collection until October. Although having a control group helps in this kind of situation, conducting panel surveys at different times of the year may lead to problems from the confounding nature of seasonal effects. 44 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) Delays in setting up the program management information system may also cause problems. Deficiencies in the delivery of program benefits may go undetected and thus be unaccounted for in the evaluation. In Mexico, Progresa’s payment records revealed that 27 percent of the eligible population in the evaluation sample had not received benefits after almost two years of program operation. This can result in a divergence between the “intention to treat� effect estimated by the evaluation and the mean effect of the program on those who actually participated.9 Finally, as Mexico’s Progresa and Honduras’s PRAF have revealed, implementing impact evaluations requires strong political support, particularly for a randomized control design. Incorporating a control or comparison group into the evaluation can generate strong criticism and lead to political and media pressure to extend program benefits to these groups. It is important to secure a solid commitment from policymakers to maintain the integrity of the program and evaluation designs and to communicate clearly the benefits of random allocation when budget constraints prevent reaching all eligible beneficiaries at once. Evaluation Results and Impact on the Ground Evaluation results are available for Progresa in Mexico, FA in Colombia, and the RPS pilot in Nicaragua. These evaluations reveal that conditional cash transfers can provide effective incentives for investing in the human capital of poor people. Impacts on Education, Health, and Consumption In education, conditional cash transfer programs have demonstrated a positive effect on enrollment rates for both boys and girls.10 In Mexico primary school enrollment rates before Progresa were 90–94 percent. Econometric estimates of program impact using a difference-in-differences model controlling for household and com- munity characteristics show an increase ranging from 0.74 to 1.07 percentage points for boys to 0.96 to 1.45 percentage points for girls (table 5). At the secondary level, baseline enrollment rates were 67 percent for girls and 73 percent for boys. Estimates of program impact show an increase ranging from 3.5 to 5.8 percentage points for boys to 7.2 to 9.3 for girls. In Nicaragua program impacts are even more impressive (table 6). From a low starting point of 68.5 percent, average enrollment rates in treatment areas increased nearly 22 percentage points. Colombia’s FA pro- gram seems to have had no effect on enrollment rates among the primary school age population (7–13 years old) while boosting secondary school enrollment rates (for 14–17 years old) 5.5 percentage points in rural areas and 14 percentage points in urban areas (table 7). Laura B. Rawlings and Gloria M. Rubio 45 Table 5. Impacts on Education, Health, and Consumption of Mexico’s Progresa Baseline Net change/program impact Education Primary school enrollment Female 90–94% 0.96–1.45 percentage pointsa Male 90–94% 0.74–1.07 percentage pointsa Secondary school enrollment Female 67% 7.2–9.3 percentage pointsa Male 73% 3.5–5.8 percentage pointsa Health Mean growth monitoring visits (in the month 0.22 0.054–0.133b prior to the survey), children ages 0–2 Illness rates (in the month prior to the survey), 0.40 −4.7 percentage pointsb children ages 0–2 Consumption Mean consumption level per household — 13.4%c per month Median food consumption per person — 10.6%c per month Median caloric acquisition per person per day — 7.8%c Source: Education, Skoufias (2001); health, Gertler (2000); consumption, Hoddinott and others (2000). a Econometric estimates using a difference-in-differences model controlling for household and community char- acteristics. b Econometric estimates of program impacts using a difference-in-differences model. c Percentage difference between beneficiary and control households at 20 months postbaseline. The evidence of impacts on attendance is mixed. The evaluation of Nicaragua’s RPS indicates a larger impact on attendance than on enrollment rates, finding a 30 percentage point increase in the share of children with fewer than six unexcused school absences in a two-month period. The evaluation of Progresa showed more pronounced effects on enrollment than attendance. Evaluations have also found improvements in child health and nutrition. The Progresa evaluation shows a significant increase in nutrition monitoring and immunization rates. Econometric estimates from difference-in-differences models accounting for individual fixed effects found that children 0–2 years old partici- pating in Progresa increased their growth monitoring visits 25–60 percent with respect to the baseline value of .22 visits during the previous month. Progresa also lowered illness rates for the same group of children by 4.7 percentage points, or 12 percent lower than the baseline value (Gertler 2000). The data also suggest that Progresa has had a significant impact on child growth, lowering the probability of child stunting for children ages 12–36 months (Behrman and Hoddinott 2000). 46 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) Table 6. Impacts on Education, Health, and Consumption of Nicaragua’s RPS Treatment areas Control areas Baseline Follow-up Baseline Follow-up Net change/ Indicator 2000 2001 2000 2001 program impacta Education (%) Share of children ages 7–13 enrolled 68.5 93.2 72 75.1 21.7 (2.7) in primary school (grades 1–4) Health (%) Share of children less than 3 years old 55.9 91.8 60.6 67.4 29.1 (4.3) participating in growth monitoring Share of children ages 12–23 35.4 81.9 40.3 68.5 18.3 (7.8) months with complete, timely immunization Consumption (cordobas) Per capita annual total expenditures 4,310 4,498 3,929 3,300 817 (231) Per capita annual food expenditures 2,922 3,165 2,684 2,175 753 (154) Source: IFPRI (2002b). a Unconditional difference-in-differences estimator. Note: Numbers in parentheses are standard errors. Table 7. Preliminary Impacts on Education, Health, and Consumption of Colombia’s Families in Action (FA) Comparison areas Effect of program in treatment areas Urban Rural Urban Rural Education Enrollment probability, ages 7–13 0.941 0.915 0.003 (0.022) 0.012 (0.028) Enrollment probability, ages 14–17 0.639 0.496 0.138* (0.066) 0.055 (0.053) Health Probability of suffering from acute 0.212 0.170 −0.102* (0.055) −0.054 (0.065) diarrhea, children under 6, over past 15 days Probability of suffering from acute 0.448 0.404 −0.032 (0.103) −0.021 (0.073) respiratory disease, children under 6, over past 15 days Consumption Number of days 2–6-year-olds ate eggs 2.4 2.67 0.705* (0.437) 0.774* (0.428) Number of days 2–6-year-olds ate 1.26 1.67 1.383* (0.437) 1.148* (0.488) vegetables Source: Attanasio and others (2003). *Significant at the 5 percent level. Note: Numbers in parentheses are standard errors. Laura B. Rawlings and Gloria M. Rubio 47 In Colombia’s FA the proportion of children under age 6 enrolled in growth mon- itoring rose 37 percentage points. The incidence of acute diarrhea in children under age 6 was reduced by 10 percentage points in urban areas, but there was no significant change in rural areas. The study applied various measures of malnutrition to children under age 6 and detected no impact on global or acute malnutrition in any of the program areas. It did find a positive impact on weight-for-height and weight-for-age in rural areas though not in urban areas (Attanasio and others 2003). Nicaragua’s RPS program generated similar improvements. Before RPS was implemented, some 60 percent of children under age 3 participated in nutrition monitoring. After several months of program operation more than 90 percent of children in RPS areas participated in nutrition monitoring compared with 67 percent in control areas. Rates of timely immunization among children 12–23 months old rose by 18 percentage points in the treatment group compared with the control group (IFPRI 2002b). Conditional cash transfer programs have also resulted in higher consumption levels. In Mexico after just over a year of program operation the average con- sumption level was 13 percentage points higher, and the value of food consumption for the median beneficiary household was 11 percent higher in Progresa households than in non-Progresa households. Higher expenditures on fruits, vegetables, and animal products accounted for much of the increase in household consumption. Median caloric intake per person in Progresa house- holds increased by 7.8 percentage points (Hoddinott and others 2000). Dietary intake also improved in FA households in Colombia. In Nicaragua consumption levels remained unchanged in RPS areas despite worsened economic conditions related to low coffee prices and a drought. By contrast, consumption declined sharply in control households (IFPRI 2002b). The net program impact translates into a 19 percent increase in per capita consumption and suggests that condi- tional cash transfer programs may help poor people protect consumption in times of crisis. The evaluation of Progresa revealed that conditional cash transfer invest- ments can be delivered cost-effectively. The administrative costs of delivering cash transfers to poor households appear to be small (Mex$8.9 of every Mex$100 allocated to the program) relative to the costs of previous Mexican programs and to targeted programs in other countries (Coady 2000). The largest cost components are those associated with household targeting (nearly 30 percent), followed by those associated with conditioning the receipt of transfers (26 percent). The evaluation also found that the cost of generating an extra year of schooling using subsidies is around Mex$10,000 in secondary school and Mex$55,000 in primary school, compared with Mex$168,000 for extensive expansion by building additional secondary schools and thus reducing travel distances. 48 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) Impacts on Program Expansion and Design These findings of encouraging human development impacts have been used to revise programs and influence policy. But the evaluations reveal little about which element of the intervention (the transfer or the conditionality) is responsible for the observed changes or whether the relatively short-term changes will be translated into long-run impacts on human capital formation and poverty. They also provide no comparative evidence on whether alternative interventions would have achieved comparable results. The impact evaluations have triggered some program modifications, guided program expansion, allowed the programs to survive changes in political admin- istrations, and generated interest in replicating the programs internationally. The positive impacts of Mexico’s Progresa helped prompt its expansion from rural to urban areas, and the program has continued with few alterations despite a change in government. Likewise, the measured achievements of FA in Colombia contributed to the program’s continuation and expansion despite a change in government. The evaluation findings for Nicaragua’s RPS showing that the program had met most of its targets and exceeded many of them trig- gered plans for program expansion. The demonstrated utility of the conditional cash transfer evaluations also increased political support for other impact evalu- ations. In Mexico a congressional mandate calls for the evaluation of all social programs, whereas in Colombia the national performance monitoring system is expanding to include impact evaluations of the country’s principal social programs. The evaluation results have also contributed to the ongoing debate on the design of conditional cash transfer programs and their potential role in broader reform of social protection systems. One issue under discussion is whether condi- tional cash transfer programs should take on income-generating activities. Although this may be a necessary condition for the sustainability of human capital investment in future generations, whether conditional cash transfer pro- grams should take this on themselves or whether separate income-generation programs are preferable is unclear. In both Mexico and Nicaragua there has been a tendency to expand conditional cash transfer programs to include train- ing and other income generation activities. Both programs are planning impact evaluations to help inform the debate. Evaluation results of conditional cash transfer programs and national education-oriented cash transfer programs in Brazil have contributed to a reform that will consolidate an array of cash trans- fer programs and triple the average monthly benefit per family. The new program is expected to reach 11.4 million families by 2006, about a quarter of Brazil’s population, making this the world’s largest conditional cash transfer program. Laura B. Rawlings and Gloria M. Rubio 49 Upcoming Evaluations: Expected New Insights New conditional cash transfer programs have recently begun in Jamaica, Turkey, and urban areas of Mexico. This second generation of programs is being imple- mented under considerably different circumstances than the first. They have benefited from the experience of the first generation, making the logistical aspects less daunting. Evidence of program impacts from the first generation has reduced uncertainty about program results and thus the need for small pilots or phased implementation. Finally, the socioeconomic and political circumstances are particularly pressing in some cases, so implementation plans incorporate nationwide expansion almost immediately. Both the PATH program in Jamaica and the SSF in Turkey have had short pilots, mainly to test program processes, followed rapidly by nationwide expansion. Consequently, new methodologies are being tested. Program pilots include only a process evaluation, reserving impact evaluations for the full-scale program. Because experimental evaluation designs are more challenging when used to evaluate a nationwide program, the second generation of conditional cash transfer programs relies on quasi-experimental designs. The evaluation of Oportunidades, the follow-on to Progresa, takes advantage of the proxy-means test used for beneficiary selection to construct a comparison group from households that applied to the program but were not selected because they fell above the cutoff point. Presumably, households immediately above the cutoff point are similar on average to program beneficiaries and can serve as a comparison group. A second comparison group will be drawn from eligible households in nonin- tervention areas, selected through propensity score matching techniques. The evaluation of Turkey’s SSF anticipates applying a quasi-experimental design using panel data with a baseline and two follow-up measures, as well as a qualitative study. Data from the first follow-up survey—to be conducted about one year after the program begins—will be used to assess poverty targeting, short-term welfare impacts, changes in utilization of health and education services, and stakeholder perspectives. A more comprehensive impact evaluation is contemplated for two years after program implementation, using the last round of panel data. Because of the reliance on quasi-experimental designs, second-generation evalua- tions are politically less sensitive and less demanding to implement. However, the results are likely to be less robust and straightforward than those generated by carefully planned experimental designs. In addition, given the rapid expansion of these programs to national scale, there is less control over the timing of the implementation schedule and a greater need for flexibility in evaluation plans. Potential contamination of the comparison group is another problem. The use of households just above the cutoff point for constructing a comparison group risks contamination of the sample from premature incorporation of comparison group households into the program should the cutoff point change. 50 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) These second-generation evaluations will address many of the same core questions about program impacts on school attendance, health care utilization, and consumption and so will help confirm the cross-program robustness of earlier results. Together with continuing evaluations in Colombia, Honduras, and Nicaragua, these evaluations will also analyze new questions prompted by program objectives in each country and by a desire to increase the global body of knowledge about conditional care transfer programs. In Nicaragua the evaluation will assess the sustainability of behavioral changes by measuring program impacts once cash transfers are phased out and only supply-side interventions remain. In Honduras the evaluation will focus on the relative importance of supply and demand factors in increasing human capital as well as program impacts on maternal and child mortality rates. In Colombia, implementation of the conditional cash transfer program as one of three emergency safety net programs will allow for a comparison of the relative effectiveness of conditional cash transfer, workfare, and train- ing programs in achieving particular outcomes. In Mexico the evaluation will examine the results of a new educational savings program for Oportunidades students. Conclusions and Recommendations for the Future In contrast to many development programs, the recent expansion of conditional cash transfer programs is based on fairly solid evidence of program impact. Evalua- tion results from the first generation of these programs in Colombia, Mexico, and Nicaragua show them to be an effective means for promoting human capital accu- mulation among poor households. In particular, there is clear evidence of program success in increasing school enrollment rates, improving preventive health care, and raising household consumption. These evaluation results have provided policy- makers with rare empirical evidence on program efficiency and effectiveness that has informed administrative reforms, prompted the expansion of programs geo- graphically and to new population groups, and contributed to their continuation despite changes in political regimes. The next generation of evaluations is building on this body of knowledge of condi- tional cash transfer programs by providing evidence on the medium-term impact of programs, the value of new elements, and the impact of new conditional cash transfer programs in Honduras, Jamaica, Turkey, and urban areas of Mexico. These evaluations will confirm or challenge earlier findings, shed light on questions of sustainability and medium-term impacts, and provide policymakers with a better understanding of the impacts of alternative combinations of program inputs and different regional circumstances. These results will be useful in understanding the capacity of condi- tional cash transfer programs to meet the new demands placed on them and ensure that these demands do not interfere with achievement of the program’s primary objectives. Laura B. Rawlings and Gloria M. Rubio 51 But even when evaluations of the new generation of conditional cash transfer pro- grams become available, many fundamental questions will remain unanswered— about the effectiveness, including long-term welfare impacts; synergies between program components; tradeoffs between transfer size and number of beneficiaries; and balance between the short-term transfer objectives and the long-term human develop- ment objectives. There is also a need to assess the effectiveness of these programs both as a permanent institution for addressing chronic poverty and as a temporary instru- ment for addressing vulnerability to shocks, and to explore ways to strengthen the links with suppliers of health and education services to improve access and quality. These long-term questions can be answered only through further evaluation. Improvements in evaluation instruments are also needed. Econometric modeling is being used to simulate the anticipated impacts of program design alternatives, such as transfer size and eligibility criteria. Although not a substitute for impact evaluations, these tools can be very useful, particularly for program design. There is also a need to go beyond impact evaluations of individual programs to improve results-based monitoring and evaluation systems of related programs, as a foundation for effective policy management. Finally, there is a need for impact evaluations to explore the development effectiveness of alternative programs and policies, particu- larly concerning long-term welfare impacts. The benefits of program evaluations go far beyond country boundaries and constitute a global public good. The experience of conditional cash transfer programs to date shows the critical role of evaluations in shedding light on success and failure in the fight against poverty. The evaluations are also contributing to the spread of condi- tional cash transfer programs, which are being replicated around the world. Even so, it should not be assumed from positive evaluation results from a handful of countries that similar successes can be achieved in other countries in different con- texts, especially in areas facing supply constraints in health and education or where the capacity to administer a conditional cash transfer program is limited. Nor do the positive results from one program imply that the program subject to the evaluation is necessarily the best approach to achieving a particular outcome. Ideally, program evaluations would compare alternative interventions for achieving a similar objective to determine the most effective and efficient approach. What evaluations of the impact of conditional cash transfer programs reveal so far is the importance of good program and evaluation design in informing policy decisions and providing evidence for achieving progress in the fight against poverty. Notes Laura Rawlings is the Country Sector Leader for Central America in the Latin America and Caribbean Human Development Department at the World Bank; her e-mail address is lrawlings@worldbank.org. 52 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) Gloria Rubio is Deputy Director of Social Programs Evaluation at the Ministry of Social Development (SEDESOL) in Mexico. This paper was written while she was a consultant at the World Bank; her e-mail address is gmrubio@sedesol.gob.mx. The authors are grateful to Orazio Attanasio, Jeanine Braithwaite, Christopher Chamberlain, David Coady, Ariel Fiszbein, Kathy Lindert, John Maluccio, Monica Orozco, Emmanuel Skoufias, Miguel Székely, Andrea Vermehren, and three anonymous reviewers for com- ments received and to Tania Gomez for editorial assistance. 1. In March 2002, Progresa changed its name to Oportunidades and changed some of its objectives and operational features. 2. A review of World Bank projects from 1998 to 2000 analyzing the quality of impact evaluation plans incorporated into the project appraisal process found that only 10 percent of projects had ade- quate plans for a rigorous impact evaluation, though the percentage of projects that included them had doubled over these years (World Bank 2001b). 3. See Kim and others (1999) for a review of Pakistan’s primary school female fellowship program; Ravallion and Wodon (2000) for an assessment of Bangladesh’s Food for Education program; Kandkher and others (2003) for an evaluation of the Bangladesh Female Stipend program; Yap and others (2001) for a discussion of Brazil’s PETI program aimed at reducing child labor; and World Bank (2001a) for a review of Brazil’s Bolsa Escola education stipend program. 4. This section draws on Rawlings (2004). For a more in-depth description of conditional cash trans- fer programs, see Ilahi and others (2000), Legovini and Regalia (2001), and Morley and Coady (2003). 5. Nicaragua initially stipulated that families would lose their grant if malnourished children did not gain adequate weight, but this requirement was dropped after the first year. 6. For example, to increase its coverage of rural areas, Mexico’s Progresa expanded in phases from August 1997 to early 2000. Nicaragua’s RPS started with a two-year pilot phase in two departments (Madriz and Matagalpa), whereas in Honduras implementation of PRAF was limited to a subset of municipalities because of funding stringencies. 7. Nicaragua’s RPS has completed follow-up measurements after one year and two years of program implementation and plans to conduct a third once demand incentives are eliminated and only the supply intervention remains. Mexico’s Progresa collects six rounds of panel data in rural areas every six months. Evaluators of Honduras’s PRAF planned to follow up after one and two years of program implementation. Colombia’s FA will have baseline data available for about half of the treatment sample (table 4). 8. Although not strictly part of the evaluation, censuses were conducted in the evaluation areas in some countries. In Mexico, a census collected data to determine household eligibility. In Honduras and Nicaragua, censuses generated a beneficiary registry and a household list from which to draw a repre- sentative sample in treatment and control areas and provided information for simulating inclusion and exclusion errors resulting from a proxy-means test targeting mechanism. 9. As Skoufias (2001) discusses, the use of the Progresa eligibility variable for program evaluation allows the evaluators to estimate the “intention to treat� effect. To the extent that not all eligible house- holds actually receive program benefits, the intention to treat effect underestimates the program mean effect on actual program beneficiaries. 10. For a comprehensive discussion of the education impacts see Schultz (2000a, b, c), Behrman and others (2000), and IFPRI (2002b). References Attanasio, Orazio, and others. 2002. “Evaluación de Impacto del Programa Familias en Acción: Primer Informe de Avance.� Institute for Fiscal Studies, London. Attanasio, Orazio, and others. 2003. “Baseline Report on the Evaluation of Familias en Acción.� Insti- tute for Fiscal Studies, London. Laura B. Rawlings and Gloria M. Rubio 53 Behrman, Jere, and John Hoddinott. 2000. “An Evaluation of the Impact of Progresa on Pre-School Child Height.� International Food Policy Research Institute, Washington, D.C. Behrman, Jere R., and Petra E. Todd. 1999. “A Report on the Sample Sizes Used for the Evaluation of the Education, Health and Nutrition Program (Progresa) of Mexico.� International Food Policy Research Institute, Washington, D.C. Behrman, Jere, P. Sengupta, and P. Todd. 2000. “The Impact of Progresa on Achievement Test Scores in the First Year.� International Food Policy Research Institute. Washington, DC Coady, David. 2000. “The Application of Social Cost-Benefit Analysis to the Evaluation of Progresa.� International Food Policy Research Institute, Washington, D.C. FISE (Fondo de Inversión Social de Emergencia). 2001. “Información General del Programa.� Red de Protección Social, ABC no. 1, Managua, Nicaragua Freeman, Howard E., and Peter Rossi. 1993. Evaluation: A Systematic Approach. Newbury Park, Calif.: Sage. Gertler, Paul J. 2000. “Final Report: The Impact of Progresa on Health.� International Food Policy Research Institute, Washington, D.C. Grossman, Jean. 1994. “Evaluating Social Policies: Principles and U.S. Experience.� World Bank Research Observer 9(2):159–80. Hoddinott, J., E. Skoufias, and R. Washburn. 2000. “The Impact of Progresa on Consumption: A Final Report. September.� International Food Policy Research Institute, Washington, D.C. IFPRI (International Food Policy Research Institute). 2000a. “Second Report: Implementation Proposal for the PRAF/IDB Project—Phase II.� Washington, D.C. ———. 2000b. “Third Report: Monitoring and Evaluation System PRAF.� Washington, D.C. ———. 2001a. “Informe: Sistema de Evaluación de la Fase Piloto de la Red de Protección Social de Nicaragua.� Washington, D.C. ———. 2002. “Informe Final—Sistema de Evaluación de la Fase Piloto de la Red de Protección Social de Nicaragua: Evaluación de Impacto.� Washington, D.C. Ilahi, Nadeem, Guilherme Sedlacek, and Emily Gustafsson-Wright. 2000. “Targeted Conditional Transfer Programs in Latin America: An Early Survey.� World Bank, Office of the Chief Economist, Latin America and the Caribbean Region, Washington, D.C. Kandkher, Shahidur, Mark Pitt, and Nobuhiko Fuwa. 2003. “Subsidy to Promote Girls’ Secondary Education: The Female Stipend Program in Bangladesh.� World Bank, Poverty Reduction and Economic Management Group, Washington, D.C. Kim, Jooseop, Harold Alderman, and Peter F. Orazem. 1999. “Can Private School Subsidies Increase Enrollment for the Poor? The Quetta Urban Fellowship Program.� World Bank Economic Review 13(3):443–65. Legovini, Arianna, and Ferdinando Regalia. 2001. “Targeted Human Development Programs: Investing in the Next Generation.� Inter-American Development Bank, Sustainable Development Department, Poverty and Inequality Advisory Unit, Washington, D.C. Morley, Samuel, and David Coady. 2003. “From Social Assistance to Social Development: A Review of Targeted Education Subsidies in Developing Countries.� International Food Policy Research Institute, Washington, D.C. Ravallion, Martin. 2003. “Targeted Transfers in Poor Countries: Revisiting the Trade-offs and Policy Options.� World Bank, Development Economics Research Group, Washington, D.C. Ravallion, Martin, and Quentin Wodon. 2000. “Does Child Labor Displace Schooling? Evidence on Behavioral Responses to an Enrollment Subsidy.� Economic Journal 110:158–75. 54 The World Bank Research Observer, vol. 20, no. 1 (Spring 2005) Rawlings, Laura. 2004. “Conditional Cash Transfer Programs: Experimenting with Service Delivery Alternatives.� World Bank, Latin America and Caribbean Region, Human Development Department, Washington, D.C. Schultz, T. Paul. 2000a. “Final Report: The Impact of Progresa on School Enrollments.� International Food Policy Research Institute, Washington, D.C. ———. 2000b. “Impact of Progresa on School Attendance Rates in the Sampled Population� Inter- national Food Policy Research Institute, Washington, D.C. ———. 2000c. “School Subsidies for the Poor: Evaluating a Mexican Strategy for Reducing Poverty.� International Food Policy Research Institute, Washington, D.C. Skoufias, E. 2001. “Progresa and Its Impacts on the Human Capital and Welfare of Households in Rural Mexico: A Synthesis of the Results of an Evaluation by IFPRI.� International Food Policy Research Institute, Washington, D.C. World Bank. 2001a. “Brazil: An Assessment of the Bolsa Escola Programs.� Human Development Department, Latin America and Caribbean Region, Washington, D.C. ———. 2001b. “Poverty Reduction and the World Bank: Progress in Fiscal Year 2000 and 2001.� Washington, D.C. ———. 2001c. “Project Appraisal Document: Social Safety Net Project� Washington, D.C. ———. 2001d. “Project Appraisal Document: Social Risk Mitigation Project for Turkey�. 22510-TU. Washington, D.C. ———. 2002. “Conditional Cash Transfer Programs: Operational Experiences Workshop.� Washington, D.C. Yap, Yoon-Tien, Guilherme Sedlacek, and Peter Orazem. 2001. “Limiting Child Labor through Behavior- Based Income Transfers: An Experimental Evaluation of the PETI Program in Rural Brazil.� World Bank, Washington, D.C. Laura B. Rawlings and Gloria M. Rubio 55