NOTES AGRICULTURAL & RURAL DEVELOPMENT 37040 Monitoring and Evaluating the Poverty Impacts of Agricultural Water Investments ISSUE 15 JUNE 2006 BY: THE WATER FOR FOOD TEAM Agricultural water projects contribute in several ways collection capacity, collection of the right type of data, to achieving the Millennium Development Goals of and a system set-up that allows analysis and interpre- eradicating extreme poverty and hunger and ensuring tation. Broad consultations and early user involvement environmental sustainability. Increased yields and in the design and implementation of the M&E system cropping area and shifts to higher-value crops help are important to build consensus and ownership. boost the incomes of farm households, generate employment, and lower consumer food prices. They POTENTIAL AREAS also stabilize incomes and employment. Community OF INVESTMENT participation and the creation of water user groups have become integral parts of these projects, which M&E systems are usually based on four steps: have empowered users and made them self-reliant. 1. Deciding what information is needed Mainstreaming monitoring and evaluating (M&E) will 2. Assessing the availability and requirements of tools help generate the data to establish the cost-effective- for collecting and analyzing the data needed ness of projects in reducing poverty and propose ways 3. Deciding on outputs of the M&E system, who will to improve it. produce them, and how they will be used 4. Determining the resources needed to set up and run Monitoring and evaluating (M&E) the poverty impact the M&E system of agricultural water investments may allow some conclusions about returns on different types of invest- Each step of M&E agricultural water projects should ments and their contributions to poverty reduction. focus on poverty, but the first two steps are especially M&E of the poverty impact should receive attention important: defining the indicators (the types of data) early in project preparation to ensure adequate data and the procedures for analyzing and evaluating them. INDICATORS. In addition to the standard indicators for inputs (resources assigned to project activities) and outputs (for example, the length of irrigation canals upgraded or built), outcome and impact indicators are needed to monitor welfare dimensions such as health status, consumption, and income levels (Box 1). These data, collected for different groups of project beneficiaries, measure which target groups receive the most benefits and these groups' satisfaction or dissat- isfaction with the benefits. Beneficiary groups may be distinguished according to income status (sometimes proxied by landholding status, if adequate data on income status are unavailable), ethnicity, indigeneity, and gender. Good assessments have also collected indicators on interventions complementary to agricul- tural water­related projects (such as the establishment of marketing infrastructure and processing facilities) Box 1: Some Outcome and Impact through a combination of household and individual sur- Indicators for Monitoring Poverty in veys and participatory methods. To allow statistically sig- Agricultural Water Projects nificant inferences from the data, the sampling framework must be appropriate. Surveys have been done in-house by Outcome indicators implementing agencies, but they are frequently contracted · Crop yields, cropping patterns, and out to a research or survey firm or an institution. This approach is generally preferred. A rigorous "with/without" ··· Output and input prices production levels design may be justified to evaluate the impact of an invest- Fisheries and livestock production ment that is critical to poverty reduction but which suffers Employment rates and wages from substantial knowledge gaps about which approach- es work best or when a new approach should be tested. Impact indicators With/without evaluation helps determine whether reduc- · Share of population below the nationally estab- tions in poverty result from project interventions or other lished rural poverty line or share of population causes. Controlled impact evaluations are demanding in terms of the analytical capacity and resource requirements, · Prevalence of underweight and stunted children with less that $US1 a day pre- and postproject and not all investments warrant them. (measured by height for age and weight-for- height) pre- and postproject. PROCEDURES FOR ANALYZING AND EVALU- ATING MONITORING AND EVALUATING Source: Mona Sur, World Bank. (M&E) OUTPUT. Initial planning for M&E includes developing the management information system. It and have monitored the quality of services affecting the should allow disaggregation of key data by social and impact of investments. Because projects do not take economic groups to allow monitoring of the poverty place in isolation, the inclusion of quantitative and qual- impact of activities. It should also enable an assessment itative information on intervening and/or external influ- of the inclusiveness of project activities. ences on the selected indicators is recommended. As shown in box 1, one would include anything else that Emphasis on poverty M&E early in project preparation might influence the output/input prices or the preva- can ensure that data collection capacity is adequate, lence of child malnutrition. that the right type of data is collected, and that a sys- tem is set up that allows analysis and interpretation. TYPES OF DATA. Well-designed baseline surveys are an Broad consultations and early user involvement in the essential tool for collecting data for investment planning design and implementation of the M&E system are purposes, monitoring, and evaluation. Poverty maps have important to build consensus and ownership. also proved valuable tools for targeting and monitoring poverty (Box 2). Depending on the project interventions, Early involvement of potential data users (typically the baseline surveys usually collect data at several levels (such project implementing agency) and broad consultations as the village, watershed, household, plot, and individual) with researchers, beneficiaries, donors, and imple- Box 2: Poverty Maps Poverty maps are spatial representations of poverty assessments.They combine survey with census data and graphically present indicators of poverty (such as per capita income or daily subsistence levels) or well- being indicators (such as life expectancy, child mortality, and literacy). In the Peruvian Social Fund project, FONCODES, poverty maps, in conjunction with community poverty assessments, helped target community-based projects, including small irrigation projects. Superimposing remote sensing data (publicly available satellite images) with poverty maps provides enormous scope for better water resources planning, poverty targeting, and impact assessments. Sources: Mona Sur, World Bank; World Bank 1998. 2 menters during the initial design stage have been help- tems set up for M&E. For example, conducting a full- ful in building consensus on what to monitor and how fledged household survey to collect data on welfare to do it, and in generating a sense of ownership among measures may be costly, but alternative rapid assess- the various stakeholders. ment methods could provide a more affordable alter- native for measuring poverty impacts. Alternatives RESOURCE REQUIREMENTS. The resources need- include approaches such as collecting "core welfare" ed for poverty M&E vary from project to project, indicators (typically assets) that help trackchanges in depending on the scope of project activities and sys- consumption and income. Box 3: India: An Example of an Impact Evaluation of Watershed Development Projects An example of an impact evaluation that also attempted to examine the poverty impact of agricultural water-related projects is a study by the International Food Policy Research Institute (IFPRI) evaluating watershed development projects in India. The primary objectives were to assess the following: (1) Which watershed projects were most successful in raising agricultural productivity, improving natural resources management, and reducing poverty? (2) What approaches enabled projects to succeed? (3) What nonpro- ject factors contributed to achieving these objectives? The study evaluated projects funded and imple- mented by several donors and state governments, including the World Bank. The study used mainly quantitative analysis, but also drew on qualitative information about the effects on interest groups (such as farmers with, and without, irrigation, landless people, shepherds, and women). The study used a nonexperimental design,relying on an instrumental variable approach to correct for the endo- geneity of program placement. Instrumental variables were first used to predict program participation, and then the variation of outcome indicators with predicted values of program participation was examined. The study covered a 10-year period and relied on baseline survey data from theWorld Bank and the Indian Council of Agricultural Research villages.The postproject situation was captured through a 1997 survey of 86 villages in Maharashtra and Andhra Pradesh conducted by IFPRI.The survey collected quantitative data at the village, plot, and household level for econometric analysis of the conditions that determined changes before and after the project.Qualitative information on project impacts was collected from interest groups through open-ended discussions. Control villages were selected and roughly matched geographically (data were insufficient for rigorous matching). The authors measured the impact on household welfare through proxy indicators including the perceived effects of the project on the household, perceived changes in liv- ing standards, changes in housing quality, change in percentage of families migrating, perceived changes in real wage, and the availability of casual employment opportunities. Lack of adequate baseline and moni- toring data in the projects was a major limitation to the study. The study found that participatory projects performed better than top-down approaches. Projects where participation was combined with sound technical inputs performed best of all.The authors also found that equity issues remained a problem and respondents perceived that project benefits rose with landholding size, and that the landless and near-landless people were most likely to report negative effects from proj- ects. For example, across projects while 45 percent of households with 2 or more hectares claimed they had benefited from watershed projects, only 12 percent of the landless and 19 percent of farmers with less than 1 hectare claimed to have benefited. On the other hand, 19 percent of the landless felt that they were harmed by the projects compared to 7 percent of larger farmers.The authors also found that projects run by nongovernmental organizations (NGOs) were associated with higher net returns; however, the econo- metric results for the model were insignificant. Source: Kerr, Pangare, and Pangare 2002. 3 POTENTIAL BENEFITS the reasons behind results observed in a quantitative analysis. A special attempt can be made to monitor the Agricultural water-related projects that monitor poverty satisfaction of the poor with the project through focus impacts have improved poverty targeting and tailored group or other qualitative methods. activities to maximize benefits during implementation. They have made end-of-project assessments of the returns and impacts from different types of investments, RECOMMENDATIONS FOR allowing more cost-effective planning and implementa- PRACTITIONERS tion of future investments. · Involve beneficiaries early in the M&E process. Broad IMPLEMENTATION consultations and early involvement of users in the design and implementation of the M&E system will do CONDUCTING AN IMPACT EVALUATION. The much to build consensus and ownership. · Use beneficiary self-assessments and other participato- objective of an impact evaluation is to measure the results of the project interventions on dimensions of ry approaches so that assessments can be made mid- poverty (Box 3). Establishing causality is important and course. There may be a long time lag in realizing a necessary for impact evaluation. To do so requires iden- reduction in poverty from agricultural water-related tifying a comparison or control group that does not projects, and the poverty impact often cannot be eval- receive the project intervention and comparing uated until well after a project ends. this group to the treatment group. The control group · Keep the number of indicators collected within man- must match the treatment group in terms of its socioe- ageable proportions. conomic aspects and the physical characteristics of · Ensure that a good baseline survey is undertaken. its site. Without a proper baseline, it is difficult to monitor progress and evaluate the impacts of investments. Experimental or quasi-experimental designs are generally used to effectively show the results of project interven- REFERENCES CITED tions on poverty. In experimental design, the intervention is allocated randomly among all eligible beneficiaries. Baker, J. L. 2000. Evaluating the Impacts of Development Projects on Poverty: A Handbook for Practitioners. Washington, DC: Experimental designs need to be set up prior to the World Bank. investment. Quasi-experimental designs, in contrast, attempt to generate control groups after the intervention Government of Karnataka. 2003. Operation Manual: Sujala by means of statistical and econometric methods, such as Watershed Project. Bangalore: Government of Karnataka Watershed Development Department. propensity score matching, computing double differ- ences, or using instrumental variables (Baker 2000). Kerr, J., G. Pangare, and V. Pangare. 2002. "Watershed Development Projects in India: An Evaluation." IFPRI Research QUANTITATIVE VERSUS QUALITATIVE DATA: Report No. 127. International Food Policy Research Institute, Integrating qualitative and quantitative approaches in Washington, DC. monitoring and evaluating poverty impacts has proven World Bank. 1998. "Implementation Completion Report." Peru very effective. Quantitative analysis results in more gen- Social Development and Compensation Fund Project, Report eralizable results, but qualitative and participatory meth- No. 18016. World Bank, Washington, DC. ods allow in-depth study of selected issues, cases, or World Bank. 2005. Shaping the Future of Water for Agriculture: events, and can provide critical insights into beneficiar- A Sourcebook for Investment in Agricultural Water ies' perspectives, the dynamics of a particular reform, or Management. Washington, DC: World Bank. This Note was prepared by Mona Sur, a Senior Economist at the World Bank. It was revised by Salah Darghouth, Water Adviser for the Agriculture and Rural Development Department (ARD) of the World Bank, and Ariel Dinar, Lead Economist in ARD at the World Bank ­both part of the Water For Food Team. The note is based on Investment Note 9.1 in the larger volume Shaping the Future of Water for Agriculture: A Sourcebook for Investment in Agricultural Water Management. The book documents a range of solutions and good prac- tices from World Bank and worldwide experience, concentrating on investments in policy and institutional reforms in technology and management to improve water productivity and farming profitability. You can download a copy of the full report at www.worldbank.org/rural or email ard@worldbank.org. THEWORLDBANK 1818 H Street. NW Washington, DC 20433 www.worldbank.org/rural