72693 Impact Evaluations in Agriculture AN ASSESSMENT OF THE EVIDENCE Impact Evaluations in Agriculture An Assessment of the Evidence 2011 The World Bank Washington, DC i © 2011 The International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved 1 2 3 4 5 14 13 12 11 This volume is a product of the staff of the Independent Evaluation Group of the World Bank Group. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. This volume does not support any general inferences beyond the scope of the evaluation, including any inferences about the World B ank Group’s past, current, or prospective overall performance. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. How to cite this report: IEG (Independent Evaluation Group). 2011. Impact Evaluations in Agriculture: An Assessment of the Evidence. Washington, DC: World Bank. Cover image: Indian women showing handfuls of their agricultural harvest. Photo by Simone D. McCourtie, courtesy of the World Bank Photo Library. ISBN-13: 978-1-60244-168-2 ISBN-10: 1-60244-168-5 World Bank InfoShop Independent Evaluation Group E-mail: pic@worldbank.org Strategy, Communication, and Learning (IEGCS) Telephone: 202-458-5454 E-mail: ieg@worldbank.org Facsimile: 202-522-1500 Telephone: 202-458-4497 Facsimile: 202-522-3125 ii Contents Abbreviations ............................................................................................................................ v Preface .....................................................................................................................................vii Summary...................................................................................................................................ix 1. Introduction ........................................................................................................................... 1 The Report and Analytical Questions .............................................................................. 1 Conceptual Framework Guiding the Impact Evaluation Search ...................................... 2 Selecting the Group of Impact Evaluations for Review ................................................... 4 2. Impact Evaluation Overview ................................................................................................. 6 Impact Evaluation Characteristics .................................................................................... 6 Challenges Facing Users of Impact Evaluations in Agriculture ...................................... 9 3. Inside the Interventions ....................................................................................................... 13 Introduction to the Taxonomy of Interventions .............................................................. 13 Characteristics of the Interventions ................................................................................ 15 Land Tenancy and Titling ........................................................................................ 15 Extension Interventions ............................................................................................ 17 Irrigation Interventions ............................................................................................. 20 Improved NRM Technology Interventions .............................................................. 22 Input Technology Interventions ............................................................................... 24 Marketing Interventions ........................................................................................... 26 Microfinance Interventions ...................................................................................... 29 Other Interventions: Infrastructure and Safety Nets ................................................. 31 4. What Does the Evidence Reveal?........................................................................................ 34 Primary Constraints for Farmers .................................................................................... 34 Summary of the Results Reported by the Impact Evaluations ....................................... 36 Results by Agricultural Indicator ............................................................................. 36 Results by Intervention Type .................................................................................... 37 Results by Agricultural Constraint ........................................................................... 40 Lessons by Design and Implementation Characteristics ................................................ 41 Some Relevant Intervention Design Features .......................................................... 41 Some Relevant Implementation Features ................................................................. 42 5. Conclusion ........................................................................................................................... 44 Appendixes A: Table of Impact Evaluations Analyzed ....................................................................... 45 B: The Analysis ................................................................................................................ 71 C: Types of Evaluation Design and Methodologies ......................................................... 77 Bibliography ............................................................................................................................ 79 iii Boxes 1.1 Concepts Underlying the Impact Evaluation Literature Search and Analysis ................... 3 2.1 Overcoming Some of the More Critical Challenges ........................................................ 10 3.1 Impact Evaluations in Agriculture at the World Bank ..................................................... 15 3.2 Interventions That Delivered Mixed Results on Productivity or Income ......................... 16 3.3 Relevance of the Information Determines the Incentives to Take Up the Project ........... 18 3.4 Irrigation Interventions ..................................................................................................... 21 3.5 Natural Resource Management Technology .................................................................... 23 3.6 Clear Synergies between Input Technology and Extension Services .............................. 24 3.7 Context Can Lessen Potential Impact of Marketing Interventions .................................. 27 3.8 Microfinance Services beyond Credit Can Increase Gains .............................................. 30 3.9 Evaluations Listed as ―Other‖ Include Roads and Safety Net Programs ......................... 32 4.1 Interventions That Delivered Mixed Results on Productivity or Income........................ 38 Figures 1.1 Impact Pathway of Agricultural Interventions ................................................................... 2 1.2 Process of Selecting the Group of Impact Evaluations ...................................................... 5 2.1 Profile of IEs....................................................................................................................... 8 4.1 Intervention Categories and Primary Constraints............................................................. 36 Tables 2.1 Advantages and Disadvantages of Various Evaluation Designs and Methodologies ..................................................................................................................... 6 2.2 Examples of Direct and Indirect Indicators Measured ....................................................... 9 3.1 Intervention Types Included in the Impact Evaluation Group ......................................... 14 3.2 Land Tenancy and Titling Interventions: Observations from the Reviewed Literature .......................................................................................................... 17 3.3 Extension Interventions: Observations from the Literature ............................................. 20 3.4 Irrigation Interventions: Observations from the Literature .............................................. 22 3.5 NRM and Non-NRM Interventions: Observations from the Literature ........................... 25 3.6 Marketing Interventions: Observations from the Literature ............................................. 29 3.7 Microfinance Interventions: Observations from the Literature ........................................ 31 3.8 ―Other‖ Interventions: Observations from the Literature ................................................. 33 4.1 Constraints Addressed by Evaluated Agricultural Interventions ..................................... 34 4.2 Results by Agricultural Indicator ..................................................................................... 35 4.3 Results by Intervention Category ..................................................................................... 38 4.4 Results by Constraint ........................................................................................................ 40 iv Abbreviations Bt Bacillas thuringiensis (cotton) CDD Community-driven development FFS Farmer field school IEG Independent Evaluation Group IFAD International Fund for Agricultural Development IFPRI International Food Policy Research Institute NRM Natural resource management OECD Organisation for Economic Co-operation and Development PSNP Productive Safety Nets Program v Preface This report was prepared by Ximena V. Del Carpio (Task Team Leader) and Mywish Maredia, with inputs from Gayatri Datar, Pamela Velez-Vega, and Andrew Warner. Kristian Lopez and Kurt Waldman assisted the team, and many colleagues inside and outside the Independent Evaluation Group (IEG) and the World Bank provided helpful comments. The work was conducted under the general guidance of Cheryl Gray (Director) and Mark Sundberg (Manager). The team is grateful for the extensive and excellent advice provided by four peer reviewers: Boris Bravo-Ureta, Michael Carter, Elisabeth Saudoulet, and Maximo Torero. Yezena Yimer and Diana Hakobyan provided administrative support. William Hurlbut edited the report. vii Summary Agriculture projects often focus on improving productivity and farm income to reduce poverty, curtail hunger, and promote environmental sustainability. Over the past decade, growing attention to aid effectiveness has increased emphasis on demonstrating measurable impacts on people and their environment. This interest in learning what is effective in promoting agricultural development has created a challenge for evaluation research and practice, as well as an opportunity to gain knowledge from a relatively young but growing body of impact evaluation work. This Independent Evaluation Group (IEG) meta-analysis examines the results of agricultural impact evaluations around the world—by diverse individuals and groups, including the World Bank—that met standard criteria for design and rigor. The report describes the state of impact evaluation literature in agriculture, provides a taxonomy of agricultural interventions to organize results, and examines discernable performance patterns for lessons to inform the design of future interventions. Worldwide, few evaluations of agricultural interventions use a counterfactual to measure change resulting from the intervention—the majority rely on quasi-experimental or nonexperimental methods. Moreover, the heterogeneity of intervention designs, implementation strategies, and outcome indicators makes it difficult to aggregate results or derive broad conclusions about which interventions work best and under what circumstances. Consequently, the only common denominator for comparing results is whether an intervention has a positive impact on the targeted outcomes. More than half of all intervention types covered in the analysis had positive impacts on various agricultural outcome indicators (mostly yields, income, or input use). Interventions that sought to improve yields or farm income by addressing market-linkage failures, easing access to technologically enhanced inputs, and promoting farmer knowledge through advisory services had the highest share of positive impacts. Impact evaluations of World Bank-supported interventions accounted for one-quarter of the evaluations covered by the analysis; of these, two-thirds had positive impacts on various agricultural dimensions. Land titling and extension services were the most common of these interventions. Objectives of the Study The study collects and analyzes lessons derived from a body of rigorous impact evaluations to begin to discern what has been effective in agricultural work. The impact evaluations analyzed were produced from 2000 to early 2009. The study is part of a broader effort by IEG to understand how impact evaluations can help improve performance and broadly disseminate lessons of experience. The study has three objectives: 1. Assess the current state of impact evaluations in the agriculture sector and highlight challenges to discerning what works best in the sector. ix 2. Derive a taxonomy of interventions that have been evaluated using IE methodology, and identify the most common constraints faced by farmers and addressed by the interventions assessed by the selected IEs. 3. Summarize what can be said about the impact of different interventions on individuals, households, or communities, focusing primarily on productivity and farm income. Future research priorities are also identified. The State of the Impact Evaluation Evidence in Agriculture Each IE has unique characteristics requiring different approaches and methods; however, some general qualities considered good practices guided the process used to select the IEs included in the analysis. Those good practices—mostly based on having credible strategies for measuring impact, plus a focus on farm performance (productivity, production, income— were used to determine which evaluations were selected for the analysis. Of 271 agriculture evaluations identified in the literature, only 86 met the criteria for inclusion. A large number of evaluations were eliminated for lack of a valid counterfactual, lack of a defined intervention, or lack of clear farm productivity indicators. The low number of IEs relative to the number of agriculture projects worldwide limits the investigation. However, the number of IEs has been growing—for example, 7 were produced in 2002 and 23 in 2008. This suggests that the analysis in this study can be expanded in the future. Despite the limitations of the analysis, some useful lessons emerge. A PROFILE OF IMPACT EVALUATIONS Of the 271 evaluations identified initially, 50 were excluded because they did not refer to a specific country. In the final group of 86 selected IEs, 39 percent were for projects in Africa. East Asia accounted for 22 percent, South Asia for 20 percent, and Latin America and the Caribbean for 17 percent. The final group includes no IEs for the Middle East and North Africa, and few from Eastern Europe and Central Asia. The largest number of IEs covered land or extension interventions. Other intervention types included market arrangements, irrigation, natural resource management, input technology, and microfinance. Experimental designs, such as randomized control trials, were rarely used to evaluate agricultural interventions (only 6 percent of the 86 IEs). This is in part because not all projects are amenable to such designs. About 44 percent of the selected IEs used quasi- experimental designs to construct plausible counterfactual groups. Many of these IEs were designed after the intervention began or implementation was completed (ex post). All land reform, natural resource management (NRM), and irrigation interventions used quasi- experimental or nonexperimental evaluation approaches. The majority of interventions evaluated were planned and implemented in the 1990s, and the time elapsed between the intervention and the IE ranged from less than six months to more than two decades. However, only a handful of IEs addressed long-term effects or sustainability concerns. x Taxonomy of Interventions and Primary Constraints for Farmers INTERVENTIONS IDENTIFIED IN THE ANALYSIS The interventions covered by agriculture IEs can be grouped into eight categories, as described below. The number in parentheses is the percentage of each type:  Land tenancy and titling (19 percent). These interventions aim to facilitate access to farm credit and promote land markets. Most IEs reviewed in this category reflected a change in law or enforcement of property rights traditions. The category includes titling (10 percent), tenancy law (8 percent), and inheritance law (1 percent).  Extension services (20 percent). These interventions seek to improve farmer knowledge and promote improved technologies and practices. They cover farmer field schools (8 percent), technical or advisory services (9 percent), and access to market information (3 percent).  Irrigation (9 percent). These interventions seek to affect productivity and farm income through improved water availability and management. They include access to water infrastructure (2 percent), management systems (3 percent), and dams (4 percent).  Natural resource management (14 percent). NRM interventions seek to improve farmer knowledge and adoption of new technologies and conservation techniques. They include soil and water conservation (9 percent), systems of crop management (4 percent), and integrated aquaculture-agriculture (1 percent).  Input technology (9 percent). These focus on the development and adoption of improved crop varieties, improved seed technology, and innovative fertilizer application techniques. The category includes improved seeds (8 percent) and fertilizer (1 percent).  Marketing arrangements (15 percent). These interventions promote linkages with buyers and sellers and provide incentives for group formation and social learning. The category includes contracts and related items (9 percent), interlinked credit-input- output arrangements (3 percent), cooperatives and social learning (3 percent).  Microfinance (7 percent). These interventions refer to the provision of small-scale financial services to farmers, from cash grants to credit-related advisory services. Objectives range from increased access to credit to consumption increases (poverty reduction). The category includes access to financial services (5 percent) and rural noncredit or insurance (2 percent)  Other (9 percent). These interventions (and their evaluations) are too varied for a single definition, but they all seek to improve farmer economic well-being and, in a few cases, agricultural performance. There are at least three types: rural roads or infrastructure (5 percent), community-driven development (2 percent), and safety net programs (2 percent). xi FARMERS’ CONSTRAINTS IDENTIFIED FROM THE IMPACT EVALUATIONS The IEs identified six primary constraints, and every intervention sought to ease at least one of them in order to improve agricultural performance. The constraints are low quality of inputs, limited farmer knowledge, low quality of land or physical farm resources (including water), a nonconducive policy environment, limited output sales or commercial assistance, and limited access to credit or insurance. The six constraints make it possible to group interventions with similar cause-and-effect relationships to discern patterns. This is particularly useful for interventions with elements that cut across categories and address more than one constraint (such as natural resource management, technology, and extension programs). CHALLENGES TO LEARNING FROM IMPACT EVALUATIONS IN AGRICULTURE A proper meta-analysis is the analysis of a large collection of individual studies for the purpose of integrating the findings. It goes beyond a literature review in two ways. First, it includes all the studies that meet the review criteria and is thus comprehensive. It provides a basis for understanding why evidence of impacts differs among studies, over time, and among types of interventions. Second, with a large sample a meta-analysis can make use of statistical techniques for amalgamating, summarizing, and reviewing quantitative research to overcome limits of size or scope in individual studies and obtain more reliable information about the impact of a treatment. Because of these advantages meta-analysis has become increasingly popular in recent decades and has been applied with increasing frequency, especially with randomized controlled trials, in health, medicine, and psychology where randomized controlled trials are the research norm. However, several features of IEs in agriculture inhibit the use of a proper meta-analysis. One constraining factor is the heterogeneity of interventions and specifications used in the IEs. This poses a special challenge when attempting to synthesize results across interventions using common and comparable units of explanatory and impact variables, thus limiting the depth of analysis. For example, not all IEs of land reform interventions deal with the same policy change or use the same units of measurement. Similarly, interventions grouped under extension or irrigation do not use the same implementation approach or measure the same development indicators. Second, while there are IEs in agriculture with well-specified counterfactuals, they are scarce and employ different approaches, making rigorous meta-analysis difficult. The scarcity of IEs is most pronounced for farm-level interventions seeking to enhance farm use through improved water access and interventions seeking to improve the quality or access to inputs such as seeds or fertilizer technology. Similarly, IEs measuring the effectiveness of value- chain arrangements and microfinance interventions directed to farmers are relatively scarce. However, the evaluation search identified several ongoing IEs in both categories. A third constraint to analysis is the lack of financial and economic costs in the impact evaluations. Even though IE methods and cost-benefit analysis can complement each other, projects rarely use them together. Less than 10 percent of the selected IEs reported this information, indicating either a lack of data or of evaluator interest (or skill) in measuring xii project costs along with impacts. This omission makes it impossible to compare the cost effectiveness or the magnitude of impacts across different intervention results. Finally, few IEs explored distributional impacts on important dimensions—poor and less poor or female- and male-headed households. Most IEs reported average impacts on the targeted groups rather than subgroups based on different socioeconomic characteristics. Impacts of Interventions on Agricultural Performance Given the heterogeneity of measurement and specifications across impact evaluations, the only common denominator for comparison is the sign of the impact. Impact magnitudes vary widely across country and intervention context, which seriously limits the conclusions one can make on the basis of aggregated evidence. While the share of evaluations reporting a positive or negative impact is reported below, this is not a reliable guide to intervention success rates in general. The small number of studies, the potential for selection bias, and the focus here on only IE studies (not evaluations using other approaches) are limiting and must be borne in mind. More evidence is needed to determine wider applicability of the results. The study analyzed impacts of the Table S.1. Results by Agricultural Indicator interventions by particular indicators. The most common indicators measured in the Positive Nonpositive IEs were yields, defined as production or Indicator impact (%) impact (%) labor per total area of cultivated land (39 Yield 62 38 percent); income, defined as earnings from Income 42 58 all activities (24 percent); production, Production 56 44 defined as the amount of farm production cultivated and farmed (9 percent); and Profit 87 13 profit, defined as marginal gains or net benefits (sales minus costs) reported by farmers (8 percent). Table S.1 shows that, across different categories of indicators, interventions generally succeeded in improving yields, production, and profits in the studies surveyed. However, income was less commonly found to have been positively affected. Table S.2 summarizes the results by intervention category. Positive impact is defined as evidence of a positive relationship between the relevant indicator and the treatment. Nonpositive impact is defined as evidence that is negative or not statistically significant. About 41 percent of all interventions led to negative or nonsignificant impacts on farm yields, farm household income, or input adoption. Positive impacts (59 percent) were most evident for input technology interventions, where impacts were consistent across several agricultural indicators. IEs of land tenancy and titling interventions, for example, showed that about 65 percent of the evaluations found improvements on crop yield, value of production per hectare, agricultural profits, and household income across regions and types of reforms. For extension services, half of the interventions evaluated indicate a positive impact on beneficiary knowledge or farm yields. Review of extension intervention IEs found that impacts through a diffusion of knowledge modality (known as ―train the trainer‖) often failed to show a positive impact. Evaluations report that the main reason provided for the lack of demonstration effects was the inability of participant farmers to convey complex decision-making skills effectively to other farmers. xiii Table S.2. Results by Intervention Category Number of IEs showing impact on an agricultural indicator Not Nonpositive Intervention category Negative significant Positive impacts (%) Land reform 2 5 13 35 Extension 1 12 13 50 Irrigation 1 3 7 36 NRM 2 5 8 47 Input technology 2 1 8 27 Marketing 2 3 9 36 Microfinance — 3 6 33 Other — 5 4 55 Total 10 37 68 41 Over one-third of irrigation interventions evaluated showed nonpositive impacts on an agricultural or farm income indicator; negative or nonsignificant impacts pertain to micro- dams or water reservoirs rather than canals or other irrigation infrastructure or technology. The most common reasons for negative or nonsignificant impacts were losses due to water disruption and increased incidence of disease among farmers caused by large-scale impounding of water. The construction of a micro-dam in Ethiopia exemplifies how, despite production increases, labor productivity among male farmers decreased due to illness. For marketing interventions such as farm-group arrangements (for example, value-chain participation), results are mostly positive (64 percent) on yields, crop prices (and profits), and value of production. Positive results are primarily due to improvements in access to modern inputs, farming technology, and wider markets resulting from participating in value-chain or contract activities. A little more than half of the NRM interventions were found to positively affected farm yields or other agricultural indicators. Interventions that promoted technology use that subsequently changed the structure or composition of the soil had the Table S.3. Results by Constraint most positive results. Examples of Positive this include the construction of stone Primary constraint addressed impact (%) bunds in Burkina Faso and Ethiopia or a system of rice intensification Access and quality of inputs 61 that applies guidelines for spacing Farmer knowledge 61 and transplanting crops. Input Quality of land/farm resources 37 technologies, such as improved Policy environment 59 seeds, on the other hand, showed Output/post-production promotion and commerce 63 clear improvements on farm Access to credit and insurance 55 outcomes, with more than 70 percent Note: Some interventions address more than one constraint. positive impacts. xiv The evidence for microfinance and the ―other‖ category (which includes infrastructure and safety net interventions) shows distinct patterns of positive impacts on rural development outcomes related to farm performance (67 and 45 percent positives, respectively). Few of the evaluations of these interventions measured impacts on agricultural outcomes, and most concentrated on household income or consumption. The thinness and diversity of objectives of both categories made it difficult to derive many meaningful patterns. Interventions that sought to improve agricultural performance by addressing policy environment constraints, market-access constraints (output promotion), access to technologically enhanced inputs, and promotion of farmer knowledge had the most impact (table S.3). The analysis also found that most interventions contained an element of technological innovation, particularly those that dealt with input enhancement and crop intensification through resource management or water. Interventions that sought to enhance land quality—mostly focusing on improving soil conditions—had the lowest reported rate of success. Although microfinance interventions exhibited mixed results on their own, credit was an important complement to the success of interventions or components in value-chain and input-access interventions. World Bank Practice of IEs in Agriculture Twenty-six percent of the selected IEs assess World Bank-supported interventions. The two most common types of interventions evaluated were land tenancy and titling, and extension services (farmer field schools). No World Bank natural resource management and input technology interventions were evaluated and included in the IE group. To address some of the evidence gaps in agricultural work and contribute to the general knowledge base, the World Bank launched a new technical assistance program of Impact Evaluation for Improving Results in Agriculture in late 2009. There are at least six impact evaluations under way that, once completed, will help enrich agricultural IE evidence. World Bank-supported interventions (or components of these interventions) using IE methodology to measure results mostly concentrated on improving the policy environment (14 percent) and enhancing farmer knowledge (9 percent). The rest were more evenly distributed among improving access to farm inputs (6 percent), dealing with access to water (5 percent), and addressing credit or insurance access (5 percent). There was a notable absence of Bank interventions that addressed output or post-production promotion (1 percent) and adoption of new farm input technologies. The results of the Bank-supported IEs show mainly positive impacts. About 68 percent of the Bank-supported interventions led to positive change. The interventions measured a wide range of indicators related to farm performance. Of the indicators this report focused on, yield was the most common indicator reported in World Bank IEs and more often showed positive impacts. For IEs that measured impact on farm income the results were more mixed. xv Conclusion The review of rigorous IEs in agriculture over the nine-year period 2000–09 shows that the evidentiary base is still very thin and the heterogeneity of indicators and approaches prevents aggregation and firm conclusions about ―what works or doesn‘t work‖ in agricultural interventions. However, lessons at the micro level about the impact of specific interventions are still quite valuable. By mapping and organizing existing IEs, the report contributes to establishing an analytical framework for future investigation of what has been effective in agriculture. Moreover, as a larger evidence base accumulates across intervention clusters, this should help guide understanding of what works in different contexts and within intervention categories. With appropriate attention to context, it will also help guide future project design and investments. xvi 1. Introduction The Report and Analytical Questions In recent years the Millennium Development Goals have emerged as key objectives in guiding the planning and implementation of a broad range of global and national development efforts. The goals that relate to agriculture focus on reducing poverty and hunger and promoting environmental sustainability. The attention to these issues has been accompanied by increasing calls for accountability and an emphasis on results, outcomes, and actions that have a real impact on people and their environments. There is also a genuine interest in institutional learning—specifically, what interventions have proven effective in promoting agricultural development. This has created a challenge for impact evaluation research and practice, as well as an opportunity to learn from a growing body of impact evaluation work. This report seizes the opportunity to learn from existing evidence by analyzing lessons derived from impact evaluations produced between 2000 and January 2009—to begin to discern what has been effective in agriculture. It is part of a broader effort being undertaken by the Independent Evaluation Group (IEG) of the World Bank to understand how impact evaluations can help improve performance and broadly disseminate those lessons. Specifically, the report has three objectives: 1. Assess the current state of impact evaluations in the agriculture sector and highlight challenges that users face when trying to answer what works best in the sector. 2. Derive a taxonomy of agriculture interventions evaluated using impact evaluation methodology, and identify the most common constraints for farmers and addressed by those interventions. 3. Highlight what can be said about the impact of different interventions on agricultural outcomes (focusing on productivity and farm income). Also, point to areas for future research of agricultural interventions to broaden the use of this analysis. The rest of this chapter provides some necessary definitions for the report, outlines the conceptual framework, and presents a brief background on the selection of impact evaluations (IEs) from the evaluation literature. Chapter 2 provides a profile of the evidence, including IE characteristics and challenges encountered in the analysis. Chapter 3 looks within the interventions and presents a formal taxonomy of all agricultural interventions represented in the group analyzed for the report. Chapter 4 delves into the evidence by presenting the primary constraints dealt with in the interventions, the results reported by the evaluations, and some lessons that may be incorporated into future project design. Chapter 5 concludes with some general remarks. 1 Conceptual Framework Guiding the Impact Evaluation Search The report focuses on broadly defined interventions in agriculture (and rural development)1 that occur in the ―real world‖ and affect people, not laboratories or nonfarm settings. Figure 1.1 illustrates a simplified and generalized impact pathway of key outcomes sought in agricultural interventions. Those outcomes are at the heart of the IEs included in this analysis. Positive agricultural impacts, such as productivity, income, and profits, can lead to macro-level developmental improvements. The focus of this report is on evaluations that measure ―impact‖ in any of these indicators (dashed circle in figure 1.1) caused by the intervention; in some cases the analysis includes indirect indicators. Long-term changes in developmental goals and macrodevelopmental impacts (such as poverty reduction) cannot usually be attributed to a single intervention. Therefore, this report focuses on IEs of interventions to begin building a body of evidence that can eventually reveal a more comprehensive picture of the role of agriculture interventions on macro-level improvements.2 Figure 1.1. Impact Pathway of Agricultural Interventions Interventions Program goal Development (Agricultural projects, programs, activities, shocks) impacts Characteristics A Increase productivity, Poverty Income, profit (Direct Environment (Objectives, type of intervention, location, scale, indicators) Sustainability time frame, components) Health C Equity B Other goals (Indirect Nutritional security indicators) Focus of the Meta-analysis Arrows indicate the direction of influence, the thickness indicates the degree of influence A = Impact evaluations that directly establish a causal link to productivity B = Impact evaluations that indirectly establish a causal link to productivity C = Hypothesized link between impact indicators evaluated in type-B IEs and productivity Defining impact indicators—especially productivity, income, and profit—is not a straightforward exercise. There are many channels to improving farm performance, especially productivity, farm income, and profit. Also, there are various ways of defining and measuring outcome indicators, such as yields, depending on how the intervention design was done and the type of intervention being evaluated. Therefore, the report takes 1 This report excludes interventions that have weak links with agricultural outcomes and are not related to productivity or farm income, for example, interventions such as fortified foods that are developed to affect the health and nutritional status of a community. 2 Research of impacts on macro-level developments often use methods other than IE methodology (such as general equilibrium analysis or macroeconomic modeling using econometric tools and techniques) for measurement. Such research is outside the scope of this report. 2 a flexible approach in defining agricultural indicators, to include as many of the diverse types of agricultural interventions as possible in the analysis.3 Box 1.1. Concepts Underlying the Impact Evaluation Literature Search and Analysis Intervention: A project, program, policy, activity, or shock that causes an action meant to generate a change. Interventions are ―actions‖ whose impacts are evaluated by the IEs included in the analysis. They are characterized by various features, such as design, objectives, and implementation strategies. All interventions included in this analysis take place in real-world settings where social, economic, and political factors play a role in influencing their results. An intervention may have several subcomponents that together contribute to achieving its objectives. Impact evaluation: An assessment of changes in outcome indicators that can be attributed to a particular intervention. The IEs identified use many different methods and approaches to examine and establish a causal link between the intervention and its results. Common across all IEs are two interrelated challenges: establishing a viable counterfactual and attributing the impact to an intervention. Measuring impacts requires that a valid counterfactual be available to compare with the group treated by the intervention. The validity of the counterfactual is related to the attribution concern. Validity entails comparability between control and treatments groups, in all relevant dimensions, to ensure that outcome changes for the treatment groups are due solely to program participation. Economic welfare, with a focus on productivity and farm income:4 The standard productivity definition (ratio of agricultural outputs to inputs such as labor or land) is not strictly applied in this report. The term ―agricultural welfare‖ is all-encompassing. It includes increases in production, income, output marketing, profits, and decreases in per unit cost, as well as other indicators related to farm household welfare improvement. Interventions with a direct link to intermediate outputs, such as knowledge of new farming practices, are also included under this term. Note: Details on impact evaluation methods are available in Ravallion 2008. Indicators tangentially linked with changes in some measure of agricultural performance are also included. The indirect indicators (arrow C in figure 1.1) measure changes in intermediate outputs or outcomes that, if achieved, can lead to changes in direct indicators of agricultural performance. Examples of such indicators include changes in farmer knowledge on input use (such as pesticides), diversification of farm- level economic activities, improvements in rural infrastructure, changes in product procurement and sales prices, reduction of risks, access to credit, and the availability of 3 In this study, “income� means earnings from all activities, and “consumption� means expenditures at the household level. “Production� is defined as the amount of farm production cultivated and farmed. Profits are marginal gains or net benefits (sales minus costs) reported by farmers. See chapter 4 for a detailed discussion of these indicators. 4 The term “economic welfare� is used as an all-encompassing term in the report. The literature treats these indicators distinctly, where agricultural income reflects the value of the sales of agricultural products while household income reflects earnings from all activities, including agricultural sales. Consumption reflects a standard measure of expenditure at the household level. Similarly, the term “agricultural indicator� is used to reflect various measures of agricultural activity, such as agricultural income, productivity, yields, and agricultural production. 3 weather insurance. Although these are not the main focus of the analysis, some IEs that measure such indirect indicators are included to broaden the analysis and its results. Selecting the Group of Impact Evaluations for Review Less than one-third of all IEs focused on an agricultural outcome or had a credible strategy for measuring impact. Of 278 evaluations identified in an extensive search effort, as of January 2009, only 83 met the established criteria for inclusion in the analysis. These were complemented by three new IEs generated for this report, making a total of 86. More than 200 evaluations were dropped because they did not focus on indicators related to agricultural performance, or did not use commonly accepted IE methodologies that properly isolate the ―impact‖ of the intervention from other potentially confounding factors. Of the 86 IEs in the group, 23 evaluated a World Bank- supported intervention in agriculture; this number includes the three new IEs produced for this analysis (appendix B presents details of the final group selected). A five-step process, based mostly on having a valid counterfactual, helped determine which IEs remained in the analysis. Although each intervention has unique characteristics requiring different approaches and methods for evaluation, some general qualities can be considered good practice for conducting an IE. The selection method applied those generally accepted good practices, drawn from the evaluation literature and the authors‘ experience, as a guide to score each evaluation on a scale of 1–3. IEs were grouped by three criteria—source of the evaluation, method of evaluation and validity of the counterfactual, and robustness of results. Figure 1.2 illustrates the selection process. The five steps involved an extensive evaluation search (step 1), application of criteria to identify relevant evaluations (steps 2 and 3), in-depth screening for quality and rigor (step 4), and application of a rating scale outlined in appendix B (step 5). Appendix B elaborates on the process. 4 Figure 1.2. Process of Selecting the Group of Impact Evaluations Steps Results Number of studies Step 1: Identify IEs focused List of potential IEs of 278 on agriculture through agriculture development literature search interventions Step 2: Eliminate IES without Pool of relevant IEs 134 a defined intervention and agricultural indicator Step 3: Eliminate IEs with no Database of qualified IEs for 110 clearly defined counterfactual review Step 4: Review each IE in List of IEs for meta-review 83 depth to ensure it meets and synthesis eligibility criterai Step 5: Apply the rating IEs for quantitative analysis 77 scale 5 2. Impact Evaluation Overview Impact Evaluation Characteristics Evaluators need to keep a balanced appreciation of the strengths and weaknesses of different methodological approaches. One could argue that the combination of randomization and panel dataset offers the potential for a rigorous impact analysis. However, no single methodology for impact evaluation is perfect under all settings—each method has advantages and disadvantages (table 2.1). Careful theoretical analysis of the impact pathway is important in guiding the choice of design and methodology to evaluate a specific program. Table 2.1. Advantages and Disadvantages of Various Evaluation Designs and Methodologies Evaluation Advantages Disadvantages design Experimental  Free from selection bias issues  May be expensive and time consuming  Ease of measurement (simple  Can be politically difficult econometric methods)  Risk of contamination of control group  Ease of interpreting results  Difficult to ensure assignment is truly random  High internal validity Quasi-  Can draw on existing secondary  Reliability of the results is often reduced, as the experimental data sources methodology may not completely solve the problem  Can be quicker and cheaper to of selection bias implement  Some techniques can be statistically complex that  Evolving econometric methods require unique skills Non-  Relatively cheap  Reliability of results is reduced as the methodology experimental  Easy to implement since it can is less robust statistically draw on existing data sources  Some techniques can be statistically complex that  Well-developed econometric require unique skills methods  Full correction of selection bias remains a challenge  Identifying good instrumental variables can be problematic Sources: Maredia 2009 and IEG. There is a growing emphasis on measuring results in agricultural interventions using IE methodology. Although the analysis for this report searched for evaluations produced since 2000, more than 40 percent of the evaluations found were produced in 2006 or later. No evaluations produced in 2000 and 2001 filtered through. This could be due to the recent emphasis on getting results in agricultural investments using IE methodology. Another reason may be that practitioners are forced to show results quickly, and these interventions do not usually yield quantitatively measurable results in the short term, so practitioners use other methodologies that do not meet the criteria applied in the filtering process. This fits well with the decision to restrict the focus of this report to IEs published in the nine-year period from 2000 to January 2009. 6 The majority of the IEs included in the final group were either published in a journal, book, or monograph or issued as a working paper in an organization’s impact-related publication series. Fewer than half were published in a peer-reviewed agricultural journal. Most of the working papers included in the selected group were issued by a development bank such as the World Bank, a research institution such as the International Food Policy Research Institute (IFPRI), or a university working paper series. All three are known to subject papers to an informal or formal peer-review process before publication. At least seven studies are dissertation chapters or unpublished IEs that are likely to be issued eventually as either an institutional monograph or a journal article. A greatest portion of IEs included in the analysis focused on Sub-Saharan Africa. Thirty-four percent of the interventions focused on Africa, specifically Sub-Saharan Africa. IEs from South Asia and East Asia accounted for 17 and 19 percent, respectively, whereas 15 percent focused on Latin America and the Caribbean and 2 percent on Eastern Europe and Central Asia. The three new IEs produced to complement this report were in Africa, East Asia and Pacific, and Latin America and the Caribbean. Unfortunately, none of the IEs that passed the filtering process pertains to the Middle East and North Africa. Although the IE search did not impose a geographic limitation, IEs of development interventions in OECD (Organisation for Economic Co-operation and Development) countries either did not fit the objective of the report or failed one of the selection criteria. The time that elapsed between the intervention and the IE varied; however, fewer than a handful of IEs addressed long-term effects or sustainability concerns. The publication cutoff date for inclusion in the report was 2000, but the intervention or shock evaluated could have taken place before 2000. For impacts to be measured, an intervention had to be either completed (preferably) or well into its implementation cycle. Also, there is usually a lag between the IE date, the data used, and the completion of the IE. Lags occur for several reasons, including methodological improvements (as seen in the past decade), data availability, resource constraints, and researchers‘ interest. About 28 percent of the IEs assessed interventions initiated in the 2000s. Some of the interventions evaluated go as far back as the 1960s and 1970s; however, only three measured medium- or long-term impacts. Randomized designs have been rare in agricultural work, and most were conducted in the last four years of the period examined. In terms of evaluation design and methodology, experimental designs, such as randomized control trials, have not commonly been used for agricultural projects. There are several reasons why experimental approaches are rare, including the difficulty in coordinating with planning and implementing teams, complex political economy that inhibits randomizing interventions, and the time commitment required to ensure that the implementation stays true to the design. Only six percent of the interventions were randomized; most of these were completed in the last four years of the period. Appendix C elaborates on various evaluation methods. Given the limited use of randomized designs, the methods used to measure impacts relied heavily on innovative ways for identifying a proper counterfactual. Most of the IEs included used quasi-experimental approaches, such as statistical matching techniques, 7 to construct plausible counterfactual groups for comparison. These were sometimes combined with instrumental variables to provide robustness checks or deal with biases. IEs categorized as having nonexperimental designs often employed data sets that were collected for purposes other than to measure impacts; these IEs relied strictly on instrumental variables and other econometric manipulations. Appendix C elaborates on the main IE methodologies used by evaluators of agricultural interventions; see figure 2.1 for a profile of the IEs.. Figure 2.1. Profile of IEs (percentage) A. YEAR PRODUCED B. SOURCE VENUE 100 Percentage of studies 100 80 Percentage of studies 80 60 53 60 35 40 40 29 18 20 4 4 5 20 7 7 10 14 14 0 2 0 Journal Book Working Dissertation Not chapter, paper series published 2002 2003 2004 2005 2006 2007 2008 2009 report Publication venue Year C. REGION D. EVALUATION DESIGN 100 100 Percentage of studies Percentage of studies 80 80 60 60 60 41 34 40 40 20 21 17 20 6 20 3 0 0 Experimental Quasi- Nonexperimental AFR SAR EAP LCR ECA experimental Region Evaluation design E. INTERVENTION START DATE a F. ANALYTICAL METHOD USED 100 100 Percentage of studies 80 Percentage of studies 80 59 60 44 60 32 40 33 40 20 20 12 11 20 0 0 Unknown 1960- 1980- 1990- Only Matching Differencing Instrumental 1979 1989 1999 2000s techniques variables Intervention start date Anaysis method used Source: Author’s calculations. Note: Columns may total more or less than 100 due to rounding. a. An IE could use multiple techniques for analysis; hence the total is more than 100 percent. 8 The mix of interventions evaluated makes it difficult to amalgamate and synthesize results across interventions. Since agriculture and rural development is a broad area, the list of IEs identified for review represents evaluations of interventions that ranged from land issues to community-driven development (CDD). The objectives of interventions and indicators measured vary within and between categories (table 2.2). Consequently, no two interventions are alike. The evaluation methodologies and analytical techniques used to evaluate program effectiveness are also very diverse, making it difficult to integrate findings across individual evaluations and derive results that can be generalized and are meaningful results, which is the goal of a meta-analysis. Table 2.2. Examples of Direct and Indirect Indicators Measured Direct impact indicators Indirect impact indicators  Yield  Price gain  Gains in assets  Productivity  New land-related  Indicators of food security  Production investments  Indicators of labor market  Reproductive Efficiency Index (livestock)  Fertilizer use participation  Value of output per hectare  Repairs of conservation  Knowledge of farming  Profits per hectare structures practices  Household income  Planting multiyear crops  Reduction in risk  Household consumption  Reduction in pesticide  Social efficiency gains  Productivity gains from land rental application  Total gross margins for all agricultural  Intensive use of land production  Access to credit  Changes in household liquid assets 65% of total observations 35% of total observations Challenges Facing Users of IEs in Agriculture Agriculture interventions exhibit considerable heterogeneity of objectives, implementation strategies, and evaluation methodologies. That heterogeneity poses a special challenge for synthesizing results across interventions using common and comparable units of explanatory and impact variables. Hence the depth of analysis is limited. For example, not all IEs of interventions grouped under land titling and reform dealt with the same policy change or used the same units of measurement. Similarly, interventions grouped under extension or irrigation did not use the same implementation approach or measure the same development indicators. Another challenge, which inhibits the use of more rigorous analytical methods for meta-analysis, is the limited number of IEs in agriculture. For a formal meta-analysis, five criteria must be present: (1) evaluations are empirical rather than theoretical; (2) they produce quantitative results rather than qualitative findings; (3) they examine the same constructs and relationships; (4) they have findings that can be configured in a comparable statistical form (for example, as effect sizes, correlation coefficients, odds ratios, or t-statistics); and (5) they are ―comparable‖ given the question at hand (Wilson 1999). With the IEs available, this report only meets three of these criteria (1, 2, and 5) and would have to make accommodations to meet the third and fourth criteria. The IE group is not only too thin, it is also diverse in too many dimensions, and especially in 9 how outcome indicators are measured. See box 2.1 for examples of overcoming challenges users of IEs face. Box 2.1. Overcoming Some of the More Critical Challenges To estimate how much of the change is attributed to the project being evaluated, IEs typically use data to measure what happened before and after the project, and with and without the project. Real-world projects present a set of complexities that force development practitioners to make difficult trade-offs. As a result, evaluators often have to innovate to be able to measure results. Though it is not always possible to have project-specific data or to establish a counterfactual before a project begins (and in a random manner), there are ways of obtaining usable data and techniques that allow evaluators to establish a credible counterfactual. Using secondary data Many of the impact evaluations analyzed for this report use secondary data, collected for some other purpose, to reconstruct a baseline or measure project outcomes. Secondary sources must include an adequate set of outcome indicators that are relevant to the project, have adequate geographical overlap with the project, and be collected during the right time periods. Some examples of secondary sources that have been used in the IEs included here are: national household surveys, such as the Living Standard Measurement Surveys (LSMS); administrative data collected by ministries, project management units, and other public institutions; data collected by other donors, private institutions, academia, or nonprofit organizations; and a combination of these. Building a counterfactual by using “quasi-experimental methods� Random assignment to a project is not always feasible; sometimes there are criteria set out for eligibility (such as geography, poverty, politics), exogenous events that determine a locality (such as floods, droughts), or people who select themselves to take up an intervention (such as for an extension program). This is true for about 60 percent of all IEs in the sample for this report. As a result, the sample contains a variety of good examples that tackle nonrandomness using techniques that ensure the results do not reflect sample biases or mismeasurement due to noncomparability between beneficiaries and the counterfactual group. Among the most commonly used techniques in agriculture IEs are matching on observables (propensity score matching or other matching techniques), regression discontinuity, pipeline sampling approach, control functions, instrumental variables, differencing (double or triple), and multivariate regression. Quasi-experimental methods attempt to reduce biases due to unobservable characteristics and allow evaluators to create an imperfect yet useful counterfactual. Each technique has a body of literature that details the pros and cons of each method (see Ravallion 2006 for further details) and their applicability in a given scenario. Source: IEG. The uncommon use of a counterfactual to measure impacts of agricultural interventions effectively limits the number of IEs to draw from for in-depth analysis. Experimental and quasi-experimental techniques have not been widely used in agricultural development work and evaluation; data from observational studies are more commonly available for evaluating interventions (Stroup and others 2000). Observational studies include desk reviews, qualitative case-control, and cross-sectional or panel studies with no comparable counterfactual. These often attempt to measure impact by using 10 before-and-after comparisons. Most evaluations lack the with-and-without project scenario of statistically comparable groups to measure impacts; thus they do not attempt to attribute causality to the intervention. Less than one-third of all studies indentified in the literature search passed the quality filter; the rest failed based on their methodological approach or lack of focus on agricultural income and productivity. This clearly limits the analysis and forces a level of aggregation that may hide patterns, which is important for beginning to understand what works. The scarcity of IEs is most pronounced for farm-level interventions seeking to enhance farm use through improved water access and for interventions seeking to improve the quality or access to inputs such as seeds or fertilizer technology. Four areas need further work. First, there is limited evidence on interventions related to input technology. For example, activities related to seed improvement and fertilizer use in developing countries are still limited.5 Second, irrigation interventions, which affect the use of the soil through improved water access, require more research. Third, evaluations related to value-chain activities (projects related to marketing and to linking farmers to markets) are also scant, but there appears to be a growth trend in the last two years of the period examined. Fourth, interventions in microfinance, such as farmer loans and noncredit services, are an area of very limited evidence. The search phase identified several ongoing projects with experimental designs, so additional impact evaluations soon should be available in this area. Finally, there is a clear gap in knowledge on the differential effects of agricultural investment on women; the only examples in the sample are in the microfinance, land, and ―other‖ categories (CDD). Another challenge is the lack of financial and economic costs provided in the impact evaluations. A recent IEG study (2010) highlights that impact evaluation methods can be a natural complement to cost-benefit analysis. However, projects rarely use both methods, and shadow prices and other cost-benefit analysis methods are little used even in projects where the beneficiaries are easy to identify. In the sample used for this report, less than 10 percent of all IEs report any cost or financial data. The small percentage is not a surprise. IEG‘s study of cost-benefit analysis finds a declining trend of presenting economic data for World Bank projects in all sectors. And even though the percentage for agriculture is relatively high (48 percent) compared with the social sectors (but not compared with transport), the combination of IE and economic analysis is extremely rare, making it impossible to interpret the significance of the magnitude of impacts relative to other intervention results. 5 A vast literature examines the impact of technologies such as improved seeds and inputs like fertilizer application and other agronomic practices on productivity; these are usually measured in terms of crop yields. However, this type of research is mostly conducted by agricultural researchers or agronomists on experiment stations or researcher-managed farmers’ fields, with the intention of studying the efficacy of a technology before recommending its use by farmers. They are rarely conducted as impact evaluation studies (that is, studies that allow credible counterfactuals) in a “real-world� setting, where these technologies are introduced as part of an intervention to study how they perform and interact with the presence or absence of other factors that may influence the decision making processes faced by producers. 11 The reason for not using cost-benefit and impact analysis to measure the effectiveness and efficiency of a project is unclear. Though it is not known why evaluators do not present both methods, one can posit a few reasons based on the requirements of the two approaches. Both analytical methods are often hampered by neglect or failure to collect relevant data, especially for underperforming projects. The lack of baseline data especially limits the accurate estimation of costs, benefits, and impacts. There may be a lack of evaluator interest in undertaking either method. Funding sources usually require that results be measured; however, they typically do not impose the method to be used and they rarely ask for two approaches. Lastly, the skills required to do cost-benefit analysis are different than the skills required to measure impact. Therefore, evaluators use the method in which they are trained. Last, few IEs explore distributional impacts on various important dimensions— poor and less poor, or female- and male-headed households. Most IEs report average impacts on the overall targeted group rather than subgroups based on some socioeconomic characteristic. This limits what can be learned from the IEs. This is of particular concern when the average results show no impact because the project may be discarded without a thorough assessment of the impacts on each subgroup. 12 3. Inside the Interventions Introduction to the Taxonomy of Interventions IEs differ greatly in the types of interventions they evaluate. The interventions can be sorted using a taxonomy of eight intervention categories, as follows:  Land tenancy and titling (19 percent). These interventions facilitate access to farm credit and land markets. Most evaluations reviewed in this category reflected a change in law or enforcement of property rights traditions.  Extension services (20 percent). These interventions seek to improve the knowledge of farmers and promote the uptake of improved technologies and practices.  Irrigation (9 percent). These are water-related interventions that seek to affect productivity and farm income through stabilization and intensity of cropping, possible due to water supply improvements.  Natural resource management (14 percent). NRM interventions seek to improve farmer learning, understanding, and adoption of new technologies, such as soil and water conservation techniques.  Input technology (9 percent). These non-NRM technology interventions focus on the development and adoption of improved crop varieties, improved seed technology, and innovative fertilizer application techniques.  Marketing arrangements (15 percent). These interventions promote linkages with buyers and sellers, enable farmers to engage in contracts, benefit consumers through better pricing, and provide incentives for group formation and social learning.  Microfinance (7 percent). These interventions refer to the provision of small-scale financial services to farmers, from cash grants to credit-related advisory services. Objectives range widely, from increased access to credit to household consumption increases (poverty reduction).  Other: infrastructure and safety nets (9 percent). These interventions (and their evaluations) are too varied for a single definition, but they all seek to improve farmer economic well-being and, in a few cases, agricultural performance. Each of these categories includes several subcategories, as shown in table 3.1. 13 Table 3.1. Intervention Types Included in the Impact Evaluation Group a Interventions by subcategory % of studies Land tenancy and titling a. Titling 10 b. Tenancy law 8 c. Inheritance law regarding property rights 1 Extension a. Farmer field schools 8 b. Technical or advisory services 9 c. Market information 3 Technology: Natural resource management (NRM) a. Soil and water conservation 9 b. System of rice intensification 4 c. Integrated aquaculture-agriculture 1 Technology-inputs a. Improved seeds 8 b. Fertilizer 1 Irrigation a. Access to irrigation 2 b. Irrigation management systems 3 c. Dams 4 Marketing arrangements a. Contracts 6 b. Interlinked credit-input-output marketing arrangements 3 c. Spillover effects on food productivity 3 d. Cooperatives or social learning 3 Microfinance a. Access and services 5 b. Rural noncredit or insurance 2 Other a. Social safety net programs 2 b. Community-driven development 2 c. Rural roads or infrastructure 5 Source: IEG. a. Some IEs have more than one observation because they analyze more than one intervention or location; hence the total is more than 100 percent. Although the intervention coverage is broad, the number in any one category is small, given the limited availability of high-quality impact evaluations. All of these IEs represent the wide spectrum of development interventions occurring in agriculture and rural development and carried out by the World Bank (box 3.1), other donors, and country initiatives. Nevertheless, these 86 IEs, as broad representatives of different types of interventions, can provide useful insights into the types of impacts achieved, the constraints addressed, and the factors contributing to the achievement of impacts. 14 Box 3.1. Impact Evaluations in Agriculture at the World Bank Twenty-six of the impact evaluations examined involved World Bank-supported interventions. Among these, land tenancy and titling and extension services (especially farmer field schools) were the two most common types of interventions evaluated. No World Bank NRM and input technology interventions were evaluated and included in the IE group It is not clear whether this is because few of these interventions are being implemented by the Bank, no NRM or non-NRM interventions have been rigorously evaluated in the past nine years, or because these interventions are components of complex interventions (multicomponent) and categorized under other intervention types. With respect to the main constraints farmers face, Bank-supported interventions (or components) using IE methodology to measure results mostly concentrated on improving the policy environment (14 percent) and enhancing farmer knowledge (9 percent). The rest were more evenly distributed among improving access to farm inputs (6 percent), dealing with access to water (5 percent), and addressing credit or insurance access (5 percent). Interventions that addressed output or post-production promotion (1 percent) and adoption of new farm input technologies were notably absent. Characteristics of the Interventions LAND TENANCY AND TITLING Land tenancy and titling interventions facilitate access to farm credit and land markets. IEs in this category predominantly addressed titling and tenure security. Titling entails a change in the law or the introduction of programs and policies that promote titling and land registration to grant ownership rights to use, sell, and rent the land. Similarly, tenure security entails legal changes, programs, and policies that promote (or modify) land security through mechanisms other than titling and without giving the land user ownership rights. Easing these constraints can improve agricultural outcomes by increasing the landowner‘s confidence in the benefits from the land investment and eliminating the risk of expropriation. Furthermore, increased land security can facilitate access to and supply of credit, as farmers can use land ownership as collateral. Last, improvements in land rights can catalyze land markets by lowering ownership risks and land transfer costs. Interventions included for this analysis reflected changes in land-related laws (such as in China, Bulgaria, India) or traditions (Ghana, Zambia) that governed property rights. Some interventions addressed changes in the legal system put in place to speed up the adoption and implementation of legal changes (Cambodia, Nicaragua, Peru, and Vietnam; see box 3.2). 15 Box 3.2. Interventions That Delivered Mixed Results on Productivity or Income Land reform in Vietnam enabled an extensive rural land titling program that led to measurable improvements in the decisions of households to undertake long-term agricultural investments and, at the same time, devote labor to nonfarm activities. These results were not very large in magnitude, however, and more work should be done to assess the long-term impacts. Another intervention in Vietnam designed to promote land consolidation in a region characterized by extensive land fragmentation did not lead to changes in productivity because the law is complex, the consolidation system (including workers) is not set up to motivate changes, and farmers do not have incentives to take advantage of the legal change. In West Bengal, a change in property rights through land tenancy reform can explain around 28 percent of the subsequent growth of agricultural productivity and overall economic improvements. Few studies looked at how credit access varies with farmers‘ ability to use land as collateral, and these studies did not reach a clear consensus on impact. For example, titleholders in Zambia have greater credit use; another study found no conclusive impact of a Peruvian titling project on obtaining credit. The third potential impact channel—the facilitation of land transactions such as renting, buying, and selling—was evaluated in a land reform in Bulgaria. The authors found that land allocation is still suboptimal because historical ownership and legal constraints can reduce effective decision making. Sources: Do and Iyer 2008; Del Carpio and others, 2009; Banerjee, Gertler, and Ghatak 2002; Smith 2004; Torero and Field 2005; Vranken and others 2008. Impact indicators measuring various definitions of productivity in this group showed that the interventions led to improvements 65 percent of the time. Direct indicators most commonly assessed in these interventions included crop yield (kilograms per hectare), value of production per hectare, profits, and household income. The most common indirect indicators were investments made on plots (including the decision to fallow), some measures of economic welfare, risk of appropriation (including the decision to grow multiyear crops), access to credit, and efficiency. Table 3.2 shows how these interventions performed on the impact indicators measured. Sixty-five percent of the IEs had a positive and statistically significant change in the desired impact indicator. Whether this reflects a ―publication bias‖ (that is, the tendency in academic journals to publish results that are statistically significant) or an unbiased sample of a population of IEs is a testable hypothesis that merits further review in future work. Another aspect worth further exploration is the differential impacts on female-headed households and various economic groups; these were rarely reported in this intervention group despite the clear implications they can have in understanding disaggregated impacts and for future policy formulation. 16 Table 3.2. Land Tenancy and Titling Interventions: Observations from the Reviewed Literature Explanatory variables found in Impact indicators Number of IEs with evidence tenancy and titling IEs measured of impact on the indicator a – 0 + Land with registered title Yield 1 1 1 Income 1 Efficiency gains 1 Investments 5 Welfare 1 Multiyear crop planting 1 More secured land tenure Yield 2 Efficiency 1 Plots under co-ownership Welfare 1 Members who hold a social or Decision to fallow 1 political office Right to give land without Profits 1 external approval or notification Value of product 1 Land with no rights Cap ratio 1 Total observations 2 5 13b a. The symbols indicate the direction of impact and level of statistical significance. (–) = negative and significant; (0) = either positive or negative but not statistically significant; and (+) = positive and significant. Statistical significance is defined as significance of an estimated impact coefficient less than or equal to 10%. b. IEs may have more than one outcome measured and are likely to be listed more than once. EXTENSION INTERVENTIONS Extension interventions seek to improve the knowledge of farmers and promote the uptake of improved technologies and practices. These interventions have an element of learning and influencing farmers‘ knowledge—broadly defined as their analytical skills, critical thinking, and creativity. There are three subcategories of interventions under the extension heading: (1) extension agents who make periodic visits to villages and present information on new techniques to the farm communities; (2) systems that develop a cadre of trained field-level extension workers who visit villages and serve as a bridge between farmers and knowledge systems and implement participatory approaches, such as the farmer field school (FFS); and (3) the provision of market information, ranging from input and sale prices in broader markets to information on buyers and general farming or fishing trends. All three intervention types promote the use of improved technologies for farm production. All the interventions included in this review that fit in the first two subcategories promoted integrated pest and crop management technologies on food crops (rice, potato) and cash crops (cotton). FFSs have evolved to include broader coverage of farm-relevant topics under the rubric of ―integrated production management.‖ However, no impact evaluation studies of such interventions were found in the filtered literature reviewed for this analysis. Other mechanisms include demonstration plots, training, group meetings, and focus group discussions. Box 3.3 provides illustrations from the selected evidence pool. Any program that did not refer to itself as FFS was grouped with technical or advisory services, a broad category of interventions that are implemented at a national 17 level or in specific locations to promote knowledge, practices, and technologies to farmers growing a specific commodity. The market information group includes IEs that do not focus on providing technical, advisory, or training services but only price information on products, information on accessing new technologies, or quality testing services and general input technology practice. A wider body of literature—not reviewed in detail for this report—assesses the effectiveness of extension projects. However, few IEs deal with potential biases that lead to measurement errors. The amount of resources invested over the years in such activities is large, and the literature on the application and effectiveness of extension programs is quite rich. A review of studies published between 1970 and 1989 by Birkhaeuser and others (1991) shows large positive rates of return to extension services. However, in the absence of random assignment to treatment and control groups, this methodology is likely to provide biased estimates of causal effects due to endogeneity of program participation and the presence of unobservable characteristics that might determine participation and be correlated with an outcome variable. All extension IEs included in this study addressed the endogeneity concern by comparing outcomes to a statistically valid counterfactual. Box 3.3. Relevance of the Information Determines the Incentives to Take Up the Project An IE of a farmer field school program in Indonesia found no conclusive evidence of improved economic performance or farming practices. The results suggest that graduates of the program, who underwent fairly intensive training, obtained some gain (although quantitatively small) in knowledge of better pest management. However, this knowledge was not diffused significantly to other members of the villages, possibly because the information in the course was too complex or the incentives to learn it were not clear. For example, farming processes and ecosystem concepts are not easily transferred in informal farmer-to-farmer communications. Some of the technologies may have been targeted at more sophisticated farmers engaging in larger production (unlike the typical farmer). In another example, grape producers in Argentina received technical assistance from extension officers seeking to improve the quality and quantity of grapes in the region. The results varied by type of producer. Services were effective at increasing yields for those with low initial yields, while content aimed at improving quality seemed to be most helpful to larger producers who increased their average quality more than others. A subset of beneficiaries experienced no change in either quality or production, indicating that the content was not suitable for all producers; only a subset was given incentives to take advantage of the project. A slightly different kind of extension service relates to the provision of market information, ranging from input and sale prices in broader markets to information on buyers and general farming or fishing trends. An intervention in India found strong impacts of providing market information to fishermen using mobile phones: price dispersion was reduced substantially, waste of fish was completely eliminated, fishermen‘s profits increased by eight percent on average, and consumer prices declined by four percent. Sources: Feder, Murgai, and Quizon 2004a; Cerdan-Infantes, Maffioli, and Ubfal 2008; Jensen 2007. 18 Farmers learn their lessons, but increased knowledge does not automatically translate into increased productivity. The impacts of extension provide a mixed picture with regard to increased productivity, broadly defined. Except for one study that found a positive impact of extension on food crop yields, no evaluations that measured productivity reported significant impacts of extension programs on yields.6 An FFS characteristic in this IE group is that trainers appear to act as facilitators rather than as instructors. Results show positive impacts on farmers attending schools; conversely, farmers exposed indirectly to the lessons—from neighbors or friends—do not experience positive changes. The FFS concept does not require that all farmers attend FFS training; instead, it relies on training a select number of farmers from a village or local farmers‘ group. These informal schools are often organized by local nongovernmental organizations. To disseminate new knowledge more rapidly, selected farmers receive additional training to become farmer-trainers and are expected to organize field schools within their communities, with some support from public sources. The cost-effectiveness and sustainability of the FFS model is based on diffusion effects from FFS graduates to nonparticipants. Though impacts of FFS on knowledge and reduction in pesticide inputs by direct participants are mostly positive, the impact on exposed farmers is not. This lack of success in promoting the farmer-to-farmer diffusion of knowledge is echoed by other assessments (Tripp 2005). The IEs reviewed provide clues about the lack of ―demonstration effect.‖ Potentially, the ability of participant farmers to convey complex decision-making skills effectively to other farmers through informal communications may be limited and, therefore, may curtail the diffusion process. Moreover, trainers are required to be facilitators rather than top-down instructors, which may pose a challenge to program effectiveness (Feder and others 2004b). There is very wide variation—in intervention design, objectives, and measurement—making it difficult to compare across interventions. The variation is partly due to the types of interventions evaluated, which range from national extension programs (such as in Africa) to location-specific and technology-focused FFS (in India, Indonesia, Peru, the Philippines, and Thailand). Technical and advisory service interventions promote the adoption of newer techniques through traditional advisory methods and often focus on specific commodities (such as programs in Argentina, Thailand, and Uruguay); these range from regional (Sub-Saharan Africa) to local interventions. In this category of interventions, like most in this study, results vary based on the setting, the extension approach implemented, the evaluation method used, and even the yardstick used to assess impact. The impacts of extension programs are unclear because about half of the evaluations reported positive impacts on a productivity-related outcome while a little less than half reported no positive impacts (table 3.3). 6 Feder and others (2004a) cites several studies that report positive impacts of FFS training on crop yields. However, all these studies were published before 2000 and were not reviewed. Therefore, no comments are made as to the comparability of the evidence presented in this study with previous studies that may have used a different method to assess impacts of FFS. 19 Few IEs report the impact of extension services on female farmers or differentiate between poor and non-poor groups. Impacts of extension programs on subgroups such as female farmers and poorer farmers have been explored in a few IEs (females in India, modality; younger farmers in Peru, FFS program; poor households in Argentina and Uganda). Table 3.3. Extension Interventions: Observations from the Literature Explanatory variables found in Impact indicators Number of IEs with evidence extension IEs measured of impact on the indicator a – 0 + Participation in FFS Yield increase 2 Pesticide reduction 1 2 Increased knowledge 3 Gains in financial capital 1 1 Exposure to FFS graduates Yield 1 Increased knowledge 1 Pesticide reduction 1 Participation in extension Yield 2 1 programs focused on a Productivity 1 commodity-specific Profit 1 technical/advisory services Efficiency gains 1 Product quality 1 Exposure to visits by an extension Consumption growth and 1 service provider poverty head count Yield 1 Market information Price of output produced 1 Yield 1 Efficiency gains (price) 2 Total observations 1 12 13b Source: IEG. a. The symbols indicate the direction of impact and level of statistical significance. (–) = negative and significant; (0) = either positive or negative but not statistically significant; and (+) = positive and significant. Statistical significance is defined as significance of an estimated impact coefficient less than or equal to 10%. b. IEs may have more than one outcome measured and are likely to be listed more than once. There is also a paucity of studies that evaluate the cost-benefit analysis. This anaylsis can help assess the efficiency of different modalities of extension programs. For example, privatization, fee-for-service, and decentralization, among others, can be adequately compared. The number of projects that undertake a cost-benefit calculation in this and other project types is low. The primary reason for the lack of such analysis is unclear. However, it is likely that many projects do not have the relevant data to undertake the necessary calculations. IRRIGATION INTERVENTIONS Water-related interventions seek to affect productivity and farm income by stabilizing and intensifying cropping through water supply improvements. Interventions in this group aim to improve access to irrigation through infrastructure 20 (dams), plot-level technologies (drip irrigation), and changes in irrigation management structures to give farmers greater leverage in the use of water. Box 3.4 provides some examples. The main ―impact channels‖ identified in the IEs are stabilization of cropping patterns, crop yields, and agricultural outputs; cropping intensity by growing several crops per year on the same plot of land under irrigated conditions; and increased reliability (quality) of water, thus enabling farmers to replace low-yielding crops with more profitable high-yield crops. Box 3.4. Irrigation Interventions An IE of an intervention in Peru found that private investments on private farms through drip irrigation technology had an immediate impact on the yields and profits of beneficiaries. The effects were diffused throughout neighboring farms that were not intended beneficiaries; spillover effects were probably possible through exposure to new technology or increased labor demand. The same program had an extensive canal rehabilitation component that aimed to increase water availability for all members of the water user association, but the impacts on productivity were less obvious in the short term. A different type of irrigation project in the Philippines involved changing management structures to give farmers greater leverage, for example, through water user associations. For this intervention, the IE found that maintenance of irrigation facilities improved with such a shift in management, also resulting in a positive indirect impact on rice production. An evaluation of a large irrigation construction project in India found that cropping intensity increased as a result of water availability in additional seasons. It also reduced year-to-year income fluctuations caused by variations in rainfall. The new construction raised net farm income by about 60 percent. Indirect benefits were possible from additional wage employment. The intervention had several negative aspects stemming from cost overruns, construction delays, and discrepancies between realized income increases and those expected at project appraisal. More negative effects were highlighted by some irrigation and dam investments in India and Ethiopia. The negative externalities were due to disruption effects in the beneficiary localities and the increased incidence of diseases in the upstream and surrounding areas. Sources: Del Carpio, Loayza, and Datar 2010; Bandyopadhyay, Shyamsundar, and Xie 2007; IEG 2008b; Duflo and Pande 2005; Amacher, Ersado, Grebner, and Hyde 2004. Most irrigation interventions seek to affect farm production and productivity. The evidence shows that irrigation interventions generally lead to positive impacts on various farm performance indicators, such as production and productivity. Although the IEs included in this category do not cover the breadth of issues confronted in assessing productivity impacts of irrigation, and while geographic coverage is limited,7 they do provide insights on different types of impact challenges associated with irrigation interventions. The evidence of impact of irrigation (either access to irrigation or decentralized management of it) is positive for most of the direct and indirect impact indicators examined by these IEs. The exceptions are the impacts of an irrigation project measured in specific ―catchment areas‖ (Duflo 2005) and areas close to an irrigation 7 Three regions are represented in this category: Africa (Amacher and others 2004; Van den Berg and Ruben 2006; Dillon 2008), South Asia (Duflo 2005; IEG 2008b), and Latin America and the Caribbean (Datar and others 2009). 21 project; negative impacts stem from increases in the incidence of diseases (Amacher and others 2004) among farmers due to waterlogging and poor maintenance. Table 3.4 presents all outcomes measured in the IEs in this category. Of these, about 20 percent deal primarily with access to irrigation, 33 percent with irrigation management systems, and 44 percent with the construction of dams. The separation of dams from general access to irrigation is for illustration purposes only. Canal rehabilitation, dam construction, or installation of a system to access water for irrigation is supposed to increase access to water and farming effectiveness (among other objectives). Table 3.4. Irrigation Interventions: Observations from the Literature Explanatory variables found in Impact indicators Number of IEs with evidence irrigation IEs measured of impact on the indicator a – 0 + Access to irrigation Yield increase 1 Production 2 Household income 1 Presence of irrigation Yield 2 (catchment area) Presence of irrigation Yield 1 (command area) Proximity to dams Production 1 Productivity 1 Participation in irrigation 1 1 management training Total observations 1 3 7b a. The symbols indicate the direction of impact and level of statistical significance. (–) = negative and significant; (0) = either positive or negative but not statistically significant; and (+) = positive and significant. Statistical significance is defined as significance of an estimated impact coefficient less than or equal to 10%. b. IEs may have more than one outcome measured and are likely to be listed more than once. IMPROVED NRM TECHNOLOGY INTERVENTIONS Natural resource management interventions seek to improve farmer knowledge and adoption of new technologies, such as soil and water conservation techniques. The broader literature shows that investments in science and technology for agricultural development has a good record of delivering benefits to farmers, consumers, and processors through new crop, livestock, fish, forest, and farming technologies that improve productivity and farmers‘ incomes (Gollin and Evenson 2003, Raitzer and Kelley 2008, Maredia and Raitzer 2006). These interventions include integrated pest management, soil fertility management, and soil and water conservation practices meant to affect the management of land and other natural resources. Like other technology-related interventions, NRM is often complemented by extension interventions. These interventions seek to improve production and natural resource systems for community use, increase the availability of environmental amenities, and improve the policies that govern NRM regimes. 22 Most assessments in the broader literature focus on the rates of return to the uptake of technologies resulting from research investments or assess adoption rates; however, few measure productivity gains using a counterfactual. The NRM technology projects included in this review do not cover all the objectives commonly identified in these types of interventions. The wider literature is replete with studies that report high returns to investment in science for agriculture (see Alston and others 2000 for a meta-analysis of this literature). But most focus on cost-benefit estimations, where benefits are conceptualized as the economic surplus resulting from the adoption of new technologies or inputs (Alston and others 1998). The supply-shift parameter is the ―impact‖ variable that is estimated using standards and approaches that differ significantly among studies—ranging from the use of data from scientific experiments or on-farm technology trials that compare various technological options, to expert solicitations and adopter surveys to estimate farm-level budgets with and without technology. However, impact data based on scientific ―lab-based‖ experiments may overestimate the true effect of a new technology because experiments are conducted in controlled environments that do not mimic real farm conditions. Therefore, despite hundreds of assessments measuring rates of return to the adoption and uptake of technologies or inputs resulting from research investments, only a few have been undertaken in real-world settings. The IEs included in this analysis mostly measured impacts of some type of soil and water conservation technique, system of crop intensification, or integrated aquaculture- agriculture technologies (see examples in box 3.5). Box 3.5. Natural Resource Management Technology A program promoting soil conservation practices, such as using hedgerows and contour ridging in Thailand and Vietnam, significantly increased levels of adoption, especially for soil conservation practices, which were linked to productivity impacts. The adoption of contour ridging was linked to a reduction of cropped area. A system of rice intensification technology intervention in Madagascar was designed to increase rice yields substantially in a country with specific microclimates and types of smallholder farmers. The technology delivered high impacts for some, but it was unattractive to farmers within the standard range of relative risk aversion. The incentives to adopt the technology were low because it requires skill and has a higher labor input requirement (increased weeding and monitoring) than other methods. Sources: Dalton and others 2007; Barrett and others 2004. The evidence for NRM technology interventions is mixed, as several IEs show nonpositive results. IEs reporting positive results involved projects that promoted technology that subsequently changed the structure or composition of the soil. The positive impacts reported by the IEs reviewed suggest that productivity and food production can be increased through improved NRM using simple but effective technologies, such as traditional practices like contour stone bunding (a soil water conservation technique), hedgerows, crop rotation, green manure, contour tillage, crop residue mulching, and minimum and zero tillage, among others. Nonpositive results indicate that interventions had negative impacts or the evaluation could not establish a significant impact of the interventions. 23 Discernable patterns related to specific technologies emerge from the analysis in this intervention group. IEs that look at the impacts of stone bunding, in Ethiopia for example, find that bunds have the highest productivity impact in low-rainfall areas. When rainfall is excessively abundant, stone bunds retain too much water, depressing yields (Dutilly-Diane and others 2003; Kassie and others 2008). This highlights the importance of developing and disseminating soil conservation technologies that are tailored to agro-ecological zones instead of making blanket recommendations that promote similar conservation measures to all farmers and all contexts. In high-rainfall areas, moisture conservation using physical structures may not be important, but appropriate drainage measures could help protect soils during extreme rainfall (Kassie and others 2008). The IE by Cocchi and Bravo-Ureta (2007) in El Salvador also found varying impacts of different types of NRM technologies, with a clear positive association between conservation practices (crop residue mulching, minimum and zero tillage, crop rotation, green manure, and contour tillage) and income, while there is a nonpositive effect of conservation structures (such as terraces, ditches, live barriers, and stone walls) on income. INPUT TECHNOLOGY INTERVENTIONS Non-NRM input technology interventions focus on the development and adoption of improved crop varieties, improved seed technology, and innovative fertilizer techniques. They lead to productivity increases and risk reduction in various contexts, partly because of their high adaptability to adverse conditions. Input interventions focus on the development and adoption of improved crop varieties, improved seed technology (for example, transgenic technologies), and innovative fertilizer application techniques, among other areas. Such technologies produce higher-yielding crops that can bring enormous benefits to the poor through reduced risks in cases of adverse weather or geographic conditions and enhanced efficiency that leads to higher incomes and lower food prices. Box 3.6 presents several examples. Box 3.6. Clear Synergies between Input Technology and Extension Services A series of interventions that demonstrated top-dressing fertilizer in western Kenya showed farmers that productivity gains were possible when the right amount of fertilizer was used. The intervention combined an introduction to the technology for some and lessons on appropriate use of the technology for most; it provided the fertilizer as well as information, which enabled farmers to be exposed to the technology and experience production gains. One of the innovations of this intervention was the demonstration feature, designed to give farmers incentives to use a hands-on approach to learn. The case for modified seed technology is strong in different agro-ecological settings. In Ghana, the impact of an intervention that distributed new varieties of cocoa to farmers resulted in a 42 percent yield increase. One of the reasons was the ease of adaptability to the Ghanaian soil. Interventions that promote the use of transgenic technology to produce Bacillas thuringiensis (Bt) cotton have also shown positive results in Argentina and India, but only when the price of the technology was affordable to a wide set of farmers and in some agro-ecological conditions where farmers were given incentives to adopt it due to alignments of their practices and the technology. Sources: Duflo, Kremer, Robinson 2008; Edwin and Masters 2005; Qaim and de Janvry 2003; 2005; Qaim, and others 2006. 24 The evidence shows that adoption of new farming technologies specifically related to production inputs (such as fertilizer) delivered positive impacts. The application of varietal technology that incorporates improved crop features, such as disease resistance, without increasing the need for additional inputs delivered positive impacts. At least three IEs in this category test the Food and Agriculture Organization‘s claim that there is great potential for new and safe agricultural biotechnologies to underpin further productivity increases and decrease the risk to food security. All three IEs show positive impacts of crop technology in increasing yields and reducing costs (Qaim and others 2006; Subramanian and Qaim 2009; Qaim and de Janvry 2003, 2005; see also table 3.5). Table 3.5. NRM and Non-NRM Interventions: Observations from the Literature Explanatory variables found in the IEs Impact indicators Number of IEs with measured evidence of impact a – 0 + Presence of stone bunds in a low- Yield 2 rainfall area Presence of stone bunds in a high- Yield 1 1 rainfall area Adoption of soil fertility/conservation Farm income 1 practices NRM Technologies Change in 2 household‘s liquid assets Adoption of soil conservative structures Farm income 1 Adoption of hedgerows in cassava field Yield 1 Adoption of system of rice Yield 3 intensification Yield risk reduction 1 Adoption of integrated aquaculture- Household income 1 agriculture technologies Participation in cassava participatory Yield change 1 research for soil fertility and conservation Fertilizer application at rate less than Net return (maize) 1 Non-NRM Input Technology official recommended rate Fertilizer application at official rate and Net return (maize) 1 at ¼ teaspoon top dressing Full package recommended (fertilizer at Net return (maize) 1 planting, top dressing, and hybrid seed) Plots growing Bt seeds Yield (cotton) 1 4 Fields planted with new varieties Yield (cocoa) 2 Planting resistant varieties Reduction in risk of 1 income loss (bean growers) Total observations 4 6 16 b a. The symbols indicate the direction of impact and level of statistical significance. (–) = negative and significant; (0) = either positive or negative but not statistically significant; and (+) = positive and significant. Statistical significance is defined as significance of an estimated impact coefficient less than or equal to 10%. b. IEs may have more than one outcome measured and are likely to be listed more than once. 25 The types of interventions included in this category are limited to development and adoption of seed varieties and application of fertilizer. The input technology or non- NRM interventions in this analysis include the development and adoption of improved varieties (beans, wheat, and cocoa), including seeds developed using transgenic technology (Bt cotton), and application of fertilizer. In light of global efforts and substantial investments in agricultural technology development, the inclusion of only two types of technologies or inputs in this review seems quite narrow. However, improved seeds and fertilizer were arguably the backbone of the technological revolution in the last century, attributed with having led increases in the production of food, feed, and fiber around the world. Even though overall impacts on a productivity-related indicator in this broad category are positive, technologies vary and may perform differently in distinct settings. Some IEs argue that the lack of adoption and integration of improved seeds and fertilizer technology in farming systems has possibly contributed to significantly lower yields than what is potentially achievable. The impacts of non-NRM technologies are positive and highly significant (in magnitude), but magnitudes vary according to the country and context. For example, physical attributes of the soil can affect effectiveness and local institutional attributes can affect efficiency. The IEs in Ghana, Honduras, and India show evidence of positive impacts of improved seed varieties and modified plants adopted by farmers (Matuschke and others 2007, Edwin and Masters 2005; and Mather and others 2003); disease resistance technology is one such example. India has been experimenting with crop varieties for many decades; one IE illustrates the benefits of giving proprietary seed technologies for subsistence (smallholder) farmers (Matuschke and others 2007). Overall impacts of input technologies on various productivity indicators are positive, especially for interventions including a farmer training component. Most IEs point to significantly positive impacts of interventions that promote input technology. An important characteristic of these interventions is that they often include extension services— technologies are often developed by a research system and then extended to farmers through various promotion channels. The observed outcome of adoption of a technology is a reflection of the desirable characteristics of the technology, as well as the effectiveness of the method of extension used. Extension is thus embedded in the implementation designs of technology projects. The observed impacts of these interventions can be viewed as joint impacts of the technology and the extension. Despite such interaction effects, however, these IEs are listed under only one category of interventions for the descriptive review; they are recategorized according to the constraint they address in follow-up analysis. MARKETING INTERVENTIONS Marketing interventions promote linkages with buyers and sellers, enable farmers to engage in contracts, benefit consumers through better pricing, and provide incentives for group formation and social learning. Access to reliable markets can promote the transformation of rural economies from low-value agriculture, characterized by ―ad hoc‖ sales, to high-value agriculture, with coordinated supply chains linking farmers, input providers, credit providers, traders, processors, and retailers. Farmers can link to productive input markets and reliable output markets that make their adoption of 26 new technologies and inputs worthwhile. Marketing arrangements generally influence productivity by enabling contracts with buyers, landowners, and suppliers; promoting linkages between various actors in the supply chain; triggering spillover effects of marketing dynamics on food productivity; and facilitating cooperative arrangements that lead to shared social learning and knowledge. Some interventions in this category aim to relieve constraints on farmers, producers, and sellers to increase productivity throughout the value chain. Interventions in this group commonly link the producer to someone else, who either facilitates or inhibits production. Some of the IEs measure the effectiveness of contracts that promote interlinked activities, such as input, credit, and output markets. The IEs in Peru (described in box 3.6) and Indonesia exemplify how smallholder farmers can be linked with buyers through a contractual arrangement. The Indonesian IE investigates the impact on gross margins of various contractual agreements between private companies and farmers involving the production of rice and maize seeds and poultry (Simmons and others 2005). Other types of contracts included in this category are between land tenants and landowners who share production outputs, as in Pakistan (Jacoby and Mansuri 2007), and contracts with cotton production buyers in Zambia (Brambilla and Porto 2005). Box 3.7 has some examples. Box 3.7. Context Can Lessen Potential Impact of Marketing Interventions The role of a complex contracting scheme for small commercial potato farmers in the Peruvian Andes had mixed outcomes for income improvement. This less-than-desirable result is likely caused by the lack of complementary interventions related to infrastructure improvements and regulatory reform. Membership in burley tobacco clubs in Malawi, where farmers are interlinked with various actors in the supply chain, had a positive impact on productivity and income. Farmers obtained higher premiums at sales auctions for their crops and had increased yields of 40–70 percent, partly due to social dynamics and incentives derived from membership. The club membership effect, however, did not extend to the amount of land dedicated to tobacco production, a surprise given the large productivity gains. This is perhaps an indication that large gains from the cash crop are necessary for producers, but they are not sufficient to forgo planting food crops in a context where poverty is high and smoothing mechanisms are so embedded in the farming culture. Sources: Escobal and Torero 2006; Negri and Porto 2007. As the agriculture sector in developing countries modernizes and commercializes, value chains for agricultural products become increasingly important. In some interventions farmers are linked with buyers abroad; in others they are linked to big national or multinational companies operating in the country. The DrumNet intervention in Kenya encourages the production of export-oriented crops by providing smallholder farmers with credit linked to agricultural extension and marketing services (Ashraf and others 2008). The IE used a randomized design to examine the productivity impacts of participation in two variations of interlinked marketing arrangements; one with all services and the other with all services except credit. The Malawian burley tobacco club (Negri and Porto 2007), highlighted in box 3.7, represents a more typical locally developed intervention where farmers access inputs, information, and markets through a production 27 club. The Malawi case is an example of a value-chain intervention that provides services (access), economies of scale (buying inputs), collective action (selling products), and furthers knowledge and access to credit through networks. Cooperatives and farmer associations function as coordinating mechanisms for linking farmers to markets. These interventions are designed to influence factors that determine farmers‘ decisions to participate in such marketing arrangements to obtain positive impacts on production prices. In Costa Rica, for example, a cooperative and a farmer association provided support to coffee growers to produce and sell their product in specialty international market niches. Two other IEs of intervention in Kenya (Jayne and others 2004) and Madagascar (Minten and others 2007) measured impacts of interventions that linked farmers to export markets through interlinked marketing arrangements. Several contract arrangements influence the use of modern production inputs more broadly—for both commercial and noncommercial crops. In general, there are positive spillovers from participating in various marketing arrangements. The most common indicators measured are increase in price and value of production. At least two IEs in this category investigated spillover effects on the productivity of food crops grown (for noncommercial purposes) by farmers participating in marketing arrangements. In the Kenya example, farmers increased their fertilizer application on their own cereal crops as a result of participating in interlinked market arrangements for growing export crops that required the use of fertilizers. In Madagascar, spillover impacts were evident from growing off-season vegetable crops that use fertilizer on the productivity of rice, which is grown in the main season. Lack of market information, financial risks, and access to markets reduce farmers’ incentives to adopt high-value crops and to participate in marketing schemes. Some of the primary motivating factors identified in IEs show that common constraints for adopting high-value agriculture are addressed through marketing-type interventions. The first constraint is a lack of information on production methods (relevant to quality), marketing opportunities, and the probable distribution of net returns. The second constraint is the risk associated with the production of a new commodity, and switching to a commercial crop implies depending on market purchases to meet food requirements. Third, location disadvantages reflecting unsuitable natural conditions can exacerbate risk and limit the marketability of production. Results are positive overall for interlinked marketing schemes (table 3.6). The interventions evaluated and included in this category tried to ease constraints farmers faced in commercializing their products. Interlinked marketing programs that were coordinated through nongovernmental organizations, cooperatives, or a contractual agreement had positive effects on various indicators, such as gross margins, value of production, the price obtained for outputs, and the productivity of food crops (as a result of spillover effects). One-third of the observed effects were negative or not significant, which highlights the heterogeneity in performance as well as interventions in this category. 28 Table 3.6. Marketing Interventions: Observations from the Literature Explanatory variables found Impact indicators Number of IEs with evidence in the IEs measured of impact on the indicator a – 0 + Participation in contract Gross margins 2 1 3 Participation in contract for Yields of food crops 1 export crops Participation in interlinked Value of harvested produce 1 input-output-credit arrangements Value of harvested produce 1 for export crops (new adopters) Fertilizer application on 1 food crops Participation in special output Price of the produce sold 3 marketing channels Household income 1 Total observations 2 3 9b a. The symbols indicate the direction of impact and level of statistical significance. (–) = negative and significant; (0) = either positive or negative but not statistically significant; (+) = positive and significant. Statistical significance is defined as significance of an estimated impact coefficient less than or equal to 10%. b. IEs may have more than one outcome measured and are likely to be listed more than once. MICROFINANCE INTERVENTIONS The provision of small-scale financial services to farmers ranges from cash grants to advisory services. Microfinance interventions provide services to needy households, which often lack collateral, through individual or group arrangements. These services usually entail the provision of credit, but they can also be cash grants, savings accounts, advisory services, insurance, and access to farmer cooperatives. The definition of this category of interventions changes depending on the objectives set out by the evaluators; however, all interventions fit within the rural finance definition, and some are specifically agricultural finance. It would be preferable if this category could include only interventions that provide financing for agriculture-related activities, such as input supply, crop production, distribution or wholesaling and marketing among other activities. The evidence of agricultural finance IEs is too thin to focus on such a narrow set of objectives. The main constraint addressed by these interventions is access to credit, but there are direct links to other interventions that address other constraints. Agricultural (or rural) credit addresses the lack of liquidity households face when they try to meet investment needs, such as inputs for general farming activities. Access to credit usually allows for greater investment beyond basic inputs, for example, in irrigation technologies, farm equipment, and transportation of products to markets. Microfinance institutions may also operate through cooperatives and exporting firms, thus allowing farmers to secure sales outlets to potentially sell their production at a higher price. Box 3.8 provides more examples of interventions in this category. 29 Box 3.8. Microfinance Services beyond Credit Can Increase Gains An evaluation of three large credit programs in Bangladesh found that access to credit and noncredit services positively affected agricultural profits. These effects were possible through various channels, including the availability of cash resources and advisory services. Easing up cash constraints made possible the acquisition of other goods (potentially farm inputs), while the advisory services or noncash benefits also had increased effects; however, the impacts were most significant for landless borrowers. The wealth impacts of having access to or receiving various forms of credit on agricultural households in Thailand were evaluated to determine the most effective forms of financial services. The evaluation found that rice and buffalo banks had negative impacts on asset growth and harmed household consumption, while production credit and women‗s groups had positive impacts. This is likely due to additional services offered from these other entities. Another financial service that is theoretically very beneficial for farmers—insurance—was not very popular even when bundled with high-yielding hybrid seeds in Malawi. Although this intervention responded to a seemingly clear market need (risk aversion), farmers did not take up the intervention as expected, probably because insurance is not commonplace in their context and is complicated to understand, especially when education levels are low and risk aversion is high. Sources: McKernan 2002; Kaboski and Townsend 2005; Gine and Yang 2007. The evidence in this category is limited, but two-thirds of the interventions included under microfinance resulted in positive impacts on rural and farm households. There are few IEs in this category—about 7 percent of all interventions covered in this report fall under microfinance, but these are usually part of interventions listed under different categories (particularly marketing value-chain interventions). The initial IE search produced a wide array of ongoing studies on microfinance and rural or agriculture populations.8 Most evaluations included in this category were originally excluded because of their weak link to productivity. Nonetheless, finding suitable microfinance evidence presented a challenge that resulted in exceptions being made to include IEs with a weak productivity link that still met the rigor criteria and had a clear economic welfare link to rural development. Microfinance interventions have positive impacts on various economic welfare measures. The thread connecting these IEs to agriculture outcomes runs through various economic welfare channels, such as household consumption, income, production, and profits. Many credit interventions seek to improve the provision of microcredit to the poor. In rural Bangladesh, for example, Khandker (2005) found a significant reduction in poverty among poor borrowers and within the local economy due to increased access to rural credit. The program helped raise per capita consumption and had spillover effects for nonparticipants. Similarly, in a rural-led credit expansion intervention, a government licensing policy promoted entry into the rural areas by lenders. This led to the opening of branches in areas previously without banks, resulting in an equalizing presence of banks 8 Most of these examples are listed in the Poverty Lab at MIT (JPAL) Web site or are currently being undertaken by the World Bank, Inter-American Development Bank (IADB), Millennium Challenge Corporation (MCC), and several academic institutions worldwide. 30 across the country and a reduction in rural poverty from increased savings and credit (Burgess and Pande 2005). Most microfinance IEs report positive results, but none directly tackle the measurement of agricultural productivity. The thinness of the evidence in this intervention category made it difficult to do more in-depth analysis or derive meaningful patterns. Nevertheless, the impact evaluations included show that the majority of the interventions had positive results on the indicators they measured (table 3.7). One-third of all interventions failed to show impact and none had negative impacts. Although none of the interventions were designed to tackle agricultural productivity directly, changes in income, consumption, and assets serve as acceptable proxies for changes in economic welfare. Table 3.7. Microfinance Interventions: Observations from the Literature Explanatory variables found Impact indicators Number of IEs with evidence in the IEs measured of impact on the indicator a – 0 + Access to services Welfare (assets, income) 2 Other welfare indicators at 1 1 the household level Local area and market 2 development indicators Noncredit Insurance adoption/use 1 Participation rates 1 1 Total observations 0 3 6b a. The symbols indicate the direction of impact and level of statistical significance. (–) = negative and significant; (0) = either positive or negative but not statistically significant; and (+) = positive and significant. Statistical significance is defined as significance of an estimated impact coefficient less than or equal to 10%. b. IEs may have more than one outcome measured and are likely to be listed more than once. OTHER INTERVENTIONS: INFRASTRUCTURE AND SAFETY NETS It is difficult to outline a comprehensive definition for such a diverse group of interventions, but their consideration in the analysis is crucial for achieving a fuller understanding. A few IEs—on infrastructure (rural roads), CDD, and social safety nets programs—that measured agricultural productivity rigorously filtered through the IE search despite their poor fit in the established categories (box 3.9). Interventions of these types generally have a broader developmental agenda and do not usually include ―productivity‖ as an explicit goal. They are included in the review as examples of broad- based development programs that often occur in rural areas and interact with productivity-enhancing activities in agriculture. Some of these interventions address the linkages among access to goods and services and improvements in agricultural production, off-farm diversification, and other income-earning opportunities. An example of such linkages are roads; the road improvement projects in this intervention group link isolated rural areas to more populous parts of the country (world), including markets. The IEs of interventions focused on rural roads measured the impacts of improvements in the quality of rural roads on consumption growth and poverty rates in Vietnam (Mu and Van de Walle 2007) and rural Ethiopia 31 (Dercon and others 2008).9 The IE of an intervention in Georgia analyzed the effect of various rural infrastructure projects on the economic welfare of rural households and found that road and bridge rehabilitation projects generated clear economic benefits at the community level. Box 3.9 Evaluations Listed as ―Other‖ Include Roads and Safety Net Programs Two IEs were of social safety net programs that sought to improve economic welfare through cash and access to credit, agricultural extension services, technology transfer, and irrigation and water harvesting schemes, both in Ethiopia (Gilligan and others 2008). One of the interventions, a cash transfer program (Productive Safety Nets Program, or PSNP), is the largest social protection program in Sub-Saharan Africa outside of South Africa. It provided cash transfers to prevent asset depletion at the household level and foment asset acquisitions at the community level. The Other Food Security Program complemented the PSNP and consisted of productivity-enhancing transfers or services seeking to increase incomes generated from agricultural activities and to build up assets. Results show that these programs have little impact on participants, due in part to transfer levels that fell far below program targets. Beneficiary households that received at least half of the intended transfers experienced a significant improvement in economic welfare (measured by food security). However, participants with access to both the PSNP and Other Food Security Program packages were more likely to experience improvements (or be food secure). This implies that the two-pronged approach of a safety net program— prevention of asset depletion and building of assets—has more impact than schemes that only rely on cash transfers as preventative measures. In Georgia, various rural infrastructure projects had mixed impacts on the economic welfare of rural households (indirect channel), but road repaving and repairing and bridge rehabilitation projects did result in some economic benefits, mostly through a 36-minute reduction in transport time (costs) and improved access to wider markets for agricultural products. An evaluation in Bangladesh found that the savings in household expenditures, possibly from decreased access costs due to a road construction project, were larger for beneficiary villages. This intervention also had positive impacts on income through better wages, decreases in agricultural inputs such as fertilizer prices, and increases in crop production. The project‘s wider impacts included a measurable effect on adult labor supply and children‘s outcomes. From a general economic standpoint, consumption increased 11 percent a year, and impacts were mostly felt by households at the 15th percentile of the consumption distribution. The largest agricultural project in Nigeria, a CDD project, gave support to communities to acquire infrastructure and productive assets, provided them with demand-driven advisory services, increased villagers‘ capacity to manage economic activities, and reduced conflicts among resource users. The IE for the project found a positive impact on the incomes of its farm household beneficiaries in the first year of operation. However, household incomes for the poorest asset tercile did not increase in the short term. Impacts were mostly possible due to reductions in time (to market and wait time) and transport costs (reduced fare prices due to increased availability) for farmers. There was also a measurable increase in the acquisition of 9 This IE also investigates this question with respect to investments in agricultural extension and was also included in the review list under the extension category. 32 productive assets by poorer farmers. Sources: Lokshin and Yemtsov 2005; Khandker and others 2006; Nkonya and others 2008. The constraints addressed by these interventions vary greatly, and their impact on agricultural outcomes is mixed. Fewer than half of the interventions included in the sample show a positive impact on an economic welfare outcome (table 3.8). However, the heterogeneity and likely ―missing‖ sample for this category makes it difficult to draw broad conclusions. Table 3.8. ―Other‖ Interventions: Observations from the Literature Explanatory variables found Impact indicators Number of IEs with evidence a in the IEs measured of impact on the indicator – 0 + Access to roads Consumption growth 2 Local area and market 1 development indicators Participation in a CDD program Household income 2 1 Participation in cash transfer Average values of indicators 2 alone of food security/consumption Participation in cash transfer 1 plus other components b Total observations 0 5 4 a. The symbols indicate the direction of impact and level of statistical significance. (–) = negative and significant; (0) = either positive or negative but not statistically significant; and (+) = positive and significant. Statistical significance is defined as significance of an estimated impact coefficient less than or equal to 10%. b. IEs may have more than one outcome measured and are likely to be listed more than once. 33 4. What Does the Evidence Reveal? Primary Constraints for Farmers The group of IEs analyzed identify six primary constraints for farmers to farm effectively, produce efficiently, and gain a profit; every intervention seeks to ease at least one of these. The six constraints are: low quality of inputs, limited knowledge, low quality of land or physical farm resources, nonconducive policy environment, limited outputs or post-production promotion assistance, and limited access to loans or insurance. Each intervention seeks to address one or more of these constraints, depending on its design and overall objectives. By breaking down interventions into their primary constraints it is possible to isolate the underpinnings of each intervention (and its subcomponents) and derive more meaningful findings. Sorting interventions into groups with similar cause-and-effect relationships makes it easier to discern patterns. This is especially useful for interventions with elements that cut across categories, such as in multicomponent interventions, addressing more than one constraint. This is especially true for interventions grouped under NRM technology, extension programs, and marketing, where various constraints are addressed with one large intervention. The section that follows outlines the main constraints identified in each intervention group and addressed in the IEs. Table 4.1 shows some examples. Table 4.1. Constraints Addressed by Evaluated Agricultural Interventions Primary constraints Examples of interventions addressing these constraints 1. Low quality of inputs Promotion and use of new input technologies (field application of research and development-derived technologies) Adoption of improved technologies and practices (high-yielding varieties, Bt seeds, wheat, disease-resistant varieties, soil conservation practices, intensive farming) Participation in herd health control program 2. Limited farmer knowledge Participation in FFS Participation in extension programs Visits from extension service Participation in project/program such as social safety net Access to extension services Participation in community-driven development programs Access to all weather road Exposure to program participants (like FFS, extension program) Exposure to inputs/practices from another program participation 3. Low quality of land or Access to irrigation and water infrastructure/technology physical farm resources Adoption of land structures (such as bunds, drainage system, and so forth) Adoption of soil conservation structures Adoption of soil fertility-enhancing technologies Participation in irrigation management transfer system 4. Nonconducive policy Policies that impact: titling, tenure security, reduce expropriation 34 Primary constraints Examples of interventions addressing these constraints environment risk, irrigation management 5. Limited output sales or Participation in project/program that promotes products (special commercial assistance marketing channels) Participation in contracts (related to output) Participation in contract for export crops Participation in interlinked input-output-credit arrangements for export crops Access to all-weather road 6. Limited access to credit or Access to loans, grants, in-kind goods, equipment insurance Access to services related to credit or insurance This analysis is most useful for Table 4.2. Results by Agricultural Indicator interventions with elements that cut across categories and address more Positive Nonpositive than one constraint. This approach is Indicator impact (%) impact (%) especially true for multicomponent interventions that fall under the NRM Yield 62 38 technology and marketing groups Income 42 58 because they are often designed to Production 56 44 address more than one constraint. Each IE is assigned to a primary constraint Profit 87 13 category based on the focus of the evaluation; this categorization draws from the IEs and the broader literature. Figure 4.1 presents a simplified illustration that positions each intervention category along the agricultural production process and links each category to the primary constraint addressed. The illustration, derived from basic agricultural production principles, indicates that farm inputs, knowledge, and land quality are essential for production to take place. Once the crop is produced (yield) it can be sold (income) in the market or consumed by the household. Both reform and financial service interventions can take place at any time in the production process, hence their position in the flow chart. The letters A–H represent the intervention categories, and the words associated with them are the primary constraints each intervention type seeks to address. For example, farm input is related to intervention category D (improved input technology); this category addresses the lack of quality inputs farmers encounter in their production process. 35 Figure 4.1. Intervention Categories and Primary Constraints POLICY ENVIRONMENT A Reform OUTPUT/POST- ACCESS AND PRODUCTION PROMOTION QUALITY OF AND COMMERCE INPUTS D H Farm Inputs H F Income or Production FARMER Consumption B H Knowledge KNOWLEDGE QUALITY OF Plot Quality C E LAND/FARM RESOURCES Financial Services ACCESS TO CREDIT AND INSURANCE G Source: Author’s illustration. Note: The box reflects the eight categories of interventions and the six primary constraints: (A) land titling and reform, (B) extension services, (C) irrigation, (D) natural resource management (NRM) technology, (E) input technology, or non-NRM, (F) marketing, (G) microfinance, and (H) other. Summary of the Results Reported by the Impact Evaluations RESULTS BY AGRICULTURAL INDICATOR The heterogeneity of the causes and effects being addressed and measured in agricultural interventions makes it difficult to assess causality. The analysis undertaken for this report focuses on impacts on productivity and economic welfare. However, the IEs selected for this study do not measure these indicators uniformly. Therefore, a step toward ensuring some comparability across observations is to concentrate on IEs that report similarly defined outcomes: yields, income, production, profits, and input use. Yields and income are the most common agricultural outcome indicators measured in the IEs. Yields in this analysis are defined as production or labor per total area of cultivated land; it was measured in about 39 percent of all interventions. ―Agricultural income‖ means earnings from agriculture, ―income‖ means earnings from all activities, and ―consumption‖ means expenditures at the household level; this group of economic welfare variables is the second most measured (24 percent). Production is defined as the amount of farm production cultivated and farmed; about 9 percent of all interventions measure this outcome. Last, profits are marginal gains or net benefits (sales minus costs) reported by farmers, found in about 8 percent of all IEs. The rest of the interventions (and some of the ones also reporting other direct outcomes) report inputs or other outcomes indirectly related to productivity. These are defined as intermediary goods such as fertilizer use, investments in agricultural tools, or knowledge acquired that may or may not be linked to changes in final outcomes in the IE but are expected to lead to improvements over time. (See table 2.2 for detailed examples.) Results show that across different categories of interventions for which IEs have been done there is general success in improving agricultural yields and profits. The results summarized in table 4.2. show that IEs of interventions that sought to improve productivity 36 (specifically yields) and profits most often reported positive results (62 and 87 percent, respectively). The evaluations that failed to establish a positive causal relationship reduced the overall efficiency and effectiveness of agricultural interventions in fostering a desired change. These were most common among interventions that sought to affect farm household income. IEs for more than half of interventions in Africa report positive impacts. When yields were the outcome indicator measured, the likelihood of success was even higher. Thirty- two countries are represented in the sample and most interventions were in Africa, with 55 percent of positive results. In terms of the outcomes targeted, a similar number of interventions aimed at improving yields and farm household income, but interventions focusing on yields had higher reported success rates. With respect to World Bank-supported interventions, results were predominantly positive; about 68 percent of all such interventions led to positive change, and most measured yield changes. A wide set of outcome indicators related to farm performance were measured. Among the ones focused on in this report, yield is most commonly reported in IEs of Bank-supported projects and the most likely to exhibit positive results. When the outcome is farm income, results are more mixed, indicating that the evaluated interventions seemed to do well when aiming to improve agricultural performance directly rather than through economic welfare. RESULTS BY INTERVENTION TYPE In the following sections, the results, in terms of positive or negative impacts, are reported across different intervention types. It should be borne in mind that this is not a guide to success rates of interventions in general but only to those measured by the IE literature. The small number of studies, the potential for selection bias, and the focus here on only IE studies (excluding evaluations using other approaches) are limiting, and hence the results are not necessarily representative or generalizable. About 41 percent of all interventions with impact evaluations show negative or nonsignificant impacts on various agricultural dimensions. Positive impacts (59 percent) are most evident for non-NRM or input technology interventions, where impacts were consistent across several agricultural indicators. Positive impact is defined as evidence of a positive relationship between the relevant indicator and the treatment; nonpositive is defined as any evidence that is either negative or not statistically significant (table 4.3). The subsequent analysis—by constraints addressed by the interventions—disentangles the findings further. Although some of the results may seem to contradict the results shown in this section—by intervention category—it is important to note that interventions are broad categories with very distinct within-group characteristics. Thus, an intervention may fall under one primary category but have several components addressing various constraints. Box 4.1 illustrates the variety of interventions and results in the evidence pool. 37 Box 4.1. Interventions That Delivered Mixed Results on Productivity or Income A land reform intervention in Vietnam enabled an extensive rural land titling program that led to measurable improvements on the decisions of households to undertake long-term agricultural investments and, at the same time, devote labor to nonfarm activities. These results are not very large in magnitude, however, and more work should be done to assess the long-term impacts. Another intervention in Vietnam designed to promote land consolidation in a region characterized by extensive land fragmentation did not lead to changes in productivity because the law is complex, the consolidation system (including workers) is not set up to motivate changes, and farmers do not have incentives to take advantage of the legal change. Few IEs look at how credit access varies with farmers‘ ability to use land as collateral, and these studies do not reach a consensus on impact. For example, titleholders in Zambia have greater credit use; another study found no conclusive impact of a Peruvian titling project on obtaining credit. The third potential impact channel—the facilitation of land transactions such as renting, buying, and selling—was evaluated in a land reform intervention in Bulgaria. The IE found that land allocation is still suboptimal because historical ownership and legal constraints can reduce effective decision making. Sources: Do and Iyer 2008; Del Carpio and others 2009; Smith 2004; Torero and Field 2005; and Vranken and others 2008. Table 4.3. Results by Intervention Category Number of IEs showing impact on an agricultural indicator Not Nonpositive Intervention category Negative significant Positive impacts (%) Land reform 2 5 13 35 Extension 1 12 13 50 Irrigation 1 3 7 36 NRM 2 5 8 47 Input technology 2 1 8 27 Marketing 2 3 9 36 Microfinance — 3 6 33 Other — 5 4 55 Total 10 37 68 41 IEs of land tenancy and titling interventions show that about 65 percent of interventions resulted in improvements; 50 percent of extension interventions showed improvements on various farm indicators. These interventions took place across regions and exhibit a large heterogeneity of characteristics. More specifically, these two broad categories of interventions include activities as diverse as titling, tenancy law, inheritance law, FFSs, farmer technical advisory services, and market information services. The land tenancy category showed only 35 percent nonpositive results. These reform-type interventions were generally at the national level and were put in place by the government in response to an existing need. Their designs were generally simple and easy to implement and did not target a specific population in the rural area. In extension services, half of the interventions had a negative impact on beneficiary knowledge or farm yields. A more in- 38 depth review of extension intervention IEs shows that impacts through the diffusion of knowledge (known as ―train the trainer‖) modality often failed to show impacts. The main reason provided for the lack of ―demonstration effects‖ is the inability of participant farmers to convey complex decision-making skills effectively to other farmers. More than one-third of all irrigation and marketing interventions with impact evaluations showed nonpositive impacts on an agricultural or farm income indicator. The nonpositive impacts in irrigation interventions pertain to micro-dams or water reservoirs rather than canals or other irrigation infrastructure or technology. The most common reasons for negative or no impacts are losses from water disruption and increased incidence of disease among farmers caused by large-scale impounding of water. The construction of a microdam in Ethiopia exemplifies how, despite production increases, labor productivity among male farmers decreased due to illness. For interventions classified under marketing, such as farm-group arrangements and value-chain participation, results are mostly positive (64 percent) on yields, crop prices (and profits), and value of production. Positive results are primarily due to improvements in access to modern inputs, farming technology, and wider markets resulting from participating in value-chain or contract activities. Results for NRM interventions with impact evaluations show a little more than half had positive impacts. Fifty-three percent of NRM interventions positively affected farm yields or another related agricultural outcome. Interventions that promoted the application of new technologies to subsequently change the structure or composition of the soil had the most positive results. Among the most common activities in this category are conservation practices and conservation structures. Unfortunately, only some of the interventions led to positive results. In El Salvador, Cocchi and Bravo-Ureta (2007) found a positive association between conservation practices (such as crop residue mulching, minimum and zero tillage, crop rotation, green manure, and contour tillage) and income, but the effect of conservation structures (such as terraces, ditches, live barriers, stone walls) on income was negative and not statistically significant. Input technology (or non-NRM) interventions with IEs report a high rate of positive results (over 70 percent). The impacts of non-NRM technologies, such as the development and adoption of improved varieties (such as beans and cocoa), fertilizers, and seeds developed using transgenic technology (such as Bt cotton), showed clear improvements on farm outcomes, with over 70 percent positive impacts. Impacts of varietal technology that incorporate improved features (such as disease resistance) without increasing the need for additional inputs are considered optimal for poor farmers with limited resources and training. Higher-yielding crops and technologically enhanced inputs bring about clear benefits to farmers by reducing their risks, enhancing their farming efficiency, improving their production and incomes, and lowering food prices. The evidence for microfinance and the ―other‖ category shows distinct patterns of impacts on rural development outcomes related to farm performance. Microfinance interventions with IEs, which include the provision of credit, access to finance services, and various saving schemes, reported positive impacts in 67 percent of interventions. Several IEs in this category mention good targeting and complementarity of services as contributors to their success. The ―other‖ category, which includes infrastructure and 39 safety net interventions, reports improving productivity or income only 45 percent of the time. An explanation is that these interventions were not designed to improve agricultural indicators; instead, their focus was on rural development more broadly. As a result, few of the evaluations of these interventions measured impacts on agricultural outcomes directly, most concentrated on household income and consumption. The thinness and diversity of objectives of both categories made it most difficult to derive many meaningful patterns. RESULTS BY AGRICULTURAL CONSTRAINT Interventions with impact evaluations that sought to improve agricultural performance by addressing access to technologically enhanced inputs, promoting post-production activities, promoting farmer knowledge, and enacting policy reforms had the more impact. Farmers face many constraints to operating productively. Activities that assist in the access to input technology such as enhanced seeds, and the promotion of farmer knowledge via training, and sales and commerce channels report overwhelmingly positive impacts (above 60 percent of the time, table 4.4). Similarly, interventions seeking to improve the policy environments and affecting institutional change (such as tenancy law, property rights, and export promotion) report high rates of success (60 percent). In contexts producing high-value products, tenancy security and export openness likely provide producers with incentives to invest more in their plots and improve their yields. Conversely, investment is unlikely to take place in environments where regulation imposes high transaction costs on farmers. Table 4.4. Results by Constraint a Positive impact by Primary constraint addressed constraint addressed (%) Access and quality of inputs 61 Farmer knowledge 61 Quality of land/farm resources 37 Policy environment 59 Output/post-production promotion and 63 commerce Access to credit and insurance 55 a. Some interventions address more than one constraint. The analysis also shows that most interventions with IEs contain an element of technological innovation, particularly those interventions that deal with input enhancement and crop intensification through resource management or water. The IEs reviewed offer many successful cases, especially related to input technology, where impacts were consistent across several outcomes. Examples of technology-modified crops are Bt cotton, hybrid maize, and rice varieties; examples of general food crops affected by increases in production made possible through fertilizer use include regular maize, rice, beans, and cassava. Also, most input-related project IEs cite the historical evolution of crop improvements and chemical fertilizers, suitable for rain-fed agriculture, as catalysts for productivity improvements. It is likely that technological improvements 40 made possible through these projects produce steady growth in total factor productivity, which can be responsible for increases in output. Interventions that enhanced land quality—mostly focusing on improving soil conditions—had the lowest reported rate of success. Interventions that addressed farm productivity and income through irrigation and NRM techniques (such as conservation structures, contour bunding, and so on) report having been successful less than 40 percent of the time. These interventions often are complex; they rely on farmers having incentives (and resources) to seek, accept, and adopt a new technology. They also require farmers‘ knowledge (and aptitude) to implement and use the technology properly. Finally, in contexts where the technology is new, other components, such as extension services, precede or seem to be necessary accompaniments to these interventions. In irrigation interventions, the IEs show that lackluster results are more due to factors related to poor project design, negative spillovers, and mismeasurement, than to the availability of water. Interventions addressing credit and insurance constraints are crucial for the success of many other components. Reported successes among interventions in this group are mostly positive when the outcome sought to be affected is farm income. The opposite is true when the outcome sought is yields, indicating that impact of financial resources is better measured through economic outcomes than farm performance. Easing the access to credit constraints is also a typical complementary component to value-chain interventions, but incentives for creditors to lend are more apparent toward the end of the production cycle, when the conversion to income is easier and repayment is more likely. Lessons by Design and Implementation Characteristics Design and implementation elements of interventions appear to influence the results. This section elaborates on how two sets of factors can influence intervention outcomes. The first factor is related to who implemented the program and the second is related to the design and scale of the intervention (single component versus multiple features and national, subnational, or local). SOME RELEVANT INTERVENTION DESIGN FEATURES Output promotion interventions report positive impacts when complemented by input technology, market reform, and access to credit components. Marketing-related interventions benefit from being implemented alongside other programs or components, especially when input technology, credit programs, and regulatory changes take place. Qualitative analysis of the evaluations found that many interventions were inspired from past experiences, particularly during the Green Revolution. A handful of interventions in this category were motivated by examples where gains were possible as a result of reforms to open markets. These IEs state that secure markets motivated innovation (property rights) and adoption by farmers, which, in turn, catalyzed productivity gains. Both government and the private sector played key roles during that period. The government enabled market reliability through secured tenure and free-market laws that fostered competition, whereas private actors engaged in technological innovation and product promotion (Dorward and others 2004). 41 Land-enhancing interventions in the evidence group were mostly implemented on their own; their complexity may require other components for them to be successful, which may help explain why they show negative or no impact. Irrigation interventions often require farmers to have knowledge or skill to use water effectively and operate the system efficiently. Lack of knowledge or training is cited as a barrier to adopting the technology or using it effectively and led to high transition costs that hindered a wider spread of benefits. Policy reform worked best when complemented by credit and output promotion interventions. The analysis shows that policy changes or reforms were most effective when farmers had access to financial services and when market promotion interventions helped farmers take advantage of the change in the law. This is evident in many market reform interventions where private firms engaged in contract farming and farmers sold their products to many private firms. SOME RELEVANT IMPLEMENTATION FEATURES Interventions implemented by the private sector report a high level of positive outcomes, particularly for input technology interventions. Evaluations show a variety of actors in this role, from the private sector to the government. Also, different types of actors often work in conjunction, including nongovernmental organizations and donor agencies such as the World Bank. The analysis shows that the role of private sector entities as primary or sole implementers was important in input technology projects and output promotion projects. One could expect private firms to have an incentive to invest when positive impacts can be assured, which, in turn, result in profits. Two examples are the promotion of high-yielding cotton seeds in Argentina and the promotion of export rice in Madagascar, where farmers were given high-value crops to plant and were trained to comply with specific export requirements. For credit projects, private sector implementers show positive results when the outcome indicator measured is income. Almost all microfinance interventions included were not strictly designed as agricultural credit but rather functioned more broadly as rural credit. The difference is important because although the projects did not target farmers specifically, some of them still resulted in measurable impacts on economic welfare, which indirectly influenced farmer behavior. The analysis reveals that the private sector plays a significant role in implementing successful credit projects, which is consistent with the incentives explanation previously posited. An important avenue for future research is to incorporate new impact evaluations of microfinance projects in agriculture and revisit this analysis. Governments are important in implementing some interventions. Policy reform projects—mostly land, marketing, and microcredit interventions—were mostly implemented by the government and showed a high rate of positive results on various agricultural indicators. There are examples of public-private collaborations that also result in positive impacts; however, results are more likely to be mixed than when the government is the sole implementer. It is not entirely clear whether the negative results are due to the complexity of some intervention designs (usually the case with public- private collaborations) or the difficulty in the collaboration between the actors. 42 Fewer than half of the interventions were implemented at the national level. The scale of the intervention is considered a relevant implementation (and design) feature that can help influence the results. Some interventions, such as policies, and some land enhancing projects, such as irrigation, are likely to be implemented at a higher scale than interventions targeted to enhance inputs, outputs, or farmers‘ knowledge. Of approximately 49 interventions implemented at the national level, 23 aimed at improving yields; of those, 61 percent report positive impacts. The rest of the interventions were implemented at subnational or local levels. There are several explanations for why large-scale projects may be more difficult to implement; the main one is capacity limitations. For example, pilot projects (generally small in scale) are often implemented in localized settings, where distributing the benefits, monitoring achievements, and adjusting the implementation are more feasible. In countries where the implementing agency has limited resources, it is more difficult to ensure the distribution of benefits in a highly decentralized setting. Similarly, large-scale interventions, such as policy reforms, are more difficult to evaluate rigorously because it is more difficult to identify a suitable counterfactual group unaffected by the intervention. 43 5. Conclusion The evidentiary base is still too thin to derive broad-based conclusions. Two concerns affect the results obtained by this report. The first is the gaps in evidence and limited number of rigorous impact evaluations in agriculture. The second concern is potential publication bias in the evaluations reviewed, particularly because newer evaluations seem more likely to report positive results, which may reflect a bias rather than improved performance. Despite these concerns, findings from this report can help establish an analytical framework, derive a taxonomy of interventions, and outline standards for IE quality. Moreover, it is hoped that this report will further motivate the use of impact evaluations to measure results in agricultural projects and identify what characterizes successful intervention in various settings around the world. Bearing these limitations in mind, differences in performance across interventions with impact evaluations can provide useful information. Well over half of all interventions that aimed to increase agricultural performance—mainly yields, production, and profit—show positive impacts; results are more mixed when income is the outcome of interest. The data set compiled from the IEs shows that the overall results on outcomes such as farm yield, production, and profits were successful more than half the time, and higher for World Bank-supported interventions. However, when the outcome is farm income, the percentage of positive impacts reduces to 42 percent. As more evidence with similar outcome indicators becomes available, more generalizable conclusions should be possible. The exercise of mapping and organizing agricultural IEs provides an analytical framework for further investigation of what the IE literature indicates has been more effective in agriculture. Ongoing and new IEs should allow greater understanding of the relative value of different types of interventions in improving agricultural performance. Having a larger evidence base will provide insight into what works in different contexts and intervention categories. It may also help guide future project design and investments. The results show that some aspects of the interventions—mainly the constraints addressed, the design, and implementation characteristics—can influence the results. Complex interventions, such as those with multiple components addressing various aspects of production, generally address complementary activities needed to improve productivity. However, the evidence shows that when interventions were too complex, and capacity to implement them was low, negative results followed. Several IEs demonstrated that most types of interventions, especially those that involved new technologies, needed to be complemented by knowledge and credit-related activities to deliver their full impact. The analysis demonstrates that only a subgroup of interventions was effective when implemented alone, but most interventions did best when complemented by other activities 44 Appendix A. Table of Impact Evaluations Analyzed Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 1 Aker, Jenny Does Digital 2008 Extension Market What is the impact of Niger 1996- QE Price Cell phones reduced price dispersion Divide or information cell phones (and 2006 dispersion across grain markets, with a larger Provide? The access to information) (profits) increase for markets far apart. Impact of Cell on grain market Phones on performance? Grain Markets in Niger 2 Amacher, GregoryDisease, 2004 Irrigation Dams How do microdams Ethiopia 1996-97 NE Productivity Microdams do, in fact, negatively impact S., Lire Ersado, Microdams and and disease affect the (crop and fuel household resources through disease. Donald Leo Natural time allocation of wood But the impacts depend on the age and Grebner, and Resources in household members production and location of microdams, as well as the William F. Hyde Tigray, to productive total marginal effects of microdams on Ethiopia: activities? productivity) disease, production, and income Impacts on generation for the household. Older Productivity dams increase production of all goods, and Labour but they also result in more time sick for Supplies household members. 3 Arjunan Village-wide 2008 Input Seeds What are the village- India 2004 QE Yield The simulation results demonstrate Subramanian, Effects of technology wide impacts of the substantial labor market effects, which in Matin Qaim Agricultural adoption of Bt Cotton particular should not be ignored. Bt Biotechnology: on some indicators of cotton is associated with a substantial The Case of Bt economic welfare? overall generation of rural employment. Cotton in India While labor requirements for pest control decrease, more labor is employed for harvesting. This has interesting gender implications. The aggregation of total wage income shows that females earn much more from Bt cotton than males. 4 Ashraf, Nava, Finding Missing 2008 Marketing Interlinked What are productivity Kenya 2004-05 E Value of The middle-income farmers were most Xavier Giné, Markets (and a credit/input/ impacts of DrumNet, a harvested likely (relative to low-income and high- and Dean Disturbing output cashless micro-credit produce income) to switch to export crop in Karlan Epilogue): marketing program that links response to this project. Comparing Evidence from arrangements commercial banks, members that were offered credit to an Export Crop smallholder farmers, those that were not, we find that credit Adoption and and retail providers of increases participation in DrumNet but Marketing farm inputs? does not translate into higher income Intervention in gains relative to the noncredit treatment Kenya group. The epilogue to this project is not good as the authors report that the project has collapsed because of the 45 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator failure to meet EurepGap certification for export to Europe. 5 Bandyopadhy Yield Impact of 2007 Irrigation Irrigation Does IMT have an Philippines 2003 QE yield The presence of IMT is associated with ay, Sushenjit, Irrigation management effect on farm-level an increase in maintenance activities Priya Management outcomes because of undertaken by irrigation associations. Shyamsundar, Transfer: Story increased timeliness Rice production in IMT irrigation and Mei Xie from the and distribution in associations is higher even after Philippines water delivery and controlling for various differences among decreased water rice farmers in IMT and non-IMT irrigation losses due to associations. IMT is associated with a improved reduction in technical inefficiencies in maintenance? production. 6 Banerjee, Empowerment 2002 Land Tenancy law What are the India 1977-93 QE yield Operation Barga explains around 28 Abhijit, V., and Efficiency: reform agricultural percent of the subsequent growth of Paul J. Tenancy productivity effects of agricultural productivity in West Bengal. Gertler, and Reform in West a major change in But to get more precise estimates, micro- Maitreesh Bengal property rights level data are required. Ghatak (tenancy reform) in the India State of West Bengal? 7 Bardhan, Productivity 2008 Land Titling What was the impact India 1981-95 QE Rice yields and Study found significant effects of Pranab and Effects of Land reform of reforms in land farm value Operation Barga on rice yields and farm Dilip Reform: A property rights on farm added per acre value added per acre, but somewhat Mookherjee Study of productivity in the smaller in magnitude compared with Disaggregated Indian state of West Banerjee and others. Farm Data in Bengal spanning West Bengal, 1982–95? India 8 Barrett, Better 2004 Technology Crop What are the Madagascar 2001 NE Mean yields The SRI technology generates significant Christopher, Technology, NRM intensification productivity gains from yield gains but the increased yield risk Christine Better Plots, or SRI technology? Are makes it unattractive to farmers within Moser, Oloro Better labor requirements the standard range of relative risk McHugh, and Farmers? increased under SRI? aversion. This is mostly due to higher Joeli Barison Identifying What is the labor inputs required by the method. Changes in explanation for 46 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator Productivity adoption and and Risk disadoption? among Malagasy Rice Farmers 9 Benin, Impact 2008 Extension Technical/advis What are the impacts Uganda 2004, QE Productivity The NAADS program has had significant Samual, Evaluation and ory services of the program on 2007 impact on crop productivity, with the Ephraim Returns to empowerment of value of gross crop output per acre Nkonya, Investment of farmers, agricultural having increased by up to 29 percent for Geresom the National productivity, market those participating directly in the NAADS Okecho, Agricultural participation, income, program, but for livestock productivity Josee Advisory assets, food and has contributed to a decline (about 27-45 Randriamamo Services nutrition security, and percent) in the value of gross output per njy, Edward (NAADS) welfare? unit of animal, as compared with their Kato, Geofrey Programs of nonparticipant counterparts. Lubade, Uganda Miriam Kyotalimye, and Francis Byekwaso 10 Bernard, Impact of 2008 Marketing Cooperatives Are cooperatives Ethiopia QE Yields, price Cooperative members get a higher price Tanguy, Cooperatives associated with better for cereals in the market; however, the Alemayehu on Small prices for their overall share of cereal production sold Taffesse, Holders members? commercially is not different for members Eleni Gabre- Commercializat and nonmembers. Small farmers Madhin ion Behavior: produce less in response to higher Evidence from prices. The opposite is true for large Ethiopia farmers. 11 Bolwig, The Economics 2009 Extension Technical What are the impacts Uganda 2006 QE revenue, yields The study finds that both, participation Simon, Peter of Smallholder TA, Input advisory, seeds on profit of per tree and the technology itself lead to Gibbon and Organic technology participating in an increases in revenue. Participation has Sam Jones Contract organic coffee contract an impact (75 percent) through access to Farming in farming scheme? And, guaranteed price premiums. The effect Tropical Africa what are the effects on of the technology is more modest (9 profits of using the percent) but significant, and it is special farming measured by yields per tree. techniques? 12 Brambilla, Farm 2005 Policy Contracts What are the impacts Zambia 1997- QE Productivity The early phase led farmers to have Irene and Productivity reform of cotton market 2002 (volume of significant reductions in farm productivity. Guido Porter and Market reforms on farm cotton The latter phase led to significant Structure: productivity and crop production per investment in land and, subsequently, to Evidence from choice? hectare yield per hectare (productivity). 47 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator Cotton Reforms produced) in Zambia 13 Burgess, Do Rural Banks 2005 Microcredit Access and Did state-led rural India 1961- NE Poverty (rural The paper finds that the licensing policy Robin and Matter? services branch expansion 2000 headcount led to the opening of branches in rural Rohindi Evidence from reduce poverty in rural ratio) areas previously not banked and this led Pande the Indian India? to an equalizing presence of banks Social Banking across the country. The reductions in Experiment rural poverty were directly linked to increased savings and credit provision in rural areas. 15 Cerdan- The Impact of 2008 Extension Technical What is the impact of Argentina 2002- QE crop yield The program was effective at increasing Infantes, Agricultural /advisory the provision of 2006 (kg/ha) yields for those with low yields before Pedro, Extension services agricultural extension participation, and large producers. In Alessandro Services: The services for grape addition, the average quality of grapes Maffioli and Case of Grape producers in Mendoza increased, especially for those at the top Diego Ubfal Production in in increasing quality of the yield distribution. However, there is Argentina and production? no evidence that the program served the needs of other beneficiaries who did not see their yields or the quality of their grapes increase. A plausible explanation might be that these producers received extension services in a menu of topics that did not match their needs. 16 Cocchi, On-Site Costs 2007 Technology Soil and water What is the El Salvador 2002- QE farm income Crop diversification and the adoption of Horacio and and Benefits of NRM conservation relationship between 2005 conservation practices are significantly Boris E. Soil farm income, adoption promoted by the length of farmers’ Bravo-Ureta Conservation of conservation involvement with PAES and by their Among Hillside technologies and participation in social organizations. Farmers in El output diversification Diversification significantly increases Salvador among PAES farm income. The positive association participants?. between conservation practices and income contrasts with the effects of conservation structures, which is negative and nonsignificant. 17 Conley, Learning about 2005 Marketing Social learning How do innovations in Ghana 1996-98 QE Change in Farmers are more likely to change input Timothy and a New farmer fertilizer use fertilizer use, levels on new technology (fertilizer for Christopher Technology: respond to news about innovation in pineapple) upon the receipt of bad news Udry Pineapple in fertilizer productivity fertilizer use about the profitability of their previous Ghana on pineapple level of fertilizer use and less likely to cultivation from his change when they get bad news that 48 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator neighbors experience? alternative levels lead to lower profitability. Inexperienced farmers are more responsive to news in their information networks. 18 Dalton, Impact of 2007 Technology Soil and water Has the intervention Thailand and 2003 NE Yield change Controlling for the treatment effects, Timothy, J., Participatory NRM conservation enhanced the Vietnam participation was positively related to Nina K. Lilja, Natural adoption of more yield increases over non-participants. Nancy Resource sustainable production The project achieved significant levels of Johnson, and Management practices through adoption, especially for soil conservation Reinhardt Research in farmer participatory practices. The adoption of hedgerows Howeler Cassava-Based methods? was linked to productivity impacts, while Cropping the adoption of contour ridging to a Systems in reduction of cropped area. Study also Vietnam and finds additional benefits to participatory Thailand research activities that are not embodied in the adoption of soil conservation or fertility management techniques. 19 X. Del Carpio, Is Irrigation 2010 Irrigation, Access to Does irrigation Peru 1998- QE Yields, Labor, Irrigation projects had a positive impact N. Loayza, Rehabilitation extension irrigation, increase crop 2007 Consumption on the poor but not by increasing and G. Datar Good for Poor irrigation mgt., diversification, production in their own small plots but by Farmers? A technical production, and overall providing them with better employment nonexperiment advisory productivity? opportunities on larger farms. al IE in Peru 20 Datar, G., X. Land, Grants, 2009 Other Land transfer Has a land Malawi 2006- QE Production, The program has made a positive impact Del Carpio and Title: The resettlement program 2009 Diversification, on production and livestock holdings, and V. Impact of a improved farmer Food access which is somewhat less pronounced for Hoffman Land productivity and those who moved great distances. The Resettlement welfare, and how does evaluation demonstrates mixed results Program on this impact vary on welfare measures, such as food Productivity across gender, security and asset accumulation, and Welfare in distance moved, and although women accumulated more Malawi farming experience? assets. Impacts hardly differed across farming experience. 49 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 21 Deininger, Investment and 2003 Land Titling Is the propensity to Nicaragua 1994, QE Value of land Registration, acquisition through Klaus and Equity Effects reform invest higher on plots 1996 investment (log purchase, and agrarian reform title are all Juan of Land that were registered, of the self- associated with signi�cant increases in Sebastian Regularization: purchased, or assessed land the value of plots, and the propensity to Chamorro the Case of received agrarian price) invest was higher for these households. Nicaragua reform titles? The program for land titling and registration also shifted investment toward land-related items with high economic returns, which were previously discriminated against. The program also caused an appreciable increase in land prices. 22 Deininger, Land Rental 2007 Land Tenancy law What are the impacts China 2001- NE productivity of Gains in productivity of land use through Klaus and Markets in the reform of rental markets on 2004 land use land rental in a growing economy can be Songqing Jin Process of households’ economic through land large, with estimated productivity Rural Structural strategies and welfare, rental increases of some 60%. These translate Transformation, and the productivity of into improvements of 25% in tenants’ Productivity land use at the plot welfare and, by facilitating occupational and Equity level? diversification, even larger increases in Impacts in landlords’ income. There is no evidence China that rental puts the poor at a disadvantage; to the contrary the factor equalization and growth effects found here imply that, as better educated individuals join the nonagricultural labor force, the poor and less educated can gain by renting additional land. 23 Deininger, Impacts of 2008 Land Tenancy law What are the Ethiopia 1999- NE new investment The positive results suggest that the way Klaus, Daniel Land reform economic impacts of a 2007 in or repairs of in which Ethiopia implemented land Ayalew Ali, Certification on low-cost registration conservation certification responded to local needs. and Tekie Tenure program as structures The fact that most disputes could be Alemu Security, implemented in the during the last resolved in the field, our Investment, Amhara region of 12 months failure to find outcomes biased in favor of and Land Ethiopia on perceived the wealthy or against women, reinforced Markets. tenure security, land- by initial evidence of positive investment Evidence from related investment, and transfer effects, all support this view. Ethiopia and land market participation. 50 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 24 Deininger, Rural Land 2008 Land Titling What are the costs, Ethiopia 2006 NE New land- The rapid speed, participatory nature, Klaus, Daniel Certification in reform and the incidence and related and low cost of Ethiopia’s land Ayalew Ali, Ethiopia: preliminary impact of investment in certification, together with the positive Stein Holden, Process, Initial Ethiopia's massive the past 12 results from this process and the and Jaap Impact, and land titling efforts months absence of bias in favor of the wealthy or Zevenbergen Implications for (under which over a lack of access to information by the poor Other African short period, demonstrate that, contrary to what one Countries certificates to more might be tempted to conclude from than 20 million plots experience in other countries, large-scale have been issued)? and rapid delivery of land certificates in a participatory way is possible. Users’ positive assessment suggest that the way in which Ethiopia implemented land certification responded to local needs. 25 Del Carpio, What Impedes 2009 Extension Information What is the impact of Vietnam 2004- E Land The pilot intervention did not lead to X., T. Do, G. Farmers From an experimentally 2008 consolidation, consolidation. On the contrary, there is Gutierrez, T. Consolidating designed pilot yields,consump quantitative evidence that further Le Dung, A. their Land? information and tion fragmentation took place, potentially due Waxman Evidence from capacity building to continuance in splitting land among an Experiment campaign on farm family members as an inheritance in Vietnam household outcomes mechanism or something related. No in a highly fragmented impacts on agriculture or income were area in Northern observed. Vietnam? 26 Dercon, The Impact of 2008 Other, Rural roads, What are the impacts Ethiopia 1994- NE consumption Receiving at least one extension visit Stefan, Daniel Agricultural extension technical of two forms of public 2004 growth reduces headcount poverty by 9.8 Gilligan, John Extension and advisory investments – percentage points and increases Hoddinott, Roads on extension and roads – consumption growth by 7.1 percent. and Tassew Poverty and on consumption Access to all-weather roads reduces Woldehanna Consumption growth and poverty in poverty by 6.9 percentage points and Growth in rural areas? increases consumption growth by 16.3 Fifteen percent. Ethiopian Villages 27 Dey, Madan Impact of 2006 Technology Integrated What are the Malawi 2004 QE Total This study is an example of a M., P. Development NRM aquaculture- economic, health and household comprehensive impact assessment Kambewa, M. and agriculture welfare effects of a income framework using a combination of Prein, D. Dissemination small-scale integrated methodological tools to assess the Jamu, F.J. of Integrated aquaculture impact of a complex natural resource Paraguas, Aquaculture- agriculture (IAA) management R&D project. For the D.E. Pemsl Agriculture technology project's analysis, technology adoption and the and R.M. (IAA) outputs in Malawi? impacts on the farm household as well as Briones Technologies in economy-wide welfare effects of the Malawi project were attributed to two separate 51 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator outputs: (i) an integrated multi-output multi-input technology; and (ii) a technology transfer model. 28 Diagne, Aliou Diffusion and 2006 Technology Crop What is the treatment Cote d'Ivoire 2000- QE Knowledge, The impact of the campaign is estimated Adoption of NRM intensification effect of a New Rice 2001 adoption of rice to be positive and significant for adoption NERICA Rice for Africa varieties of new rice varieties. Varieties in dissemination effort on Cote D'Ivoire adoption of improved rice varieties? 29 Dillon, Andrew Access to 2008 Irrigation Access to 1) Do the gains in Mali 1998, QE Agricultural Regardless of the estimation method Irrigation and irrigation agricultural production 2006 Production used to evaluate irrigation investments in the Escape induced by irrigation northern Mali, significant positive from Poverty yield higher household increases in total household savings, and (2) Do consumption, agricultural production, and intra-village transfers caloric and protein intakes are estimated from irrigators to non- for households who have access to irrigators contribute to irrigation. These results reinforce informal social previous studies on smallholder irrigation insurance? investments by showing that, in an area with low agricultural potential, welfare gains can be realized with targeted investment. 30 Do, Quy-Toan Land Titling 2008 Land Tenancy law What are the impact of Vietnam 1992, QE proportion of Vietnam’s 1993 Land Law made land and Lakshmi and Rural reform the 1993 Land Law of 1998 multiyear crops rights secure, pledgable and tradable, Iyer Transition in Vietnam on increases cultivated and was implemented by means of an Vietnam in the share of total extensive rural land titling program. area devoted to long- Results of this IE show that this reform term crops and in had a statistically signi�cant impact on labor devoted to the decisions of households to undertake nonfarm activities? long-term agricultural investments and at the same time devote labor to nonfarm activities. However, these results are not very large in magnitude. 52 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 31 Duflo, Esther Dams 2005 Irrigation Dams What is the impact of India 1971-87 QE yield Results provide a consistent picture of and Rohini dams on aggregate the impact of dams on agricultural Pande agricultural production, outcomes. In the districts where they are poverty, disease and a built, dams do not significantly alter range of related overall agricultural production. In outcomes? downstream districts, they enhance overall agricultural production, and production of some water-intensive cash crops (sugar) and staples that have seen the advent of HYV (wheat and rice). 32 Duflo, Esther, How High are 2008 Input Fertilizer Although fertilizer and Kenya 2000- E net return from A series of demonstration plot Michael Rates of Return technology hybrid seed may 2003 fertilizer input experiments in which treatment and Kremer, and to Fertilizer? increase yield on control plots were randomly allocated Jonathan Evidence from model farms, are they within farms suggests that top dressing Robinson Field profitable on many fertilizer, when used in appropriate Experiments in small farms, where quantities, is highly profitable. However, Kenya conditions are less other levels of fertilizer use, including the than optimal? official recommendations of the Ministry of Agriculture, are unprofitable for the average farmer in our sample. Two reasons postulated for low adoption of fertilizer are: lack of information and savings difficulties. Simple interventions that affect neither the cost of nor the payoff to fertilizer can substantially increase fertilizer use. 33 Dutilly-Diane, Household 2003 Technology Soil and water What are the impacts Burkina 2000 QE grain yields Farm households in Burkina Faso’s Celine, Behavior Under NRM conservation of the introduction of Faso Sahelian zone have a comparative Elisabeth Market contour stone bunds advantage in livestock production as a Sadoulet, and Failures: How on yield? How NRM cash crop and, at the same time, need to Alain de Natural and cooperation affect produce their food consumption largely Janvry Resource households’ income through home production due to very Management in generation strategies high transactions costs on food markets. Agriculture and the levels of Food production can be increased Promotes income achieved through improved NRM, in particular, Livestock according to their water harvesting and soil conservation Production in particular market using the remarkably simple and effective the Sahel participation regime? traditional practice of contour stone bunding. 53 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 34 Edwin, J. and Genetic 2005 Input Improved seed What are the impacts Ghana 2002 QE yield We find that the multiplication and William Improvement technology of new varieties (post distribution of new cocoa varieties Masters and Cocoa 1980) on farmers’ developed in Ghana have raised Yields in Ghana fields, controlling for farmers’ yields by at least 42%, other factors? controlling for other factors. That yield difference is attributable to almost 50 years of breeding effort, followed by large-scale multiplication and distribution of new planting materials to farmers. The economic policy reforms of 1984 were critical to raising both private and public investment in cocoa production. 35 Escobal, Access to 2006 Marketing Contracts What are the impacts Peru 2003 QE HH total In a nutshell, the results obtained in this Javier and Dynamic of new and more income study indicate that appropriate Maximo Markets for complex contracting investment policies in infrastructure need Torero Small schemes, as opposed to go together with well-functioning Commercial to traditional marketing market institutions in order to take Farmers: The channels, on small advantage of market opportunities, Case of Potato farmers’ welfare? sustain increased agricultural output, and Production in raise rural incomes. the Peruvian Andes 36 Feder, Sending 2004 Extension Farmer field Has program Indonesia 1991, QE yield The empirical results presented do not Gershon, Farmers Back school participation improved 1999 provide evidence of signi�cant Rinku Murgai, to School: The yields and reduced improvement in economic performance and Jaime Impact of pesticide use among or indicate a program effect on pesticide Quizon Farmer Field graduates and their use. The study shows that it is risky to Schools in neighbors? extrapolate the results of small and early Indonesia pilots, and the nonrigorous analyses that typically accompanied them. The impact of the FFS training can be much smaller than envisaged, when the program is applied on a large scale, rendering the economic, environmental, and health bene�ts much less attractive than what decision makers were expecting. 54 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 37 Feder, The Acquisition 2004 Extension Farmer field What is the evidence Indonesia 1991, E Knowledge of The empirical results suggest that Gershon, and Diffusion of school regarding the diffusion 1999 pest and graduates of FFS, who undertake a fairly Rinku Murgai, Knowledge: of knowledge on a key disease intensive training, benefit from a and Jaime The Case of theme of the FFS problems and statistically significant (although Quizon Pest program, namely pest solutions quantitatively small) gain in knowledge of Management management? better pest management in the course of Training in the FFS. However, such knowledge does Farmer Field not diffuse in a significant way to other Schools, members of their villages. This is Indonesia potentially due to the fact that the information is complex, entailing decision-making processes and ecosystem concepts, and is not easily transferred in informal farmer-to-farmer communications. 38 Frank Place, The Impact of 2005 Technology Soil and water How do agroforestry- Kenya 1997- NE Changes in Their conclusions are broad: they find Michelle Agroforestry- NRM conservation based SFR practices 2003. HH's liquid that poverty is hard to define and that Adato, Paul Based Soil facilitate people's assets households are moving away from Hebinck, Mary Fertility investment in assets agriculture, so it is difficult to measure Omosa Replenishment and lead to reduction SFR. The poor are adopting at rates Practices on in poverty in western similar to the non-poor. SFR does not the Poor in Kenya? appear to have much effect on yields Western Kenya although the other social impacts are important. 39 Gilligan, The Impact of 2008 Other Social safety What is the impact of Ethiopia 2006 QE Indicators of This paper ―should be considered as an Daniel, John Ethiopia's net PSNP on food food security, interim assessment of program impact.‖ Hoddinott, Productive security, consumption consumption, Assessment of the program depends on and Safety Net levels, asset growth, input use, labor how participation is defined. Matching Alemaeyhu Programme investment, and use of market estimates find little evidence of program Seyoum and its linkages income? participation, impact when participation is defined in Taffesse assets, etc. terms of receiving any payment for undertaking work on PSNP-supported public works, although we find no evidence that asset levels shrank, which was a key program objective. Stronger evidence of impact emerges when participation is defined in terms of households receiving at least half of their intended transfers. A more positive picture emerges when participation in both the PSNP and OFSP is considered. 55 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 40 Gine, Xavier Land Security 2005 Land Titling How do rental rates Thailand 1997 QE Cap ratios (as This paper shows empirically how a in Rural reform compare between an indicator of government policy created an unforeseen Thailand: secured and risk of negative externality. The issuance of land Evidence from unsecured plots in expropriation) reform and STK titles was not only a Property reform and nonreform ineffective because it provided small Rights Reform areas? private bene�ts to holders but that, in fact, it distorted the land market by triggering a sense of insecurity among landowners. 41 Gine, Xavier Group Versus 2006 Microcredit Access and Does group credit Philippines 2004, E loan size, Individual lending is more attractive to and Dean Individual services create excess 2006 excess savings, new clients and is no better in ensuring Karlan Liability: A Field pressure on borrowers payment repayment than group loans. Individual Experiment and discourage them performance, loans require less exchanges of from the from accessing credit? time invested information among members and may Philippines Is group lending better (lend, advice, indirectly reduce transaction costs to the than individual monitor) borrower. On the other hand, lenders lending? rely more on having well-functioning formal institutions to enforce individual contracts. The individual modality eases access, reduces indirect costs and provides more flexible products to the poor. 42 Gine, Xavier Insurance, 2007 Microcredit, Cash, seeds If income-raising Malawi 2006 E Credit- This randomized experiment finds that and Dean Credit, and Input technology adoption is insurance take- farmers offered credit to purchase high- Yang Technology technology hindered by risk, how up for hybrid yielding hybrid maize/improved Adoption: Field much does credit- seeds groundnut seeds were less likely to take Experiment insurance raise a loan when bundled with insurance. Evidence from demand for loans to Low education of farmers had a negative Malawi obtain improved seeds effect on take-up; insurance is a difficult (high yield hybrid product to understand. This has an maize and improved effect on designing products aimed at groundnut seeds)? reducing risks to encourage adoption of income-improving technologies. 43 Godtland, Erin The Impact of 2004 Extension Farmer field What is the impact of Peru 1999 QE Knowledge of Using data on a small-scale pilot FFS M., Elisabeth Farmer-Field- school FFS on knowledge of pest and program targeted toward Peruvian potato Sadoulet, Schools on FFS participants disease farmers, this article �nds that FFS Alain de Knowledge and compared with a problems and participation signi�cantly enhances Janvry, Rinku Productivity: A matched control group solutions knowledge about pests, fungicides, and Murgai, and Study of Potato of nonparticipants? resistant varieties—all instrumental in Oscar Ortiz Farmers in the implementing IPM practices. The Peruvian Andes robustness of the positive results of FFS participation on knowledge is 56 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator demonstrated by the fact that two separate approaches (and several variations on each of them) used for estimating the effect of FFS yield the same result. 44 Goldstein, The Profits of 2008 Land Tenancy law What is the effect of Ghana Not NE Decision to Insecure land tenure in Ghana is Markus and Power: Land reform an individual's position reported fallow land (as associated with greatly reduced Christopher Rights and in local political and an indicator of investment in land fertility. Individuals Udry Agricultural social hierarchies on investment in who are not central to the networks of Investment in his or her fallowing soil fertility) social and political power are much more Ghana choices on a plot? likely to have their land expropriated while it is fallow. Their reduced confidence of maintaining their rights over land while it is fallow induces such individuals to fallow their land less than would be technically optimal. As a consequence, farm productivity for these individuals is correspondingly reduced. There is a strong gender dimension to this pattern as women are rarely in positions of sufficient political power to be confident of their rights to land. 45 Goyal, Information 2008 Extension Market What is the impact of India 2005 QE price of soy in A change in the procurement strategy of Aparajita Technology information this intervention on the mandi a private buyer of soybean in Madhya and Rural price received by Pradesh has had signi�cant spillover Markets: soybean farmers in effects on the movement of prices across Theory and the mandis and on agricultural mandis in the state. The Evidence from their subsequent immediate bene�t to ITC Limited of this a Unique planting decisions? intervention was the improvement in Intervention in procurement efficiency of soybeans Central India resulting from the creation of a direct marketing channel and a reduction in its transaction costs. 46 Hall, David, Economic 2004 Extension Technical What are the farm and Thailand 1998-99 E Private profits Basic herd health programs to control C., Simeon K. Analysis of the advisory social level impacts of diseases associated with intensive Ehui, and Impact of the adoption of basic production are low-cost, low-input Barry I. Adopting Herd veterinary herd health technologies that contribute to increased Shapiro Health Control programs on private profits. Results from this study Programs on smallholder dairy show that dairy farmers of central Smallholder farms in Central Thailand have incentives to adopt herd Dairy Farms in Thailand? health management programs; following 57 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator Central the adoption of such programs there is a Thailand reduction in social inefficiency resulting from dairy public policy. 47 Independent An Impact 2008 Irrigation Irrigation What are the India 2005 QE yields Access to irrigation increases both yields Evaluation Evaluation of management economic and social and cropping intensity. Cropping intensity Group of the India's Second transfer impacts of Irrigation is increased through the extension of World Bank and Third Projects II and III in growing into additional seasons. It also Andhra the State of Andhra reduces year-to-year income fluctuations Pradesh Pradesh? caused by variations in rainfall. Gaining Irrigation access to irrigation raises net farm Projects income by about 60 percent. Indirect benefits come as well, from additional wage employment. The ex-post rate of return is only 2 percent. The low level results from cost overruns, construction delays, and discrepancies between realized income increases and those expected at appraisal. 48 Jacoby, H., G. Hazards of 2002 Land Tenancy law What is the impact on China 2002 NE social efficiency Empirical results strongly support the Li, and S. Expropriation: reform farm HH's investment gains view that heightened expropriation risk Rozelle Tenure behavior of the puts a damper on investment in rural Insecurity and variation in China. Farmers living in villages where Investment in expropriation risk as a expropriation risk is higher, use organic Rural China result of tenure fertilizer less intensively. Despite having insecurity under a significantly negative impact on one China's current form of plot-specific investment (i.e., system of collective organic fertilizer use/periodic land land management? reallocations do not appear to entail a substantial social cost). 58 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 49 Jacoby, Land Titles, 2006 Land Titling What are the Madagascar 2005 QE yield The consistent message emerging from Hanan and Investment, reform magnitude of private this study is that the private economic Bart Minten and Agricultural benefits of land titles benefits from extending land titling in Productivity in in the presence of Madagascar would be minor, especially Madagascar: A expropriation risk? relative to its cost. The median rice plot in Poverty and the Lac Alaotra region is worth about Social Impact US$ 1,000 per hectare, and titling it Analysis would raise its value by no more than US$ 60 per hectare. The estimates also provide a threshold for the costs of any new land rights system in Madagascar above which it would not make economic sense to implement. 50 Jacoby, Incentives, 2007 Marketing Contracts Does land tenancy Pakistan 1993, NE Average yield The productivity of land cultivated by Hanan and Supervision matter for 2001 differential sharecroppers does not differ from that of Ghazala and productivity? And, owners or renters. However, the yields Mansuri Sharecropper does a different level of sharecroppers that are more heavily Productivity of supervision by monitored by landlords are 28% higher landlords lead to than unmonitored share-tenants. The different productivity costs of supervision make it less possible outcomes? for regular monitoring. Therefore, redistributing land rights can potentially generate efficiency gains. 51 Jayne, T.S., Interlinked 2004 Marketing Spillover effects What are the effects of Kenya 1997, QE Fertilizer use Results indicate that households Takashi Credit and arrangeme of high value participation in 2000 on non-ILC engaging in interlinked marketing Yamano, and Farm nts agriculture on alternative cash crops programs for selected cash crops applied James Nyoro Intensification: food cropping schemes on considerably more fertilizer on other Evidence from productivity food crop productivity? crops (primarily cereals) not directly Kenya Can such cash crop purchased by the cash crop trading firm. programs serve as an These findings suggest that, in addition indirect vehicle for the to the direct stimulus that interlinked cash promotion of food crop crop marketing arrangements can have productivity? on small farmer incomes, these institutional arrangements may provide spill-over benefits for the productivity of farmers’ other activities such as food cropping. 52 Jensen, The Digital 2007 Extension Market How do improvements India 1996- QE price Price dispersion was reduced by use of Robert Provide: information in information 2001 dispersion, mobile phones. The mean coefficient of Information technology impact waste, welfare variation declined across markets from (Technology) market performance 60–70 percent to 15 percent. Waste of Market and welfare of fish was completely eliminated. Performance fishermen? Fishermen's profits increased, on and Welfare in average, by 8 percent and consumer 59 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator the South price declined by 4 percent, and Indian Fisheries consumption of sardine increased. Sector 53 Kabir, Results of 2007 Technology Crop What benefits can be Myanmar 2001, QE yield Three years after one-third of the farmers Humayun and Disseminating NRM intensification obtain from SRI 2002, in a community received FFS training, Norman the System of methods, especially 2003 almost all of its farmers were using SRI Uphoff Rice when introduced methods. This study confirmed many Intensification through FFS previously reported benefits from SRI with Farmer methodology? practices, particularly important for Field School limited-resource households. Methods in Northern Myanmar 54 Kaboski, Policies and 2005 Microcredit, Access and What is the (wealth) Thailand 1997 NE assets, Rice and buffalo banks have negative Joseph and Impact: An extension services impact of microfinance probability of impacts on "wealth" (proxy by asset Robert analysis of institutions on reducing growth) and harm consumption. Townsend Village-Level agriculture-households consumption, Production credit and women's groups Microfinance that get credit? prob. getting have positive impacts, perhaps due to Institutions credit, additional services (e.g., advice, training switching labor and savings accounts). Of all four, activity women's groups have largest and positive impact on member households. Savings-related policies have beneficial impacts. 55 Kassie, Estimating 2008 Technology Soil and water How do the impacts of Ethiopia 1999, QE value of crop Stone bunds have a positive and Menale, John Returns to Soil NRM conservation stone bunds on crop 2000 production per statistically significant productivity impact Pender, Conservation production value hectare (yield). in low rainfall areas. But the yield effect is Mahmud Adoption in the compare in high not observed in high rainfall areas, Yesuf, Gunnar Northern (Amhara region) and suggesting that the productivity impact of Kohlin, Randy Ethiopian low rainfall (Tigray stone bunds is agro-ecology specific. Bluffstone, Highlands region) areas of the This highlights the importance of and Elias Ethiopian highlands? developing and disseminating soil Mulugeta conservation technologies that are appropriately tailored to agro-ecological zones, instead of making blanket recommendations that promote similar conservation measures to all farmers. For instance, in high rainfall areas, moisture conservation using physical structures may not be important, but 60 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator placing appropriate drainage measures could help protect soils during extreme rainfall events. 56 Khandker, Microfinance 2005 Microcredit Access and Does access to Bangladesh 1991-92, NE HH Microfinance led to a reduction in poverty Shahidur and Poverty: services microfinance lead to 1998-99 consumption among poor borrowers and within the Evidence Using poverty reduction? local economy. It raises per capita Panel Data consumption, and has spillover effects for from nonparticipants. There is an increase in Bangladesh women borrowing, but due to diminishing returns to additional borrowing, the increases in consumption in this group are not large enough to reduce poverty as had been expected in previous predictions. 57 Khandker, The Poverty 2006 Other Rural roads What are the impacts Bangladesh 1997, QE transport costs, The results show that the savings for Shahidur, Impact of Rural of rural road projects 2001 fertilizer price, household costs average 36–38 percent Zaid Bakht Roads: on income, agricultural daily agric in treatment villages. There are also and Gayatri Evidence from production, wages, wage, HH p/c significant agricultural wage Koolwal Bangladesh input costs, transport expenditure, improvements, decreases in fertilizer costs and output schooling, prices and rises in crop indices. In terms prices? employment of access, the projects have a measurable impact on adult labor supply and children's access to school. Lastly, consumption per year is positively impacted by 11 percent per year. Quintile regressions show that impact is mostly felt for households at the 15th percentile of the consumption distribution. 61 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 58 Lokshin, Has Rural 2005 Other Rural How do several Georgia 2000, QE Access (time), Road and bridge rehabilitation projects Michael and Infrastructure infrastructure infrastructure (water, 2001 road quality generated clear economic benefits at the Ruslan Rehabilitation rehabilitation education and roads) (subjective), community level. The importance of Yemtsov in Georgia rehabilitation projects water supply, barter trade fell in the area, while small- Helped the in rural Georgia impact Sales of agric. and medium-size enterprises increased. Poor? welfare of Products, off- Other health and education benefits are beneficiaries? farm also measured. When looking at the employment, poor and non-poor, the non-poor costs, other benefited the most from the investments. (health and Access to services also increased for all, education) but most notably for the poor. Impacts of water investments were ambiguous and did not result in measurable impacts. 59 Lopez, Technology 2008 Extension Technical/advis What are the project Uruguay 2001, QE Reproductive The results of the impact evaluation show Fernando and Adoption, ory services impacts on two 2003 Efficiency Index that the LPP-1 positively affected the rate Alessandro Productivity specific outcomes and the of adoption of managerial practices. Maffioli and indicators: (i) the Percentage of Results also show some evidence that Specialization Reproductive Weaning the project could have positively affected of Uruguayan Efficiency Index (REI), the productivity of livestock producers Breeders: and (ii) the rate specialized in the breeding stage. There Evidence from adoption of were no differential impacts associated an Impact managerial practices? with the amount of the solicited subsidy. Evaluation 60 Macours, Land Titles and 2007 Land Titling Do the effects of land Guatemala 2007 QE Efficiency- We find that the effects of titles on the Karen Conflicts in reform titles on plot use and whether each efficiency of plot use, depend on the Guatemala credit access vary with plot was used conflict-context of the community. This is the prevalence of according to it’s not surprising as the value and the effect conflicts and different potential of a title, is likely to depend on whether a types of conflict formal title helps to secure property resolution rights, and on whether there are mechanisms? alternative mechanisms that might secure such rights. While it is hard to specifically identify which aspects of the conflict-context might matter the most, the results indicate some intriguing patterns, and clearly suggest that community context is key to understanding the potential value of a registered title. 62 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 61 Mancini, Increasing the 2008 Extension Farmer field What are the changes India 2001-04 QE Yield The study shows that the adoption of Francesca, A. Environmental school in agronomic integrated pest management (IPM) can J. and Social practices, input use significantly reduce the use of pesticides Termorshuize Sustainability of (fertilizer, pesticides on cotton in Andhra Pradesh. Given that n, Janice L.S. Cotton Farming and physical labor) farmers trained in IPM through FFS used Jiggins, and through Farmer and yield levels of one-sixth of the pesticides used by the Ariena H.C. Education in farmers trained in control group to obtain the same yield van Bruggen Andhra FFSs? levels. This reduction was possible Pradesh, India because of the strong correlation between knowledge level (promoted by FFS) and reduction in pesticide use. The farmers who learned more about biological control principles were able to manage the largest decrements in pesticide usage. 62 Mancini, Evaluating 2006 Extension Farmer field What are the India 2002-04 NE gains in IPM FFSs in the two cases studied were Francesca, Cotton school additional benefits of Financial found to have had broader effects than Ariena H.C. Integrated Pest FFSs on livelihood Capital simply those relating to pest Van Bruggen Management indicators in the social management. By use of a method that and Janice (IPM) Farmer and economic arena? investigates these effects in terms of the L.S. Jiggins Field School components of sustainable livelihoods, it Outcomes has been shown that farmers do place Using the values on these effects, and do perceive Sustainable the inter-dependency of impacts in one Livelihoods domain and others. The results revealed Approach in a number of surprises, such as perceived India impacts on fodder quality and animal health, which might result from reduced pesticide use and merit further study. 63 Markussen, Property 2008 Land Titling 1) Do government- Cambodia 2003-04 QE value of output The introduction of formal property rights Thomas Rights, reform issued land ownership per hectare to land in Cambodia have an Productivity, documents have an economically and statistically signi�cant, and Common effect on the value of positive effect on agricultural productivity Property output and on land and land values of owner-operated plots. Resources: values? 2) What the This suggests that land titling and Insights from causes and certi�cation programs can be effective Rural consequences of this policy instruments, even when the state Cambodia effect? is weak. 63 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 64 Mather, The economic 2003 Input Improved seed What are the farm- Honduras 2001 NE Reduction in Study indicates the positive outcome of David, R. impact of bean technology level bene�ts of risk of income resistant varieties of beans in Honduras Bernsten, J.C. disease resistant bean variety loss over the past 30 years. The ex-post rate Rosas, A. resistance adoption in Honduras? of return to disease-resistant bean Viana research in research is 41.2%. Ruanoc, and Honduras D. Escoto 65 Matuschke, Adoption and 2007 Input Improved seed What is the India 2004 NE Yields per acre, Hybrid wheat has a significant yield Ira,Ritesh Impact of technology technological impact income advantage over traditional wheat. The Mishra and Hybrid Wheat of hybrid wheat on technology does not require large inputs Matin Qaim in India adopter farmers? and is not limited to large-scale farmers, Particularly small- despite high seed costs. The paper also scale farmers. finds that individual networks play a role in adoption, and that price influences adoption–lower prices extends the reach. 66 McKernan, The Impact of 2002 Microcredit Rural noncredit Do microcredit Bangladesh 1991-92 QE Monthly profit The results on the three large credit Signe-Mary Microcredit effects programs increase programs show a large positive and Programs on productivity? And, significant total effect of participation on Self- whether the non-credit profits. The noncredit effects of Employment aspects of the participation are also large and positive. Profits: Do program have an The effects of participation are Noncredit impact beyond the decreasing in the amount of assets held; Program provision of credit? households with less assets benefit the Aspects most. Effects are for agriculture and Matter? nonagriculture work; the landless benefit more than landed borrowers. 67 Minten, Bart, Spillovers from 2007 Marketing Spillover effects What are the Madagascar 2004 NE yields Strong spillover effects of these trade Lalaina High-Value of high value spillovers from high opportunities on land use are found to Randrianariso Agriculture for agriculture on value agriculture for exist. This increase in yields seems n, and Johan Exports on food exports? Specifically, especially linked to an increase of soil Swinnen Land Use in productivity what are the effects of fertility due to the application of compost, Developing contract farming for which most farmers would not use prior Countries: the export of to the contracts. Although agricultural Evidence from vegetables on land output goes up signi�cantly, labor Madagascar use? productivity stays the same. There is greater labor absorption on existing land, and the diffusion of this technology throughout Madagascar would be expected to substantially decrease incentives to deforest by boosting productivity of existing lands relative to newly deforested ones. 64 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 68 Mu, Ren and Rural Roads 2007 Other Rural roads What are the average Vietnam 2001 E local area and There are indications of significant Dominique and Poor Area impacts of the market impacts on the development of markets van de Walle Development in rehabilitation of rural development and commercialization. Some outcomes, Vietnam roads on local area indicators (but such as food goods availability, and market none on responded rapidly to the new and development agricultural improved roads. Others, such as the indicators? production or presence and frequency of markets and productivity) nonfood goods and services availability, took two years more, on average, to emerge. The role of the initial level of local and market development appears to be crucial. 69 Nakasone, The Impact of 2008 Land Titling What are the Peru 2004 QE household On one hand, increases in tenure Eduardo Land Titling on reform alternative channels income security should have a negative effect on Labor through which land the number of on-farm hours of work allocation: titling might affect (Field effect). On the other hand, stronger Evidence from labor allocation in rural property rights should decrease Rural Peru areas? expropriation risks, promote land- attached investments, increase productivity in agricultural activities, and increase the number of on-farm hours of work (productivity effect). Based on the analysis of returns in different activities, results of this study suggest that there might be some shift in investments from nonagricultural to agricultural self- employed activities. 70 Negri, Burley Tobacco 2007 Marketing Interlinked What is the role of Malawi 2004-05, QE Output per Burley tobacco clubs show positive Mariano and Clubs in credit/input/out non-market institutions 2007 acre, sales per impacts on productivity of members; Guido Porto Malawi: put marketing for increasing acre, area yields are higher than of nonmembers by Nonmarket arrangements productivity, linkages dedicated, crop 40–74 percent. They also earn more Institutions for to market and variety income from sales with higher premiums. Exports international trade However, membership does not lead to and/or exports? success at negotiating better prices at auctions. Club members dedicate equal amounts of land to tobacco as nonmembers despite productivity gains and due to their need to cultivate subsistence crops as well. 65 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 71 Nkonya, From the 2008 Other Community What are impacts of Nigeria 2004, QE HH income Using propensity score matching and Ephraim, Ground Up - driven the Fadama II 2006 double-difference methods to control for Dayo Phillip, Impacts of a development projecton income project placement and self-selection Tewodaj Pro-Poor poverty, access to biases, we found that Fadama II reduced Mogues, John Community- infrastructure and beneficiaries’ distance and travel time to Pender, Driven productive assets, and the nearest town and reduced the waiting Muhammed Development provision of demand- time and fares for transportation services, Kuta Yahaya, Project in driven advisory relative to nonbeneficiary households in Gbenga Nigeria services? Fadama II LGAs. Household access to Adebowale, productive assets increased dramatically, Tunji Arokoyo, especially for the poorest households, and Edward largely because of the subsidy provided Kato to help finance acquisition of such assets. 72 Owens, The Impact of 2003 Land Tenancy law What is the impact of Zimbabwe 1993-97 Nonexpe yield This article finds that after controlling for Trudy, John Agricultural reform extension on the value rimental innate productivity characteristics and Hoddinott, Extension on of crop production per farmer ability, access to agricultural and Bill Farm hectare, with and extension services, defined as receiving Kinsey Production in without controls for one to two visits per agricultural year, Resettlement unobservable raises the value of crop production by Areas of household �xed about 15%. However, the impact is Zimbabwe effects? markedly different in drought and nondrought years. Collectively, these results suggest that although access to farm-level extension visits does increase productivity, even after controlling for innate productivity characteristics and farmer ability, results from single-year cross-sectional studies should be treated with caution. 73 Place, Frank Land Tenure 2002 Land Tenancy law What are the impacts Uganda 1996-97 NE Value of This study finds evidence that incentives and K. Otsuka Systems and reform of weak individual land production to plant and grow commercial trees, such Their Impacts rights, on planting as coffee, may not be thwarted by weak on Agricultural coffee and other trees, individual land rights under customary Investments fallowing of crop fields, tenure institutions, which prevail at the and and productivity of Bupadengo site, because of the land Productivity in mixed coffee and crop rights enhancing effect of tree planting. Uganda farming? Fallowing is less frequent on the customary land than on public land, which indicates that individual land rights are significantly weaker under this customary institution. Farm size was not 66 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator related to profits per hectare or productivity, so there is no indication that the tenure systems have a significant indirect impact on efficiency through effects on land holdings. 74 Praneetvataku The Impact Of 2006 Extension Farmer field What are the Thailand 2000-03 NE change in The findings suggest that a wider l, Suwanna & Farmer Field school environmental and pesticide cost dissemination of Bt cotton technology at Hermann Schools On economic impacts of reasonable seed prices could lead to Waibel Pesticide Use Farmer Field School considerable productivity gains and And on crop and pest income increases in smallholder Environment In management practices agriculture (high technology fee charged Thailand of rice in Thailand? by the monopoly seed supplier is currently the major adoption constraint). 75 Qaim, Matin Genetically 2003 Input seeds Does seed expensive Argentina 199, QE Costs, yields The higher price of seeds negatively and Alain de Modified Crops, technology (seeds) technology 2001 impacts adoption and yields of cotton. Janvry Corporate impact input use Costs for technology are double the price Pricing and/or subsequently of regular cotton and may outweigh Strategies and yields? monetary benefits. There are clear Farmers foregone economic gains for firms at Adoption: The current prices. This has led to less Case of BT formal adoption, illegal seeds are traded Cotton in and increased resistance in the pest Argentina population to the technology develops. 76 Qaim, Matin Bt Cotton and 2005 Input Seeds What are the Argentina 1999, QE Yield (kg/ha) The findings suggest that a wider and Alain de Pesticide Use technology economic, social, and 2001 dissemination of Bt cotton technology at Janvry in Argentina: environmental reasonable seed prices could lead to Economic and repercussions of Bt considerable productivity gains and Environmental cotton in Argentina? income increases in smallholder Effects agriculture (high technology fee charged by the monopoly seed supplier is currently the major adoption constraint). 67 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 77 Qaim, Matin, Adoption of Bt 2006 Input Seeds What is the evidence India 2003 QE Yield Results from the first season of Bt cotton Arjunan Cotton and technology of impacts of Bt cotton adoption in India show that the Subramanian, Impact on average yields and technology leads to significant pesticide Gopal Naik, Variability: variability across reductions, yield gains, and income and David Insights from regions? increases on average. Yet, heterogeneity Zilberman India among farmers causes significant variability in impacts. Agro-ecological conditions and farmers’ spraying habits are important determinants for spatial differences in technology outcomes. 78 Rola, Agnes, Do Farmer 2002 Extension Farmer field Do Farmer Field Philippines 1995, QE Knowledge The results show that FFS graduates Serlie Jamias Field Schools school Schools Graduates 2000 score on learn the material and use it. They also and Jaime Graduates Retain and Share farming retain the material and keep using it in Quizon Retain and what they learn? practices the long term. However, the information Share what sharing through informal farmer-to-farmer they learn? An channels does not show an impact on investigation in knowledge or use. Iloilo, Philippines 79 Romani, M. The Impact of 2003 Extension Technical/advis What is the impact of Cote d'Ivoire 1997-00 NE Yield of food The impact of extension services on Extension ory services agricultural extension crops yields in the period from 1997 to 2000, Services in on farm productivity? which coincides with some of the worst Times of Crisis. years in the history of independent Côte Cote d’Ivoire d'Ivoire, has been mixed. The main 1997-2000 conclusion of this paper is that food crops production seems to have benefited significantly from extension services. At the same time the analysis did not show any significant impact of extension on the production of export crops. The influence of the crisis in international coffee and cocoa prices on the export levels and crop-mix choice of farmers has played an important role in the trends of the yields of these crops. 68 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 80 Simmons, An analysis of 2005 Marketing Contracts What are the impacts Indonesia 2002 QE total gross The empirical results show that reasons for Phil, Paul contract arrangeme of contracting on the margins for all contract participation varied widely across Winters, and farming in East nts gross margins of agricultural the contracts. Unexpectedly, family size did Ian Patrick Java, Bali, and smallholders who production not effect contract participation and credit Lombok, participate in such constraints were not influential in selection, Indonesia marketing except for the broiler contract. The seed arrangements? corn contract enhanced female labor demand, while neither of the other two contracts had any effect and only the rice contract affected demand for family labor. Finally, the contracts positively affected welfare and no evidence was found for any pernicious effects. 81 Smith, Robert Land Tenure, 2004 Land Titling (i) Do titled farms have Zambia 2001 NE productivity This study �nds �rm evidence through E. Fixed reform greater fixed (production per the traditionally hypothesized causal path Investment, investment than hectare and per of �xed investment (though not credit) and Farm untitled farms? and (ii) unit farm labor leading to increased productivity. Title- Productivity: Do titled farms have force) holders and (to a lesser extent) lease- Evidence from greater productivity holders have greater �xed investment Zambia’s than untitled farms on and credit use than other categories; and Southern either State or titleholdings association with �xed Province customary land? investment is independent of an array of socioeconomic control variables. Cattle ownership emerges as a critical independent correlate of productivity per hectare and per capita. 82 Torero, Impact of Land 2005 Land Titling What is the impact of Peru 2004 QE welfare We find evidence for a significant Maximo and Titles over reform having access to a dimension; the improvement in risk of expropriation Erica Field Rural PETT title on 1) the value of the resulting in a composition effect, i.e.; in Households reduction of risk of dwelling; risk of secure investment in assets less subject expropriation, 2) gains expropriation to expropriation such as metallic and from trade of land, 3) cement fences. Similarly, we find a access to credit, and significant increase in the value of the 4) provision of public change and in the current market value goods? of the plot. With respect to the access to credit the results are not conclusive— informational asymmetries are still a key feature of the micro-credit environment in rural Peru. Finally, with respect to security and the provision of public goods in rural areas, the results suggest a heterogeneous response to strengthening individual property rights on community-level collective action. 69 Study Publi- Type of Intervention Evalua- Relevant numb Broad Data Author(s) Title cation Evaluation question Country tion impact Major conclusion category Subcategory years er year design indicator 83 Van Den Small-Scale 2006 Irrigation Dams What are the Ethiopia 1998-99 QE Total HH Study suggests that past development of Berg, Marritt Irrigation and distributional impacts consumption irrigation has stimulated growth without & Ruerd Income of irrigation in (cash deepening inequality. Local mechanisms Ruben Distribution in Ethiopia, a country expenditures for distribution have not discriminated Ethiopia that has plus home against the poor. Irrigation decreased put irrigated consumption dependence on food-for-work programs. agriculture at the heart valued at of its development average market strategy? prices) 84 Vranken, Property Rights 2008 Land Inheritance law (i) How, after Bulgaria 2003 QE HH welfare The evidence in this paper indicates that Liesbet, Karen Imperfections, reform regarding privatization, the property rights imperfections can remain Macours, Asset property rights remaining a serious constraint, even after a Nivelin Noev, Allocation, and imperfections of massive land privatization process aimed and Johan Welfare: Co- property rights affect at restituting complete property rights. As Swinnen Ownership in allocation of assets such, it sheds new light on the different Bulgaria and welfare? (ii) dimensions of property rights that can be Whether the allocation important for economic growth and of, and the returns to, development. In particular, historical land differ depending ownership and legal constraints can on whether plots are in reduce effective decision-making. This co-ownership? can result in sub-optimal land allocation even after complete land titles have been established. 85 Wollnia, Do Farmers 2007 Marketing Cooperatives What are the factors Costa Rica 2003 QE Marketing Regression results reveal that marketing Meike and Benefit from that determine performance performance, measured by the average Manfred Participating in farmers’ participation (measured in price farmers received for their coffee, is Zellerb Specialty in specialized markets terms of the considerably improved by participation in Markets and and whether average coffee both specialty and cooperative marketing Cooperatives? participation in these price obtained channels. The Case of markets leads to by the farmer at Coffee higher prices for the end of the Marketing in farmers? season) Costa Rica Source: IEG. Note: Full citations of these studies are provided elsewhere. A study with a number in bold was included in the reduced pool of studies for meta-analysis. E = experimental, NE = nonexperimental, and QE=quasi- experimental. 70 Appendix B. The Analysis Selecting the Pool of Impact Evaluations The study used a five-step filtering process to identify and select impact evaluations for the meta-analysis, as shown in figure 1.2 in the main text. The process consisted of a literature search (step 1) and the application of a set of criteria to identify relevant studies (steps 2 and 3). These were then subjected to in-depth analysis to further identify studies for review and synthesis (step 4) and selected for inclusion in the meta-analysis based on a rating scale (step 5). This appendix provides details on each of the five steps. Step 1: Literature search to identify a pool of impact evaluation studies The production of rigorous impact evaluations has been, and still is, riddled with problems both, methodological, such as econometric techniques and data availability, and practical, such as ethical concerns, funding, weak incentives (Ravallion 2008). However, in the past decade substantial donor support has been provided for data collection (over 50 developing countries have regular household surveys) and the emphasis on measurable results has increased. In this climate, the IE toolkit has been evolving rapidly, particularly with regard to econometric methods to deal with complex settings. This has enabled an increasing number of practitioners to invest in IEs that provide an evidence base for meta-analysis. For this meta-analysis a comprehensive inventory of all possible impact evaluations pertaining to agriculture and rural development was developed using the following sources:  The World Bank‘s impact evaluation database  Web search of electronic databases (such as JSTOR) using the keywords production, agricultural productivity, impact, analysis, effectiveness, assessment, and evaluation; Web search (by going through all the journal issues, one by one, since 2000) based on titles and abstracts of articles published in: World Development, Journal of Political Economy, Journal of Development Studies, Journal of Development Economics, American Journal of Agricultural Economics, Australian Journal of Agricultural Economics, Canadian Journal of Agricultural Economics, Crop Science, Agricultural Systems, Experimental Agriculture, and Science. Other economic development and agricultural journals were consulted during the search process but are not listed here because they yielded no impact evaluation studies that fit the stated criteria;  The impact Web site of the Consultative Group on International Agricultural Research (http://impact.cgiar.org/) and selected other centers (such as IFPRI).  Recommendations made by personal contacts with impact evaluation professionals in this area and studies cited in the reference lists of relevant studies. Two main criteria were used to build the inventory of potential impact evaluation studies: publication since 2000 and broad relevance to impact evaluation of one of the categories of agricultural development interventions. The latter consist of infrastructure development that affects agriculture directly—roads, irrigation, drainage systems; policy changes related to land, labor, and capital; market access in rural areas; the provision of fertilizer, improved 71 technologies, or other inputs (water, pesticides, seeds); and the provision of extension services to producers. The studies selected using these sources and criteria form the pool of IEs that were considered for inclusion in steps 2–5. A limitation of this pool is that it may not be an exhaustive list of all potential impact evaluations on this subject. There are, no doubt, some published and unpublished papers, reports, conference presentations, and especially dissertations that were not captured by the approach used. Also, because the search was limited to English-language journals, impact evaluations that were published solely in other languages were likely missed. However, since most quality dissertations do eventually get published and many non-English language publications at least have an abstract or a summary in English (which would have been captured by Web-based searches), we believe that the number of studies missed by the literature search is not likely to be substantial. Step 2: Identifying qualified studies In order to qualify for inclusion in the meta-analysis an impact evaluation was required to measure the effectiveness of an intervention in influencing a change in some direct measure or indicator of productivity or a change in some measure or indicator of intermediate outcomes that are hypothesized to increase productivity. Ensuring that a study met this minimum criterion was therefore a critical step in the filtering process. To be selected a study had to have an identifiable activity that was being assessed (either an intervention or a shock) and productivity had to be one of the objectives of the impact evaluation (indicators of productivity may be as defined by the study). Duplicate studies and studies outside the scope of agriculture were eliminated at this step. In all cases, the suitability of the impact evaluations were assessed on the objectives outlined in the papers or what the authors presented to be the evaluation criteria. It is acknowledged that there may be disconnects if the initial project development objective was not related to productivity improvement but the evaluation author measured it in their study. Step 3: Applying the necessary condition of an impact evaluation—a defined counterfactual Impact evaluation relies on the construction of a counterfactual situation to examine the outcome of a group in two states at the same time, in and out the program, or affected or not affected by a shock. Although the counterfactual outcome is never actually observed as people cannot simultaneously participate and not participate in a project, determining the counterfactual is at the core of evaluation design and necessary for a quality impact evaluation. Therefore, a study selected for the meta-analysis had to demonstrate that it carefully selected a group of nonparticipants that were equally needy or deserving of the program and were the same with regard to most characteristics and outcomes of interest or the study had to deal with the differences econometrically. Another valid method allowed in the pool of evidence was the use of an instrumental variable. Although the counterfactual method assumes an exogeneity of program placement (where either group could have received the program), instrumental variable techniques aim to isolate a part of the variation in program placement (deemed exogenous) while not altering the outcomes meant to be affected by the intervention. Few 72 papers with instrumental variable methodology were included. In this filtering process some studies were dropped and the remaining ones were promoted for further consideration in step 4. Step 4: In-depth review to ensure eligibility (double check) Each of the remaining studies was reviewed again to ensure that studies that filtered through steps 2–3 did indeed meet the qualification criteria. Based on the in-depth review of each study and application of the criteria established for steps 2–3, some studies were rejected at this step. Most of these studies were rejected either because they were the same or similar to another study in the pool (in which case the more recent and peer-reviewed studies were kept) or, on closer scrutiny, were outside the scope of the objectives of this study. The list of studies filtered through this step forms the pool of studies used for the overview and synthesis. Appendix A lists the studies in the pool, identified by their author(s), title, year of publication, and other pertinent information related to the intervention and impact evaluation. Step 5: Applying the quality and rigor criteria for selecting studies for the meta- analysis Given that the studies in the pool are produced independently by different authors at various times and locations, and use different designs, methods, and datasets, it is very likely that there is considerable heterogeneity in the quality of the impact evaluations. Much traditional meta- analysis attempts to assemble as broad a pool of cases as possible from which to draw inferences, under the assumption that a sufficient population of cases will balance out individual methodological flaws during regressions. In this study we have taken the alternative ―best evidence‖ approach to meta-analysis, in which methods are scrutinized and screened for quality and rigor before results are accepted as part of the analysis (Slavin 1995). Although each impact evaluation has unique characteristics requiring different approaches and methods, a few general qualities can be considered best practices for conducting impact evaluation. A set of such generally accepted best practices, drawn from the literature and personal experience, was used as a guideline to score each study on a three-point scale (see, for example, Baker 2000). These guidelines were grouped into three criteria—type of publication, method of impact evaluation, and robustness of results. The best practice guidelines for each of these three criteria, interpretation of the rating scale used and its rationale are described in table B.1. The goal of the rating exercise was to ensure that studies selected for the meta-analysis met certain minimum quality criteria, albeit based on a subjective assessment by two of the authors, who independently reviewed and scored each study. A simple average score was calculated for each study across these three criteria and two reviewers.10 A study that scored below 1.5 was rejected to maintain an overall quality of IEs. Several studies report IE results for multiple locations, crops or commodities, or impact indicators. Therefore the number of 10 A subset of the studies was only reviewed by one of the authors because of time constraints; the same criteria for inclusion were applied for this subset and should be well within the boundaries of acceptance. 73 observations used in the meta-analysis is more than the total number of studies included in the final pool. 74 Table B.1. Best Practice Guidelines, Rating Scale, and Rationale for the Criteria Used to Score Studies Filtered in Step 4 Criteria used Best practice guidelines Explanation/interpretation of the Rationale for scoring rating scale used (1–3) Type of Peer-reviewed publication. 1 = Unpublished monographs. A study filtered through a publication process of critical review Between 1 and 3 = Published as a before publication is monograph (that is, institutional deemed to meet report/series, dissertation). A professional standards of subjective score based on the quality (broadly defined) by reputation of the series/publishing its peers. organization. 3 = Published in a recognized peer- reviewed journal. Method of Estimate counterfactual by (a) using random 1 = The method lacks in rigor or does A study that meets best impact assignment to create a control group (experimental not meet the best practice guidelines. practice guidelines in the evaluation design), (b) appropriate and careful use of other method of conducting methods, such as matching, to create a comparison Between 1 and 3 = The study meets impact evaluation (that is, group (quasi-experimental design), or (c) valid some but not all the best practice quality and rigor used to instrumental variable. guidelines. identify treatment and Control for pre- and post-program differences in comparison groups, method participants. 3 = A rigorous and well-done impact of analysis, justification Relevant data collected at baseline and follow-up to evaluation as per best practice provided, data used, etc.), estimate program impacts. guidelines. and robustness of results Sufficient time frame allowed for program impacts. (sample size used, Qualitative techniques are incorporated to allow for techniques used to infer the triangulation of findings. results) conveys confidence 75 Criteria used Best practice guidelines Explanation/interpretation of the Rationale for scoring rating scale used (1–3) Robustness of More than one technique used to infer patterns of 1 = The study does not meet the best in the results and inferences results impact from data collected (that is, obtain robust practice guidelines. drawn from those results. results). The treatment and comparison groups are of sufficient Between 1 and 3 = The study meets sizes to establish statistical inferences with minimal some but not all the best practice attrition. guidelines. 3 = A rigorous and well-done impact evaluation as per best practice guidelines. 76 Appendix C. Types of Evaluation Design and Methodologies To generate counterfactual data it is necessary to establish a control or comparison group (for example, those who do not participate or receive benefits) to compare it with the group receiving the intervention. How this group is identified is at the heart of ―evaluation designs,‖ which can be broadly classified into three categories: experimental, quasi-experimental, and nonexperimental (Baker 2000).  Experimental designs are based on a lottery system of randomly allocating the intervention among eligible beneficiaries. The random assignment process itself creates comparable treatment and control groups that are statistically equivalent. This is considered a powerful approach because, in theory, a control group generated through random assignment serves as a perfect counterfactual, free from the troublesome selection bias issues that often plague evaluations (Kerr and Chung 2001; Baker 2000).  Quasi-experimental studies also examine outcomes, but they do not involve randomly assigning participants to treatment and control groups. A quasi- experimental study might compare outcomes for individuals receiving program activities with outcomes for a similar group of individuals not receiving program activities (either through matching or regression techniques). This type of evaluation also might compare outcomes for one group of individuals before and after the group‘s involvement in a program (known as ―pre/post‖ or ―reflexive‖ designs). Quasi-experimental studies can inform discussions of cause and effect, but, unlike true experiments (randomization), they require more econometric manipulation to establish this link.  Nonexperimental evaluation designs can be used when it is not possible to randomly select a control group, identify a suitable comparison group through matching methods, or use reflexive comparisons. In such situations, program participants are compared with nonparticipants using statistical methods to account for differences between the two groups. A common approach under this category is the use of instrumental variables.11 Good instruments help predict program participation and allow the researcher to predict the outcome of program participants and nonparticipants. These evaluation designs are mostly based on existing data sources and are thus relatively cheap and easy to implement. However, identifying good instruments is very difficult and limits the use of this method generally. These three evaluation design types differ in the identification of a treatment/participant group, control/comparison group (and thus the counterfactual), how and when the evaluation is planned, and other elements of evaluation design that determine its ―rigor.‖ The choice of an evaluation design to determine the counterfactual depends largely on how and when the evaluation is planned. The earlier an evaluation is planned, the greater 11 Instruments are sometimes used to control for biases stemming from unobservable characteristics. These variables determine program participation but do not affect outcomes. 77 the methodological flexibility in terms of the choice of using experimental (and quasi- experimental) designs. In addition to these three basic types of evaluation designs, the method of analysis depends on the type of data used to infer the causal link between an intervention and its impact. A cross-sectional dataset consists of a sample of units of impact analysis (that is, individuals, households, firms, cities, states, countries, or a variety of other units) taken at a given time. Such data allow comparison of treatment and control/comparison groups at a given time (that is, after the intervention). A panel dataset (or longitudinal data) consists of a time series for each cross-sectional member in the dataset. Panel data are distinct from a pooled cross-section. The key feature of panel data that distinguishes it from a pooled cross-section is the fact that the same cross-sectional units are followed over a given period. Because panel data require replication of the same units over time, panel datasets, especially those on individuals and households, are more difficult to obtain than pooled cross-sections. Not surprisingly, however, observing the same units over time leads to several advantages over cross-sectional data or even pooled cross-sectional data. One of the benefits is that having multiple observations on the same units allows the analyst to control certain unobserved characteristics of individuals/households (or the unit of analysis). The use of more than one observation can facilitate causal inference in situations where inferring causality would be difficult if only a single cross-section were available. A second advantage of panel data is that it often allows the analyst to study the importance of lags in behavior or the result of decision making. This information can be significant since many development interventions (including policies) can be expected to have an impact only after some time has passed. Due to these advantages, economists and impact analysts now recognize that some questions are difficult, if not impossible, to answer satisfactorily without panel data (Wooldridge 2002). 78 Bibliography (References shown in bold are in the sample of impact evaluations in the analysis) Aker, J.C. 2008. ―Does Digital Divide or Provide?: The Impact of Cell Phones on Grain Markets in Niger.‖ Working Paper 154. Center for Global Development, Washington, DC. Alderman, H. 2007. ―Improving Nutrition through Community Growth Promotion: Longitudinal Study of the Nutrition and Early Child Development Program in Uganda.‖ World Development 35(8): 1376–1389. Alemayehu, M., F. Yohannes, and P. Dubale. 2006. ―Effect of Indigenous Stone Bunding (kab) on Crop Yield at Mesobit-gedeba, North Shoa, Ethiopia.‖ Land Degradation and Development 17(1): 45–54. Alston, Julian M., Barbara J. Craig, and Philip G. Pardey. 1998. "Dynamics in the Creation and Depreciation of Knowledge, and the Returns to Research:," EPTD Discussion Paper 35, International Food Policy Research Institute (IFPRI), Washington, DC. Alston, Julian M., Michele C. Marra, Philip G. Pardey, and T. J. Wyatt. 2000. ―Research Returns Redux: A Meta-Analysis of the Returns to Agricultural R&D.‖ The Australian Journal of Agricultural and Resource Economics 44(2): 185–215. Alston, Lee J., Gary D. Libecap, and Robert Schneider. 1996. ―The Determinants and Impact of Property Rights: Land Titles on the Brazilian Frontier.‖ Journal of Law, Economics and Organization 12 (1): 25–61. Amacher, G.S., L. Ersado, D.L. Grebner, and W.F. Hyde. 2004. ―Disease, Microdams and Natural Resources in Tigray, Ethiopia: Impacts on Productivity and Labour Supplies.‖ Journal of Development Studies 40(6): 122–145. Ashraf, N., X. Gine, and D. Karlan. 2008. ―Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya.‖ Policy Research Working Paper 4477, World Bank, Washington, DC. Baker, J.L. 2000. Evaluating the Impact of Development Projects on Poverty: A Handbook for Practitioners. Washington, DC: World Bank. http://siteresources.worldbank.org/INTISPMA/Resources/Impact-Evaluation-Handbook-- English-/impact1.pdf. Bandyopadhyay, S., P. Shyamsundar, and M. Xie. 2007. ―Yield Impact of Irrigation Management Transfer: Story from the Philippines.‖ Policy Research Working Paper 4298, World Bank, Washington, DC. Banerjee, A.V., P.J. Gertler, and M. Ghatak. 2002. Empowerment and Efficiency: Tenancy Reform in West Bengal.‖ Journal of Political Economy 110(2): 239–280. Bardhan, P., and D. Mookherjee. 2008. ―Productivity Effects of Land Reform: A Study of Disaggregated Farm Data in West Bengal, India.‖ Working Paper, Dept. of Econ., University of California, Berkeley and Dept. of Econ., Boston University. Barrett, C.B., C. Moser, O.V. McHugh, and J. Barison. 2004. ―Better Technology, Better Plots, or Better Farmers?: Identifying Changes in Productivity and Risk among Malagasy Rice Farmers.‖ American Journal of Agricultural Economics 86(4): 869–888. 79 Benin, S., E. Nkonya, G. Okecho, J. Randriamamonjy, E. Kato, G. Lubade, M. Kyotalimye, and F. Byekwaso. 2008. ―Impact Evaluation and Returns to Investment of the National Agricultural Advisory Services (NAADS) Program of Uganda.‖ Discussion Paper 724, International Food Policy Research Institute (IFPRI), Washington, DC. Bentley, J.W. 1987. ―Economic and Ecological Approaches to Land Fragmentation: In Defense of a Much-Maligned Phenomenon.‖ Annual Review of Anthropology 16: 31–67. Bernard, T., Alemayehu Seyoum Taffesseb, and Eleni Gabre-Madhina. 2008. ―Impact of Cooperatives on Smallholders’ Commercialization Behavior: Evidence from Ethiopia.‖ Agricultural Economics 39 (2008) 147–161. Birkhaeuser, D., R.E. Evenson, and G. Feder. 1991. ―The Economic Impact of Agricultural Extension: A Review.‖ Economic Development and Cultural Change 39(3): 607–650. Blarel, B., P. Hazell, F. Place, and J. Quiggin. 1992. ―The Economics of Farm Fragmentation: Evidence from Ghana and Rwanda.‖ World Bank Economic Review 6(2): 233–254. Bolwig, S., P. Gibbon, and S. Jones. 2009. ―The Economics of Smallholder Organic Contract Farming in Tropical Africa.‖ World Development 37(6): 1094–1104. Borras, Jr., Saturnino. 2005. ―Can Redistributive Reform Be Achieved via Market-Based Voluntary Land Transfer Schemes?: Evidence and Lessons from the Philippines. Journal of Development Studies 41(1) January: 90–134. Brambilla, I., and G.G. Porto. 2005. ―Farm Productivity and Market Structure: Evidence from Cotton Reforms in Zambia.‖ Economic Growth Center, Discussion Paper No. 919, Yale University, New Haven, CT. http://ssrn.com/abstract=770786. Burgess, R., and R. Pande. 2005. ―Do Rural Banks Matter?: Evidence from the Indian Social Banking Experiment.‖ The American Economic Review 95 (3): 780–795. Card, D., and A. B. Krueger. 1995. ―Time-Series Minimum–Wage Studies: A Meta-analysis.‖ The American Economic Review 85(2): 238–243. Carter, Michael R., and Pedro Olinto. 2003. ―Getting Institutions Right for Whom?: Credit Constraints and the Impact of Property Rights on the Quantity and Composition of Investment.‖ American Journal of Economics 85(1): 173–186. Cerdán-Infantes, P., A. Maffioli, and D. Ubfal. 2008. ―The Impact of Agricultural Extension Services: The Case of Grape Production in Argentina.‖ Working Paper 5/08, Inter- American Development Bank, Office of Evaluation and Oversight (OVE). Cocchi, H., and B.E. Bravo-Ureta. 2007. ―On-Site Costs and Benefits of Soil Conservation among Hillside Farmers in El Salvador. Working Paper 4/07, Inter-American Development Bank, Office of Evaluation and Oversight (OVE). Conger, J., and G. Toegel (2002). ―A Story of Missed Opportunities: Qualitative Methods for Leadership Research and Practice.‖ In K. Parryand and J. Meindl, eds. Grounding Leadership Theory and Research: Issues, Perspectives, and Methods, pp. 175–197. Greenwich, CT: Information Age Publishing. Conley, T.G., and C. Udry. 2005. ―Learning About a New Technology: Pineapple in Ghana.‖ unpublished. Dalton, T., N. Lilja, N. Johnson, and R. Howeler. 2007. ―Impact of Participatory Natural Resource Management Research in Cassava-Based Cropping Systems in Vietnam and 80 Thailand.‖ In H. Waibel and D. Zilberman, eds. The Impact of Natural Resource Management Research in the CGIAR, chapter 6. Rome, Italy: FAO and Wallingford, UK: CAB International. Datar, G., X. Del Carpio, and V. Hoffman. 2009. ―Land, Grants, and Title: The Impact of a Land Resettlement Program on Productivity and Welfare in Malawi.‖ Background paper, World Bank, Washington, DC. Deininger, K., A. Ali, and T. Alemu. 2008. ―Impacts of Land Certification on Tenure.‖ Deininger, K., A. Ali, S. Holden, and J. Zevenbergen. 2008. ―Rural Land Certi�cation in Ethiopia: Process, Initial Impact, and Implications for Other African Countries.‖ World Development 36(10): 1786–1812. Deininger, K., and J.S. Chamorro. 2004. ―Investment and Equity Effects of Land Regularisation: The Case of Nicaragua.‖ Agricultural Economics 30 (2):101–116. Deininger, K., and S. Jin. 2007. ―Land Rental Markets in the Process of Rural Structural Transformation: Productivity and Equity Impacts in China.‖ Policy Research Working Paper 4454, World Bank, Washington, DC. Del Carpio, X., Q. Do, G. Gutierrez, L. Trung, and A. Waxman. 2009. ―What Impedes Farmers from Consolidating Their Land?: Evidence from an Experiment in Vietnam.‖ Background paper, World Bank, Washington, DC. Del Carpio, X., N. Loayza, and G. Datar. 2011. ―Is Irrigation Rehabilitation Good for Poor Farmers? An Impact Evaluation of a Non-Experimental Irrigation Project.‖Journal of Agricultural Economics 62 (2). Forthcoming. Deininger, K., D.A. Ali, and T. Alemu. ―Impacts of Land Certification on Security, Investment, and Land Markets: Evidence from Ethiopia.‖ Policy Research Working Paper 4764, World Bank, Washington, DC. Dercon, S., D.O. Gilligan, J. Hoddinott, and T. Woldehanna. 2008. ―The Impact of Agricultural Extension and Roads on Poverty and Consumption Growth in Fifteen Ethiopian Villages.‖ Discussion Paper 840, International Food Policy Research Institute (IFPRI), Washington, DC. Dey, M.M., P. Kambewa, M. Prein, D. Jamu, F.J. Paraguas, D.E. Pemsl, and R.M. Briones. 2007. ―Impact of Development and Dissemination of Integrated Aquaculture- Agriculture (IAA) Technologies in Malawi.‖ In H. Waibel and D. Zilberman, eds. The Impact of Natural Resource Management Research in the CGIAR, chapter 7. Rome, Italy: FAO and Wallingford, UK: CAB International. Diagne, A. 2006. ―Diffusion and Adoption of Nerica Rice Varieties in Cote D’Ivoire.‖ The Developing Economics 44(2): 208–231. Dillon, A. 2008. ―Access to Irrigation and the Escape from Poverty: Evidence from Northern Mali.‖ Discussion Paper 782, International Food Policy Research Institute (IFPRI), Washington, DC. Do, Q., and L. Iyer. 2008. ―Land Titling and Rural Transition in Vietnam.‖ Economic Development and Cultural Change 56 (3): 532–579. Dorward, A., J. Kydd, J. Morrison, and I. Urey. 2004. ―A Policy Agenda for Pro-Poor Agricultural Growth.‖ World Development 32 (1): 73–89. Duflo, E., and R. Pande. 2005. ―Dams.‖ Economic Growth Center, Discussion Paper 923, Yale University, New Haven, CT. 81 Duflo, E., M. Kremer, and J. Robinson. 2008. ―How High Are Rates of Return to Fertilizer?: Evidence from Field Experiments in Kenya.‖ Paper presented at the American Economic Association Meetings, New Orleans, LA, January 4–6. Dutilly-Diane, C., E. Sadoulet, and A. De Janvry. 2003. ―Household Behavior under Market Failures: How Natural Resource Management in Agriculture Promotes Livestock Production in the Sahel.‖ CUDARE Working Paper 979, University of California, Berkeley. Edwin, J., and W.A. Masters. 2005. ―Genetic Improvement and Cocoa Yields in Ghana.‖ Experimental Agriculture 41 (4): 491–503. Egger, Matthias, and George Davey Smith. 1997. ―Meta-analysis: Potentials and Promise.‖ Education and Debate 315: 1371–1374. Ellis, F. 1992. Agricultural Policies in Developing Countries. Cambridge, UK: Cambridge University Press. Escobal, J., and M. Torero. 2006. ―Access to Dynamic Markets for Small Commercial Farmers: The Case of Potato Production in the Peruvian Andes.‖ Markets Trade and Institutions (MTID) Discussion Paper 99, International Food Policy Research Institute (IFPRI), Washington, DC. Espey, M., J. Espey, W.D. Shaw. 1994. ―Price Elasticity of Residential Demand for Water: A Meta- analysis.‖ Water Resources Research 33 (6): 1369–1374. Feder, G., and T. Onchan. 1987. ―Land Ownership Security and Farm Investment in Thailand.‖ American Journal of Agricultural Economics 69 (2): 311–320. Feder, G., R. Murgai, and J.B. Quizon. 2004a. ―Sending Farmers Back to School: The Impact of Farmer Field Schools in Indonesia.‖ Review of Agricultural Economics 26 (1): 45–62. Feder, G., R. Murgai, and J.B. Quizon. 2004b. ―The Acquisition and Diffusion of Knowledge: The Case of Pest Management Training in Farmer Field Schools, Indonesia.‖ Journal of Agricultural Economics 55 (2): 221–243. Finan, F., E. Sadoulet, A. de Janvry. 2005. ―Measuring the Poverty Reduction Potential of Land in Rural Mexico.‖ Journal of Development Economics 77 (1): 27– 51. Galasso, E., and N. Umapathi. 2007. ―Improving Nutritional Status through Behavioral Changes: Lessons from Madagascar.‖ Policy Research Working Paper 4424, World Bank, Washington, DC. Gilligan, D.O., J. Hoddinot, and A.S. Taffesse. 2008. ―The Impact of Ethiopia’s Productive Safety Net Programme and its Linkages.‖ Discussion Paper 839, International Food Policy Research Institute (IFPRI), Washington, DC. Gine, X. 2005. ―Land Security in Rural Thailand: Evidence from a Property Rights Reform.‖ Policy Research Working Paper 3734, World Bank, Washington, DC. Gine, X., and D. Yang. 2007. ―Insurance, Credit, and Technology Adoption: Field Experimental Evidence from Malawi.‖ Policy Research Working Paper 4425, World Bank, Washington, DC. Godtland, E., E. Sadoulet, A. De Janvry, R. Murgai, and O. Ortiz. 2004. ―The Impact of Farmer Field Schools on Knowledge and Productivity: A Study of Potato Farmers in the Peruvian Andes.‖ Economic Development and Cultural Change 53 (1): 63–92. 82 Goldstein, M., and C. Udry. 2008. ―The Profits of Power: Land Rights and Agricultural Investment in Ghana.‖ Journal of Political Economy 116 (6): 981–1022. Goldstein, M., E. Skoufias, and A. Fiszbein. 2008. ―Assessing Our Work on Impact Evaluation.‖ PREM Notes No. 126. World Bank, Washington, D.C. http://www1.worldbank.org/prem/PREMNotes/premnote126.pdf. Gollin, D., and R. Evenson. 2003. ―Valuing Animal Genetic Resources: Lessons from Plant Genetic Resources.‖ Ecological Economics 45 (3): 353–363. Gorg, H., and E. Strobl. 2001. ―Multinational Companies and Productivity Spillovers: A Meta- analysis.‖ The Economic Journal 111 (475): F723–F739. Goyal, A. 2008. ―Information Technology and Rural Markets: Theory and Evidence from a Unique Intervention in Central India.‖ Working Paper, Dept. of Econ., University of Maryland, College Park. Gutierrez, G., and P. Velez-Vega. 2009. ―Qualitative Impact Assessment of the Peru Subsectorial Irrigation Project – Phase I.‖ Background paper, World Bank, Washington, DC. Hall, D.C., S.K. Ehui, and B.I. Shapiro. 2004. ―Economic Analysis of the Impact of Adopting Herd Health Control Programs on Smallholder Dairy Farms in Central Thailand.‖ Agricultural Economics 31 (2–3): 335–342. Heckman, J., H. Ichimura, and P. Todd. 1997. ―Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme.‖ Review of Economic Studies 64 (4): 605–654. Hussain, I., and M.A. Hanjra. 2004. ―Irrigation and Poverty Alleviation: Review of the Empirical Evidence.‖ Irrigation and Drainage 53 (1): 1–15. IEG (Independent Evaluation Group). 2008a. ―Approach Paper: The World Bank‘s Support for Land Reform, Administration, and Related Procedures.‖ World Bank, Washington, DC. IEG. 2008b. An Impact Evaluation of India’s Second and Third Andhra Pradesh Irrigation Projects: A Case of Poverty Reduction with Low Economic Returns. Washington, DC: World Bank. IEG 2010. Cost-Benefit Analysis in World Bank Projects. Washington, DC: World Bank. Instituto Peruano de Estudios Sociales (CEPES). 2004. ―Evaluación del desempeño institucional del Proyecto Sub-sectorial de Irrigaciones- PSI‖ Informe Final del Centro, Lima, Peru. International Fund for Agricultural Development (IFAD). 2003. IFAD Evaluation Policy. Executive Board – Seventy-Eighth Session, Rome, Italy, April 9–10. .Document #: 327978. http://www.ifad.org/gbdocs/eb/78/e/EB-2003-78-R-17-REV-1.pdf . Jacoby, H., and B. Minten. 2006. ―Land Titles, Investment, and Agricultural Productivity in Madagascar: A Poverty and Social Impact Analysis.‖ The World Bank Economic Review. Published online on June 30, 2007. http://siteresources.worldbank.org/INTPSIA/Resources/490023- 1120841262639/Madagascar_Land_Titling_PSIA_Final_2006.pdf. Jacoby, H., G. Li, and S. Rozelle. 2002. ―Hazards of Expropriation: Tenure Insecurity and Investment in Rural China.‖ American Economic Review 92 (5): 1420–1447. Jacoby, H.G., and G. Mansuri. 2007. ―Incentives, Supervision, and Sharecropper Productivity.‖ Policy Research Working Paper 4125. World Bank, Washington, DC. 83 Jalan, J., and M. Ravallion. 2003. ―Estimating the Benefits Incidence of an Antipoverty Program by Propensity Score Matching.‖ Journal of Business and Economics Statistics 21 (1): 19–30. Jarrel, S.B., and T.D. Stanley. 2004. ―Declining Bias and Gender Wage Discrimination? A Meta- Regression Analysis.‖ The Journal of Human Resources 39 (3): 828–838. Jayne, T.S., T. Yamano, and J. Nyoro. 2004. ―Interlinked Credit and Farm Intensi�cation: Evidence from Kenya.‖ Agricultural Economics 31 (2–3): 209–218. Jensen, R. 2007. ―The Digital Provide: Information (Technology), Market Performance, and Welfare in the South Indian Fisheries Sector.‖ The Quarterly Journal of Economics 122(3): 879–924. Kabir, H., and N. Uphoff. 2007. ―Results of Disseminating the System of Rice Intensification with Farmer Field School Methods in Northern Myanmar.‖ Experimental Agriculture 43 (4): 463–476. Kaboski, J.P., and R. M. Townsend. 2005. ―Policies and Impact; An Analysis of Village-Level Microfinance Institutions.‖ Journal of the European Economic Association 3(1): 1–50. Kassie, M., J. Pender, M. Yesuf, G. Kohlin, R. Bluffstone, and E. Mulugeta. 2008. ―Estimating Returns to Soil Conservation Adoption in the Northern Ethiopian Highlands.‖ Agricultural Economics 38 (2): 213–232. Kerr, J., and K. Chung. 2001. ―Evaluating Watershed Management Projects.‖ CAPRi Working Paper 17, CGIAR Systemwide Program on Collective Action and Property Rights, International Food Policy Research Institute (IFPRI), Washington, DC. Khandker, S. 2005. “Microfinance and Poverty: Evidence Using Panel Data from Bangladesh.� The World Bank Economic Review 19 (2): 263–286. Khandker, S., Z. Bakht, and G. Koolwal. 2006. ―The Poverty Impact of Rural Roads: Evidence from Bangladesh.‖ Policy Research Working Paper 3875, World Bank, Washington, DC. Lokshin, M., and R. Yemtsov. 2005. ―Has Rural Infrastructure Rehabilitation in Georgia Helped the Poor?‖ The World Bank Economic Review 19 (2): 311–333. Lopez, F., and A. Maffioli. 2008. ―Technology Adoption, Productivity and Specialization of Uruguayan Breeders: Evidence from an Impact Evaluation.‖ Working Paper 7/08, Inter-American Development Bank, Office of Evaluation and Oversight (OVE). Lopez, R., and G. Galinato. 2007. ―Should Governments Stop Subsidies to Private Goods?: Evidence from Rural Latin America.‖ Journal of Public Economics 91(5-6): 1071–1094. Macours, K. 2007. ―Land Titles and Conflicts in Guatemala.‖ School of Advanced International Studies (SAIS), Johns Hopkins University, Washington, DC. Macours, K., A. De Janvry, and E. Sadoulet. 2007. ―Insecurity of Property Rights and Social Matching in the Tenancy Market.‖ Unpublished. Malawi, Ministry of Agriculture and Food Security. 2008. Evaluation of the 2006/7 Agriculture Input Subsidy Programme, Malawi: Final Report. Mancini, F., A.H. Van Bruggen, and J.L.S. Jiggins. 2007. ―Evaluating Cotton Integrated Pest Management Farmer Field School Outcomes Using the Sustainable Livelihoods Approach in India.‖ Experimental Agriculture 43 (1): 97–112. 84 Mancini, F., A.J. Termorshuizen, J.L.S Jiggins, and A.H. Van Bruggen. 2008. ―Increasing the Environmental and Social Sustainability of Cotton Farming through Farmer Education in Andhra Pradesh, India.‖ Agricultural Systems 96 (1–3): 16–25. Maredia, M. K. 2009. ―Improving the Proof: Evolution of and Emerging Trends in Impact Assessment Methods and Approaches in Agricultural Development.‖ Discussion Paper 929, International Food Policy Research Institute (IFPRI), Washington, DC. Maredia, M., and D. Raitzer. 2006. CGIAR and NARS Partner Research in Sub-Saharan Africa:Evidence of Impact to Date. Report commissioned by the Science Council’s Standing Panel on Impact Assessment. Rome, Italy: Science Council Secretariat. Mariano, N., and G. Porto. 2007. ―Burley Tobacco Clubs in Malawi: Nonmarket Institutions for Exports.‖ Munich Personal RePEc Archive (MPRA) Paper No. 6210. http://mpra.ub.uni-muenchen.de/6210/. Markussen, T. 2008. ―Property Rights, Productivity, and Common Property Resources: Insights from Rural Cambodia.‖ World Development 36(11): 2277–2296. Mather, D.L., R. Bernsten, J.C. Rosas, A. Viana Ruano, and D. Escoto. 2003. ―The Economic Impact of Bean Disease Resistance Research in Honduras.‖ Agricultural Economics 29(3): 343–352. Matuschke, I., R.R. Mishra, and M. Qaim. 2007. ―Adoption and Impact of Hybrid Wheat in India.‖ World Development 35(8): 1422–1435. McKernan, S-M. 2002. ―The Impact of Microcredit Programs on Self-Employment Profits: Do Noncredit Program Aspects Matter?‖ The Review of Economics and Statistics 84(1): 93-115. Minten, B., L. Randrianarison, and J. Swinnen. 2007. ―Spillovers from High-Value Agriculture for Exports on Land Use in Developing Countries: Evidence from Madagascar.‖ Agricultural Economics 37 (2–3): 265–275. Mookerjee, Rajen. 2006. ―A Meta-analysis of the Export Growth Hypothesis.‖ Economics Letters 91 (3): 395–401. Mu, R., and D. van de Walle. 2007. ―Rural Roads and Poor Area Development in Vietnam.‖ Policy Research Working Paper 4340, World Bank, Washington, DC. Mutume, Gumisai. 2001. ―Whose Land Reform?‖ Multinational Monitor. Published June 1. Nakasone, E. 2008. The Impact of Land Titling on Labor Allocation: Evidence from Rural Peru.‖ Working Paper, Agri. and Res. Econ. Department, University of Maryland, College Park. Negri, M., and G. Porto. 2007. ―Burley Tobacco Clubs in Malawi: Nonmarket Institutions for Exports,‖ MPRA Paper 6210, University Library of Munich, Germany. Nkonya, E., P. Dayo, T. Mogues, J. Pender, K. Yahaya, G. Adebowale, T. Arokoyo, and E. Kato. 2008. ―From the Ground Up: Impacts of a Pro-Poor Community-Driven Development Project in Nigeria.‖ Discussion Paper 40, Agriculture and Rural Development, World Bank, Washington, DC. Owens, T., J. Hoddinott, and B. Kinsey. 2003. ―The Impact of Agricultural Extension on Farm Production in Resettlement Areas of Zimbabwe.‖ Economic Development and Cultural Change 51 (2): 337–357. Phillips, J.M. 1994. ―Farmer Education and Farmer Efficiency: A Meta-analysis.‖ Economic Development and Cultural Change 43 (1): 149–165. 85 Place, F., and K. Otsuka. 2002. ―Land Tenure Systems and Their Impacts on Agricultural Investments and Productivity in Uganda.‖ Journal of Development Studies 38 (6): 105– 128. Place, F., M. Adato, P. Hebinck, and M. Omosa. 2005. ―The Impact of Agroforestry-Based Soil Fertility Replenishment Practices on the Poor in Western Kenya.‖ Research Report 142, International Food Policy Research Institute (IFPRI), Washington, DC. Praneetvatakul, S., and H. Waibel. 2006. ―The Impact of Farmer Field Schools on Pesticide Use and Environment in Thailand.‖ Paper presented at German Association of Agricultural Economists (GEWISOLA) 46th Annual Conference, Giessen Germany, October 46. Qaim, M., and A. de Janvry. 2003. ―Genetically Modified Crops, Corporate Pricing Strategies and Farmers Adoption: The Case of BT Cotton in Argentina‖ American Journal of Agricultural Economics 85 (4): 814–828. Qaim, M., and A. de Janvry. 2005. ―Bt Cotton and Pesticide Use in Argentina: Economic and Environmental Effects.‖ Environment and Development Economics 10 (2): 179–200. Qaim, M., A. Subramanian, G. Naik, and D. Zilberman. 2006. ―Adoption of Bt Cotton and Impact Variability: Insights from India.‖ Review of Agricultural Economics 28 (1): 48–58. Raitzer, D.A., and T.G. Kelley. 2008. “Benefit-Cost Meta-analysis of Investment in the International Agricultural Research Centers of the CGIAR.� Agricultural Systems 96 (1–3): 108–123. Rao, V., and M. Woolcock. 2004. ―Integrating Qualitative and Quantitative Approaches in Program Evaluation.‖ In F. Bourguignon and L.A. da Silva, eds. The Impact of Economic Policies on Poverty and Income Distribution: Evaluation Techniques and Tools, pp. 165–190 (chapter 8). Washington, DC: World Bank. Ravallion, Martin. 2006. ―Evaluating Anti-Poverty Programs.‖ In R. Evenson and P Schultz, eds., Handbook of Agricultural Economics, Vol. 4. Amsterdam: North-Holland. Ravallion, Martin. 2008. ―Evaluation in the Practice of Development.‖ Policy Research Working Paper 4547, World Bank, Washington, DC. Reinikka, R., and J. Svensson. 2006. ―Fighting Corruption to Improve Schooling: Evidence from a Newspaper Campaign in Uganda.‖ Journal of the European Economic Association 3 (2-3): 259–267. Romani, M. 2003. ―The Impact of Extension Services in Times of Crisis. Cote d’Ivoire 1997- 2000.‖ Working Paper 190, The Centre for the Study of African Economies, Oxford University, Oxford, UK. Simmons, P., P. Winters, and I. Patrick. 2005. ―An Analysis of Contract Farming in East Java, Bali, and Lombok, Indonesia.‖ Agricultural Economics 33 (s3) issue supplement: 513–525. Slavin, R.E. 1995. ―Best Evidence Synthesis: An Intelligent Alternative to Meta-analysis.‖ Journal of Clinical Epidemiology 48 (1): 9–18. Smith, J., and P. Todd. 2005. ―Does Matching Overcome Lalonde‘s Critique of Nonexperimental Estimators?‖ Journal of Econometrics 125 (1–2): 303–353. Smith, R.E., 2004. ―Land Tenure, Fixed Investment, and Farm Productivity: Evidence from Zambia’s Southern Province.‖ World Development 32 (10): 1641–1661. Spielman, D., and K. von Grebmer. 2004. ―Public-Private Partnerships in Agricultural Research: An Analysis of Challenges Facing Industry and the CGIAR.‖ Discussion Paper No. 113, International Food Policy Research Institute (IFPRI), Washington, DC. 86 Stanley, T.D. 2001. ―Wheat from Chaff: Meta-Analysis as Quantitative Literature Review.‖ The Journal of Economic Perspectives 15 (3): 131–150. Stroup, D.F., J.A. Berlin, S C. Morton, I. Olkin, G.D. Williamson, D. Rennie, D. Moher, B.J. Becker, T.A. Sipe, S.B. Thacker. 2000. ―Meta-analysis of Observational Studies in Epidemiology: A Proposal for Reporting.‖ The Journal of the American Medical Association 283 (15): 2008– 2012. Subramanian, A., and M. Qaim. 2009. ―Village-wide Effects of Agricultural Biotechnology: The Case of Bt Cotton in India.‖ World Development 37(1): 256–267. Thiam, A., B.E. Bravo-Ureta, T. E. Rivas. 2001. ―Technical Efficiency in Developing Country Agriculture: A Meta-analysis.‖ Agricultural Economics 25(2–3): 235–243. Torero, M., and E. Field. 2005. ―Impact of Land Titles over Rural Households.‖ Working Paper 7/05, Inter-American Development Bank, Office of Evaluation and Oversight (OVE). Tripp, R., M. Wijeratne, and V.H. Piyadasa, 2005. ―What Should We Expect from Farmer Field Schools?: A Sri Lanka Case Study.‖ World Development 33(10): 1705–1720. Van de Walle, D., and D. Cratty. 2002. ―Impact Evaluation of a Rural Road Rehabilitation Project.‖ Working Paper. World Bank, Washington, DC. Van Den Berg, M., and R. Ruben. 2006. ―Small-Scale Irrigation and Income Distribution in Ethiopia.‖ Journal of Development Studies 42(5): 868–880. Vranken, L., K. Macours, N. Noev, J. Swinnen. 2008. ―Property Rights Imperfections, Asset Allocation, and Welfare: Co-Ownership in Bulgaria.‖ Discussion Paper 180, LICOS Centre for Institutions and Economic Performance, Katholieke Universiteit Leuven. Wegerif, M. 2004. ―A Critical Appraisal of South Africa‘s Market-Based Land Reform Policy: The Case of the Land Redistribution for Agricultural Development Program in Limpopo.‖ Nkuzi Development Association, Research Report No. 19, December. Wilson, D. B. 1999. ―Practical Meta-analysis.‖ Presentation made at the American Evaluation Association meeting, Orlando, Florida, October. Accessed on February 8, 2009. http://mason.gmu.edu/~dwilsonb/downloads/overview.ppt#256,1,Practical Meta-Analysis. Wollni, M., and M. Zeller, 2007. ―Do Farmers Bene�t from Participating in Specialty Markets and Cooperatives?: The Case of Coffee Marketing in Costa Rica.‖ Agricultural Economics 37 (2–3): 243–248. Wooldridge, J., 2002. Econometric Analysis of Cross Section and Panel Data. Cambridge, MA: MIT Press. World Bank. 2008. World Development Report 2008: Agriculture for Development. Washington, DC: World Bank. Wright, K., and J. Copestake. 2004. ―Impact Assessment of Microfinance Using Qualitative Data: Communicating between Social Scientists and Practitioners Using the QUIP.‖ Journal of International Development 16 (3): 355–367. Zhou, J.M. 1999. ―How to Carry Out Land Consolidation: An International Comparison.‖ Working Paper eco99/1, European University Institute, Department of Economics. San Domenico, Italy. 87