20881 AGRICULTURAL KNOWLEDGE & INFORMATION SYSTEMS (AKIS) GOOD PRACTICE NOTE Ex Ante Economic Analysis in AKIS Projects Methods and Guidelines for Good Practice Based on a report by Gesa Horstkotte-Wesseler, Mywish Maredia, Derek Byerlee, and Gary Alex with inputs from Jock Anderson, Marie-Hélène Collion, Madhur Gautam, J. Price Gittinger, Robert D. Hunt, Jaakko Kangasniemi, Matthew McMahon, Ridley Nelson, David Nielson, Michel Petit, Trayambkeshwar P.N. Sinha, Chandrashekhar Ranade, Dina Umali- Deininger, and Willem Zijp March 2000 The World Bank Work in progress Rural Development Family for public discussion Agricultural Knowledge & Information Systems (AKIS) Agricultural Knowledge & Information Systems is a thematic team focusing on agricultural extension, education, and research within the Rural Development Department of the Environmentally & Socially Sustainable Development Network of the World Bank. Contents Preface iv Abbreviations v Summary and Key Recommendations 1 Introduction and Background 2 Overview of Economic Analysis 3 What Is Economic Analysis of Agricultural R&E? 3 Special Problems 4 Methods for Assessing Agricultural R&E Projects 6 Economic Surplus Approach 6 Cost–Benefit Analysis: A Simplified Economic Surplus Approach 7 Efficiency Index: A Simplified Cost–Benefit Analysis 7 Cost-Effectiveness Analysis 7 Break-Even Analysis 8 Risk Assessment 8 Guidelines for Good Practice 8 Activities During Project Preparation 10 Activities During Project Implementation 11 Special Considerations for Competitive Grants Programs 12 References 13 Annex 1: Data Required for Economic Analysis of Agricultural R&E 15 Annex 2: Resources for Economic Analysis of Agricultural R&E 18 Annex 3: How to Conduct Retrospective Technology Impact Studies 20 Annex 4: How to Conduct Prospective Technology Case Studies 21 iii Preface A 1996 World Bank OED review of agri- AKIS is the Agricultural Knowledge and cultural research projects emphasized Information Systems Thematic Team, composed the importance of Borrowers’ use of of World Bank staff working in or interested in economic analysis as an input to prioritizing research, extension, and education programs. research. This led to discussions on the use of The overall team objective is to enhance the ef- economic analysis as a basis for Bank project fectiveness of Bank support to agricultural investments in research and extension projects knowledge and information system develop- and the conclusion that a review of options and ment and thus contribute to the Bank’s objec- good practice in this area would be useful to tives of alleviating poverty, ensuring food AKIS TTLs. Gesa Horstkotte-Wesseler prepared security, and improving sustainable manage- an initial concept paper which was reviewed ment of natural resources. The AKIS team em- and revised with input from other Bank staff in phasizes policy, institutional, and management a series of discussion meetings. This paper sum- issues associated with agricultural research, ex- marizes these discussions and is intended as a tension and education, recognizing that other contribution to the exchange of ideas and expe- thematic teams will focus on technical issues. rience within the Agriculture Knowledge and The Team mission is to “promote the develop- Information Systems Thematic Team. The pa- ment of sustainable and productive agricultural per concludes that while economic analysis of research, extension, and education systems in World Bank projects for AKIS system capacity Bank client countries.� building per se is often not warranted, increased This “AKIS Good Practice Note� was pre- attention to and use of economic analysis within pared by Gesa Horstkotte-Wesseler, Mywish research and extension programs is useful and Maredia, Derek Byerlee, and Gary Alex, based to be encouraged. on inputs from Jock Anderson, Marie-Hélène “AKIS Good Practice Notes� are to dissemi- Collion, Madhur Gautam, J. Price Gittinger, nate views, experience, and ideas, which may Robert D. Hunt, Jaakko Kangasniemi, Matthew assist World Bank Team Leaders, national coun- McMahon, Ridley Nelson, David Nielson, terparts from Borrower counties, and other part- Michel Petit, Trayambkeshwar P.N. Sinha, ners with preparation and implementation of Chandrashekhar Ranade, Dina Umali-Deininger, projects to strengthen agricultural research, ex- and Willem Zijp. tension, and education programs. They attempt to disseminate lessons from innovative experi- ences in World Bank projects and elsewhere and make this information readily available for com- David Nielson ment and use by project teams. Chairman iv Abbreviations IRR Internal rate of return M&E Monitoring and evaluation NARO National agricultural research organization NARS National agricultural research system NPV Net present value OED (The World Bank’s) Operations Evaluation Department R&E Research and extension v Summary and Key Recommendations T his note outlines an approach to eco- • Agricultural extension investments (again nomic analysis in agricultural research including those with non-World Bank fund- and extension (R&E) projects financed ing) are often best evaluated using rough under World Bank loans and credits. Because measures of cost-effectiveness or unit costs Bank-funded R&E projects allocate considerable (such as costs per farmer reached or hectare resources to institutional development, it makes affected). The evaluation of extension im- little sense to attempt an economic analysis for pacts separately from research impacts is such projects per se. Furthermore, although it is justifiable if recommended technologies conceivable and often desirable to undertake ex have been available for some time but have ante economic analysis of direct investments in not been adopted. agricultural research programs, the time and re- • Alternatively, break-even analyses can be sources available during project preparation usu- conducted for extension programs to show ally preclude a comprehensive, high-quality the minimum impact needed to justify pro- analysis. This note provides guidelines for alter- posed expenditures. Break-even analyses native approaches, recommending steps to take might also be used for research programs during project preparation and implementation. or for Bank-financed projects as a whole. In designing and conducting an economic • During project preparation, economic analy- analysis, the emphasis should be on building ses should focus on (1) retrospective tech- capacity within national agricultural R&E sys- nology adoption and impact studies to tems. Rather than purely serving the needs of demonstrate that the current technology World Bank or bilateral donors, economic analy- system has been productive in some areas; sis of R&E should become an integral part of and (2) prospective technology case studies management processes within national systems to demonstrate that R&E investments can for project evaluation, priority setting, and im- produce substantial economic benefits. pact assessment. Accordingly, this paper makes These analyses can be based on existing the following recommendations: studies and information or—preferably, at • Agricultural research investments should be least for the prospective studies—on field- evaluated at the level of research programs oriented case studies. Break-even analyses for important commodities or production and overall assessments of profitability can systems (such as maize, beef, or irrigated help in evaluating whether proposed invest- rice). Evaluators should consider all pro- ments are justified. grams, even those with non-World Bank • During project implementation, the main funding. At the level of individual research emphasis should be on establishing systems projects within a research program, that incorporate economic analysis on an however, economic analysis is not recom- ongoing basis into R&E evaluation, plan- mended. The analysis of research expendi- ning, and priority setting. This requires im- ture impacts will almost always have to take proving R&E management within national extension expenditures into account. systems. Training, support for socioeco- 1 2 Ex Ante Economic Analysis in AKIS Projects nomic programs, partnership building, and underlying model, the economic surplus stakeholder participation are needed to en- method or a simplification thereof, such as sure effective economic analysis within pro- a cost–benefit analysis. The specific type of gram management systems. More attention analysis chosen should be determined by the should also be paid to collecting baseline data and analytical capacity available within information and designing performance in- the national system. dicators (issues that are treated in separate Economic analysis of agricultural R&E guidelines). should not be designed to calculate a single sum- • Evaluators within the national agricultural mary measure (which is normally subject to a research system should conduct the eco- wide range of assumptions). Instead, the nomic analyses whenever possible in order mainstreaming of economic analysis should pro- to build local capacity and foster local own- vide a framework for research managers and ership of the results. The role of the World scientists to think about the impact of their ac- Bank or other external parties should be lim- tions in terms of costs and benefits, thereby lead- ited to providing support. ing to a more efficient allocation of scarce • All economic analyses should use, as an resources. Introduction and Background For projects funded through World Bank loans Today, in many fields of agricultural re- and credits, economic analysis has many dimen- search, the “easy gains� have already been sions. It can include the assessment of economic, made. It no longer seems obvious that invest- fiscal, social, and environmental impacts; the ments in agricultural R&E can sustain contin- relationship of the project to broader develop- ued high rates of return. Indeed, the pervasive ment objectives of the sector and country; the funding crisis and the poor performance of rationale for public-sector involvement; and the many national agricultural R&E systems have contribution to poverty reduction. This note fo- placed the efficiency of agricultural R&E invest- cuses on a single dimension—assessing how ments in doubt. Moreover, thanks to recent ad- investments in agricultural research and exten- vances in the methodology of economic analysis, sion (R&E) affect agricultural productivity and most aspects of agricultural R&E are now mea- growth in the agriculture sector. A comprehen- surable. These factors have prompted calls for sive treatment of the other dimensions would a “fairer� treatment of agricultural R&E in the require developing separate guidelines. World Bank’s project cycle. For most Bank-funded projects, preparation In 1997, in a key report on agricultural R&E includes a mandatory economic analysis of the (see box 1), the World Bank’s Operations Evalu- expected returns on investments. In the past, ation Department (OED) recommended agricultural R&E projects were exempt from this integrating economic analysis into the priority- requirement because their benefits are difficult setting and project preparation/appraisal pro- to quantify and their rates of return worldwide cesses for national agricultural research systems were widely acknowledged to be high. (NARSs) and into the ex ante and ex post evalu- Overview and Economic Analysis 3 Box 1. OED Recommendations on Agricultural R&E nomic analysis, especially in setting program priorities�(p. 12). In 1997, the OED published a report under the title, • To build the capability for economic analysis of Agricultural Extension and Research: Achievements and research within NARSs, “an ex ante economic Problems in National Systems (Purcell and Ander- assessment of programs within a borrower’s re- son 1997). The report includes these recommen- search agenda should be completed during the dations: project preparation/appraisal process� (p. 279). However, “to be consistent with ownership and Research: capacity-building objectives, the scale and com- • Economic analysis should be integrated into prehensiveness of ex ante analysis will depend project preparation for NARSs and into the ex upon the availability of economic expertise within ante and ex post evaluation processes of World NARSs� (p. 279). Bank-supported research programs. “[E]conomic analysis (normally using an economic surplus Extension: model or a simplified derivative) should be fos- • For agricultural extension projects, ex ante eco- tered in borrowers’ processes for prioritizing re- nomic analysis is not always necessary. Because search. By the same logic, economic analysis of the difficulties of linking cause and effect, “the should be used by the Bank in ex ante and ex expenditure of a large amount of scarce resources post evaluation of supported research programs� on extensive surveys to quantify extension im- (p. 278). pact and economic benefits is of dubious value • Instead of analyzing the full economic rate of re- for many countries� (p. 197). turn on investments, the Bank should build • Instead, “the Bank should place much greater NARS capacity to conduct economic analysis. emphasis during implementation on assessing the “[The Bank’s] focus should not be on estimating effectiveness of the extension process in meeting the economic return on the projects it supports. the program objectives in the targeted farming sys- […] Rather, the Bank should focus on institution- tems—accelerating the rate of adoption of relevant alizing the capacity within NARSs to use eco- technology in a cost-effective manner� (p. 270). ation processes for Bank-supported research Clear guidelines are needed to help World programs. Accordingly, the importance of eco- Bank operational staff perform economic analy- nomic analysis is growing in Bank-funded agri- sis of agricultural R&E projects. This note is de- cultural R&E projects. Some Bank managers signed to clarify some of the issues involved and now encourage or even require economic analy- to provide guidelines for good practice in ana- sis in agricultural research projects, both to jus- lyzing the potential impact of R&E investments tify the investment and to prioritize programs on agricultural productivity and on growth in and projects funded by the Bank. the agriculture sector. Overview and Economic Analysis What Is Economic Analysis of Agricultural R&E? cultural R&E activities, either before they are undertaken (ex ante evaluation) or after they are Economic analysis of agricultural R&E uses eco- completed (ex post evaluation). In either case, nomic principles and models to evaluate agri- the main purpose of economic analysis is to help 4 Ex Ante Economic Analysis in AKIS Projects select and design R&E activities that contribute or equal to the expected NPV of mutually to the welfare of a country. Ex post analysis is acceptable investment alternatives; or done to justify R&E budgets and to identify ar- 2. the IRR must be greater than the opportu- eas with likely future research payoffs. Ex ante nity cost of capital. analysis is designed to help set priorities, allo- Projects can be ranked using either NPV or cate resources, and decide whether or not to IRR. However, NPV is preferred because it can proceed with a specific R&E program. be normalized in a way that takes into account Economic analysis is only one part of the the size of the research investment. Although the overall analysis of a World Bank-funded project. IRR is not always a satisfactory measure (for It takes for granted that the project is technically example, when comparing and ranking mutu- sound, socially acceptable, and financially viable ally exclusive project alternatives), it is widely for all participants; that institutional arrange- understood and may be used for the purpose of ments will be effective during implementation presenting the results of economic analysis. (Belli and others 1998); and that environmental For investments with benefits that are diffi- and natural resource impacts are acceptable (al- cult to measure in monetary terms (for example, though these can sometimes be costed and in- reductions in soil erosion), physical indicators cluded in more complex economic analyses). of achievement in relation to costs (such as cost- In the early phases of project preparation, effectiveness and unit costs) are appropriate economic analysis should help answer several measures for determining the acceptability of the questions of strategic importance, such as: What investment. In all cases, however, the economic will happen if the project is undertaken? What analysis should give a persuasive rationale for if it is not? Is the project the best alternative for why the benefits of the project are expected to achieving the objective? Do some components outweigh its costs—that is, why the net devel- of the project perform better than others? Should opment impact of the project investment is ex- the public or the private sector undertake the pected to be positive. project? And who are the gainers and losers? For investments with benefits that are mea- Special Problems surable in monetary terms, the appropriate yard- stick for judging whether the investment is The main goal of investments in agricultural acceptable is its expected net present value R&E is to increase the efficiency of agricultural (NPV). The NPV is the sum of all costs and ben- production. Important secondary goals are to efits over the life of the investment, discounted provide equity and to enhance security. In other at the opportunity cost of capital. The expected words, successful investments in agricultural NPV takes account of the entire range of pos- R&E increase total income, improve income dis- sible NPVs by weighting all possible project tribution, and reduce income variability. The outcomes with their corresponding relative fre- trick is to measure these benefits and to link them quencies or probabilities. to specific investments in agricultural R&E. That A related yardstick for determining whether poses several problems. an investment is acceptable is the project’s in- Measurement of research benefits is compli- ternal rate of return (IRR). The IRR is the inter- cated because (1) benefits are spread geographi- est rate at which the NPV equals zero—that is, cally and vertically in markets for goods and at which the investments break even. To be ac- services, (2) research can affect product quality, ceptable on economic grounds, an investment (3) some research is not commodity oriented, must meet one of two conditions: (4) some research (such as social science) is 1. The expected NPV of the investment must aimed at modifying institutions, (5) some re- be (a) zero or positive, and (b) higher than search generates externalities, and (6) research Overview and Economic Analysis 5 can be relatively basic without immediate ap- considerable analytical challenges in determin- plication (Alston and others 1995). World Bank ing the potential returns from research invest- lending for agricultural R&E increasingly in- ments. cludes institutional development and capacity Many of the same uncertainties apply to building, natural resources management, inte- extension as to research. Uncertainties include grated farming systems, and social sciences re- the relevance of available technologies and ex- search, all of which produce benefits that are tension “messages� to producer needs; the im- difficult to quantify. pact of technologies at different locations and Time lags pose additional problems in mea- on different farm situations; and technology suring research benefits (see box 2). Research adoption patterns and lag times under differ- takes a long time to affect production and then ent economic, social, technical, and other con- affects production for a long time (20–30 years). ditions. Variations in the available technology, The analyst usually estimates the length and in the efficiency of the extension system, and in shape of time lags arbitrarily, because there are the receptivity of farmers all render the impact no proven theories on how to structure time lags of extension highly uncertain. in an economic analysis model. Erroneous time Another complication is that many exten- lag structures can result, biasing the estimates sion programs address a broad range of com- of rates of return on research investment. modities and production and management Another problem is the inherent uncertainty activities. This makes it difficult to estimate and in predicting the outcomes and impact of re- aggregate costs and benefits. On the cost side, search. Uncertain parameters include the impact extension expenditures are not easily allocated and adoption of research results; the natural, to specific crops, farming systems, or recommen- economic, and political environments in which dations; on the benefit side, separating the agricultural commodities are produced; the time project from nonproject impacts is challenging. it takes to find solutions to research problems, For example, productivity increments for a which can easily exceed the timeframe of a typi- group of farmers might result from any of vari- cal World Bank project cycle; and the factors that ous extension providers or other sources of in- influence farmers’ decisions to adopt new tech- formation; from new technologies, which in turn nologies, such as market conditions, domestic might be due to research by the NARS, by inter- agricultural policies, and the effectiveness of national agricultural research centers, or by the agricultural infrastructure in linking local prices private sector; from infrastructural improve- to border prices. These uncertainties present ments; or from institutional innovations. In a Box 2. Time Lags in Realizing Benefits uptake phase; and the depreciation phase (the time it from Agricultural R&E takes for research knowledge to become obsolete). As a rule of thumb, the time lag for extension is Investments in agricultural R&E can take decades to three years, for adaptive research six years, for ap- realize any benefits. Such time lags are critical in ana- plied research fifteen years, and for strategic research lyzing economic returns. The shape and length of the twenty-five years. Of course, lag times might be longer time lag distribution in the early years of a project, in rain-fed and less commercial agricultural systems important in calculating research benefits, vary among and shorter in more commercialized areas. For exten- research programs and among technological options sion projects, the main benefit comes from speeding within programs (Alston and others 1995). Three re- up a technology adoption that will likely occur any- lated lag components need to be estimated: the re- way, only at a slower rate. If adoption without project search lag (the time between a research expenditure investment will occur ten years later, then the annual and the release of new technologies); the adoption or net project benefits ten years later would be zero. 6 Ex Ante Economic Analysis in AKIS Projects dynamic world with many players, where tech- R&E and many other sectors. Quantifying ben- nology is not the only factor that is changing, it efits from those programs is challenging enough; is difficult to measure the impact of a project’s to go further by attempting to allocate benefits activities. to specific program components (such as staff Benefits from investments in agricultural training), and then to estimate the value of those education are even more difficult to estimate. components, poses nearly insurmountable Education and training support programs in obstacles. Methods for Assessing Agricultural R&E Projects Several different economic methods can be used average rate of return over time) by (1) calcu- to evaluate investments in agricultural R&E lating the change in consumer and producer sur- projects. Methods range in complexity from pluses that results from technological change simple checklist and scoring models to complex brought about through research, and (2) using mathematical programming and simulation estimated economic surplus together with re- models. Annex 1 lists data requirements for vari- search costs to estimate the NPV or IRR. ous methods. The theory underlying the economic surplus No single approach is best for every situa- approach is based on the material benefits to tion. Evaluators should choose the method or society from technological change. The adoption combination of methods best suited to the type of new technology reduces the unit cost of pro- of data available; to the resources, skills, and duction, shifting the supply curve to the right time available; and to the economic importance and increasing consumer and producer sur- of the decision. Annex 2 outlines the available pluses. Consumers gain from the new technol- references, manuals, computer programs, Web ogy because they can consume more at a lower sites, and other resources for assessing agricul- price, and producers gain because their unit tural R&E and for planning and priority setting costs of production fall. The distribution of ben- in NARSs. efits between producers and consumers de- pends on the elasticities of the demand and Economic Surplus Approach supply curves and on the magnitude and na- ture of the supply shift. The combined total ben- The most widely used methods in project evalu- efit to consumers and producers, measured in ation are based on measures of economic sur- monetary units, is the change in economic plus. This evaluation methodology is preferred surplus. The basic formula for estimating the because economic surplus concepts underlie the change in economic surplus in year t (∆ESt) is: conventional economic rationale for government 0 . 5 K t εη intervention in providing agricultural research ∆ ES t = K t Pt Q t (1 + ) ε +η through the public sector (Alston and others 1995). The economic surplus approach estimates where Pt is the price of a commodity affected by returns on investment (generally, a weighted research in year t, Qt is the quantity of production Methods for Assessing Agricultural R&E Projects 7 in year t of the commodity affected by research, an efficiency score) for a research program or e is the elasticity of supply, η is the elasticity of individual project. The index is used to rank a demand, and Kt is the proportionate downward given project in relation to other research shift in the supply curve in year t due to re- projects. For project i, the net efficiency index Ii search. The most critical parameter, the variable is calculated as: K, is calculated as the net change in the cost of production due to new technology (which is K imax ⋅ Pi ⋅ Qi Ii = ( ) sometimes approximated by the yield increment Ci due to the new technology), weighted by the rate of adoption of the new technology in year t. where K imax is computed for the maximum The economic surplus approach has been adoption rate, C i is the research cost of the widely used to calculate benefit–cost ratios, project, Pi is the price of the commodity affected IRRs, and NPVs for benefits generated from by project i, and Qi is the quantity of produc- agricultural research. The method is best used tion of the commodity affected by project i. The by those with at least some training in econom- value of the net efficiency index by itself should ics. Ex ante analysis of future benefits requires not be used to economically justify a project, but information on expected values of production, only as a scoring measure to rank and prioritize expected yield increases, reduction of unit costs research projects based on the economic effi- and/or the maintenance effects of research, ciency criterion. probabilities of research success, adoption rates, research and adoption lags, and the appropri- Cost-Effectiveness Analysis ate discount rate for converting future benefits and costs into present values or for comparing The methods described above are appropriate them against an IRR. for R&E activities where benefits are measur- able in monetary terms and where output has a Cost–Benefit Analysis: A Simplified market price that is relatively easy to assess. Economic Surplus Approach However, the principal benefits of many activi- ties, especially in extension, are not easily quan- Depending on the type of data available and the tified in monetary terms. For such activities, a purpose of economic analysis, the economic straightforward cost–benefit assessment to get surplus approach is often simplified by calcu- economic rates of return can be very demand- lating research benefits in year t as the product ing, if not impossible. of Kt ´ Pt ´ Qt. The cost–benefit analysis assumes An appropriate method to use for projects that demand and supply elasticities are polar— where the principal benefits are not easily quan- that is, that demand is infinitely elastic and sup- tified in monetary terms is the cost-effectiveness ply is completely inelastic, or vice versa. If the analysis. In this analysis, the benefits are mea- purpose of the economic analysis is to measure sured in nonmonetary units (such as the num- only the total economic surplus and not its dis- ber of farmers receiving services or the number tribution between consumers and producers, of trainees) and compared to the cost of the then this simplified approach gives reasonable project to arrive at the cost-effectiveness ratio estimates of the economic benefits. (or cost per unit of output). If the benefits con- sist of improvements in several areas (such as Efficiency Index: A Simplified in farmer education, technology adoption by Cost–Benefit Analysis farmers, and information dissemination), then the various benefit areas are weighted and re- Another simplification often made in ex ante duced to a single measure in a “weighted cost- analysis is to calculate a net efficiency index (or effectiveness analysis.� 8 Ex Ante Economic Analysis in AKIS Projects Cost-effectiveness analysis is always com- Tools for assessing risk include sensitivity parative. It is used to compare different ways of analysis. Using this tool, the analyst identifies achieving the same objective. Cost-effectiveness the variables that most influence a project’s net analysis can determine the alternative that will benefits, such as aggregate costs and benefits, accomplish the same output at lowest cost, or critical cost and benefit items, and the effects of the alternative that generates the highest out- delays. Then the analyst quantifies the extent of put with a given level of inputs. their influence. The preferred approach to sen- sitivity analysis is based on the switching value Break-Even Analysis of a variable—the value at which the project’s NPV becomes zero (or the IRR equals the dis- If benefits from an R&E activity are unknown count rate). Switching values are usually given or difficult to estimate ex ante, a break-even (or in terms of the percentage change in the value threshold) analysis can be conducted as a first of the variable needed to turn the project’s NPV step towards quantification. Using this ap- to zero. The switching values of the more im- proach, an analyst determines the minimum portant variables may be presented in order of benefits that an R&E activity should generate declining sensitivity. For variables that are to justify its costs. This method permits ex ante expected to vary together (such as price and analysis of a project when only its costs are quantity sold), the analyst should examine the clearly known. However, it is based on guess- sensitivity of the outcome to changes in combi- work by the analyst as to the likelihood of nations of those variables (such as in total achieving the minimum necessary benefits, and revenues). it says nothing about the distributional impacts The major shortcoming of sensitivity analy- of the activity. sis is its failure to account for the probabilities of future outcomes. Using more elaborate ap- Risk Assessment proaches to risk assessment, such as simulation techniques, analysts can assign probabilities to Agricultural R&E are inherently risky. Because the a range of outcomes for important variables. economic analysis of R&E activities is based on However, with the data commonly available for uncertain future events, the measurement of costs agricultural R&E, these probabilities are usually and benefits inevitably involves explicit or im- highly subjective. Simulation exercises are there- plicit probability judgments (Belli and others 1998). fore not recommended. Guidelines for Good Practice Given the OED recommendations and the grow- economic analysis purely for a World Bank loan ing emphasis on quantitative evaluation by or credit would be of little value. Bank-funded World Bank management, economic analysis agricultural R&E projects are no longer limited will increasingly be required for Bank-funded to supporting specific technologies in extension agricultural R&E projects. However, a summary or well-defined programs in research. Instead, Guidelines for Good Practice 9 many projects now focus on institutional re- appropriate for extension and adaptive re- structuring, capacity building, and farmer em- search, less relevant to applied research, and powerment, areas where benefits are difficult to very speculative for basic and strategic re- quantify in the short to medium term. It there- search. fore makes little sense to attempt a full economic • For individual projects within a research analysis of a World Bank project per se. The most program, economic analysis is not recom- important role for economic analysis today is in mended because (1) the benefits from such priority setting, planning, evaluation, and im- fine-tuning might not justify the cost, (2) pact assessment for agricultural R&E within a measurement problems rise as the degree of national system. The emphasis should be on data disaggregation increases, and (3) building capacity for economic analysis within micromanaging creative endeavors such as the national system. research can be counterproductive. Detailed From this central conclusion, several prin- allocation decisions are probably best ciples follow: guided by well-structured incentive systems • Both ex post and ex ante economic analyses and farmer demands (Alston and others of direct expenditures for agricultural R&E 1995). operations are feasible and often desirable • For agricultural education and for extension under the particular circumstances of a bor- activities that do not focus on productivity rowing country. Therefore, ex ante analysis enhancement or specific commodities, ex- should become an integral part of the plan- penditures are best evaluated through rough ning and priority-setting process in agricul- measures of cost-effectiveness or unit costs tural R&E, and ex post analysis should (such as costs per farmer reached or hectare become a regular part of evaluating R&E affected). activities. These analyses should be inte- • A break-even analysis showing the mini- grated into the country’s ongoing programs, mum impact needed to justify an investment not performed specifically for World Bank is often useful in evaluating a proposed loan projects. in terms of risk and viability. Extension im- • Investments that directly support agricul- pacts may be evaluated separately from re- tural research should be evaluated at the search impacts if the recommended level of research programs for important technologies have been available for some commodities or production systems (such time without having been adopted. as maize, beef, or irrigated rice). Evaluators • To build local capacity and foster local own- should consider all programs, even those ership of the analytical process (including with non-World Bank funding. The analy- its results), evaluators within the NARS sis of research expenditure impacts will al- should conduct the economic analyses most always have to take extension whenever possible, with the World Bank or expenditures into account. other external parties providing support. • For R&E projects that are designed to in- • Some technology adoption occurs even crease productivity, relatively simple cost– without agricultural R&E investments. benefit or economic surplus methods can be Therefore, especially for extension pro- used to estimate NPV and IRR as a basis for grams, investment decisions should be investment decisions. Costs should include based on a comparison of outcomes with all direct costs for the activity; management and without investment. overheads; and, for research activities, ex- • Economic analysis should be based on the tension costs. Such simple analyses are most full duration of a project’s impact, not the 10 Ex Ante Economic Analysis in AKIS Projects duration of a specific Bank-funded project. of ex ante analysis can be reduced by build- The analyst should choose a timeframe from ing on ex post studies to define key param- the beginning of project activities (such as eters of agricultural technology generation the start of extension activities or of a breed- and dissemination, such as adoption rates. ing program) to the end of the period when All available information should be used to benefits are likely to accrue. make the case that future investments in • For any particular country, the uncertainties agricultural R&E will be profitable. Activities During Project Preparation During project preparation, the limits on avail- box 3). The analyst should also indicate able time and resources will usually preclude a whether there are reasons to expect that comprehensive, high-quality economic analysis. future returns might differ from past expe- However, the analysis should indicate the over- rience. all magnitude of expenditures and the historic • Retrospective studies on technology adoption rates of return on R&E investments in the par- and impact: The analyst should conduct ticular setting. Good practices during project studies on the adoption and impact of pre- preparation include: vious technologies to show results from the • A summary of previous studies: As a first step, current technology system and to assess the the analyst should summarize previous impacts expected from past R&E efforts, studies (both ex post and ex ante) on the based on early adoption and on-farm veri- impact of agricultural R&E in the particular fication trials. For research projects, such country and/or setting. To help indicate fu- studies should concentrate on major com- ture impacts, the analyst might include a modities (such as maize, beans, or cassava); synopsis of the international experience (see for extension, they should focus on key tech- Box 3. Worldwide Experience with Returns The report found that returns averaged 88 per- on Agricultural R&E Investments cent on investments in research alone, 79 percent on extension alone, and 45 percent on R&E combined. Many ex post analyses have shown that agricultural The lower estimate for R&E combined might be be- R&E investments yield high rates of return. A com- cause the corresponding studies captured more of the prehensive 1998 report by the International Food Policy total costs of the technology innovation process. The Research Institute (Alston and others 1999) reviewed report found no evidence of a decline in rates of re- 294 studies of returns on investments in agricultural turn in recent years and little consistent difference research and development (including extension). Af- among regions. Rates of return were similar across all ter eliminating extremes, the report found that the re- research categories except for natural resources re- maining 1,760 estimates indicated an average return search, where lower rates of return were mainly due of 73 percent per year, confirming the conventional to longer production cycles. view that returns on investments in agricultural re- search and development are relatively high. Activities During Project Implementation 11 nologies (such as integrated pest manage- • A break-even analysis: The information avail- ment). To maintain objectivity and credibil- able during project preparation will rarely ity, the analyst should avoid selecting only suffice for a rigorous ex ante assessment. success stories (for step-by-step guidelines, However, the analyst should determine the see annex 3). threshold magnitudes of productivity gains • Case studies on prospective technologies: The that would be needed to justify total project analyst should conduct field-based case allocations. These figures should be com- studies, for selected crops and regions, on pared to historic figures from similar situa- the technologies that could be developed tions to assess their probability. (For national during the project (for step-by-step guide- programs, an estimate of the minimum lines, see annex 4). These studies should growth of agricultural gross domestic prod- begin as early as possible to help in project uct needed to justify proposed expenditures planning and priority setting and to estab- might help determine whether an invest- lish a core team in the national system with ment is realistic under the particular circum- experience in applying the proposed stances. The design of this type of analysis methods. is still under discussion.) Activities During Project Implementation During project implementation, the main em- • Partnership building: While its socioeconomic phasis should be on establishing a system that capacity is still limited, the NARO should incorporates economic analysis on an ongoing build partnerships to obtain the analytical basis into R&E evaluation, planning, and prior- capabilities needed for impact assessments ity setting. This requires improving R&E man- and evaluation studies. Potential partners agement within the national systems. It also include institutions in the same country implies the need to pay more attention to col- (such as agricultural universities and eco- lecting baseline information and designing per- nomic research institutes) and other coun- formance indicators (issues addressed in other tries (such as international agricultural guidelines). research centers). All partnerships should be Capacity building for improved R&E man- designed to increase the NARO’s own ca- agement touches on a broad range of issues: pacity for economic analysis. • Increased support for socioeconomics programs: • Shifts in donor support: To encourage analysts In many developing countries, the national within the national systems to conduct eco- agricultural research organizations nomic analyses for R&E investments, donor (NAROs) neglect the socioeconomics disci- support for priority setting should shift from pline. To be able to conduct economic analy- master planning by donor consultants to in- sis for agricultural R&E investments, stitutional capacity building within the NAROs should provide more staffing, train- NARS. ing, and support for their socioeconomics • Training for NARS staff: To encourage think- programs. ing in terms of costs and benefits, all research 12 Ex Ante Economic Analysis in AKIS Projects scientists and extension workers within the information used and to help build politi- NARS should have basic training in the fun- cal support for the strategy as well as for damental concepts of economic analysis. In the R&E system. addition, those who will conduct the eco- • Monitoring and evaluation (M&E): M&E nomic analyses should have a thorough should be implemented in a highly decen- training in methodology. Staff training tralized manner, with R&E staff participat- should start during project preparation, ing in data collection and reporting on when ex post analyses of past R&E efforts progress and impact indicators. A small cen- and field-based case studies of prospective tral unit is needed to promote M&E, develop technologies are prepared. standards, provide training, and consolidate • Stakeholder participation: The project should reports (Byerlee and Alex 1998). Compre- involve major stakeholders (such as farmer hensive monitoring plans with indicators groups and nongovernmental organiza- and targets should be developed during tions) as full partners in priority setting and project preparation and then updated rou- impact evaluation to increase the quality of tinely throughout the life of the project. Special Considerations for Competitive Grants Programs Competitive grants programs for agricultural sults because the time and skill needed for research are proliferating, yet no body of good good estimates are not available for all sub- practices is generally accepted for the use of eco- mitted projects. The low-quality results in nomic analysis in the operations of these pro- turn undermine the credibility of competi- grams, let alone in measuring their performance tive grants programs in the eyes of both re- (Alex 1998). The competitive grants programs searchers and research funders. in some countries (such as Australia) have re- However, the critics do agree that research- quired that all projects submitted for funding ers should be explicit in their proposals about include an ex ante cost–benefit analysis. Critics the area targeted by the proposed research, the make the following points: estimated number of farmers to be reached, and • For many projects (for example, in basic re- the expected change in yields or costs per hect- search or natural resources research), a cost– are resulting from the research. Dividing the benefit analysis is inappropriate because the costs of the proposed project by any of these quantity or value of outputs is difficult to indicators provides an impact estimate (a mea- estimate. sure of cost-effectiveness) potentially useful to • Cost–benefit analyses yield low-quality re- both researchers and proposal evaluators. References Alex, G. 1998. Assessing Agricultural Research: To- Gittinger, J.P. 1982. Economic Analysis of Agricul- wards Consensus on a Framework for Perfor- tural Projects. Baltimore, MD: John Hopkins mance and Impact Assessment. ESDAR University Press. Special Report No. 6. The World Bank, Horton, D., P. Bellantyne, W. Peterson, B. Uribe, Washington, DC. D. Gapasin, and K. Sheridan. 1993. Monitor- Alston, J.M., M.C. Marra, P.G. Pardey, and T.J. ing and Evaluating Agricultural Research: A Wyatt. 1999. “Research Returns Redux: A Sourcebook. CAB International, Wallingford. Meta-Analysis of the Returns to Agricultural Janssen, W., and A. Kissi. 1997. Planning and Pri- R&D.� EPTD Discussion Paper No. 38. In- ority Setting for Regional Research. Research ternational Food Policy Research Institute, Management Guidelines 4. International Washington, DC. Service for National Agricultural Research, Alston, J.M., G.W. Norton, and P.G. Pardey. 1995. The Hague. Science Under Scarcity: Principles and Practice Masters, W.A., B. Coulibaly, D. Sanogo, M. for Agricultural Research Evaluation and Pri- Sidibé, and A. Williams. 1996. The Economic ority Setting. Ithaca, NY: Cornell University Impact of Agricultural Research: A Practical Press. Guide. Department of Agricultural Econom- Belli, P., J. Anderson, H. Barnum, J. Dixon, and ics, Purdue University, West Lafayette, IN. J.-P. Tan. 1998. Handbook on Economic Analy- [Available by e-mail: Masters@AgEcon. sis of Investment Operations. Operations Core Purdue.edu] Services Network, Learning and Leader- Orlando, C.G. 1991. Financial, Economic and So- ship Center, The World Bank, Washington, cial Evaluation: Manual for Analysis and Evalu- DC. ation of Projects. Banco Centroamericana de Byerlee, D., and G. Alex. 1998. “Strengthening Integración Económica, San José, Costa Rica. National Agricultural Research Systems: Purcell, D.L., and J.R. Anderson. 1997. Agricul- Policy Issues and Good Practice.� ESSD, The tural Extension and Research: Achievements and World Bank, Washington, DC. Problems in National Systems. The World Collion, M.-H., and A. Kissi. 1995. Guide to Pro- Bank, Washington, DC. gram Planning and Priority Setting. Research USAID. 1987. A.I.D. Manual for Project Economic Management Guidelines 2E. International Analysis. Bureau for Program and Policy Co- Service for National Agricultural Research, ordination, U.S. Agency for International The Hague. Development, Washington, DC. 13 Annex 1: Data Required for Economic Analysis of Agricultural R&E Economic Surplus Approach Advantage: Benefits from research can be disaggregated among different groups of producers and consumers. Disadvantage: Highest data requirements (including the need to establish price elasticities of supply and demand for each identified group of consumers and producers). Research-Related Data Required Market-Related Data Required • Key technologies generated by each selected research • Quantities produced and consumed. program over the period of the analysis. • Prices received and paid. • Proportionate yield change due to new technologies. • Price elasticities of supply and demand • Proportionate change in variable input costs per hect- for each identified group of producers are, if any, to achieve the expected yield change. and consumers. • Probability of research success. • Exogenous output growth rate: antici- • Research lag: observed or expected time from the be- pated proportionate change in output ginning of research activities to the release of a new not due to research in each year (growth technology. rate of area + growth rate of yield not • Development and adoption lag: observed or expected due to research). time between the release of new technology and first • Discount rate. adoption by farmers. • For an ex ante analysis, three to four • Adoption path: observed or expected time between first years of price and quantity data (as a adoption and full adoption (for simplicity, a linear benchmark). adoption path is often assumed). • For an ex post analysis, detailed data on • Adoption ceiling: maximum number of farmers likely prices and quantities for a single com- to adopt the new technology or maximum number of modity aggregate of interest on an an- production units expected to be affected by the new nual basis, for all past years for which technology. benefits are assessed. • Period of time that the new technology will generate • For traded commodities— full benefits. – Prices and quantities of exports and • Depreciation factor: the rate at which the benefits of imports, and new technology will depreciate after the period of full – Exchange rates. benefits. • For more refined analyses— • Annual total research expenditures for each selected – Rates of population and income research program (including overheads and non- growth, and World Bank funds), from the beginning of research – Information on government poli- activities to the time of new technology release. cies and any other relevant market • Annual extension expenditures that can be related to interventions. new technology (difficult—a frequently used prag- matic approach is to allocate total extension expendi- tures among crops by relative area planted). (continued on next page) 15 16 Ex Ante Economic Analysis in AKIS Projects Annex 1 (continued) Cost–Benefit Analysis (simplified economic surplus approach with extreme assumptions about elasticities) Advantages: Lower data requirements (demand and supply elasticities are assumed to be polar, elimi- nating the need for estimates); lower analytical skills required. Disadvantage: Ignores all regional and international price effects due to research, as well as any distri- butional effects. Research-Related Data Required Market-Related Data Required Same as for economic surplus approach. Same as for economic surplus approach, except no need to estimate price elasticities for supply and demand. Efficiency Index Advantage: Lowest data requirements. Disadvantages: Insufficient alone to economically justify a project (usefulness limited to ranking and prioritizing research projects); ignores differences in the timing of benefits and their accrual over many years. Research-Related Data Required Market-Related Data Required • Anticipated proportional yield increase or reduction • Benchmark quantity of production for per unit cost that would follow from a successful pro- each commodity. gram with fully adopted results. • Benchmark price for each commodity. • Estimated probability of success. • Proportion of farmers likely to eventually adopt the technology. • Research costs. Annex 1 17 Annex 1 (continued) Cost-Effectiveness Analysis (“simplified� cost benefit analysis when benefits are difficult to quantify in monetary terms) Advantages: enables economic analysis of programs that either do not have a readily accessible market price or are not easily measurable in monetary terms. Disadvantages: can only be used to compare alternative programs that try to achieve a given result. Not applicable to programs with different outcomes. Research-Related Data Required Market-Related Data Required •· Annual total research expenditures for each selected None. research program (including overheads and non- World Bank funds), from the beginning of research activities to the time of new technology release. •· Annual extension expenditures that can be related to new technology (difficult—a frequently used prag- matic approach is to allocate total extension expendi- tures among crops by relative area planted). • Benefits measured in some nonmonetary unit (such as number of vaccines delivered). Break-Even Analysis (calculating the minimum benefits necessary to justify project costs) Advantage: Lower data requirements, especially for research programs where the size of the benefits is unknown. Disadvantages: Little information on the likelihood of achieving the minimum benefits; no consider- ation of distributional impacts. Research-Related Data Required Market-Related Data Required Same as for cost–benefit analysis, except that cost reduc- Same as for cost–benefit analysis. tion/output gain and/or rate of adoption need to be calculated. Annex 2: Resources for Economic Analysis of Agricultural R&E Resource Description References Alex (1998) Reviews approaches to research performance and impact assess- ment; makes recommendations for a practical yet rigorous ap- proach to impact assessment. Alston and others (1995) Presents the principles and practice of ex post and ex ante eco- nomic analysis methods and their use in research priority set- ting. Reviews, synthesizes, and assesses a wide range of approaches using a unifying conceptual framework. Belli and others (1998) Provides a comprehensive overview of methods, techniques, and tools of economic analysis of investment projects in the areas of education, health, and transportation. Gittinger (1982) Provides guidelines for the economic analysis of agricultural projects. Long used in World Bank training courses. Horton and others (1993) Synthesizes literature and experience on principles, processes, and methods of research M&E. Manuals Collion and Kissi (1995) Outlines an approach to research program planning by objective, based on a series of steps that include cost–benefit analysis as an approach to priority setting. Janssen and Kissi (1997) Outlines a practical approach to integrating productivity concerns into natural resource management. Introduces a methodology for prioritizing research projects based on economic consider- ations. Masters and others (1996) Summarizes tools for conducting persuasive impact studies that quantify the economic benefits and costs of research. Three spreadsheet-based computer exercises and an accompanying dis- kette help apply the methods described. Orlando (1991) Describes the financial, economic, and social analysis of public- sector investment projects. USAID (1987) Provides guidance on economic analysis for various projects, in- cluding agricultural research and development. (continued on next page) 18 Annex 2 19 Annex 2 (continued) Resource Description Software Dream© - Dynamic Research Facilitates the application of the economic surplus model under a EvaluAtion for Management variety of market situations. Developed by the International Ser- vice for National Agricultural Research, this menu-driven com- puter program is available on the International Food Policy Research Institute Web site (see below under Web Sites). [For more information, contact Stanley Wood at s.wood@cgnet.com] Masters and others (1996) See above under Manuals. MODEXC—Modelo de Análisis Helps calculate the NPV, IRR, and benefit–cost ratio for invest- de Excedentes Económicos ments in agricultural research for both ex ante and ex post eco- nomic analysis of technical change under different market scenarios. This spreadsheet-based model is available in Spanish on the International Center for Tropical Agriculture Web site (see below under Web Sites). Web Sites Site: International Center for http://www.ciat.cgiar.org/projects/bar_bp1.htm Tropical Agriculture Page: The Impact of Agricultural Features abstracts on the impact of agricultural research, data Research bases, trends, and a download version of MODEXC (see above under Software). Site: International Food Policy http://www.cgiar.org/ifpri/dream.htm Research Page: Dynamic Research Provides a description and free download of the Dynamic Research EvaluAtion for Management EvaluAtion for Management program (see above under Software). Site: International Service for http://www.cgiar.org/isnar/Fora/Priority/index.htm National Agricultural Research Page: Information and Discussion Describes the process, steps, and methods of priority setting for Forum on Priority Setting in research. Agricultural Research Site: The World Bank http://essd.worldbank.org/essd/rdv/rdvhome.nsf/AKISHolder? OpenView Page: Agricultural Knowledge Provides good practice examples, case studies, and useful links and Information Systems on planning and priority setting, as well as on M&E of agricul- tural R&E. Annex 3: How to Conduct Retrospective Technology Impact Studies During project preparation, retrospective tech- start date of current research activities (such nology impact studies should be conducted for as the start date for a current breeding pro- different types of technologies generated by the gram) and should extend for as long as ben- NARS to document that the current technology efits are likely to accrue from research system has been productive in some areas and products (thus, even a retrospective impact to justify continued support for the system. The study might have an ex ante element). How- studies should be based on already available ever, the analysis should emphasize recent information, with additional input from pro- impacts and might even focus on early gram scientists and other experts. Step-by-step adoption as the most relevant factor in as- guidelines for conducting these studies follow. sessing recent impacts. • Select appropriate research programs for • Define the appropriate level in the produc- economic analysis based on strategic con- tion/marketing/consumption system for siderations, such as the importance of a com- applying the analysis (that is, farm level modity in the agricultural gross domestic versus wholesale level—for traded goods, product or to certain groups of consumers the wholesale level is more appropriate, and or producers, the impact of planting an area world market prices should be used). to different crops, or the tendency of a com- • Identify costs and benefits for outcomes modity to lag behind regional averages. both with and without the research pro- Clearly state the selection criteria to help gram. avoid the temptation of selecting only • Compute summary measures such as the success stories. Aim for a mix of crop (expected) NPV and the (expected) IRR for management, livestock, natural resource the research program. management, and commodity varietal de- • To justify investments in agricultural re- velopment research programs. Try to con- search, compare the NPV of the various re- duct an economic analysis for every selected search programs to alternative investment program. (If not possible during project opportunities and to the opportunity cost preparation, try to finish during project of capital. To justify the allocation of re- implementation.) sources among research programs or to set • Depending on objectives, data availability, research priorities, take additional factors and analyst skills, select a method for the (such as objectives and constraints of the analysis (for the data requirements associ- system) into account. ated with each approach, see annex 1). • Conduct a sensitivity analysis to determine • For each selected research program, define whether the results stand up to changes in a timeframe for study (not necessarily linked the parameters. Present switching values to to the timeframe of the World Bank-funded show which parameters have the greatest project). The timeframe can begin on the impact on the results. 20 Annex 4: How to Conduct Prospective Technology Case Studies Project preparation should include a series of programs and useful in planning extension case studies to assess technology prospects. approaches.] Studies should be based on important commodi- ties or production systems (such as maize, beef, Step 2: For the same commodities and geo- bananas, or rice/wheat/dairying). They may be graphic areas as in step 1, estimate potential for summarized from available documentation on research. current or recent projects operating in the same • Based on available knowledge, identify general environment as the proposed project, major limitations on productivity. supplemented by field studies using rapid ru- • Identify potential technologies that could be ral appraisal or other informal information-gath- developed during the project period to over- ering techniques at the district or community come key constraints. level. Case studies often require teams of at least • Based on experience, estimate the probabil- three analysts—a research specialist for the com- ity that the research will succeed. modity or production system, a socioeconomist, • Based on prior examples and experience, and an extension specialist familiar with the estimate the likely adoption path for the area. Each study normally requires two to three technologies developed through the re- weeks. Step-by-step guidelines follow. search program under the project. Step 1: Estimate potential for transferring ex- Step 3: Conduct economic analyses based on isting technology. steps 1 and 2. • Describe current production systems and • Estimate the costs of a typical extension pro- problems, and estimate current production gram needed to diffuse the available tech- and income for a typical producer. nologies. Compare the costs to the benefits • Inventory available technologies that could estimated in step 1 and compute an NPV or be immediately transferred and are appro- IRR for the program. priate for target farmers. • Estimate the costs of a typical research pro- • Based on prior examples and experience, gram to develop the technologies identified estimate the likely adoption path for tech- in step 2. Because the research program is nologies introduced with support from the likely to benefit farmers beyond the imme- project extension program. diate area of study, scale up the estimated • Based on on-farm experimental data, estimate benefits of the technologies to include the the likely economic benefit from adoption. total farm population that is likely to ben- [Note: Case studies can provide substantial in- efit from the research. Compare these ben- formation that is both relevant to social, institu- efits to the cost of the research and compute tional, technical, and financial analyses of R&E an NPV or IRR for the program. 21