57037 FAST TRACKBRIEF June 4, 2010 The IEG report "Cost-Benefit Analysis in World Bank Projects" was discussed by CODE on June 4, 2010 Cost-Benefit Analysis in World Bank Projects Cost-benefit analysis used to be one of the World Bank's signature issues. It helped establish its reputation as the knowledge Bank and served to demonstrate its commitment to measuring results and ensuring accountability to taxpayers. It was the Bank's answer to the results agenda long before that term became popular. This report takes stock of what has happened to cost-benefit analysis at the Bank, based on analysis of four decades of project data, project appraisal and completion reports from recent fiscal years, and interviews with current Bank staff. The percentage of projects that are justified by cost-benefit analysis has been declining for several decades, due to both a decline in standards and difficulty in applying cost-benefit analysis. Where cost-benefit analysis is applied to justify projects, there are examples of excellent analysis but also examples of a lack of attention to fundamental analytical issues such as the public sector rationale and comparison of the chosen project against alternatives. Cost-benefit analysis of completed projects is hampered by the failure to collect relevant data, particularly for low-performing projects. The Bank's use of cost-benefit analysis for decisions is limited because the analysis is usually prepared after making the decision to proceed with the project. This study draws two broad conclusions. First, the Bank needs to revisit the policy for cost-benefit analysis in a way that recognizes legitimate difficulties in quantifying benefits while preserving a high degree of rigor in justifying projects. Second, it needs to ensure that when cost-benefit analysis is done it is done with quality, rigor, and objectivity, as poor data and analysis misinform, and do not improve results. Reforms are required to project appraisal procedures to ensure objectivity, improve both the analysis and the use of evidence at appraisal, and ensure effective use of cost-benefit analysis in decision-making. Current Bank policy states that cost-benefit analysis quantitative cost and benefit flows, are rarely provided in should be done for all projects at appraisal--with the a straightforward manner, such as a simple table. Such a single exception of projects for which benefits cannot table could be provided along with a discussion of the be measured in monetary terms, in which case a cost- main assumptions or empirical evidence that lies behind effectiveness analysis should be performed. Ultimately, the numbers. As was pointed out in a World Bank the requirement to conduct cost-benefit analysis stems report 20 years ago, ex-ante project analysis at the Bank from the mandate in the Articles of Agreement that the is usually based on the working assumption that Bank should strive to increase the standard of living in everything will go as planned. This imparts an upward member countries. When countries borrow--and have bias to the cost-benefit estimates because there are to repay--funds for projects in which costs exceed frequently disruptions along the way. An alternative-- benefits, the standard of living of the country declines. more in line with Bank policy to present the expected economic return--would be to employ the working Using the presence of an ex-ante economic rate of assumption that new projects would achieve the average return estimate as an indicator of whether cost-benefit results measured in previous similar projects, unless analysis was performed, the percentage of projects with changes are made to the project design that warrant such analysis dropped from 70 percent to 25 percent revision. between 1970 and 2008. Further examination of project documents reveals this to be a reliable indicator of the The weak points in economic analysis of projects are presence of cost-benefit analysis. A little over half of fundamental issues such as the public sector rationale, this decline was due to an increase in projects in sectors comparison against alternatives, and measurement of at the Bank that tend not to apply a cost-benefit analysis benefits against a without-project counterfactual. to their projects. About half of the sectors, which have Project justification rarely includes a discussion of tended to decline as a share of activity in recent years, whether the project is producing a public good, and if often apply cost-benefit screening to their projects, while alternatives are considered, they tend to be minor the other half, the growing half, rarely do. In addition to alternatives, such as alternative funding mechanisms, this shift away from sectors that apply cost-benefit rather than truly alternative projects. Counterfactual analysis, there has been a general decline in all sectors in analysis tends to be good for projects in sectors in which the application of such analysis. Most of the this analysis is hardwired into standard spreadsheets, improvement in project performance ratings that has such as transport. Impact evaluations, which are occurred at the Bank in the past twenty years is in the designed to address the counterfactual issue and thus are five sectors that tend to apply cost-benefit analysis. a natural complement to cost-benefit analysis, have rarely been used in the past, though their use is now World Bank policy notwithstanding, many appraisal growing in some sectors. There is low usage in cost- documents for new projects in recent years do not benefit analysis of shadow prices and other technical present cost-benefit analysis. How is this omission adjustments to capture some of the social benefits and explained? How are the projects justified? Of the 93 costs. investment projects that closed in 2008 without reporting cost-benefit information (either at appraisal or Projects that have easy-to-identify beneficiaries, such as at closing), 60 provided no explanation or asserted that agriculture and community-based development projects efficiency considerations were not applicable. Eighteen (albeit ex-post), could provide better poverty analysis. cited inadequate data. Nineteen projects provided some This often requires a special baseline household survey. relevant information, but the information tended to be Lack of baseline data is a key weakness undermining ex- positive anecdotes, with no attempt to address potential post cost-benefit analysis in many projects. Overall, the selection bias. Twenty-four project documents invoked economic analysis in appraisal documents in 2007-08 is cost-effectiveness as the standard they were to be judged found to be acceptable or good in 54 percent of the by, but of these, none actually applied cost-effectiveness cases. This compares with 70 percent found by a similar analysis, which entails a comparison between specific rating exercise in the 1990s. alternatives on the basis of costs. One project claimed This report also examines whether there is evidence of such an analysis had been done but did not show the bias in the economic rates of return that are reported. It results in the document. finds that the "everything goes according to plan" Of projects that do provide cost-benefit analysis, there scenario is still the working assumption underlying cost- are several examples of excellent analysis, but often a benefit analysis at appraisal. The report also finds that lack of transparency. The most important data, the the likelihood that the economic rate of return is 2 recalculated at the close of projects is lower for projects evidence is provided, and few institutional checks to with low outcome ratings. Moreover, interviews with counteract the influence of advocacy for projects that staff indicate that project cost-benefit analysis is undermines rigor in project appraisal, including cost- conducted after the decision to go ahead with the benefit analysis. projects, which puts the analysis under considerable pressure to reach conclusions consistent with the The Bank needs reforms to ensure objectivity and decisions already taken. address conflicts of interest in ex-ante project analysis. It needs to use cost-benefit analysis evidence to improve The lack of attention to cost-benefit information is decisions in a context where decisions are increasingly surprising given the positive story that emerges on driven by borrowing countries. trends in the reported rates of return in the declining subset of projects that apply this approach: reported The policy for cost-benefit analysis needs to be defined economic rates of return have doubled in 20 years, from in a way that recognizes legitimate difficulties in a median of 12 percent in the late 1980s to 24 percent in quantifying benefits in some types of projects while 2008. If reflective of the larger group of projects, this preserving a high degree of rigor in justifying projects. could signal a large rise in the effectiveness of these This report closes with suggestions on how the Bank can development projects. address these institutional issues. Some discount this rise, believing that it indicates nothing more than an increase since 1987 in the upward bias in the measurement of economic returns. The About Fast Track Briefs available evidence does not confirm this belief, but it Fast Track Briefs help inform the World Bank Group (WBG) cannot be dismissed because the evidence is thin. managers and staff about new evaluation findings and recommendations. The views expressed here are those of IEG Another possible explanation for this large rise in returns and should not be attributed to the WBG or its affiliated is growth-oriented reforms. Reforms -- comprising both organizations. Management's Response to IEG is included in a retreat of anti-market approaches to projects and the published IEG report. The findings here do not support any improvements in investments and institutional support general inferences beyond the scope of the evaluation, including in the economic environment -- could account for some any inferences about the WBG's past, current or prospective of the rise in economic returns for this subset of overall performance. projects. A review of project documents from the pre- reform 1970s and 1980s suggests that project execution was frequently frustrated by high transactions costs or The Fast Track Brief, which summarizes major IEG unavailability of imported spare parts and hampered by evaluations, will be distributed to selected World Bank Group unresponsive state entities. Examination of 47 countries staff. If you would like to be added to the subscription list, please where the available data permit the impact of such email us at ieg@worldbank.org, with "FTB subscription" in the subject line and your mail-stop number. If you would like to factors to be tested reveals that 43 had higher economic stop receiving FTBs, please email us at ieg@worldbank.org, returns in projects after reforms. with "FTB unsubscribe" in the subject line. External factors could also be responsible. Economic conditions facing countries have improved in Bank client countries, and project returns correlate with growth Contact IEG: Director-General, Evaluation: Vinod Thomas rates. But much of the growth improvement occurred Director: Cheryl Gray (IEG-WB) rather late in the 1987-2008 period, and thus is not Manager: Mark Sundberg (IEGCG) sufficient to account for the sustained rise in returns Task Manager: Andrew Warner (IEGCG) during the entire period. Copies of the report are available at: A review of economic analysis at the World Bank 20 http://www.worldbank.org/ieg/ years ago found many of the same shortcomings IEG Help Desk: (202) 458-4497 documented here in economic analysis in the Bank. Yet E-mail: ieg@worldbank.org that report's recommendations did not go far enough in confronting underlying causes: a decision-making process that often makes decisions before adequate 3