63685 Overlooked Links in the Results Chain 14 Overlooked Links in the Results Chain Vinod Thomas and Xubei Luo Evaluation Brief 14 2011 The World Bank http://www.worldbank.org/ieg Washington, D.C. © 2011 Independent Evaluation Group Communications, Learning, and Strategy The World Bank 1818 H Street, NW Washington, DC 20433 Email: ieg@worldbank.org Telephone: 202-458-4487 Fax: 202-522-3125 http://www.worldbank.org/ieg All rights reserved This Evaluation Brief is a product of the staff of the Independent Evaluation Group (IEG) of the World Bank. The findings, interpretations, and conclusions expressed here do not necessarily reflect the views of the Executive Directors of the World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denomina- tions, and other information shown on any map in this work do not imply any judgment on the part of the World Bank or IEG concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. IEG encourages the dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to ieg@ worldbank.org. ISBN-13: 978-1-60244-176-7 ISBN-10: 1-60244-176-6 Contents v Abbreviations vii Acknowledgments ix Executive Summary 1 Introduction 3 I. Aiming for the Right Results 3 Short-Term Objectives Can Impair Long-Term Results 6 Country-Level Results Are Different from Project-Level Results 8 Cross-Sectoral Linkages Are Crucial for Outcomes 11 II. Measuring Results the Right Way 11 Composite Indicators May Mislead 13 Achieving Intermediate Outcomes Does Not Ensure Desired Results 15 Averages Can Mask Outcomes for Crucial Target Groups 19 III. Using Evaluative Findings for Results 19 The Future May Not Resemble the Past 21 Identifying Missed Opportunities Can Help in Crafting Better Strategies 23 Timing and Process Can Make the Biggest Difference 27 Bibliography Abbreviations AAA Analytical and advisory activities CAS Country Assistance Strategy CPIA Country Policy and Institutional Assessment DBI Doing Business Indicator HDI Human Development Indicator HNP Health, nutrition, and population IDA International Development Association IEG Independent Evaluation Group IFC International Finance Corporation MIGA Multilateral Investment Guarantee Agency PSR Public sector reform v Acknowledgments The authors thank IEG management and staff Soniya Carvalho, Kenneth Chomitz, Shahrokh who gave generous support for this paper. They Fardoust, Cheryl Gray, Daniela Gressani, Monika thank peer reviewers Alan Gelb (Center for Global Huppi, Jaime Jaramillo-Vallejo, Ali Khadr, Ray Rist, Development), Alejandro Soriano (Inter-American Mark Sundberg, Marvin Taylor-Dormond, Stoyan Development Bank), and Patrick Grasso (Indepen- Tenev, Jiro Tominaga, Christine Wallich, and dent Evaluation Group) for helpful discus- Shahid Yusuf. The authors would like to thank sions and input. They gratefully acknowledge Silke Heuser for excellent research assistance. valuable comments from the following individu- William B. Hurlbut and Heather Dittbrenner als: Martha Ainsworth, Amitava Banerjee, Geeta provided editorial support, and Helen Chin Batra, Jaime Biderman, Hans-Martin Boehmer, edited the paper for publication. vii Executive Summary Evaluation often confirms existing knowledge, improved governance, and reduced poverty— but sometimes it also brings out factors that were not met. otherwise get short shrift in discussions and actions. Some of these factors can be crucial Ignored or missed cross-sectoral linkages links in the chain connecting actions and results may lead to shortfalls in outcomes and in development. This paper focuses on such impacts: Outcomes in a sector are often linked overlooked, but vital, dimensions, drawing to results in other sectors, and it pays to take from the Independent Evaluation Group’s them into account. For example, reconstruction (IEG) work and other development evalua- after natural disasters is not merely rebuilding tions. The lessons follow a logical path from physical infrastructure; it is also a way to preserve what results should be the focus of attention, social relationships and provide survivors with to how those results should be measured, to employment and cash transfers. the use of information about those results in improving development effectiveness. Measuring Results the Right Way Focusing on the Right Results Focusing on the right results is of limited value if we are not measuring those results properly. Focusing exclusively on short-term Poor monitoring and evaluation can risk the objectives can impair long-term results: achievement of desired outcomes and weaken Responses to natural disasters often target development effectiveness by misallocating reconstruction rather than mitigation or scarce resources from higher-value activities. prevention. But disasters should be treated Several problems may result from suboptimal as risks to development, not interruptions in measurement. development. IEG found that among countries that had received World Bank support in the Composite indicators may mislead: past to deal with disasters, fewer than half Composite measures are appealing for their of their Country Assistance Strategies even apparent simplicity, but they can send wrong discussed their disaster response. signals when what is being measured does not match the claim. For example, the Doing Country-level results are different from Business Indicators claim to measure the project-level results: Country objectives tend overall state of business regulation in a country, to be broader than those at the project level; if not the state of country reform, but it places country-level results are also affected by other a value on only the lower cost to business from World Bank interventions, such as analytical and lighter regulation and lower taxes, and not on advisory activities. In addition, they are likely to any benefit to society from a degree of regula- be influenced by factors well beyond the scope tion. Thus the prescriptions drawn for regula- of any project, or even the portfolio as a whole. tory reform could be misleading. For example, the Chad-Cameroon oil pipeline project was technically well implemented and Achieving intermediate outcomes does not a financial success; but the main country-level ensure desired results: Individual operations objectives—capacity building in the sector, often are designed to achieve intermediate ix EVALUATION BRIEF 14 outcomes—like increasing access to safe water The future may not resemble the past: or raising school enrollments. But these achieve- Context matters. Changing environments and ments may not be sufficient to ensure better emerging challenges sometimes limit the direct health or learning outcomes. For example, application of findings from past work to future Tanzania won a United Nations award for its efforts. For example, the Bank’s work in water progress in attaining universal education ahead has improved steadily, but the past lessons of the 2015 goals, but a survey led by nongovern- on water availability do not provide much mental organizations found that many children guidance for new challenges such as coastal who had completed seven years of schooling zone management, pollution reduction, and could not read at a second-grade level or solve groundwater conservation. simple multiplication problems. Identifying missed opportunities can help Averages can mask outcomes for crucial craft better strategies: The role of evaluation target groups: A project or program may be in identifying missed opportunities often is successful, on average, but fail to address the overlooked. For example, evaluative evidence right constraint or reach targeted beneficiary shows that reducing energy subsidies, which groups. For example, the Bank’s community- often go disproportionately to the better off, based and community-driven development leads to efficiency gains, creating a win-win projects aimed to reach the poor, but an evalua- situation in which conservation can be linked tion found that benefits tended to be greater to better targeting of subsidies to the poor. for the better off than for the poorest, and in some cases the poor were worse off. Timing of monitoring and evaluation are crucial: Evaluative information is particularly Using Evaluation to Improve effective if it is delivered when it can affect key Development Effectiveness decisions. Thus, early evaluations of Mexico’s conditional cash transfer program showed Even when the right results have been measured positive results on schooling, health, labor adequately, the information has to be used supply, and consumption just as a new adminis- properly if it is to lead to sound results. But this tration was coming into office, convincing it to may not happen for a number of reasons. retain the program. x Introduction C ountries and agencies are increasingly finding that evaluation, espe- cially when done independently, can make a difference to the effec- tiveness of programs. Mexico has passed legislation to require impact evaluations of social development programs. South Africa and India have taken steps to establish an independent evaluation function, reporting to the coun- try’s leadership. The United Kingdom has recently appointed the first Chief Commissioner for its Independent Commission for Aid Impact. Germany is reviewing its evaluation function for global and domestic programs. Where monitoring and evaluation can add real point to problems that can break the connec- value is in assessing whether the interventions tions from actions to outcomes, including the lead to desired outcomes, in what context, and critical assumptions and context required for the through what channels. Good monitoring and theory of change to work. Often these important evaluation has been found to be associated with elements are neglected or simply taken for good project outcomes (Thomas and Tominaga granted. By pointing out these crucial but 2010). Evaluation can provide vital information neglected areas and providing timely informa- about the effects of government policies and tion to change development thinking and guide programs. It can be an objective basis for assess- policy decisions, evaluations have the potential to ing what works and what does not, as well as help push policy interventions out of what is generally foster accountability in program delivery and accepted but perhaps a harmful state of inertia, facilitate learning from experience. onto a more effective course. But evaluations, particularly experimental impact This paper focuses on evaluations with findings evaluations, can also be costly. The key question that challenge important assumptions of the then is whether, and in what circumstances, the development field. Its objective is to pinpoint the benefits of revealing the effectiveness of programs areas to watch for in operations or policies that outweigh the costs. In this connection, it is worth are decisive for results. By drawing the attention asking when evaluation can be uniquely helpful of development practitioners, policy makers, and and how the resources spent on carrying out the evaluators to these seemingly obvious but often evaluation and applying the lessons learned can neglected areas, it aims to improve development have the most impact. effectiveness by better connecting interventions to desired results. The lessons follow a logical Many evaluation findings confirm existing path from which results should be the focus of knowledge. The value they add mainly lies in attention, to how those results are measured, summarizing lessons learned and perhaps in to how to use evaluation for better results. The suggesting improvements in future interven- examples are mainly drawn from the work of tions. Some subsets of evaluations, however, the Independent Evaluation Group (IEG) of the generate unexpected or surprising results that World Bank Group. 1 I. Aiming for the Right Results F ocusing on the right results seems straightforward in principle. However, it is often challenging or neglected in practice. In fact, interventions can translate into results at different levels, which are interdependent but not perfect substitutes. Properly focusing results inter-temporally, across sec- tors, and between the project portfolio and country program requires a right balance of multiple perspectives. Assuming a simple translation or aggregation of results from one level to another can risk not setting out or pursuing the true desired results. We might think of a three-dimensional space, Achieving short-term and long-term objectives within which development practitioners need should not be a problem of “either-or.� Evalua- to identify results to be met. In this multi- tive findings suggest that addressing the root dimensional space, one axis focuses on time, causes with an eye on the future can solve the the second on the sector, and the third on the dilemma of the seeming trade-offs between project and country program. The interactions quickly dealing with symptoms and building and trade-offs within and across each of these toward long-term results. three dimensions lead to the complex challenges of getting the right focus for results with any Responses to Natural Disasters Should Not given intervention. Be One-Offs Short-Term Objectives Can Impair Evaluation of the effects of responses to natural Long-Term Results disasters clearly shows the trade-offs between passively focusing on responding to short-term Short-term objectives are not always aligned with needs and actively pursuing long-term objectives. long-term results. Focusing only on short-term, Even in the midst of an emergency, actions immediate needs can sometimes compromise should be taken with an eye on the future. results in the long run. Disasters can wipe out development gains and A quick response to urgent needs is often eclipse years of development investment. For required in the midst of an emergency. But steps example, the Kashmir earthquake in October that are taken immediately, although essential in 2005 caused an estimated $5 billion in damage in a crisis, also feed into longer-term solutions— Pakistan, roughly equivalent to the total official just as the responses provided and the protocols development assistance for the preceding 3 years followed in the emergency room are essential to a and equivalent to the amount the World Bank patient’s long-term care. In other words, develop- had lent to the country over the preceding 10 ment is path dependent. If the initial steps do not years (IEG 2006b). Poor countries typically suffer contribute to better outcomes in the long run, the greatest loss of life from disasters—nearly they could do more harm than good. 1 million people have died from Africa’s droughts 3 EVALUATION BRIEF 14 alone. With the acceleration of urbanization usually are not among the reconstruction and an increase in the frequency and severity objectives. Project objectives mainly provide for of disasters, the need for disaster prevention is short-term fixes and rarely address root causes. increasing. Although the Bank has demonstrated consider- able agility in its approach to natural disasters, Distributing supplies by helicopters and building and natural disaster projects financed by the temporary homes are often headline news in Bank have had higher ratings for outcomes and postdisaster situations. Useful as they are in sustainability than the Bank’s portfolio, disaster addressing urgent needs, however, their contri- assistance has tended to be reactive. bution to improving long-term results is limited. Being reactive without improving preparedness Among those countries that received Bank in noncrisis time can only leave people and support for natural disasters during 1984–2005, property at risk and limit the scope to maneuver 44 percent of the Country Assistance Strategies when disasters hit. While meeting immediate (CASs) did not mention the disasters. Even in needs, countries prone to disasters could find the 40 countries that had four or more disaster themselves in a near-permanent state of recovery projects, one-third of the strategies did not if they lose sight of the long-term priorities related mention disasters; and for those that had more to reducing their vulnerability to disasters. than eight such projects, almost a third did not mention disasters (table 1). Disasters should be treated as a risk to develop- ment rather than an interruption in develop- The lessons identified by IEG feature in a new ment. The tendency to treat disasters as strategy for natural disasters at the Bank. A one-time, random events is now being rethought new Operational Policy implemented in 2007 (World Bank 2010b); a more proactive and strate- recognized the importance of integrating risk gic approach will have longer-term benefits. reduction and crisis prevention into the develop- Measures are to be taken to strengthen preven- ment strategies of countries at high risk for tion and mitigation in noncrisis times to minimize disasters or rising conflict. Prevention and mitiga- the negative impact of natural disasters and tion measures are now expected to be included as alleviate the intertemporal trade-offs of reacting part of CASs, Interim Strategy Notes, and Poverty to immediate needs after a disaster. Reduction Strategy Papers for such countries. The reality, however, is that immediately after a Whenever massive reconstruction is needed disaster, efforts often target reconstruction, but following a disaster, the pressure for haste is high. mitigating or even preventing future disasters But haste can result in incomplete reconstruc- Table 1: Many CASs That Should Discuss Natural Disasters Do Not Number of Percentage of Number of Number of their CASs with countries with no disaster projects countries with no discussion discussion of in a country this count of disasters disasters in CASs More than 8 16 5 31 4–7 24 8 33 2–3 33 15 45 1 24 15 62 Total 97 43 44 Source: IEG 2006b. 4 OVERLOOKED LINKS IN THE RESULTS CHAIN tion, which accounts for much of the longer- cases—Jakarta, St. Lucia, Istanbul, and Rohini Basin term gross domestic product cost of disaster (India)—the estimated benefit-cost ratios range (Linnerooth-Bayer and Mechler 2004). Lack of from 1.5 to 5.7 (World Bank 2010b). maintenance, also a consequence of a short- term view, has often been a major constraint on The comparison between the scenarios after the sustainability of structures rebuilt by natural earthquakes in Haiti and Chile shows how greater disaster projects. Lack of community consulta- prosperity and preparedness helped prevent tion can also create problems, as occurred in the massive casualties and economic paralysis in Bank’s Bangladesh Coastal Embankment Project, Chile, whereas this did not happen in Haiti. The during which public opposition to the alignment experiences in Colombia and Turkey also show of embankments led to significant implementa- that earthquake-resistant building codes, enforce- tion delays (IEG 2006b). ment of construction standards, and oversight of materials procurement practices pay off. Prevention and Mitigation Help Ways to Enhance Preparedness Prevention and mitigation can lower the damage caused by a future disaster. Building anticipatory Understanding the reasons why a prevention readiness over time can facilitate an effective system is not in place helps address the root issues immediate response that will make a vital differ- for results. If the key constraint is that people do ence to recovery. Reconstructing housing with not know about the importance of prevention disaster-resistant techniques, and according to or the ways of doing it, sharing knowledge and the needs of occupants, can reduce vulnerability disseminating information are necessary. If the to future disasters and minimize potential costs, key constraint is people choosing not to follow especially in poorer countries where the quality of the building codes because of an excessively high construction, land registration, and other regula- preference for present value, pursuing a proper tory mechanisms are weak. Enforcing building discount rate and building up rational expectations standards and improving information manage- are necessary. If the key constraint is that people ment, such as geographic data on hazards and would like to take prevention measures but are vulnerability, could lower the potential costs. financially forced to use substandard building These measures can be difficult to implement, materials, developing the financial system and but with appropriate design their benefits can providing access to resources to help overcome outweigh the cost. the liquidity constraint are necessary. The effects of a disaster are conditioned by the Better spending, not necessarily more spending, extent of a community’s vulnerability to a given is crucial for achieving the desired results. hazard (or conversely, its ability or capacity to Measures could include providing greater cope with it). Improving preparedness can reduce access to information, better sharing of data vulnerability. Too often urgent care could not be across borders, and reallocating existing public provided because critical-care facilities were no spending toward better maintenance of roads longer functioning, or there was no way to access and bridges. Many countries are not taking services. But there has also been clear neglect— advantage of the technological improvements of 59 completed World Bank emergency projects in weather and related forecasting. Even modest in disaster-prone settings, only 10 have had increases in spending—if supplemented by follow-on projects. Maintenance, follow-up, and international data sharing—can have enormous preventive investments need greater attention. benefits, especially to warn people of impend- ing hazards. Several countries, some very poor, Prevention pays, but the costs are justified. A have made large and quick gains from such benefit-cost ratio of over 1.0 is evidence that preven- spending. Preparing a strategy or action plan for tion is cost effective. At the household level, in four natural disaster assistance that spells out the real 5 EVALUATION BRIEF 14 long-term objectives of disaster prevention and by April 2009, 24 (27 percent) have satisfactory mitigation, and includes an assessment of each aggregate project outcomes, whereas the country country’s level of disaster risk, can improve the assistance programs were rated unsuccessful effectiveness of the Bank’s interventions. (table 2)—noting that the project ratings refer to those completed during the period, whereas Country-Level Results Are Different the country ratings may reflect a broader set of from Project-Level Results projects as well as AAA.1 The percentages of country programs and projects with satisfactory Project outcome measures are often quite differ- outcomes are higher after the implementation ent from aggregative measures at the country of the results-based CAS. For CASs started after program level. There are several reasons why the 2005, some 70 percent have both satisfactory results framework at the project level does not aggregate project outcomes and country program directly translate into that at the country level. outcomes, which is 10 percentage points higher Project outcomes may be narrow or specific (such than CASs that started before 2005. as access to schooling), whereas results beyond the project concern broader objectives (such Achieving satisfactory project outcomes is in a as competitiveness). Country-level results are number of ways not the same as achieving satisfac- more often conditioned by interventions besides tory country outcomes. There is substantial room projects, such as analytical and advisory activities for improvement in development effectiveness (AAA) and donor coordination. Project outcomes through more coherent, well-tailored country will only translate into positive country outcomes programs as well as through project improve- if they are relevant and bring about change. ments. Among the issues are the relevance of country strategy and how the different kinds Project and Country Program Ratings of Bank interventions come together; policy dialogue; complementarities among sectors and Project ratings do not add up as country program with AAA, policy, lending, and global initiatives; ratings, because each of them is evaluated against and exogenous factors such as global shocks its respective objectives. Even when project (IEG 2010f). ratings are high, outcomes at the country level may not be satisfactory, or vice versa. In review- Objectives at Project and Country Levels ing all project and country evaluations since 1993, IEG found that project outcome ratings Project ratings and country program ratings are often higher than country program outcome are set to measure different objectives. There ratings (IEG 2009a). is no fixed relationship between the results frameworks at the project level and at the country This difference concurred with the findings from level. Outcome objectives at the country level, comparing the country program ratings with the which take a broader perspective, are not a direct project ratings in the same countries. Out of a total translation of those at the project level. of 88 CAS Completion Report Reviews completed A country evaluation must assess overall Bank Table 2: Country and Project Outcome Ratings strategy, including the size, composition, and type of lending, as well as the other types of Bank interven- IEG CASCR Reviews IEG ICR Reviews Satisfactory (%) Unsatisfactory (%) tions, such as AAA. It yields a more complete picture Satisfactory 64 27 Unsatisfactory 3 6 1. ICR Reviews include only investment lending and development policy lending. The outcomes of Source: World Bank database. some recently completed projects are not included Note: CASCR = Country Assistance Strategy Completion Report; ICR = Implementation Completion because of the time lags between project comple- Report. tion and review. 6 OVERLOOKED LINKS IN THE RESULTS CHAIN of the outcome of the Bank’s assistance programs at accounts for a third of the World Bank’s outlays the country level in as much as it has comprehensive in country services, exceeding outlays in lending coverage of the Bank’s activities in a country during or supervision. Economic sector work, techni- a given period. The country program outcome may cal assistance, and country dialogue profoundly be unsatisfactory if there are critical omissions in contribute to country knowledge and performance the Bank’s assistance strategy, even if the project directly and indirectly through different channels. outcomes are rated satisfactory. Project interventions are more successful when they are based on in-depth analytic work (IEG 2008b). Country evaluations take into account the relevance of the objectives of the program, the One example of impactful technical assistance achievement of those objectives against standards is the assistance the Bank provided to Sri Lanka set in the Bank’s CAS, the quality of interac- to institute standardized small power purchase tion with the government, and the quality and agreements that facilitated access to the power relevance of analytic work. Project evaluations grid. Another example is the well-timed, assess whether the (narrowly) set objectives of good-quality knowledge products that assisted the project are met. Whether the objectives in Egypt in policy formulation, poverty reduction, the CAS are achieved often depends on the whole and development of human resources in the set of interventions that the Bank brings to bear early 2000s, despite the Bank’s small contribu- in support of the country’s chosen objectives and tion in financial terms at that time. program. In addition, country outcomes often also depend on the country’s own initiatives, such as Yet another example is how analytic work, policy changes, other development partners, or capacity building, and demonstration have exogenous developments, such as natural disasters contributed to Chinese and Mexican adoption of or financial crises—and not solely on the success favorable renewable-energy payment schemes, or failure of the projects supported by the Bank. which have stimulated more than 20 gigawatts of installed wind capacity in China and hundreds of The Chad-Cameroon oil pipeline project is an megawatts under construction in Mexico. example of the disconnect between project and country program outcomes, because they are External Factors measured against different objectives. The project itself was technically well implemented and a External factors often play a large role in achiev- financial success. But the main objectives at the ing country impact. An IEG evaluation of public country level—capacity building to manage the sector reform (PSR) showed a large differential petroleum sector and helping Chad reduce poverty between countries that borrowed for PSR from and improve governance—were not met. the Bank and those that did not (table 3). Overall, there was a 73 percent improvement rate for To the contrary, the oil revenue windfall was associ- borrowers and a 48 percent rate for nonborrow- ated with a resurgence of civil conflict and a worsen- ers. Across regions, the incidence of lending and ing of governance. The principal reason was the lack the correlation of PSR lending with changes in of government ownership of the objectives, with governance scores varied. repeated violations of the basic agreements (IEG 2009b). No alternative program design or closer The differential is high across all Regions except supervision would have achieved the program’s Europe and Central Asia, where the improve- objectives without government commitment. ment for countries getting PSR lending is the highest—90 percent—but the rate of improve- Knowledge Services ment for nonborrowers is almost as high. The explanation seems to be in requirements for Analytical and advisory activities, like lending, can accession to the European Union. Almost all the also drive country program outcomes. Such activity countries in Europe and Central Asia that did not 7 EVALUATION BRIEF 14 borrow for PSR in 1999–2006 were among the school enrollment rates and better health first from the East to join the European Union outcomes. Rural electrification improves the and had completed reforms before 1999. quality of life in many dimensions: lighting alone brings benefits, such as increased study time Cross-Sectoral Linkages Are Crucial and improved study environments for school for Outcomes children, extended hours for small businesses, and greater security. Television—the second To achieve sectoral outcomes, putting all efforts most common use of electricity—brings both into a given sector may not be optimal or adequate. entertainment and information. Often outcomes in a sector cannot be attributed solely to projects in that sector but instead result A review of the performance of the fisheries from outcomes of other sectors through multisec- portfolio of the African Development Bank toral linkages. This means that programs in one (2008) indicates that a sectorwide approach to sector may change the outcomes in other sectors fisheries development must not ignore the links through spillover effects, and the relationship may between fisheries and other areas, such as oil be reciprocal. Cross-sector linkages that are ignored and gas, forestry, agriculture, industry, tourism, or missed can lead to shortfalls in performance. and the environment. For example, both inland and marine fisheries have important implications Development solutions need to recognize complex for the environment, including biodiversity and interactions among multiple factors. The general the health of coral reefs and other fragile marine impact of economic growth and rising incomes on environments. The evaluation concludes that a range of social indicators illustrates the broad an environmental and social management plan perspective of cross-sectoral linkages. Interven- should be developed during project design, and tions in infrastructure and human development environmental costs should be included in the show that an approach that accommodates analysis. such relationships within and among proliferat- ing public and private development partners is The impact of an infrastructure project also needed to optimize sectoral outcomes. depends on the performance of other sectors. For example, postdisaster reconstruction is Infrastructure affected by cross-sectoral linkages. In the case of the recent Haiti earthquake, the breakdown of An infrastructure program may not only enhance social order and a fragile security situation, the road accessibility but also contribute to increased near-complete loss of governance structures, Table 3: Percentage of Countries with Improved Governance CPIA Scores, 1999–2006 With Bank Without Bank PSR lending PSR lending Region Percent Number Percent Number Sub-Saharan Africa 70 30 47 15 East Asia and Pacific 70 10 56 9 Europe and Central Asia 90 20 86 7 Latin America and the Caribbean 75 20 25 8 Middle East and North Africa 57 7 0 2 South Asia 50 6 0 1 Total 73 93 48 42 Source: IEG 2008c. Note: CPIA = Country Policy and Institutional Assessment; PSR = public sector reform. 8 OVERLOOKED LINKS IN THE RESULTS CHAIN and the failure to impose even minimum quality in place can the improvement of road access to standards on the construction industry added to hospitals ensure better health outcomes. the task of recovery (IEG 2010l). A mother’s education affects her children’s The rebuilding of homes and communities health in myriad ways.2 A systematic analysis in requires the safe transportation and storage of 175 countries between 1970 and 2009 found building materials and, often, the formation of that a significant share of the reduction in child community groups that work together to rebuild mortality over the past 40 years can be attributed houses and infrastructure. Reconstruction is not to the better education of women (Murray and merely rebuilding physical infrastructure, it is others 2010). For every one-year increase in the also a way to preserve social relationships and average education of reproductive-age women, provide survivors with employment and cash countries experienced a 9.5 percent decrease in transfers. child mortality. The recent Pakistan floods offer similar lessons The study found several conditions that link (IEG 2010e). Restoring rural livelihoods requires education to health, including: better-educated not only restarting cropping and livestock activi- women are more likely to understand and use ties but also addressing land rights issues (as the disease-prevention measures, such as vaccines topography in some areas has changed and, in and mosquito nets; they are more likely to some cases, land-rights documents have been take a sick child to a clinic early and to follow lost by both households and the administration). treatment instructions; and they are more likely Flood response programs should not focus only to understand germ theory and to make clean on rebuilding infrastructure but also on better water and sanitation household priorities. adaptation and preparedness for the future in complementary investments, such as water and Poor sanitation and hygiene behaviors have been flood management, cropping pattern adjustment, found to wipe out any potential benefits from rural finance, enhancing capacities of water users health-related development projects. According to groups, and early warning systems. the World Health Organization (2004), poor sanita- tion, lack of access to clean water, and inadequate Greater use of multisector approaches can be hygiene account for approximately 90 percent effective in achieving the gender equality and of childhood diarrhea. A meta-evaluation of 10 other human development Millennium Develop- rigorous studies found that the impact of improved ment Goals—as in Peru, where increased hygiene is associated with a median reduction of availability of transportation was associated with 33 percent in diarrheal morbidity, though the increasing numbers of pregnant women seeking impact ranged from 11 percent to 89 percent medical help (IEG 2010c). (Huttly and others 1997). Human Development An IEG evaluation of the health, nutrition, and population (HNP) sector found that cross-sectoral Contribution from one sector can be a necessary synergies have yet to be tapped. From 1997 to condition for achieving outcomes in another 2006, the Bank invested about $5 billion in HNP sector. Ignoring the cross-sectoral linkages can components in 350 projects managed by other risk damaging sectoral outcomes. Education, for sectors, such as social protection, education, example, may be important to improving public public sector management, water supply, and awareness in a way that ensures the outcomes of transport. During 1997–2001, half of approved other sectors. Only when people learn to wash water supply and sanitation projects cited their hands and to use clean water to prepare food can improved access to water result in better 2. http://www.thelancet.com/journals/lancet/article/ health outcomes. Only when safety measures are PIIS0140-6736(10)61257-3/fulltext#article_upsell. 9 EVALUATION BRIEF 14 potential health benefits, and close to 90 percent public and private sectors can hinder agriculture financed infrastructure that would contribute to and agribusiness productivity (ECG 2011). Given improved health (Overby 2008). the private-good nature of agricultural activities and public-good nature of agricultural services, But water supply and sanitation and transport particularly agricultural research and services, projects with health components rarely involved the extent to which interventions have used collaboration with a health ministry or the Bank’s linkages among government and private produc- HNP sector. Projects approved later (fiscal years ers makes a difference in performance. The 2002–06) were less likely to have been justified by impact of interventions by the governments and health benefits, to have explicit health objectives, international institutions will only be as good as or to plan to collect health indicators. They were the links made with private producers. also less likely to target behavior change, which is critical in transforming infrastructure improvements On safeguard policy issues, across the World into sustainable health improvements. Among the Bank Group, drawing lessons from the public 26 completed projects, only 4 had documented and private arms can help harmonize thematic changes in the incidence of disease. Fewer than coverage and guidance. Adopting strong features half of closed projects included behavior change from each approach can improve implemen- objectives or activities. tation, results, and benefits. Hence, IEG has encouraged the International Finance Corpora- The Public and Private Sectors tion (IFC), the Multilateral Investment Guarantee Agency (MIGA), and the World Bank jointly to Multisector approaches are not limited to collab- adopt and use a shared set of objective criteria to orations between public sectors, they are also assess social and environmental risks to ensure relevant to the linkages between the public and adequacy and consistency in project categoriza- private sectors. More than half of Millennium tion across the World Bank Group (IEG 2010g). Development Goal–related mother and child health services in Sub-Saharan Africa and South The Bank especially needs to strengthen supervi- Asia are privately provided. Public-private partner- sion, monitoring, and evaluation, (drawing on ships offer new approaches to service delivery, IFC’s recent experience in emphasizing these although evaluative evidence on institutional and functions) increasingly through its own clients. financial sustainability is still limited (IEG 2010c). By the same token, IFC and MIGA must especially ensure third-party verification and full and timely The involvement of multiple sectors in the public disclosure—as the World Bank is poised complex crop production chain indicates that to do—for credibility and better results in the weakness at any point within and between the environmental and social areas. 10 II. Measuring Results the Right Way F ocusing on the right results could be of limited value if we are not measuring those results properly. Poor monitoring and evaluation can send wrong signals and risk the achievement of desired outcomes by misallocating scarce resources from higher-value activities. However, measur- ing results the right way is less obvious in practice than it is thought. There are at least three possible areas that can tion in a country, if not the state of country reform, render measurement inappropriate. First, when but place a value only on one side of the ledger, there is weakness in the assumptions and method- namely, lower cost to business from lighter regula- ologies, a composite indicator can be a poor proxy tion and lower taxes, and not on the other side, for what it aims to measure. Second, even if the that is, benefit to society from a certain degree of indicators are well defined, when the elements in regulation. Thus the indicators can lead to wrong the results chain are missing or linkages broken, prescriptions for regulatory reform. achievement of intermediate outcomes may be a poor measure of the likelihood of reaching the final The DBI are built on the premise that firms are desired results. Third, even if the right measure- more likely to flourish if they have to abide by ment exists for the desired results, when the distri- fewer, cheaper, and simpler regulations. Its bution is skewed, the targeted population can be creators hypothesized that lighter regulation and left out or wind up worse off, even if the results are less taxation encourage informal firms to move achieved, when measured in averages. to the formal economy. But the literature is in- conclusive about whether these factors can cause Composite Indicators May Mislead such change and whether they can create more jobs and lead to higher growth (IEG 2008f). Composite measures are appealing for their apparent simplicity, but they can send wrong Furthermore, the de jure rather than de facto signals when what is being measured does not nature of the measured DBI implies that there match the claim. There can be a gap between may be large differences between what is listed what a composite indicator claims to measure in the rules and what is happening in the field, and what it actually measures. They can send the especially in areas where implementation is wrong signals and lead to misdirected develop- often a big challenge. Economic outcomes, be it ment activities. The soundness of the underlying growth or poverty reduction, are determined by premises is a prerequisite for a good indicator, many factors besides investment; investment, be and the theoretical foundation behind the indica- it domestic or foreign, is affected by many factors tor determines its relevance. besides firm performance; and firm performance, be it profit or sales growth, is influenced by many Inconclusive Premises factors besides the business environment. The Doing Business Indicators (DBI) claim to There are specific aspects of DBI too that have measure the overall state of the business regula- been of concern. First, DBI is constructed as an 11 EVALUATION BRIEF 14 average of 10 dimensions of the cost to firms of A simulation in the above evaluation revealed business regulations, and it is not clear which that the effects of the much larger weight on regulations matter, as compared with other governance in the Performance-Based Allocation determinants of the business environment, are not due just to the governance rating but to such as infrastructure, labor skills, and competi- how different the governance rating is, compared tion policies (Dollar, Hallward-Dreimeier, and with ratings on other clusters (IEG 2009c). Some Mengistae 2006). Second, regulations generate countries, though with low governance rating social benefits, such as safety, environmental compared with other clusters, will gain due to a protection, worker protection, and transparency, larger weight on governance for a given total of as well as private costs. Depending on whether IDA resources.3 the country starts with a little or a lot of regula- tion, reducing regulation is not always better for More generally, all core IDA countries (excluding society. Third, the same constraints may have small states) have worse governance ratings than different implications for different types of firms, ratings on other clusters, yet some countries depending on countries’ income levels, legal gain but other countries lose from the larger regimes, and other characteristics. weight on governance. Whether they gain or lose depends on how much worse is the ratio of their Finally, while DBI draws attention to the burden governance ratings to ratings on other clusters, of business regulation and offers a consistent compared with other countries. The choice of yardstick for comparing countries on regulation, weights, along with the ratings of each element, as seen from the firm’s point of view, it provides a has a critical effect on the value of the indicator. very incomplete picture of the business environ- ment, let alone country reforms. Attractive as it is Adding, Rating, Ranking to have a measure of the ease of doing business, the use of DBI as the measure of business regula- Adding up multiple indices of different dimensions tion, if not country reform, overstates the scope does not always get closer to a complete picture. of coverage and the explanatory power of the The rationale behind the choice of coverage indicators and can be misleading. and methods conditions the relevance of the exercise. Rigor often can be compromised due Clustering and Weights to arbitrary weights and clustering and through rescaling from cardinal to ordinal series. Evaluations give a more positive picture of the consistency between the content of the World Bank’s Country Policy and Institutional Assess- 3. If a country’s governance rating is much worse ment (CPIA) index and what it claims to reflect. than its ratings on other clusters, whereas another At the same time, the CPIA (along with the IDA country’s ratings on governance are only slightly [International Development Association] alloca- worse than its ratings on the other clusters, the tion formula) shows how changes in clustering former will lose and the latter will gain because of and weights can have large implications (IEG a larger weight on governance, given that the total 2009c). Both indicators have the same clusters: IDA resource is unchanged. For example, country CPIA applies equal weights to each of the four X has a better governance rating (3.7) than country clusters—economic management, structural Y (3.2). Country X also performs better on all the policies, policies for social inclusion and equity, other clusters than country Y, with the ratings for and public sector management and institutions. clusters A–C averaging 4.4, compared with country Y’s 3.3. Yet country X suffers a loss in Performance- The IDA allocation formula gives equal weight (8 Based Allocation of 13 percent under the current percent) to the first three clusters and to portfo- formula (versus a formula with equal weights on lio performance, but a much higher weight (68 all four clusters), whereas country Y actually gains percent) to the governance cluster. 20.5 percent. 12 OVERLOOKED LINKS IN THE RESULTS CHAIN Until 2010, the Human Development Indica- This arithmetic means that countries at the ends tor (HDI) was an equally weighted mean of of the distribution have to work harder to change uniformly scaled attainments in life expectancy, their overall ranking. Moreover, the magnitude education, and income. The 2010 HDI relaxes of the difference between the countries is not the its past assumption of perfect substitutability same on all points of the distribution. A country’s among its three components and switches from location in the distribution affects how a given the original additive aggregation function (the reform will change its ranking. Countries can arithmetical mean of the three components) to a make significant changes, yet fail to improve their multiplicative function (their geometrical mean). rankings, if they are at the ends of the distribution This change results in a significant reduction of for that indicator. The change in ranking for any the weight on longevity in poor countries. Based country is driven largely by where the country on the new construction method, a poor country is located on the distribution of countries on a experiencing falling life expectancy due to the specific indicator. Small changes can produce collapse of its weak health care system could large ratings jumps, and vice versa, which contrib- still see its HDI improve with even a small rate of ute to anomalies in the rankings. economic growth (Ravallion 2010b). Using rankings, which can be highly volatile over The HDI shows how arbitrariness can lead to time, to measure regulatory reform of a country wide difference in the indicators when theory can therefore be misleading. Rerankings can be offers no evidence to justify any particular set generated by even very small differences in the of weights. Simply lumping several available underlying measure of interest (Høyland and data sets and assigning them unjustified weights others 2010). without transparent documentation can result in “mashup indices� (Ravallion 2010a) that are Being clear on the data and the methodology opaque to users and risk distorting develop- behind the formulation of each indicator and ment policy making. Other examples include providing detailed information of each subcom- past efforts to combine multiple social indica- ponent may allow users to better understand what tors into one. The World’s Best Countries Index, the data actually measure. Recently, a Quality for example, averages five groups of (often of Official Development Assistance assessment themselves composite) indicators measur- constructed in four dimensions of aid quality, ing education, health, quality of life, economic built up from 30 separate indicators (Birdsall and competitiveness, and political environment, but Kharas 2010), is one step in that direction. So the does so without conclusive justification on the enthusiasm for composite indicators needs to be choice of any element. balanced by warnings, and more critical scrutiny from users. Reliance on successive stages of ordinal rankings is another approach that obscures the underly- Achieving Intermediate Outcomes Does ing cardinal values. In the DBI, each ranking is Not Ensure Desired Results translated from an indicator using cardinal values from its 10 subindicators: time, cost, number of Meeting intermediate outcome objective does procedures, and so on. These cardinal values not always mean progress toward desired results. are ranked according to their respective percen- Money can buy outputs, but to achieve develop- tiles in each of the subindicator distributions. ment outcomes, policy needs to induce behavior The subindicator percentiles are then averaged change. Controlling and measuring the inputs to come up with an indicator-level percentile; and outputs of a program—for example, how the 10 indicator percentiles are then averaged much money is spent and how many textbooks to generate the overall ease of doing business are distributed—is important, but not sufficient ranking (IEG 2008f). for achieving the intended results. Whether 13 EVALUATION BRIEF 14 or not the desired results—for example, how Tanzania won a United Nations award in learning outcomes have improved—are achieved September 2010 for its impressive progress is crucial for policy makers. However, the results toward attaining universal primary education chain from having access to schools with text five years ahead of the 2015 deadline set under books, to enrollment, to graduation, and to the Millennium Development Goals.4 But the learning, is complex. results of a recent survey, led by nongovernmen- tal organizations, of 40,000 children in Tanzania To be sure, projects often need to target interme- is sobering—about 20 percent of the children diate outcomes to make things manageable, who had completed seven years of primary and they often (rightly) do so, keeping in mind school could not read their own language, frameworks of assumed links among inputs, Kiswahili, at the grade-2 level; half could not read outputs, and outcomes. Although these interme- English, the medium of instruction in secondary diate outcomes are easier to monitor, they may education; and about 30 percent could not solve not lead to the desired final outcomes if critical a simple (grade-2) multiplication problem.5 channels in the causal chain are missing. And without the desired results spelled out and Tanzania exemplifies the hazards of an exclusive properly monitored, program achievements will focus on raising enrollment (IEG 2010h). In remain unknown. As the desired results are not Tanzania even significant increases in domestic measured, success cannot be distinguished from budget and development partner support failure and cannot be properly rewarded. were not enough to finance all the goals of primary and secondary education development. Access versus Learning in Education Rapid expansion of secondary education was constrained by a shortage of qualified teachers. There are well-known benefits in putting school- Learning outcomes were undercut by a one-third age children in classrooms, but focusing on access decline in the capitation grant for nonsalary to schools alone is not sufficient to improve items such as books and learning materials, and children’s competency. How much a child learns the doubling of teacher-pupil ratios. matters more for future productivity and quality of life than how many years he or she spends in Higher teacher attendance also cannot be school. It is true that improved infrastructure can presumed to lead inevitably to more learning. contribute to higher enrollment rates, and greater Banerjee and others (2001) evaluated a program availability of school supplies can contribute to in which an Indian nongovernmental organization higher test scores. It is also true that improved placed a second teacher in nonformal education student retention, along with enhanced teacher centers the organization runs in Indian villages. attendance, can help the achievement of learning The study found that, in 21 of the 42 centers outcomes. However, measures necessary to randomly selected to receive a second teacher, achieve the desired outcome of improved learning teacher attendance increased, as hoped, but test are often missing. scores remained the same. The absence of planning for improving learning Lessons Feeding into Strategy outcomes and lack of political commitment for it can result in unnecessary trade-offs between The broad lesson is that focusing only on interme- improved access and student learning gains, diate outcomes puts at risk the achievement of especially among the poor (IEG 2006a). If desired results. The discrepancy between access primary school completion rates are raised by, for example, automatically promoting children 4. http://www.dailynews.co.tz/home/?n=13281 to the next grade or without attending to student &cat=home. learning outcomes, higher completion rates will 5. http://blogs.worldbank.org/africacan/is-our- not reflect improved knowledge and skills. tanzanian-children-learning. 14 OVERLOOKED LINKS IN THE RESULTS CHAIN to schooling and learning outcomes indicates (IEG 2008a). Training is widely used as a tool the high cost of such neglect. Two-thirds of for helping countries build the capacity of their Bank primary education projects during the institutions to achieve development objectives 1990–2005 period focused on increasing enroll- and most Bank-financed training resulted in ment and reducing dropout rates. But only individual participant learning. However, only one-fifth of projects had objectives explicitly about half the time did such training improve the covering the expected results, such as improv- capacity of client institutions and organizations to ing reading, writing, and math skills and other achieve the development objectives. For training learning outcomes. to go from individual learning to workplace outcomes and organizational capacity, trainees This lesson has had an impact on the Bank and must not only learn but also be able to apply country strategies, which increasingly include what they have learned in their workplaces, and learning outcomes as a goal. The 2005 Education this learning must be relevant for the achieve- Sector Strategy Update substantially increased the ment of organizational goals. focus on results and learning outcomes, in part responding to IEG evaluation findings. A quality Averages Can Mask Outcomes for review framework for education programs was Crucial Target Groups introduced in fiscal 2007 to benchmark and report annually on the degree to which lending and analytic In many situations, averages may miss important activities measure learning outcomes and focus on constituencies and hide disparities between improving the quality of teaching and learning. population groups. If growth accelerates by In fiscal 2009, 50 percent of operations specifi- moving the concentration of wealth from the cally included support for countries to introduce poor to the rich, a higher average may coexist learning assessment systems (IEG 2010f). with reduced wealth or income for the poorest segment of the population. Even if the project Evidence from Bank education projects suggests has a successful outcome at the average level, that to improve learning outcomes it would be the targeted beneficiaries may still be left out or helpful to have accountability-oriented reforms made even worse off. focusing on school-based management, informa- tion for accountability, teacher incentives, and A crucial assumption for an intervention to yield leveraging the private sector (World Bank 2010a). the intended impact is that it is addressing the A study by the U.K. Department for International right constraint or reaching the right target Development (White and Masset 2004) concurs group—neither of which may be captured by with the findings and shows that, in Ghana, the averages. The good intention of targeting promoting community engagement and enhanc- the poor and vulnerable may not always yield the ing the effectiveness of teaching contributed to expected results. learning outcomes. Microfinance A recent study by the Center for Global Develop- ment (2010) suggests that cash-on-delivery aid Microfinance is often considered a panacea for is one way to promote results-based programs. poverty reduction. The Consultative Group Tying aid to desired results, rather than to to Assist the Poor considers microfinance a outputs or intermediate outcomes, can help powerful tool to help achieve the Millennium strengthen accountability among funders, recipi- Development Goals. However, the fundamental ents, and their respective constituencies, which argument for microcredit—that access increases helps improve development effectiveness. profits, business scale, and household consump- tion—is not supported by some evaluations. The In a similar vein, strengthening organizational channels through which women’s empowerment capacity requires more than individual learning improves children’s education and health status 15 EVALUATION BRIEF 14 may not hold because the assumed linkages could with high levels of social capital are more likely be missing. Microfinance is targeted at women, to apply for social funds for community develop- but it may, in fact, be used by male household ment programs, and hence the ex post difference members. is more likely a reflection of the ex ante differ- ences rather than the impact of the programs. Banerjee and others (2009) found that opening a microfinance institution in Hyderabad, India, Social funds projects worked in Malawi but had no impact on measures of health, education, were found to have limited impact in Zambia or women’s decision making. In the short term, (IEG 2008d). Looking at how social funds have households with an existing business at the time operated at the village level, just holding public of the program invested more in durable goods, meetings is not sufficient for the community to while their nondurable consumption did not participate actively in the decision making. In change. Households with a high propensity to Zambia, the leaders often make the decision; become new business owners increased their in Malawi they tend to be more inclusive, for durable goods spending and saw a decrease in example, by mobilizing the parent-teacher nondurable consumption to pay the fixed costs association.6 to enter entrepreneurship, whereas households with low propensity to become business owners Moreover, in Zambia, there is little room for increased their nondurable spending. dissension at a public meeting because the meeting only takes place after considerable work Furthermore, the channels through which has been done and the leaders have sought the microcredit works could be different from backing of the chief. As a result, the majority what is expected. A randomized evaluation by of the community participates actively in the Karlan and Zinman (2009) found that microcre- preparation work but rather more passively in dit, rather than working directly through the decision making. This casts light on why social targeted businesses, might work broadly through fund projects have limited impact on building risk management. There is no evidence that social capital despite their participatory model; increased access to credit in Manila, Philippines, it is because a degree of social organization is improved subjective well-being. Due perhaps to required to apply for social fund resources. the higher returns to capital for men, male and higher-income entrepreneurs benefited more Community-Based and Community-Driven from microcredit than women operating small- Development scale businesses, who were traditionally targeted. Business investment did not increase; instead, The importance of good targeting has been increas- the size and scope of “treated� businesses shrank ingly recognized as part of poverty reduction by shedding unproductive workers, similar to programs. However, targeting efforts may not the way that increased access to credit reduces be enough to reach the very poor. For example, the need for favor-trading within family or Bank-financed, community-based and community- community networks. driven development projects have aimed to reach the poor through targeting, but there is limited Social Funds evidence to show that they have done this more successfully than any other Bank investments, and It is often assumed that participants in the programs affect the outcome. But selection bias 6. In Zambia, only one-third of people knew of may reverse this assumption. For example, it is the meeting held for the subproject selection, often believed that social fund programs achieve one-fourth attended the meeting, and one-seventh their goal simply by observing ex post differences spoke at the meeting. In Malawi, almost four-fifths in the level of social capital among “treatment knew about the meeting, three-fifths attended, villages� and “control groups.� But communities and one-sixth spoke. 16 OVERLOOKED LINKS IN THE RESULTS CHAIN their administrative costs are significantly higher Two factors underpin this pattern on the supply than for other investments. The projects improved side: which communities get connected and living standards, but the improvement was greater which households can afford the connection once for those better off among the communities than the grid is available (IEG 2008e). To start with, for the poor. There are even cases where the across villages, communities to be connected to situations of the poor actually worsened in the the grid are identified on a least-cost basis, which context of some community-based and -driven favors larger communities nearer to the existing development projects. grid, roads, and towns. Although off-grid connec- tions can serve remote communities that may not For example, under the Borgou pilot project be connected to the grid for some years, they do in Benin, the community contribution typically not necessarily reach the poor better than grid required in Bank interventions created hardship extension does. Within villages, in most countries, for the poor. It is very difficult for the poorest to increases in coverage come from extensive make a cash contribution, so they usually have growth (extending the grid to new communi- to contribute time and labor, which takes them ties) rather than intensive growth (connecting away from income-earning activities. In situations the unconnected in already electrified villages). where the rich contribute on behalf of the In Lao People’s Democratic Republic, even in community, the position of the elite is strength- villages that have been connected for 15–20 ened relative to that of the poor (IEG 2005). years, it is not uncommon for 20–25 percent of Furthermore, reaching the poorest requires households to remain unconnected. fundamental social and cultural changes, which takes considerable time and sustained effort— Inappropriate design of a price scheme targeted something that is unusual in the implementation at the poor could even have the unintended result period of a Bank-supported project. of keeping them from reaping the full benefits. Keeping the fixed charge low for the initial limited Even if the poor are formally included in a amount of usage and imposing a significantly participatory role due to project requirements, higher rate after the threshold is passed can in fact their views, and their priorities are likely to become an obstacle for the poor. The objective of be excluded from collective decision-making this lifeline tariff setting is to encourage the poor processes (Kumar and Corbridge 2002; Turton to get connected to electricity. But the unintended and Farrington 1998). For example, in the result is that to benefit from a low tariff rate, the Matrouh Project in Egypt, where there had been poor restrict consumption. a substantial focus on women, the percentage of women who believed that they had benefited On the one hand, this results in a false savings from the project was highly variable. There were and limits the real benefit electricity can bring no elected women leaders or women’s associa- to the poor; on the other hand, it unintention- tions and there were substantial concerns about ally lowers the financial viability of providing the marketing products produced by women. electricity because of low load factors resulting from consumption being heavily concentrated in Rural Electrification the evening peak hours. Measures encouraging lower demand may aggravate the constraint on A similar concern holds for the benefit distri- the financial viability of rural electrification, which bution of rural electrification, which can have can put unnecessary pressure on the sustainabil- neutral or even negative effects on the poor. ity of the projects. Bank-supported projects that claim to have the objective of bringing rural electrification Education and Health to the poor have typically neglected to include components that would help to achieve this Chile’s voucher for private school programs was objective. intended to improve the quality of education by 17 EVALUATION BRIEF 14 providing low-income students access to private car, a large-scale community-based nutrition education and stimulating public schools to program (Expanded School and Community perform better. But, contrary to public belief, Food and Nutrition Surveillance and Education there is no evidence to suggest that vouchers and Program, or SEECALINE) tended to benefit increased choice improved education outcomes. the nutritional status of children in better-off However, the sorting mechanisms that resulted communities even though it was targeted at the from the scholarship programs meant that better- poorest areas (Galasso and Umapathi 2009). qualified students tended to move to private But overall, evidence on who is benefiting and schools—an outcome that was not intended who is not, as well as on the cost effectiveness and that had negative consequences for public of interventions, is scant. Fewer than half of the schools and particularly for low-income students 46 evaluations on nutrition programs measured (World Bank 2009). the distribution of impacts by gender, mother’s education, poverty status, or availability of Similarly, the assumption that because malnutri- complementary health services (IEG 2010h). tion is more common among the poor, children Only nine evaluations assessed the impact on living in poverty will disproportionately benefit nutritional outcomes of the poor compared from an intervention is misleading. In Ethiopia, with the nonpoor. Among the evaluations that free distribution of food raised the weight-for- did examine variation in results, several found height z-score of children younger than five years that the children of better-educated mothers in high-asset households, but not in low-asset or children living in better-off communities are households (Quisumbing 2003). In Madagas- benefiting the most. 18 III. Using Evaluative Findings for Results F ocusing the results at the right level and measuring them in the right way can still be of limited value if we are not using the evaluative find- ings properly. For evaluative findings to have the desired results, it would pay to present them at the right time, in the right format, to the right audience, and apply them in the right context. We might think of at least three circumstances Water in which even good evaluative findings can be under- or mis-used. First, in a rapidly changing One example is the Bank’s interventions in water. context, simply replicating what has worked in Water stress is about more than water availabil- the past may not be helpful for future work. ity; rapid economic growth increases water use Focusing on the underlying condition and and pollution. Sustainable management of water adapting to a dynamic situation are key. Second, resources has acquired a new urgency in the face focusing only on what worked and what did not of growing populations, increasing urbanization, may still overlook a potential area where evalua- and economic development. Changes in develop- tion can bring value: capturing missed opportu- ment patterns, increases in population pressure, nities. Third, doing the right thing in the right and the demand for better livelihoods across the way is most effective when lessons are learned globe all contribute to a looming global water and applied at the right time. Seizing the crucial crisis. The Bank’s involvement in water is facing moment to influence policy is of the highest heightened challenges due to climate change, importance for evaluation to contribute toward migration of people to coastal zones, and the improved development effectiveness. declining quality of the water resources available to most major cities and businesses. The Future May Not Resemble the Past Although water has long been a major focal area for Bank lending to developing countries and The uncertainties and complexities of develop- water project performance has improved steadily, ment situations are increasing. Environments the approach taken thus far has underempha- in the developing world keep changing and sized some of the most difficult challenges, such new challenges are emerging. High perfor- as coastal zone management, pollution reduction, mance ratings for a portfolio do not necessar- and groundwater conservation (IEG 2010i). New ily lead to high development impact if activities ways need to be found to help countries make undertaken are not being adjusted to a country’s water sustainability a cornerstone of their develop- current and emerging needs. More investment ment plans. Just repeating successful projects and is beneficial only if it responds to these changes, not tackling the toughest challenges is not a way otherwise it results in inefficient resource use. to achieve better investment results. Meeting today’s needs while putting in place innovative and sustainable strategies to address Coastal zone management and pollution control future requirements is a worldwide challenge. illustrate the urgent need for the Bank to shift 19 EVALUATION BRIEF 14 toward addressing complex environmental However, Bangladesh, Morocco, and Vietnam, challenges. In the 1980s and 1990s, the Bank among others, have experienced severe degrada- financed investment in shrimp farming in tion of their water resources due to pollution. coastal areas to create jobs and alleviate poverty. Hence, water must now be pretreated to improve Meanwhile, the world has lost 5 million hectares its quality before connecting it to households. For (or one-fourth) of mangrove forest since the 1980s. example, since the early 1990s, the city of Hanoi One of the major reasons was the conversion of has had a problem with the presence of ammonia mangrove swamps into aquaculture ponds. As a in the groundwater it taps for system use from a result of depleted mangrove forests and drained series of wells near the Red River. The 1997 Vietnam wetlands, pollution has reduced the water quality. Water Supply Project installed special ammonia- removing units at the treatment plant to remove In India and Vietnam, shrimp farming increased this contaminant from the water before connecting the vulnerability of local communities to to households (IEG 2006b; World Bank 2006b). storms and floods and degraded freshwater resources (Primavera 2005). Two cyclones— Meeting today’s water needs while putting in one in Andhra Pradesh in 1997 and one in place innovative strategies to address future Orissa in 1999—completely destroyed the newly requirements is a worldwide challenge. Bank constructed sites for shrimp farming. New instru- projects have largely met their stated goals in ments, such as replanting mangrove forests and recent years. The continuing challenge is to improving water quality, are required to protect reinforce and scale up innovative ways to confront and restore the environment and improve the mounting problems. Other country examples people’s livelihoods. include paying more attention to wetlands, as in Vietnam; being cautious in expanding irrigation The sustainability of large-scale irrigation that relies on falling underground water tables, infrastructure highlights the need for alternative as in Yemen; or confronting agricultural water environmentally sustainable approaches. During pollution, as in Morocco. The direction taken the 1970s, the Bank supported large-scale irriga- regarding dams must continue to take into tion infrastructure to improve agricultural produc- account environmental and social impacts. tivity. For example, three large pumping stations were built in China to use water from the Yellow Transport River to irrigate farmland. At times, river water in the China scheme was completely dried up, and Another example is the Bank’s intervention in concerns are increasing that the Yellow River will transport. The sharp increase in population in no longer reach the sea. the next decades, especially in urban areas of the developing world—the number of cities exceed- In many other countries, due to limited and unreli- ing 1 million inhabitants is on track to surge able access to river water, irrigation schemes relied from 268 in 2000 to 358 in 2015—coupled with heavily on extracting groundwater. In the long continuing globalization and trade liberalization run, continuous overexploitation of groundwater is expected to accelerate significantly the demand may result in salinization and depletion of ground- for transportation of both people and goods. water resources, as has occurred in the Republic of Yemen. Initial results indicate that investing in Although the Bank’s past assistance to the new technologies, such as sprinkler, drip, and pipe transport sector has been well managed and irrigation and implementing regulations, such as a effective, with above-average project ratings, water quota system, to raise the efficiency of water greater attention is needed on efficiency, safety, usage are promising. health, and the environment to address the tougher issues, such as growing population Connecting households to water supply systems density and rising environmental vulnerability used to be a way to provide access to safe water. (IEG 2007). The Bank’s experience in transport— 20 OVERLOOKED LINKS IN THE RESULTS CHAIN narrowly focusing on roads—although success- policies. Yet to bring out the value of evaluation ful, will be insufficient to provide for the Bank’s in a rapidly evolving and interconnected world, future response to the emerging challenges. evaluators’ tasks are becoming more complex. The next generation of projects is expected to Cost-Benefit Analysis have a much more urban focus. Rapid urbaniza- tion, congestion, pollution, and resource overuse Cost-benefit analysis can be a valuable tool in exacerbate the negative impacts of transport prioritizing allocation of resources to high-return investments. The removal of the highly polluting areas. However, the use of cost-benefit analysis three-wheeled taxis with two-stroke engines in as a tool to help measure results and ensure Dhaka, under the Bank’s Air Quality Management accountability has been largely missed. Overall, Project, is one example of addressing the increas- the percentage of projects with cost-benefit ing concern of worsening urban air quality. analysis dropped from 70 percent to 25 percent between the early 1970s and the early 2000s. Given the increasing and vital linkages with The low and declining quantity and quality of energy, land use, urbanization, the environ- cost-benefit analysis have made it less relevant in ment, and climate change, transport will require providing information for decision making. The programmatic, cross-cutting, and multisectoral World Bank Group’s ability to identify high-return approaches. The Bank may have to reconsider investments could have been improved had its priorities and shift resources for multimodal cost-benefit analysis been better used (IEG transport, rural linkages, and urban transport. 2010b). The Jamuna River Bridge in Bangladesh is an example how a multifunctional structure can Furthermore, not just the use but also the formula- stimulate development (IEG 2007). The bridge, tion and timing of the analysis have undermined its which connects Bangladesh’s less-developed effectiveness. The estimates of the economic rates northwestern region with its more developed of return are often not only biased but also delayed. eastern region, reduces journey times and Out of the 51 project leaders randomly chosen from transport operating costs. projects closed in fiscal 2006–07 and 2008–09, only 5 reported that cost-benefit analysis is given signifi- The need to shift focus is strong from the cant weight at the project identification stage, and demand side. For example, in China, where 18 reported that it is given significant weight at the traditional expressway projects were among preparation stage. At appraisal, “everything goes the most successful in the sector, the govern- according to plan� is often the underlying assump- ment has enlisted Bank support for transporta- tion of cost-benefit analysis. The likelihood that the tion projects connecting interior regions with economic rate of return is recalculated at the close the coast. The project pipeline is shifting toward of projects is lower for projects with low outcome railway, inland waterway, and urban transporta- ratings. Many cost-benefit analyses were conducted tion projects (World Bank 2006a). Accordingly, after the decision to proceed. This put the analysis the share of urban transportation, railways, and under considerable pressure to reach conclusions inland waterways in technical assistance and consistent with the decisions already taken, rather investment lending continues to increase. than providing critical information for decision making about whether the project is justified. Identifying Missed Opportunities Can Help in Crafting Better Strategies Environmental Protection An underappreciated role of evaluation is to Two evaluative findings on climate change identify missed opportunities in operational related to energy efficiency and protected areas work. Traditionally, the evaluator’s role has been show how identifying missed opportunities can to draw lessons from the past and to inform future provide practical guidance (IEG 2009e). Potential 21 EVALUATION BRIEF 14 win-win strategies can help address simultane- in areas where there is economic use of forests, ously the urgent environmental concerns and where the counterfactual would have been more economic needs. deforestation, (as compared with pristine areas, where the counterfactual to providing protection The Bank has long supported reductions in would have been less deforestation). energy subsidies, coupled with improvements in energy efficiency, as well as improving energy Identifying the missed opportunities helps availability for the poor (IEG 2009e). Evidence address the urgent need to combine forest protec- from European transition countries shows tion with economic development in the Reduced that cutting energy subsidies quickly leads to Emission from Deforestation and Degradation efficiency gains as prices to end users rise toward agenda. It suggests that there is some compatibil- their full production cost. In addition, a number ity between environmental goals (carbon storage of studies show that subsidies tend to go dispro- and biodiversity conservation) and support for portionately to the better off, providing few local livelihoods. This sheds light on the debate benefits for the poor. of the relative effectiveness in deforestation reduction of strictly protected areas versus areas A win-win strategy could be built around reducing that allow some degree of sustainable use by local subsidies and targeting them to the poor. This people. Reduced Emission from Deforestation would simultaneously reduce the strain on and Degradation can be win-win, by not merely government budgets, free up resources to allow putting up fences to conserve pristine forests extension of energy sources to the poor, and but also addressing the fundamental interests of promote more efficient energy use. But opportu- communities and promoting local environmen- nities were missed because little effort has been tal and developmental goals. made to use the introduction of energy efficiency as an adjustment vehicle for higher tariffs (IEG Structural Issues 2009e). Sometimes missed opportunities result from On average, protected areas significantly reduce structural issues that need to be addressed before tropical deforestation, preserving carbon and operations can be undertaken. For example, biodiversity (IEG 2009e). Yet, contrary to IEG’s annual report on MIGA (IEG 2009f) found common expectations, evaluations find that that restrictions imposed by MIGA’s conven- these impacts are greater when the protected tion on eligible risks had hampered the agency’s areas allow sustainable use by local populations effectiveness, including in IDA countries, and (table 4). The value of protection is even greater mandated restrictions were limiting its ability Table 4: Summary of Estimated Protected Area Impacts on Fire Incidence (%) Mean reduction Mean reduction Mean reduction Mean fire due to strict due to multiuse due to Area incidence protected areas protected areas indigenous areas Latin America and Caribbean 7.4 2.7–4.3 4.8–6.4 16.3–16.5 3.8–7.7 6.2–7.5 12.7–12.8 Africa 6.1 1.0–1.3 (0.1)–3.0 Not applicable 4.4–4.5 Not calculated Asia 5.5 1.7–2.0 4.3–5.9 Not applicable 2.9–3.1 5.1–6.7 Source: IEG 2009e. Note: Italics indicate estimates for protected areas established between 1990 and 2000. 22 OVERLOOKED LINKS IN THE RESULTS CHAIN to respond to country needs during the global learn faster what works and what does not, and financial crisis. to focus on results at the right time. Evaluators need to be cognizant of the political dialogue and The report came out just as the crisis was height- inform the dialogue with evidence when critical ening the need for MIGA to enhance its range of decisions are about to be made. Timeliness in products to better serve its client countries. MIGA the provision of evaluative findings is crucial for has begun to address institutional effectiveness, ensuring its relevance and impact. including amending its Operational Regulations and some policies, conducting a Business Process Crucial Timing Review, and taking a first step toward implement- ing self-evaluation. The timely completion of The value of proper evaluation in the early stages IEG’s report meant that the evaluation findings of project implementation is high, particularly could inform these changes. for programs that are expected to be scaled up, because the risk of wasting funds on ineffective World Bank Group safeguards policies have been programs is acute. Even popular programs can found to have important gaps (IEG 2010g). The prove to be ineffective. Good evaluation can narrow coverage of social safeguards in Bank avoid costly mistakes and prevent real harm. projects, compared with IFC and MIGA, leads to an For example, in the United States, the Drug underestimation of risks and, in some instances, Abuse Resistance Education program, which was to risk avoidance. Equally, project categorization believed to be effective, had been adopted in 75 at IFC and MIGA leads to an underestimation of percent of school districts. But evaluation with environmental risks, compared with the Bank. random assignment found that it was a waste When social and environment risks are underesti- of financial resources and school time (Lynam mated, or when communities are excluded from and others 1999). Another example is a U.S. project benefits to avoid dealing with safeguard program called Scared Straight, which sought risks, development benefits are significantly to reduce juvenile delinquency by taking at-risk lower. youths to visit prisons. Randomized experiments found this well-intentioned program was actually Instances such as these highlight the role that harmful, leading to higher delinquency among independent evaluation can play in getting participants (Petrosino, Turpin-Petrosino, and outside the assumptions and received wisdom Finckenauer 2000). that inevitably affect most organizations, particu- larly mature ones with long histories, such as the Early results of the evaluation of Mexico’s World Bank Group. By looking with a fresh eye conditional cash transfer program Progresa- and in depth at what the organization is doing on Oportunidades (since 2002) showed impacts on particular issues, evaluation can shine a light into schooling, health, labor supply, and consump- the dark corners where inertia lurks and provide tion. The evaluation helped convince a new new insights that can help shift behavior, or even administration not to cancel the project and, entire ways of doing business. instead, to expand the program to new areas and extend eligibility to additional children within the Timing and Process Can Make the original areas. It also encouraged the administra- Biggest Difference tion to embrace a program of rigorous impact evaluation, more generally, in developing its social Focusing on the right issues and employing sound safety net programs. Similarly, the evaluation of methodologies are not enough. Evaluations must the early childhood development program in the be conducted at the right time, delivered in an Philippines was used to justify expansion of the appropriate format, and be based on collabo- program. The findings reaffirmed the existing ration and follow-up with those evaluated and strong support for the program and played a role stakeholders. For higher impact, it is crucial to in the decision to expand the innovations. 23 EVALUATION BRIEF 14 In a rapidly changing environment, delays in corporate learning, assess the relevance of the action could disproportionately increase the response, and measure the effectiveness of the eventual costs. In fact, options may disappear mode of implementation and appropriateness of entirely if actions are not taken soon enough. operational policies. However, if efforts are unduly scaled up and generalized before their true impacts are reliably On climate-related development work, the estimated, it could result in waste of scarce costs of waiting to evaluate until projects are resources and add to the burden in the future. In fully completed are high. IEG recently did two an uncertain world, it is important for real-time phases of evaluation on climate change, and a inputs to inform policies and programs based third phase is currently being completed (IEG on the best available evidence. With the right 2010a). The evaluation stresses five measures timing, even minor course corrections, based on that offer attractive local benefits while fighting emerging evaluation evidence, can have dispro- climate change: energy efficiency, forest protec- portionate impacts. When tens of billions of tion, appropriate project finance, technology dollars are to be used to tackle the global financial transfer, and accelerated learning. The study crisis and climate change, traditional evaluation notes that carbon finance has yet to realize its cycles are too slow, leading to lost opportunities promise of catalyzing large-scale, new invest- for learning. ments in renewable energy. The Bank also needs to actively assist clients to move away from coal, The challenge of being timely is to ensure that using analyses that span entire energy systems to evaluation work is available when it can influence find cleaner, more cost-effective, and financeable decision making. The recent global financial alternatives. crisis, for example, put a premium on the speed with which evaluative findings could be used. A constant problem for established independent Based on the most recent estimate from Global evaluation organizations is they have internal Economic Prospects 2010 (World Bank 2010b), processes and procedures resulting in evalua- the crisis is having serious cumulative impacts on tions that address the right issues but are not well poverty, with 64 million more people expected timed. To counter this problem, IEG has timed to be living in extreme poverty by the end of 2010 its sector evaluations to provide inputs for sector than would have been the case if the crisis had strategies, and its country assistance evaluations not occurred. The World Bank Group’s lending aim to contribute to new CASs. More recently, to financial and private sector development it also has produced rapid-response evaluations has increased substantially and has reached an providing relevant findings on current issues all-time high of $80 billion in the past two fiscal such as the Haiti earthquake, Pakistan floods, years. The series of evaluation briefs and reports and West Africa floods (IEG 2010d). on crisis response (IEG 2008h, 2009h, 2010k) provided timely assessment of the World Bank Effective Process Group’s activities. Translating evaluative lessons to development Another example of real-time evaluation results requires not just quality standards but performed during a crisis is the World Food also effective processes. Evaluation recommen- Programme’s assessment (2003) of its Southern dations should be presented in a way that is Africa Emergency Operation, in response to the attractive to key constituents or stakeholders. Southern Africa crisis in 2002–03. The study Messages need to be tailored to the needs of the evaluated the humanitarian relief operations as targeted audience and demonstrate the points they unfolded from inception through implemen- with enough solid evidence to demonstrate their tation, and to closing. Evaluative lessons validity, but not overly much so as to obscure the captured at several stages aimed to promote message (Grasso 2003). The recommendations 24 OVERLOOKED LINKS IN THE RESULTS CHAIN of evaluations must be relevant and specific to and evaluation, disclosure of findings, and verifi- be useful and implementable. Thus, IEG has cation of results (IEG 2010g). developed a wide range of print, online, and in-person communication strategies to ensure It pays to stay engaged on an issue after an evalua- that key messages are delivered effectively to tion by keeping track of how recommendations those most likely to implement them, whether have been incorporated. The impact of evalua- it is within the World Bank Group, member tions would be short-lived with only a one-off governments, other donor agencies, civil society, engagement at the completion of an evalua- or nongovernmental organizations. tion report. Real change takes place only when findings are accepted and learning takes place. Timely dissemination of evaluation results may enable new projects to incorporate the lessons Weak follow-up on the implementation of results learned into their designs. Publicly disclosed and recommendations and poor dissemination monitoring of carbon projects shows the gains of findings have been identified as key factors such feedback can deliver. Landfill gas projects explaining why some technical assistance projects proliferated with the advent of the carbon market, of the Asian Development Bank are less success- but monitoring reports soon showed that these ful than others (Adhikari 2007). Similarly, a review projects were underperforming relative to their of World Bank economic and sector work and design expectations. This feedback revealed that technical assistance over 2000–06 showed that in the appraisal models used were based on U.S. addition to high technical quality, close collabora- experience, which is not applicable to the waste tion with clients during the process mattered for streams of developing countries. effectiveness, whether clients actually produced part of the work or not (IEG 2008b). Collaboration with clients and stakeholders is essential at the design stage of evaluations, during ******* the evaluation process, and in the formulation and follow-up to recommendations for an evalua- The examples from evaluations by IEG and other tion to be effective. In explaining why intended development evaluators cited in this paper show outcomes and impacts were not achieved, it is that evaluation findings often add the most value essential to know whether a lack of results was when shedding new light on policy directions due to design failure or implementation failure and bringing out new content or novel ways of because projects are often not implemented looking at issues. The value added would seem as planned (Bamberger and others 2010). For to be especially great when evaluations highlight example, on safeguard issues, the effectiveness underemphasized but crucial factors, and of the regulatory regime depends not only on challenge conventional wisdom. Although it may up-front risk assessments that the World Bank not be possible to set such outcomes as a goal, Group and countries carry out but also on having frameworks that promote innovation and effective implementation and supervision and on risk taking in evaluation and following up on the the checks and balances provided by monitoring findings would seem to have a high payoff. 25 Bibliography Adhikari, R. 2007. “Lessons from the Evaluations of Cairncross, S., and V. Valdmanis. 2006. “Water ADB’s Technical Assistance Operations.� Presen- Supply, Sanitation, and Hygiene Promotion.� tation, Operations Evaluation Department of the In Disease Control Priorities in Developing Asian Development Bank. Countries, 2nd ed., ed. Dean T. Jamison et al. African Development Bank. 2008. “Review of the New York: Oxford University Press and the World Performance of the Current Fisheries Portfolio Bank, 771–92. of the African Development Bank.� Brief to the ———. 2006. When Will We Ever Learn? Improving Board, African Development Bank. Lives through Impact Evaluation. Report of the African Development Bank and the International Evaluation Gap Working Group. Fund for Agricultural Development. 2009. Towards Center for Global Development. 2010. “Cash on Purposeful Partnerships in African Agriculture. Delivery: A New Approach to Foreign Aid.� http:// A joint evaluation of the agriculture and rural www.cgdev.org/section/initiatives/_active/codaid. development policies and operations in Africa of Dollar, David, Mary Hallward-Driemeier, and Taye African Development Bank and International Fund Mengistae. 2006. “Investment Climate and Interna- for Agricultural Development. http://www.ifad.org/ tional Integration.� World Development 34(9): evaluation/jointevaluation/docs/africa/africa.pdf. 1498–1516. Banberger, M., and Angeli Kirk. 2009. “Making Smart ECG (Evaluation Cooperation Group). 2011. “Evalua- Policy: Using Impact Evaluation for Policy Making— tive Lessons from Agriculture and Agribusiness.� Case Studies on Evaluations that Influenced ECD Paper No. 3, World Bank, Washington, DC. Policy.� Doing Impact Evaluation No. 14, World Galasso, E, and N. Umapathi. 2009. “Improving Bank, Washington, DC. Nutritional Status through Behavioral Changes: Bamberger, M., V. Rao, and M. Woolcock. 2010. “Using Lessons from Madagascar.� Journal of Develop- Mixed Methods in Monitoring and Evaluation: ment Effectiveness 1(1): 60–85. Experiences from International Development.� Grasso, P. 2003. “What Makes an Evaluation Useful? World Bank Policy Research Working Paper No. Reflections from Experience in Large Organiza- 5245, Washington, DC. tions.� The American Journal of Evaluation Banerjee, Abhijit, Esther Duflo, Rachel Glennerster, 24(4): 507–14. and Cynthia Kinnan. 2009. “The Miracle of Microfi- Høyland, Bjørn, Karl Ove Moene, and Fredrik Willum- nance? Evidence from a Randomized Evaluation.� sen. 2010. “The Tyranny of International Index IFMR Research Working Paper Series No. 31, Rankings.� Department of Economics, University Chennai, India. of Oslo. Banerjee, A. V., S. Jacob, and M. Kremer, with J. Huttly, S.R.A., S.S. Morris, and V. Pisani. 1997. “Preven- Lanjouw and P. Lanjouw. 2001. “Promoting School tion of Diarrhea in Young Children in Developing Participation in Rural Rajasthan: Results from Countries.� Bulletin of the World Health Organi- Some Prospective Trials.� Harvard University and zation 75(2):163–74. Massachusetts Institute of Technology, Cambridge, IEG (Independent Evaluation Group). 2002. “Grant MA. Programs: Improving Their Governance.� Précis Benson, Charlotte. 1997. “The Cost of Disasters.� In number 224. World Bank, Washington, DC. Development at Risk? Natural Disasters and the ———. 2005. World Bank Support for Community- Third World, ed. John Twigg. Oxford: The U.K. Based and -Driven Development. Washington, National Co-ordination Committee for the Interna- DC: World Bank. tional Decade for Natural Disaster Reduction.. ———. 2006a. From Schooling Access to Learning Birdsall, N., and H. Kharas. 2010. Quality of Official Outcomes: An Unfinished Agenda—An Evalua- Development Assistance (QuODA) Assessment. tion of World Bank Support to Primary Education. Washington, DC: Center for Global Development. Washington, DC: World Bank. 27 EVALUATION BRIEF 14 ———. 2006b. Hazards of Nature, Risks to Develop- Group, Phase I: An Evaluation of World Bank ment: An IEG Evaluation of World Bank Win-Win Policy Reforms. Washington, DC: World Assistance for Natural Disasters. Washington, DC: Bank. World Bank. ———. 2009f. Independent Evaluation of MIGA’s ———. 2007. A Decade of Action in Transport. Development Effectiveness 2009: Enhancing Washington, DC: World Bank. MIGA’s Risk Mitigation in IDA and Conflict- ———. 2008a. Using Training to Building Capacity for Affected Countries. Washington, DC: World Bank. Development. Washington, DC: World Bank. ———. 2009g. “The World Bank Group’s Response ———. 2008b. Using Knowledge to Improve Develop- to the Global Crisis: Update on an Ongoing IEG ment Effectiveness: An Evaluation of World Evaluation.� Evaluation Brief 8, World Bank, Bank Economic and Sector Work and Technical Washington, DC. Assistance, 2000–2006. Washington, DC: World ———. 2009h. Improving Effectiveness and Outcomes Bank. for the Poor in Health, Nutrition, and Population. ———. 2008c. Public Sector Reform: What Works and Washington, DC: World Bank. Why? Washington, DC: World Bank. ———. 2009i. Independent Evaluation of IFC’s ———. 2008d. IEG Impact Evaluation—The Experi- Development Results 2009: Knowledge for Private ence of the Independent Evaluation Group of the Sector Development. Washington DC: World World Bank. Washington, DC: World Bank. Bank. ———. 2008e. The Welfare Impact of Rural Electrifi- ———. 2010a. Climate Change Phase II: The Challenge cation—An IEG Impact Evaluation. Washington, of Low Carbon Development. Washington, DC: DC: World Bank. World Bank. ———. 2008f. Doing Business: An Independent ———. 2010b. Cost-Benefit Analysis in World Bank Evaluation: Taking the Measure of the World Projects. Washington, DC: World Bank. Bank-IFC Doing Business Indicators. Washington, ———. 2010c. Priorities in Meeting the MDGs: Lessons DC: World Bank. from Evaluation. Washington, DC: World Bank. ———. 2008g. Maintaining Momentum to 2015? An ———. 2010d. “Responding to Floods in West Africa: Impact Evaluation of Interventions to Improve Lessons from Evaluation.� Washington, DC: World Maternal and Child Health and Nutrition in Bank. Bangladesh. Washington, DC: World Bank. ———. 2010e. “Response to Pakistan’s Floods: Evalua- ———. 2008h. Lessons from World Bank Group tive Lessons and Opportunity.� Washington, DC: Responses to Past Financial Crises. Washington, World Bank. DC: World Bank. ———. 2010f. Results and Performance of the World ———. 2008i. Independent Evaluation of MIGA’s Bank Group: IEG Annual Report 2010. Washing- Development Effectiveness 2008: Evaluating ton, DC: World Bank. MIGA’s FY05–08 Strategic Directions. Washington, ———. 2010g. Safeguards and Sustainability Policies DC: World Bank. in a Changing World: An Independent Evalua- ———. 2008j. Supporting Environmental Sustain- tion of World Bank Group Experience. Washing- ability: An Evaluation of World Bank Group ton, DC: World Bank. Experience, 1990–2007. Washington, DC: World ———. 2010h. “Tanzania: Primary Education Develop- Bank. ment Program Project, Secondary Education ———. 2009a. Annual Review of Development Development Program Project, and Human Effectiveness: Achieving Sustainable Develop- Resources Development Pilot Project.� Project ment. Washington, DC: World Bank. Performance Assessment Report. Washington, DC: ———. 2009b. The World Bank’s Country Policy and World Bank. Institutional Assessment: An Evaluation. Washing- ———. 2010i. Water and Development: An Evalua- ton, DC: World Bank. tion of World Bank Support, 1997–2007. Washing- ———. 2009c. The World Bank’s Country Policy and ton, DC: World Bank. Institutional Assessment (CPIA). Washington, DC: ———. 2010j. What Can We Learn from Nutrition World Bank. Impact Evaluations? Washington, DC: World ———. 2009d. The World Bank Group Guarantee Bank Instruments 1990–2007: An Independent Evalua- ———. 2010k. “The World Bank Group’s Response to tion. Washington, DC: World Bank. Global Economic Crisis.� Washington, DC: World ———. 2009e. Climate Change and the World Bank Bank. 28 OVERLOOKED LINKS IN THE RESULTS CHAIN ———. 2010l. “World Bank Group Response to the Ravallion, M. 2010a. “Mashup Indices of Develop- Haiti Earthquake: Evaluative Lessons.� Washing- ment.� World Bank Policy Research Working Paper ton, DC: World Bank. No. 5432, Washington, DC. Imas, L., and R. Rist. 2009. The Road to Results: ———. 2010b. “Troubling Trade-offs in the Human Designing and Conducting Effective Development Development Index.� World Bank Policy Research Evaluations. Washington, DC: World Bank. Working Paper No. 5484, Washington, DC. Karlan, Dean, and Jonathan Zinman. 2009. “Expanding Thomas, V., and R. Parker. 2010. “Make Relief Networks Microenterprise Credit Access: Using Randomized Disaster-Resilient.� Miami Herald, March 15. Supply Decisions to Estimate the Impacts in Manila.� Thomas, V., and J. Tominaga. 2010. “Evaluation for Economic Growth Center, Center Discussion Paper Better Development Results.� Journal of Develop- No. 976, Yale University, New Haven, CT. ment Effectiveness 2(3): 371–86. Kaufmann, D., A. Kraay, and M. Mastruzzi. 2005. Turton, C., and J. Farrington. 1998. “Enhancing Rural “Governance Matters IV: Governance Indica- Livelihoods through Participatory Watershed tors for 1996–2004.� World Bank Policy Research Development in India.� Natural Resource Perspec- Working Paper No. 3630, Washington, DC. tives 34. Kraay, A., and N. Tawara. 2010. “Can Disaggregated White, Howard, and Edoardo Masset. 2007. “Assess- Indicators Identify Governance Reform Priorities?� ing Interventions to Improve Child Nutrition: A World Bank Policy Research Working Paper 5254, Theory-Based Impact Evaluation of the Bangladesh Washington, DC. Integrated Nutrition Project.� Journal of Interna- Kumar, Sanjay, and Stuart Corbridge. 2002. tional Development 19(5): 627–52. “Programmed to Fail? Development Projects World Bank. 2006a. “Country Partnership Strategy and the Politics of Participation.� The Journal of Partnership Report for the People’s Republic of Development Studies 39(2) 830–51. China for the Period 2006–2010.� Washington, DC: Linnerooth-Bayer, J., and Mechler, R. 2004. “Financ- World Bank. ing Disaster Risks in Developing and Emerging- ———. 2006b. “Implementation Completion Report Economy Countries.� Paper presented at the on a Credit in the Amount of Sdr 71.3 Million Organisation for Economic Co-operation and (US$98.61 Million Equivalent) to the Socialist Development conference, “Catastrophic Risks and Republic of Vietnam for a Water Supply Project.� Insurance,� Paris, November 22. Report No. 32365, World Bank, Washington, Lynam, D.R., R. Milich, R. Zimmerman, S.P. Novak, DC. T.K. Logan, C. Martin, C. Leukefeld, and R. Clayton. ———. 2008. “Impact Evaluation of Rural Road 1999. “Project DARE: No Effects at 10-Year Follow- Projects.� Doing Impact Evaluation No. 12, Up.� Journal of Consulting Clinical Psychology Washington, DC 67(4):590–3. ———. 2010a. “Discovering What Works in Education: Morley, S., and D. Coady. 2003. From Social Assistance Informed Policy Making through Impact Evalua- to Social Development: Targeted Education tions.� Washington, DC. http://go.worldbank.org/ Subsidies in Developing Countries. Washington, KEHGC6W0E0. DC: Center for Global Development. ———. 2010b. Global Economic Prospects 2010: Overbey, L. 2008. “The Health Benefits of Water Fiscal Headwinds and Recovery. Washington, Supply and Sanitation Projects: A Review of the DC: World Bank. World Bank Lending Portfolio.� IEG Working ———. 2010c. Natural Hazards, Unnatural Paper 2008/1. Disasters. Washington, DC: World Bank. Petrosino, A., C. Turpin-Petrosino, and J. Finckenauer. WFP (World Food Programme). 2003. “Summary 2000. “Well-Meaning Programs Can Have Harmful Report of the Real-Time Evaluation of WFP’s Effects! Lessons from Experiments of Programs Response to the Southern Africa Crisis, 2002–2003 Such as Scared Straight.� Crime Delinquency 46(3): (EMOP 10200.3).� 354–79. World Health Organization. 2004. Family and Quisumbing, A. R. 2003. “Food Aid and Child Nutrition Community Practices that Promote Child in Rural Ethiopia.� World Development 31(7): Survival, Growth, and Development—A Review 1309–24. of Evidence. Geneva: WHO. 29