THE WORLD BANK W O R L D B A N K O P E R AT I O N S E VA L U AT I O N D E PA R T M E N T 34467 Country Assistance Evaluation Retrospective An OED Self-Evaluation OPERATIONS EVALUATION DEPARTMENT ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank’s Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country’s overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank’s work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. WORLD BANK OPERATIONS EVALUATION DEPARTMENT Country Assistance Evaluation Retrospective An OED Self-Evaluation 2005 The World Bank http://www.worldbank.org/oed Washington, D.C. © 2005 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved Manufactured in the United States of America First edition May 2005 The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in the work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The World Bank encourages dissemination of its work and will normally grant permission promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org. Cover photos courtesy of the World Bank Photo Library. ISBN-10: 0-8213-6316-6 ISBN-13: 978-0-8213-6316-4 e-ISBN: 0-8213-6317-4 Library of Congress Cataloging-in-Publication data have been applied for. World Bank InfoShop Operations Evaluation Department E-mail: pic@worldbank.org Knowledge Programs and Evaluation Capacity Telephone: 202-458-5454 Development Group (OEDKE) Facsimile: 202-522-1500 Room # 19-352 MSN # 19-903 (202) 458-0382 Facsimile: (202) 522-3125 Printed on Recycled Paper Contents v Acknowledgments vii Foreword ix Executive Summary xiii Acronyms and Abbreviations 1 1 Introduction 1 Background 2 CAE Retrospective Evaluation Approach 3 2 Part I: What Have We Learned? 3 Findings 5 Lessons Learned 11 3 Part II: How Can the CAE Instrument Be Improved? 11 Impact of CAEs on Country Strategy Formulation 13 Methodology 15 Ratings 17 Counterfactuals 18 Timing 19 Interaction with Country Teams 20 Interaction with Beneficiaries 20 Improving the Presentation of CAEs 23 Annexes 25 A: The CAE Methodology 29 B: Completed Country Assistance Evaluations, Fiscal Years 1995–2004 31 C: Summary of Board/CODE Interviews 35 D: Board CAS Discussions and CAS Documents 37 E: Summary of Bank Staff Interviews 41 F: Summary of Country Evaluation Methods Used by Other Agencies iii COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION 43 Attachment 1: Discussion Summary, CODE Subcommittee 45 Endnotes 47 References Box 17 3.1: Unsatisfactory Outcome Ratings Do Not Imply Poor Bank Performance Figure 36 1: Distribution of Related References Tables 4 2.1: Characteristics of the 25 CAE Countries 5 2.2: CAE Outcome Ratings and Project Outcome Ratings 5 2.3: Sectoral and Project Ratings for Fiscal 2001–03 CAEs 41 F.1: Use of Key Methodological Tools iv Acknowledgments T his Country Assistance Evaluation (CAE) Director). David Shirley interviewed CODE and Retrospective was written by Poonam Bank staff for this retrospective. Svenja Weber- Gupta (task manager) and Kyle Peters (se- Venghaus reviewed CAS transcripts, and Brandie nior manager). Support and suggestions were re- Sasser and Weber-Venghaus surveyed the meth- ceived from René Vandendries, Chandra Pant, ods used by other donors in their country pro- John Johnson, and Asita de Silva. gram evaluations. Janice Joshi and Agnes Santos A number of external, independent evaluators provided administrative support. and consultants were used to assist in the prepa- Marcelo Selowsky of the Independent Evalua- ration of this self-assessment. Mike Hendricks tion Office of the International Monetary Fund evaluated CAE methods. Sharon Felzer prepared and Ulrich Zachau (LCRVP) were peer-reviewers guidelines for the structured interviews in collab- for this report. External peer review was pro- oration with Patrick Grasso (OED, Office of the vided by Vijay Kelkar and Faik Oztrak. Acting Director-General, Operations Evaluation: Ajay Chhibber Acting Director, Operations Evaluation Department: R. Kyle Peters Senior Manager, Country Evaluation and Regional Relations: R. Kyle Peters Task Manager: Poonam Gupta v Foreword T his report is a self-evaluation of OED’s sis, reducing planned assistance levels more Country Assistance Evaluations (CAEs). In sharply when faced with policy slippages, and fiscal 2002, the Informal Subcommittee of lending more prudently in turnaround situations. the Committee on Development Effectiveness The second part of this retrospective is based (CODE) discussed CAE methodology and the on extensive interviews with CODE members, role CAEs play in assessing Bank interventions at their alternates and advisers, and Bank Regional the country level, as well as the need to continue staff, as well as assessments of the CAE method- to refine the methodology. This retrospective ology and process by external evaluators. These extends that discussion and addresses two ques- assessments find that the methodological frame- tions: what have we learned, and how can the work used in the past few years, while funda- CAE instrument be improved? It addresses the mentally sound, has some shortcomings. The first question by compiling lessons relevant for structure tends to place too much emphasis on Country Assistance Strategies from the most re- Bank instruments rather than on achievement of cent CAEs, and the second by assessing the CAE program objectives, and confusion has been cre- process, methodology, and presentation. ated when different sections of the CAE appear This retrospective finds that an evaluation at to lead to different conclusions. These criticisms the country level yields a more complete picture are consistent with the perceptions of CODE of the outcome of the Bank’s assistance programs members who felt that CAEs focused too much than individual project or program evaluations on formal compliance issues and the impact of can offer. In a third of the country programs eval- Bank instruments to the neglect of the Bank’s uated, aggregate outcomes from projects were development impact and the results achieved. satisfactory, but the country assistance program This retrospective concludes that the way for- was unsuccessful overall. A review of recent CAEs ward is the adoption of a more results-based ap- suggests that the Bank needs to understand bet- proach that clearly links the Bank’s objectives and ter the country context, deepen its country knowl- instruments with the outcomes achieved. This ap- edge, and pursue adjustment or development proach would outline the key objectives and re- policy lending only in situations where govern- sults that the Bank has been trying to achieve in a ment ownership is strong. It also indicates that the country, based on the Country Assistance Strate- outcomes of the Bank’s country programs can be gies (CASs). For each objective and intended improved by undertaking more robust risk analy- result, the CAE would evaluate the efficacy and vii COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION relevance of the instruments used, as well as the were introduced for all CASs, then ratings for relative contribution of the Bank (as opposed to CAEs might become redundant. that of others) to that objective. This approach Second, the retrospective concludes that the would address the main recommendation given timing of a CAE is crucial to its usefulness. OED by the Bank’s Board of Directors that the CAEs will continue its policy of delivering CAEs to the should focus more on the actual impact of pro- Board and Bank staff about three to nine months grams at the country level. ahead of planned CAS discussions. The retrospective also examines several other Finally, this report notes a widespread per- issues that have arisen regarding CAEs. ception that more interactions with country First, it examines the debate surrounding CAE teams and with in-country beneficiaries during ratings. CODE members in general favored rat- CAE preparation would enhance both the qual- ings, while the majority of Bank staff interviewed ity of the CAEs and the acceptance of CAE find- were critical of them. OED will review its use of ings. OED, therefore, proposes to make an ratings for the CAEs and is also discussing with effort to ensure that in-country beneficiaries are management the standardization of self-ratings consulted more uniformly and to involve coun- for Country Assistance Strategy Completion Re- try teams in midterm reviews of future CAEs to ports (CASCRs) validated by an OED review. If receive early feedback on emerging findings such a system, akin to the project rating system, and recommendations. Ajay Chhibber Acting Director-General, Operations Evaluation viii Executive Summary T his report is a self-evaluation of the Ope- In a third of the country programs evaluated, ag- rations Evaluation Department’s (OED) gregate outcomes from projects were satisfactory, Country Assistance Evaluations (CAEs). In a but the country assistance program was unsuc- fiscal 2002 discussion of OED’s methodology for cessful overall. This indicates the importance of as- Country Assistance Evaluations, the Informal Sub- sessing the country program in aggregate through committee of the Committee on Development a CAE. Otherwise, evaluation may not capture crit- Effectiveness (CODE) noted the role CAEs play in ical omissions in the country assistance programs; assessing Bank interventions at the country level for example, that project interventions may not be and the need to continue to refine the methodol- addressing key development constraints. ogy. This retrospective responds to that discussion The retrospective also notes that the Bank’s and has two primary purposes. First, it compiles success varied by sector. The outcomes in health lessons relevant for developing Country Assis- and education have been relatively good. But in tance Strategies from the most recent batch of private sector development, public sector man- CAEs. Second, it assesses potential revisions to the agement, and rural development, the Bank’s ef- CAE process, methodology, and presentation. forts have been less successful overall. Factors responsible for the unsuccessful outcomes in What Have We Learned? these sectors are: The first part of this retrospective summarizes some major findings and lessons from the 25 • Reforms in these sectors seem to face stronger CAEs completed during fiscal 2001–03, comple- opposition from vested interests and potential mented by additional insights gained from CAEs losers. completed in fiscal 2004. Overall, the lessons sug- • Institutional capacity constraints take time to gest that the Bank needs to understand the coun- resolve and many projects in these sectors at- try context better, deepen its country knowledge, tempt to improve upon the legal, institutional, and better calibrate its assistance programs to ob- and regulatory framework, but implementing servable progress on reform implementation. change requires overcoming inertia and ad- This retrospective finds that an evaluation at the verse incentives in the bureaucracies of many country level yields a more complete picture of the countries. outcome of the Bank’s assistance programs than • These sectors suffer larger adverse effects from do evaluations of individual programs or projects. exogenous events and macroeconomic shocks ix COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION than others. Improving outcomes in all sectors Fourth, adjustment lending can be successful, would imply focusing more on measuring and especially when combined with a strong govern- supporting results-based indicators. ment commitment to macroeconomic stabiliza- tion and structural reform; however, adjustment The retrospective has developed a number of lending in the absence of sustained progress on lessons for improving the formulation and im- stabilization and reform saddles the country with plementation of the Bank’s Country Assistance debt and weakens incentives to reform. The Bank Strategies. needs to resist pressures to persist with adjust- First, successful country programs are tai- ment lending in the absence of government com- lored to the country context and an under- mitment to reforms and a satisfactory track record standing of the political economy of reform is in implementing them. The evidence from OED’s essential. Domestic politics and vested interests evaluations of country assistance programs, how- largely determine the pace and content of re- ever, provides only limited proof that the Bank forms in countries. The retrospective found a resisted these pressures. In many countries, number of cases where an insufficient under- accommodating pressure to lend resulted in a standing of the political economy of reforms led number of unsatisfactory outcomes and, more the Bank to push reforms that stood little chance important, weakened the incentive to reform. of success. More active dialogue with national Finally, the retrospective recommends several governments, local government, and stakehold- strategies that would improve the outcomes of ers enhances the Bank’s understanding of politi- Bank assistance programs: cal economy considerations. Second, country knowledge is strongly asso- • Undertake more robust risk analysis to care- ciated with success. It is well understood that fully assess borrower commitment to reform project interventions are more successful when and implementation capacity. This should be they are based on in-depth analytical work. The informed by the analysis undertaken by others same finding conveys to the success or failure of and by the feedback obtained through wider country programs. The retrospective finds that dialogue with stakeholders. in more than two-thirds of successful country • Reduce the level of planned assistance when programs, the Bank’s analytical work was timely. faced with clear evidence of policy slippage. Analytical and advisory activities (AAA) can also The Bank needs to set clear and meaningful be an effective vehicle for engaging governments triggers for its assistance programs and cut in policy dialogue and informing civil society, but back its level of assistance when warranted. adequate attention needs to be paid to dissemi- • Lend more prudently in turnaround situations. nation. In many cases, the attention paid to dis- This is especially the case in the presence of semination has been inadequate. This failing can long-standing issues of implementation failure. also be a feature of participatory analytical work, Levels of assistance should be initially prudent as findings still may not be widely shared outside and calibrated to measurable outcomes and the government ministries that collaborated in meeting concrete benchmarks. The outcomes the analytical work with the Bank. of Bank lending are better when lending goes Third, a number of CAEs provide clear evidence up the ladder with the reform program, not of the role technical assistance and investment ahead of it. loans can play in promoting institutional develop- ment and capacity building. But the sustainability To summarize, optimistic projections or expecta- of benefits requires that these operations be part tions with inadequate risk analysis often weaken of a broader macroeconomic stabilization and re- the performance of country strategies. Programs form program. Linking technical assistance and in- should not be based on the best possible fore- vestment loans with policy reforms supported by casted outcomes. Finally, country strategies need adjustment loans also improves the probability of to be flexible, not rigid and narrow with only one success. path to follow. x EXECUTIVE SUMMARY How Can the CAE Instrument be Improved? retrospective is that the CAEs are playing a use- The primary purpose of an OED Country Assis- ful role, but revisions to the methodology, rating tance Evaluation is to draw lessons from the system, and interactions with involved parties Bank’s past activities to guide future strategies, would lead to a higher payoff in terms of accep- illustrating the usefulness of the CAE instrument. tance and utilization. Nevertheless, a Country Assistance Evaluation is OED’s own assessment of CAE methodology a complex task and while many multilateral and and an independent external evaluator concluded bilateral agencies now undertake country pro- that the methodological framework, while funda- gram evaluations, there is no “standard” method- mentally sound, had several shortcomings. First, ological approach. This second part of the CAE the structure tends to place too much emphasis retrospective analyzes OED’s current methodol- on the Bank’s instruments rather than on achieve- ogy and perceptions of the strengths and weak- ment of the program’s objectives. Second, differ- nesses of CAEs with a view to making further ent sections of the CAE can appear to lead to improvements in the methodology, evaluation different conclusions. For example, the outcome process, and structure of CAEs. rating of the overall assistance program may de- OED formalized its methodology for CAEs viate from the aggregate of project ratings, as in fiscal 2000 and the CAE approach became more noted earlier. Finally, the organization of the CAE standardized and consistent. The existing method- tends to shift focus from the country to the Bank ology starts with an evaluation of the relevance of and back to the country, which makes it hard to the Bank’s assistance strategy given the country’s follow the story line. OED’s assessment is also development needs and challenges, including consistent with the perceptions of CODE mem- whether any key development constraints were bers who felt that CAEs focused too much on for- omitted. It then evaluates the assistance program mal compliance issues and the impact of Bank across three dimensions: a bottom-up review of instruments to the neglect of the Bank’s devel- the Bank’s products and services (lending, analyt- opment impact and the results achieved. ical and advisory activities, and partnerships); a This retrospective concludes that the way for- top-down review of whether the Bank’s program ward is the adoption of a results-based approach, achieved its objectives or planned outcomes and which clearly links the Bank’s objectives, instru- had a substantive impact on the country’s devel- ments, and the outcomes achieved. The approach opment; and analysis of the contributions of the would outline the key objectives and results that Bank, borrower, other donors, and exogenous the Bank is trying to achieve in each country, based events to the outcomes. on the CASs. The strategy discussion would be For the retrospective, OED undertook a num- based on a simple logic model that links inputs ber of usage surveys. All objective measures—a with expected results based on the information tracer study of CAE recommendations, discussion presented in the CAS itself and related docu- of CAE findings and recommendations in sub- ments. For each objective (and intended result), sequent Country Assistance Strategies (CASs), the CAE would evaluate the efficacy (and re- and Board discussions of subsequent CASs— levance) of the instruments used, as well as the indicated that CAEs are frequently referred to relative contribution of the Bank (as opposed to and their recommendations figure prominently other contributions) to that objective. This ap- in the Bank’s CASs. Members of CODE, who proach would address the Board members’ main were interviewed for this retrospective, felt that recommendation that the CAEs should focus on CAEs were useful, but were skeptical that Bank actual impact at the country level. staff were using the CAE findings to the maxi- CAEs currently contain ratings of the out- mum extent. This was reflected in Bank staff come of Bank assistance, its sustainability, and views, which were on the whole much less pos- institutional development impact. The CAE rat- itive, but were more positive when they per- ings have been the subject of considerable de- ceived that a CAE had had an impact on the bate and criticism, which to some extent reflects Bank’s strategy. The conclusion reached in the the long-standing debate within the evaluation xi COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION community on learning versus accountability. subsequent CAS Board discussions had CODE CODE members generally favored the ratings, discussions within six months of the CAS, and arguing that they held the greatest potential for two-thirds had discussions within a year. More- ensuring management attention to the CAE re- over, despite the perception of a lack of time- sults. The majority of Bank staff were critical of liness by Bank staff, all 22 of the CAEs discussed the ratings, at times citing apparent inconsisten- by CODE before Board discussion of the CAS cies between the ratings and the narrative, or, in were mentioned in those CASs. OED will con- other instances, citing a failure to rate different tinue its policy of delivering CAEs to the Board periods differently. But the key issue that arose and Bank staff approximately three to nine months is that many readers incorrectly perceive the ahead of planned CAS discussions, but notes that outcome rating as either a rating of the country’s the unpredictable scheduling of CAS discussions development performance or as a rating of the is a major challenge in meeting this performance Bank’s performance. More needs to be done to target. make clear the nature and basis of the CAE Finally, there was a widespread perception ratings. that more interactions with country teams dur- OED is reviewing the CAE ratings and is also ing CAE preparation and with in-country benefi- discussing with management the regularization ciaries would enhance both the quality of the of self-ratings for CAS Completion Reports CAEs and the acceptance of their findings. While (CASCRs) validated by an OED review. If such a this retrospective finds that there have been many system, akin to the project rating system, were more interactions with in-country beneficiaries introduced for all CASs, then ratings for CAEs than perceptions indicate, OED will make an ef- may become redundant. fort to ensure more uniformly that in-country The timing of the CAE is crucial to its useful- beneficiaries are consulted and their views in- ness. Bank staff prefer a CAE early in the CAS corporated. Similarly, the retrospective also finds process, and CODE members prefer a CODE dis- that a more systematic interaction with country cussion close to the discussion of the relevant teams during the process of preparing the evalu- CAS. This retrospective found that the large ma- ation may help clarify and improve the accep- jority of CAEs had been timed to appropriately tance of CAE findings and their incorporation in inform CAS preparation and discussions. OED’s country programs. Therefore, OED proposes to review of the timing of the CAEs completed dur- involve the country team in midterm reviews of ing fiscal 2001–03 indicates that more than 50 per- future CAEs to receive early feedback on emerg- cent of the CAEs for which there have been ing findings and recommendations. xii Acronyms and Abbreviations AAA Analytical and advisory activities AfDB African Development Bank APLs Adjustable Program Loans ARDE Annual Review of Development Effectiveness AROE Annual Report on Operations Evaluation CAE Country Assistance Evaluation CAS Country Assistance Strategy CASCR Country Assistance Strategy Completion Report CASPR Country Assistance Strategy Progress Report CDs Country directors CODE Committee on Development Effectiveness CPIA Country Policy and Institutional Assessment DAC Development Assistance Committee DFID Department for International Development (U.K.) EDs Executive directors ESW Economic and sector work FTB Fast Track Brief HIPC Heavily indebted poor country IBRD International Bank for Reconstruction and Development IDA International Development Association IDI Institutional development impact IDB Islamic Development Bank IMEP Integrational Monitoring and Evaluation Plan LICUS Low-income countries under stress LILs Learning and Innovation Loans NGO Nongovernmental organization OECD Organisation for Economic Co-operation and Development OED Operations Evaluation Department PSD Private sector development PSM Public sector management UNICEF United Nations Children’s Education Fund xiii 1 Introduction T his report is a self-evaluation of the Operations Evaluation Department (OED) Country Assistance Evaluations (CAEs). In a 2002 discussion of CAE methodology, the Informal Subcommittee of the Committee on Development Effectiveness (CODE) noted the important role CAEs play in as- sessing Bank interventions at the country level and the need to continue to refine the methodology. This retrospective responds to that discussion linkages between these results and the program and has two primary purposes. First, it addresses supported by Bank initiatives. the question “What have we learned?” by com- In keeping with the Bank’s shift from a piling lessons relevant for developing Country project to a country focus, OED first began Assistance Strategies from the most recent batch evaluating country programs in fiscal year of CAEs. Second, it assesses proposed revisions 1995. The methodological approach evolved, to CAE process, methodology, and presentation building upon OED’s project methodology. By fis- to answer the question “How can the CAE in- cal 2000, the methodological approach was for- strument be improved?” malized,1 and in fiscal 2002, as noted above, the CAE methodology was discussed at a CODE sub- Background committee meeting. As a result, the CAE approach The focus of the Bank’s development assis- became more standardized and consistent. tance is now firmly anchored at the coun- The existing methodology starts with an eval- try level. Country Assistance Strategies (CASs) uation of the relevance of the Bank’s assistance have become the centerpiece of the Bank’s coun- strategy given the country’s development needs try programs and place the Bank’s operational and challenges, examin- work in the context of the country’s own strategy ing, among other mat- Country Assistance and development objectives. The Bank is cur- ters, whether any key Strategies (CASs) have rently strengthening its strategy formulation pro- development constraints become the centerpiece of cess through the adoption of a results framework were overlooked. It then for CASs to define more clearly the country re- evaluates the assistance the Bank’s country sults that the Bank seeks to influence, as well as program across three di- programs. 1 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION mensions. The first is a bottom-up review of the offset by transaction costs and added time needed Bank’s products and services (lending, analytical for consultation and coordination. and advisory activities [AAA], and aid coordina- tion), which are used to achieve Bank objectives. CAE Retrospective Evaluation Approach The next step is a top-down review of whether To achieve the two objectives of the retrospective, the Bank’s program achieved its objectives or OED commissioned a number of background planned outcomes and had a substantive impact studies to assess the lessons learned, the CAE on the country’s development. The first and sec- methodology, and the usage of CAEs by the ond steps test the consistency of findings Board, CODE (and the CODE subcommittee), regarding the product and services and devel- and Bank staff. opment impact dimensions. Finally, the CAE at- tributes responsibility for the country strategy • OED staff reviewed the 25 CAEs completed be- outcomes to the Bank, the borrower, other tween fiscal 2001 and 2003 to ascertain lessons donors, and exogenous events. applicable across CAEs. The analysis and find- By the end of fiscal 2004, OED had issued ings of this review are included in Part I. CAEs assessing the impact of Bank assist- • OED commissioned an independent consul- ance to 64 countries.2 These evaluations cover tant to conduct qualitative research, consisting roughly 47 percent of borrowers and 76 percent of in-depth interviews with Board members, of total gross commitments. This retrospective CODE and CODE subcommittee members, focuses primarily on the 25 CAEs completed in and other participants in CAE discussions at fiscal 2001–03, but also includes some insights CODE or subcommittee meetings. In total, gained from the additional eight CAEs completed 30 people were interviewed. during fiscal 2004. • Two tracer studies were undertaken. Tran- Five joint CAEs have been undertaken scripts of Board discussions and written state- with other multilateral institutions to fos- ments by executive directors about CASs or ter the sharing of experiences and evalua- CAS Progress Reports (CASPRs) were reviewed tion harmonization. The Lesotho and Rwanda to assess the impact of CAEs on subsequent CAEs were prepared jointly with the African De- CAS discussions. Also, desk reviews of 25 CASs velopment Bank (AfDB); the Jordan and Tunisia (or CASPRs) produced following the CAEs CAEs were prepared jointly with the Islamic De- were analyzed to determine the timeliness velopment Bank (IDB), and the Peru CAE was of CAEs and the utilization of their recom- prepared in parallel with the Inter-American mendations. Development Bank. • OED commissioned the same independent The primary purposes of these joint exercises consultant who conducted the CODE inter- have been to promote learning on both sides and views to conduct a similar qualitative exercise encourage the development of common evalua- tion standards, consistent with the broader har- with Bank staff. In-depth interviews were con- monization agenda of the Bank and other donors. ducted with 20 current and former country di- For example, the AfDB recently revised its guide- rectors associated with one or more of the lines for Country Assistance Evaluations, closely 25 CAEs completed in fiscal 2001–03, five cur- aligning them with OED’s existing methodology. rent and former Regional vice presidents, and Joint country evaluations have also been effective 18 additional Bank staff. in identifying key constraints and gaps in donor • A professional evaluator examined the current assistance. For example, the AfDB evaluation of CAE methodological approach and proposed its Rwanda program highlighted the lack of a methodological changes, which would, in par- post-conflict policy. Finally, joint country evalua- ticular, focus CAEs on results related to CASs. tions are one avenue to reduce the burden of • A desk review was undertaken of country eval- multiple, separate donor evaluation efforts on re- uation methods of other bilateral/multilateral cipient countries. But these benefits are partially agencies to identify good practices. 2 2 Part I: What Have We Learned? T his chapter summarizes the findings and lessons from the 25 CAEs completed in fiscal years 2001–2003. It also includes some insights gained from the additional 8 CAEs completed during fiscal 2004.3 These 25 CAEs are not a random or a rep- performance (OED 2004a, page 15). One weak- resentative sample. CAE countries are selected ness is that the coverage of Latin American and based on several factors. The most important is South and East Asian countries is scant, primar- the timing of the Bank’s next Country Assistance ily because evaluations of countries in these Re- Strategy (CAS), but other factors are also consid- gions were fairly complete prior to fiscal 2001. ered, such as OED’s ability to collaborate with other international financial institutions, Regional Findings balance, and whether a previous CAE had already CAE outcome ratings can deviate from the covered a country. So, while the countries cannot aggregate of project outcomes. In about a third be called a random sample of Bank borrowers, of the CAEs, the outcome rating of the Country As- these 25 CAEs constitute a reasonable basis for this sistance Strategy was unsatisfactory, but the aggre- “findings and lessons” discussion (see table 2.1). gation of project outcomes in the country during There is a good mix of low- and middle-income the CAE period was satisfactory (table 2.2). This is countries, of International Bank for Reconstruc- not surprising. The CAE is a comprehensive evalu- tion and Development (IBRD) and International ation of the Bank’s program in a country that com- Development Association (IDA) borrowers, and prises both projects and analytical and advisory of Regions. Four post-conflict countries are in- activities. Moreover, CAEs cluded. The sample also contains a good mix of must make an assessment CAE outcome ratings can countries with outcome ratings that were satis- of overall Bank strategy, deviate from the aggregate factory (about three-fifths), and of those with rat- including size, sectoral of project outcomes. ings that were unsatisfactory. It is fairly evenly composition, and type of divided between good performers and weak per- lending. For example, the CAE outcome may be formers, as measured by the Bank’s country pol- unsatisfactory if there are critical omissions in the icy and institutional assessment (CPIA) rating. As Bank’s overall assistance strategy, even if the out- noted in the 2003 Annual Review of Develop- comes of individual projects are rated satisfactory. ment Effectiveness (ARDE), there is a positive cor- CAEs found that Bank programs were relation between CAE outcome ratings and policy more successful in education and health 3 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION Characteristics reflected in aggregate project ratings for the Table 2.1 of the 25 CAE same set of 25 CAEs (table 2.3). The only signif- Countries icant divergences between sectoral ratings in CAEs and project outcome ratings were in the Income level areas of private sector development (PSD), rural Low 10 development, and public sector management Lower-middle 12 (PSM), where the relatively unfavorable out- Upper-middle 3 comes in the CAEs are in contrast to the high 25 proportion of satisfactory outcomes at the proj- IBRD, IDA, or Blend ect level. IBRD 14 The divergence in success ratings at the IDA 8 project and sector levels in the areas of pri- Blend 2 vate sector development, rural development, and public sector management could re- 24a flect several factors. First, the political eco- Post-conflict or not nomy of reforms in these sectors is more Yes 4 problematic, and opposition from poten- No 21 tial losers and vested interests is likely to 25 be more focused and sustained. Individual b CPIA level projects may be successfully implemented, but it Above average 13 will take a longer and more sustained effort to Below average 11 overcome political opposition and achieve suc- 24c cessful outcomes at the macro level. For example, Region in transition economies, the historical antipathy Africa 12 to private business on the part of the nomen- East Asia and the Pacific 2 klatura has strong ideological roots and is not South Asia 1 easily overcome. Despite legal and regulatory re- forms, there remains an ingrained bias against Europe and Central Asia 5 private business in many of these countries. In Middle East and North Africa 3 the less-developed economies of Asia, Africa, and Latin America and the Caribbean 2 Latin America, major rural development initia- 25 tives often encounter conflicts of interest among OED outcome ratings powerful groups (such as big farmers versus la- Highly satisfactory 2 borers, or rural dwellers versus urban population) Satisfactory 9 that take time and political acumen to resolve. Moderately satisfactory 8 Second, even if there was no strong poli- Moderately unsatisfactory 3 tical opposition, these areas are crucially Unsatisfactory 9 dependent on institutional reforms and ca- 31d pacity building, which take time and are a. For West Bank and Gaza, the financing comes through a World Bank Trust Fund and is on IDA or not always captured even in the time frame grant terms. of a CAE. For example, private sector develop- b. CPIA = Country Policy and Institutional Assessment. ment depends on an effective judicial and court c. No CPIA for West Bank and Gaza. system, which takes time to develop. Civil service d. Four countries received different ratings for subperiods. reform takes time to implement and the results take even longer to materialize. Thus successful than in other sectors, and least successful outcomes at the sector level will take time to in private sector development, rural devel- materialize. opment, environment, and social protec- Finally, outcomes in these areas are more tion (table 2.3). These results are also largely dependent on economy-wide developments 4 PART I: WHAT HAVE WE LEARNED? and exogenous factors. The growth of the private sector depends not just on the legal and CAE Outcome regulatory framework for private sector develop- Table 2.2 Ratings and Project ment, but also on progress in other areas of Outcome Ratings economic policy such as macroeconomic sta- bilization, infrastructure, and the financial sec- CAE outcome ratings (%) tor. Noneconomic conditions within the country, Country portfolio outcomes Satisfactory Unsatisfactory such as law and order, and external events, such Satisfactory 53 33 as prevailing market sentiment in the region, Unsatisfactory 7 7 also play a major role in private sector develop- Source: World Bank database. ment. Rural development and reforms in the public sector are also influenced significantly by economywide developments. (This is not to say because the government underestimated oppo- that outcomes in other sectors do not depend sition to reforms and was unable to carry them to some extent on such factors. It is a matter of through. An insufficient understanding of the po- degree.) litical economy of reforms and the nature of the state may have led the Bank, in some cases, to Lessons Learned push reforms that stood Lesson one: An understanding of the political little chance of suc- An insufficient economy of reforms, including government com- cess. For example, in Zim- understanding of the mitment and ownership of reforms and the de- babwe, one of the princi- political economy of gree of political support for or opposition to pal motivations behind reforms and the nature of them, is essential to develop realistic Country As- the Government’s eco- sistance Strategies, specific assistance programs nomic and social policy the state may have led the and projects, and analysis of risks. was to ensure indigenous Bank, in some cases, to Economic reforms often failed either because ownership of productive push reforms that stood the government was not committed to them, or assets. A proper appre- little chance of success. Sectoral and Project Ratings for Table 2.3 Fiscal 2001–03 CAEs Project outcomes CAE sector outcome ratings (%)b Sector (% satisfactory)a Satisfactory Unsatisfactory Mixed Not assessed Education 100.0 48 8 4 40 Health 86.5 40 20 8 32 Social protection 37.9 12 24 20 44 Environment 25.5 16 16 24 44 Rural 87.2 12 36 28 24 Financial sector 81.6 24 16 32 28 Infrastructure 87.7 36 16 24 24 Public sector 99.9 24 32 28 16 Private sector development 60.1 12 24 44 20 a. This is the percentage of satisfactory project outcomes by sector (commitment value) for the countries where CAEs were completed during fiscal 2001–03. b. These columns are ratings by CAE task managers and a desk review of CAEs of the outcomes by sectors. In most cases, explicit ratings by sectors were not included in the CAEs and thus, these ratings should be treated as indicative only. Source: World Bank database and OED staff estimates. 5 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION Institutional reforms and ciation for the im- forms in a sector often depend on complemen- portance of this issue tary reforms and success in other sectors. The capacity building for might have led the Bank sequencing and packaging of reforms need to effective governance are to give priority to land re- take account of this complementarity. critical to successful forms. The Govern- Several CAEs noted that more successful out- outcomes. ment’s reluctance to comes could have been achieved had more undertake parastatal and attention been given to the complementarity of civil service reform would also have been better reforms in different sectors. Private sector de- understood. Privatization and civil service re- velopment cannot be assured simply through pri- forms in Peru were threatened by opposition vatization. It also depends on reform of the public from the middle class. Their opposition could sector through deregulation, changes in taxation policy and administration, and anti-corruption have been lessened by a different sequencing of reforms. For example, in Mongolia, important reforms and social protection measures to miti- measures were implemented to promote private gate the adverse consequences of privatization sector development—such as enabling private and civil service reforms. property, removal of price and margin controls, Lesson two: Institutional reforms and ca- reduction in trade barriers, simplification of tax pacity building for effective governance are regime, and improved framework for foreign in- critical to successful outcomes.Since capacity vestment—but not enough was done to lighten building takes time, these reforms need to start the heavy hand of the state on the economy. In- early, and be followed through over several efficient government regulatory and oversight years. functions continued to hamper private sector Institutional development is at the core of de- development. velopment effectiveness. Stronger institutions As several CAEs noted, the growth of the pri- are associated with a 20-percentage point in- vate sector depends very much on an effective crease in the likelihood of a project’s outcome judiciary. The effectiveness of financial sector re- being rated satisfactory (OED 1999). Successful forms in improving financial intermediation has reform outcomes are often undermined by weak also been shown to depend critically on reforms institutions or the absence of important institu- in the enterprise sector and the public sector. tional arrangements. For example, experience shows that as long as For example, in the transition economies of enterprises are being bailed out by the public central and eastern Europe and the former Soviet sector (through explicit or implicit subsidies and Union, the existing institutions were not designed build-up of tax arrears), they have less incentive to borrow from banks. And on the supply side, for a market economy. As the command economy banks are reluctant to lend to enterprises that collapsed, and market-oriented reforms were survive only because they are subsidized by the implemented, there was a lag before the right public sector. Either way, financial intermedia- institutions developed, which prevented the full tion is stunted. benefits of the reforms from emerging.4 Institu- Lesson four: ESW must be timely, of good qual- tional reform goes beyond changing organi- ity, and be fully integrated into the design of zational structures and rules; it also involves the Bank strategy, programs, and projects. Timely discarding of long-established habits and pat- ESW is particularly important for first-time or re- terns of behavior—a complicated and lengthy newed borrowers and for stop-go reformers. ESW must be timely, of process. Major institu- ESW can play an important role in developing good quality, and be fully tional changes may have the Bank’s assistance strategy and in enhancing to be spread over several its effectiveness. Two-thirds of the CAEs that re- integrated into the years.5 ported favorable outcomes also reported that design of Bank strategy, Lesson three: Success- the ESW was timely. The same CAEs reported un- programs, and projects. ful outcomes from re- favorable outcomes when the ESW was not. 6 PART I: WHAT HAVE WE LEARNED? In Brazil, timely ESW helped the Bank to direct Lesson five: Specific investment and technical lending toward education, health, and rural po- assistance loans can be useful vehicles for pro- verty projects in the Northeast with maximum im- moting institutional reforms, but for benefits to pact on poverty reduction. And in Vietnam, ESW be sustainable, these operations should be part demonstrated that rural poverty was strongly of a broader stabilization and economic re- associated with inadequate economic infrastruc- form strategy, and must be supported by strong ture, helping orient the Bank’s lending program governmental commitment to reform. toward economic infrastructure. About 22 of the 25 CAEs provide strong evi- Examples of less satisfactory contributions of dence of the potential for specific investment ESW include the case of Kazakhstan, where a lending and technical assistance loans in pro- social protection project sought to mitigate the moting institutional development in both low- social impact of privatization by strengthening income and middle-income countries.6 Twenty the institutional capacity of unemployment ser- of these countries also received adjustment loans. vices to streamline procedures for registration In many countries, specific technical assistance and payment of unemployment benefits. The and institution building loans were linked closely poverty assessment that came two years later to adjustment loans, helping build capacity to showed that policies to facilitate labor mobility formulate and implement policy changes sup- and to equip workers for changed circumstances ported in adjustment loans. Strong government commitment and clearly were required, rather than a strengthening of articulated priorities remained a key factor in capacity of unemployment services. In Bulgaria, the effectiveness of these instruments in pro- a timely poverty assessment might have en- moting institutional development. In this envi- abled a social protection loan to more effec- ronment the longer time frame of investment tively address targeting of social assistance to loans allowed the Bank to build relationships the needy. with counterparts, and to combine advice with In a few countries, ESW was timely and of high financial assistance especially to sector minis- quality, but findings were either not fully taken tries. Conversely, when strong and sustained into consideration in designing strategies, or used government commitment to institutional reform selectively, thus reducing their relevance and is absent, Bank assistance is not likely to be effectiveness. For example, in Jordan, Bank lend- successful.7 ing during the 1990s did not address the high Lesson six: Adjustment lending in the absence level of government expenditures, despite ana- of sustained progress on reforms only saddles the lytical work identifying how these expenditures country with additional debt and can weaken could be reduced and better targeted. In Peru, incentives for future reforms. the Financial Sector Reform Loan (fiscal 1999) did Adjustment lending can be successful, espe- not address fundamental problems in the sector, cially when combined with a strong government which had been correctly identified in an ongoing commitment to macroeconomic stabilization and assessment of the financial sector. structural reform. However, the Bank should re- ESW can build a good knowledge base and sist pressures to persist with adjustment lending help the Bank engage governments in its policy di- in the absence of the government’s commitment alogue. It can also assist a government to under- to reforms, and a satis- Adjustment lending in the stand the costs and benefits of reforms and help factory track record in im- design its reform program. The long-term impact absence of sustained plementing them. The of ESW should also be borne in mind, as it initi- rationale for adjustment progress on reforms only ates debate on policy issues and serves to famil- loans was to provide fi- saddles the country with iarize civil society with reform issues. But it goes nancing to alleviate the additional debt and can without saying that ESW cannot catalyze policy cost of adjustment that oc- change when governments lack commitment to curred when structural re- weaken incentives for reform. forms were implemented. future reforms. 7 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION However, the review of CAEs for this retro- tic growth projections in several countries (Jordan, spective showed that adjustment lending was Kazakhstan, Kyrgyz Republic, and Zambia). Unre- appropriately delayed in only two countries (Bul- alistic growth estimates contributed to inappropri- garia and Brazil) when reforms stalled. In as ate Bank assistance strategies and entailed real many as seven other countries, the Bank went costs for the countries. Had growth estimates been ahead with adjustment loans even though little realistic, the Bank would have likely concentrated progress was being made towards the Bank’s more analytical work on the sources and con- assistance objectives and corporate goals.8 straints to growth, as well as on poverty reduction In these countries there were pressures to measures. Realistic growth projections would also lend for a variety of reasons: to exploit a unique have illustrated more clearly debt sustainability is- “window of opportunity” (Kenya), show support sues and greater efforts would have been made to to the government (Morocco), maintain relation- seek debt relief or other forms of concessional ships (Zimbabwe), avert a potential crisis (Peru), financial assistance, in order not to impose too avoid a return to communism (Russia), and pre- high a burden of external debt;12 and countries vent negative net flows (Zambia). While these fac- may have been persuaded to undertake deeper re- forms to accelerate economic growth. tors may well have influenced the Bank, there is Lesson eight: The Bank should be more pre- little doubt that in some of these countries the pared to reduce the level of planned assistance pressure to reform may have been further di- when faced with clear evidence of policy slippages. luted by the Bank’s decision to lend, saddling In addition to identifying risks, the Bank needs these countries with debt.9 to be prepared to modify its assistance program Lesson seven: Thorough, hard-headed, and to reflect wavering government commitment or realistic risk analysis is important to increase policy slippage. A positive example is Bulgaria in the realism of country strategies. the mid-1990s, where lending was scaled down The Bank needs to be more realistic and hard- and a major adjustment loan was put on hold in headed in its country assessments and country the face of rising macroeconomic risks and a lack strategy formulations, including assessing a bor- of government commitment to address reform rower’s commitment to reforms and its imple- issues. But the review of CAEs suggests this did not mentation capacity, receptiveness to Bank advice, happen in a number of cases. In Peru, the Bank’s and the impact of reforms on growth and poverty program did not contain triggers to reduce lend- alleviation. Consistent errors of over-optimism on ing if risks materialized. In Kenya, the 1998 the part of the Bank regarding borrower recep- strategy also recognized risks and the program tiveness to Bank advice, willingness to undertake contained CAS benchmarks, but the bank did not difficult reforms, and government’s capacity to follow through when the benchmarks were not implement reforms has plagued Bank strategies in met. In Morocco, the Bank provided a policy re- many of the countries evaluated (Haiti, Jordan, form support loan in the late 1990s as a way of re- Kazakhstan, Kenya, Kyrgyz Republic, Lesotho, Mo- warding the country’s movement toward a more rocco, Paraguay, Peru, Zambia, Zimbabwe). This open political system and commitment to reform. optimism often persisted in the face of contrary The loan was too unfocused to have a major im- evidence10 and contributed to lending decisions pact on any of the critical reform areas identified by the Bank that failed to meet their objectives.11 in the country strategy. Many of the actions taken Country assistance strategies in many coun-tries prior to Board presentation were first steps, some- The Bank should be more assumed a much stronger times in the form of studies or plans, and many growth performance than others did not show concrete results. prepared to reduce the warranted by past coun- Lesson nine: The Bank should be especially level of planned assistance try experience or experi- cautious in turnaround situations. Realistic when faced with clear ence of other countries country assessments rather than wishful think- facing similar constraints ing should inform its assistance strategy. evidence of policy and prospects. Evalua- Typically, the Bank has difficulty in identifying slippages. tions pointed to unrealis- “turning points” and in calibrating its response to 8 PART I: WHAT HAVE WE LEARNED? changing country conditions. It tends to react Missions, its contacts with Typically, the Bank has slowly to deterioration and too quickly to im- civil society, and relevant provements. In the Dominican Republic, for ex- ESW work to gain a better difficulty in identifying ample, the Bank failed to recognize an upturn understanding of the po- “turning points” and in during fiscal 1992–95 and failed to support the litical economy. As far as calibrating its response to government during a crucial period of successful possible, a government’s changing country economic reform. This mistake cost the Bank its commitment should be as- influence in the country. The Bank failed to sessed on the basis of its conditions. It tends to recognize the downturn in Peru in 1997 and track record in imple- react slowly to continued the support it was rendering earlier menting reforms, not dec- deterioration and too when reforms were implemented. In Kenya, ini- larations of intent. Levels tial steps toward reforms in mid-2000 were pre- of assistance should be quickly to improvements. maturely identified as an upturn and rewarded initially prudent and calibrated to measurable with increased lending; reforms ultimately outcomes and meeting concrete benchmarks. stalled. This is especially the case in situations where To overcome this, the Bank needs to keep its there are long-standing issues of implementation ears closer to the ground through its Resident failures. 9 3 Part II: How Can the CAE Instrument Be Improved? T he primary purpose of an OED CAE is to draw lessons from the Bank’s past activities to guide future strategies. Part I of the retrospective syn- thesized a number of lessons and findings generated by the most re- cent set of OED CAEs. These lessons, which accord well with the findings of other OED evaluations, illustrate the usefulness of the CAE instrument. Never- theless, a CAE is a complex task, and while many multilateral and bilateral agencies now undertake country program evaluations, there is no “standard” methodological approach.13 This second part of the CAE retrospective ana- studies have been conducted to inform the analy- lyzes OED’s current methodology and percep- sis. Detailed findings from the surveys of CODE tions of the strengths and weaknesses of CAEs members are contained in Annex C; from the with a view to improve the methodology, evalua- analysis of CAS Board discussions in Annex D; tion process, and structure of CAEs. This chapter from the interviews of Bank staff with familiarity discusses the impact of CAEs on country strategy of CAEs in Annex E; and from the analysis of the formulation, the rating system, the use of counter- methodology of country program evaluations by factuals, the timing of CAEs, interactions with other agencies in Annex F. Each annex details the country teams and in-country beneficiaries, and objectives of the survey and describes the partic- the presentation of CAEs. Each section describes ipant sample, as well as the views and findings. the perception of CAE users, the results of tracer studies and OED’s own review, and concludes Impact of CAEs on Country with suggestions for improvements. Strategy Formulation The analysis in this chapter is based on survey Earlier studies have indicated that CAEs do research regarding the usage of and perceptions influence Bank Country Assistance Strate- of CAEs by the Bank’s Board of Directors and gies and there is consistency between CAE Bank staff, as well as the assessment of an external recommendations and CASs. OED’s 2002 An- evaluator, who reviewed the CAE methodology nual Report on Operations Evaluation (AROE)14 and a set of recently completed CAEs. As noted in undertook a survey of Bank country teams on the introduction, a number of surveys and tracer CAEs. About 71 percent of respondents thought 11 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION Recent analysis confirms that the CAEs had influ- this retrospective, OED reviewed Board tran- enced Bank policies at scripts from 13 CAS discussions, for which the that CAE recommendations most recent CAEs were available as of April 2003.16 the country level, 80 per- are well reflected in cent thought that a CAE About one-third of chairmen’s statements, either subsequent CASs. is relevant to their work, written or oral, explicitly referred to the CAE at and 93 percent said that least once. In total there were 85 references to if they were to start work on a new country, they CAEs in the 13 discussions. About a quarter (21) would find a CAE to be quite useful. For the same of these were very general references, while the AROE, OED carried out a tracer study to assess remaining 75 percent (64) were issue-oriented. the extent to which CAE recommendations on References to CAEs were more frequent during a country were incorporated into the subsequent full CAS discussions or when the outcome rating CAS, and were reflected in the assistance actually was unsatisfactory; chairs representing Part I con- delivered. The study reviewed 15 CAEs, all com- stituencies made the largest number of references pleted in fiscal 1999 or earlier to ensure that to CAEs. In total, references to CAEs comprised enough time had passed to permit a review of only a small fraction (6 percent) of the multitude assistance actually delivered. The study found of issues raised during CAS Board discussions. strong consistency between CAE recommenda- Nevertheless, even when a CAE was not refer- tions and the CAS, reinforcing the message from enced, there was alignment of the issues raised in the survey. the CAEs and the Board discussions of the re- spective CASs and Progress Reports. Recent analysis confirms that CAE re- There is a divergence between the results commendations are well reflected in sub- of the tracer surveys and the perceptions sequent CASs. For this retrospective, OED by the Board and Bank staff of CAE usage undertook a new tracer study for the 25 CAEs and usefulness. The usage surveys indicate that completed during fiscal 2001–03. This analysis CAEs are known to both Board members and reviewed the CASs which immediately followed Bank staff. CODE members interviewed for the the CAEs to assess to what extent the formal rec- retrospective noted that they used CAEs to pre- ommendations included in the CAE’s manage- pare for CAS Board discussions, to place sectoral ment action records are supported. Of the 22 CAEs and thematic issues in a country context, and to for which subsequent strategy documents have assess CASs. Some CODE members cited specific been prepared,15 two-thirds of the CAE recom- instances in the interviews when CAEs positively mendations are fully supported in the Country influenced Bank country strategies. Nevertheless, Assistance Strategies. Another quarter are par- a majority doubted that CAEs exerted a signifi- tially supported and fewer than 10 percent were cant influence on management decisions or pol- not supported at all. Also, all of the 22 CASs con- icy issues, noting that a lack of endorsement by tained some reference to the relevant CAE; in 16, senior management of CAE findings served to the CASs contained extended discussions of CAE undermine their impact. This “lack of endorse- findings and recommendations. Thus, it does ap- ment” was apparent in interviews with Bank staff, pear that the CAEs are having an impact on the including regional vice presidents and country Bank’s strategy formulation, although it is not directors. Their perceptions of the usefulness of possible to attribute the support for any individ- CAEs were less positive, but were more positive ual recommendation to the analysis contained in when they perceived that a CAE had had an im- pact on CAS preparation. Bank staff identified sev- CAEs also appear to have the CAE. CAEs also appear to eral factors inhibiting the effectiveness of CAEs: a had an impact on Board have had an impact on lack of timeliness, inadequate consultation with discussions of subsequent Board discussions of country teams, and a lack of useful insights or lessons for Bank involvement in a country. CASs, and the issues raised subsequent CASs, and the issues raised in Reducing the divergence and improving in CAEs are aligned with CAEs are aligned with the utilization of CAEs could be enhanced Board discussions. Board discussions. For by changes to the methodology, presenta- 12 PART II: HOW CAN THE CAE INSTRUMENT BE IMPROVED? tion and timing of CAEs, as well as to the fre- outcome to four cate- Reducing the divergence quency of interactions with country teams. gories of actors: (a) the and improving the Despite the fact that CAEs figure prominently in client; (b) the Bank; CASs and CAS Board discussions, our surveys (c) partners and other utilization of CAEs could raise a number of issues about CAEs. Board mem- stakeholders; and (d) be enhanced by changes bers expressed some skepticism over whether exogenous forces (e.g., to the methodology, operational staff were utilizing CAE findings to events of nature, in- the maximum extent possible. Bank staff have also ternational economic presentation and timing raised concerns about the underlying methodol- shocks, and the like.). of CAEs, as well as to ogy, the timing of CAEs and the balance between the frequency of accountability and learning in their content. All With the introduction of this formal method- interactions with country these factors require careful review, and, where appropriate, resolution, so that CAEs can have a ology, CAEs became teams. higher payoff in terms of utilization and accep- more similar in struc- tance of their findings and recommendations. ture and content. Each CAE begins with an The remainder of this Chapter will address the examination of the overall development situa- most salient issues identified from the usage sur- tion and the relevance of the CAS objectives. This veys: the CAE methodology; CAE ratings; use of is followed by an in-depth evaluation of all prod- ucts and services used to achieve Bank objec- a counterfactual; timing; interaction with coun- tives. The third section evaluates whether the try teams; interaction with in-country beneficia- Bank’s objectives have been achieved and con- ries; and CAE presentation. tains the CAE ratings. The contributions of the different partners to the results obtained are as- Methodology sessed, and the CAEs conclude with recommen- The adoption of a formal methodology for dations and lessons. CAEs in fiscal 2000 has led to a more con- Despite the adoption and dissemination sistent approach to Country Assistance of a formal CAE methodology, interviews Evaluations.17 The CAE methodology uses the revealed a perception that the CAE method- Country Assistance Strategy (CAS) for its initial ology could be improved and made more evaluative framework, supplemented by judg- transparent and consistent. A majority of the ments about whether the strategy omitted ad- CODE participants felt that CAEs were consis- dressing any key development constraints. The tently of high quality and contained substantive CAE gauges the extent to which the CAS’s major analysis of the Bank’s country programs. Never- strategic objectives were relevant and achieved, theless, concerns were raised about inconsisten- without any shortcomings. cies in the findings and variations in quality. CODE The current CAE methodological approach is members expressed concerns that the documents to examine the country assistance program across are sometimes too focused on monitoring com- three dimensions: pliance with formal institutional objectives to the neglect of the Bank’s development impact on • A Products and Services Dimension, involv- the country. They felt that a CAE should focus on ing a “bottom-up” analysis of major program the results achieved and on how instruments were inputs—loans, AAA, and partnerships/aid employed to achieve the results. Bank staff also coordination raised concerns about the CAE methodology. • A Development Impact Dimension, involving They felt that the methodology was not clear, a “top-down” analysis of the principal program standardized, consistent, or streamlined. More- objectives for relevance, efficacy, outcome, over, they contended that too few CAEs were sustainability, and institutional impact generating findings and lessons that provided • An Attribution Dimension, in which the eval- pragmatic guidance for future program manage- uator assigns responsibility for the program ment. Some Bank staff expressed a preference 13 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION The CAE structure tends to for evaluations focusing those objectives were to achieving the overall on a limited number of program goals in force at that time) the CAE place more emphasis on specific sectors, rather structure is not conducive to assessing the un- the Bank’s instruments than attempting to eval- derlying reasons for this disconnect. (lending, AAA, aid uate an entire country • The structure shifts focus from one section to coordination) than on the program. the next, which makes it hard to follow the OED’s own assess- story line. The current CAE structure alternates impact or results of the ment also finds that from a focus on the client (first section), the Bank’s interventions. the current CAE meth- Bank (second section), and back to the client. odology could be It is, therefore, difficult to follow the link be- modified in several ways to increase trans- tween objectives (which are discussed in the parency, consistency, and readability. To de- first section) and the outcomes (which are dis- termine the accuracy of the perceptions collected cussed later in the third section), and it is diffi- from CODE and Bank staff, OED contracted an in- cult to see the linkage between the Bank’s dependent, external evaluator to undertake an inputs and services (which are evaluated in the evaluation of the current CAE methodology with second section) and specific outcomes. no restrictions or preconditions. Program evalu- • CAE terminology lacks clarity and consis- ation methods adopted by other multilateral and tency. OED’s internal review of CAEs found bilateral donors were also reviewed.18 From this that the terminology used in different CAEs analysis and the usage surveys, a number of at times is seemingly inconsistent, as different observations emerged. The current three- task managers may interpret the same concept dimensional approach to CAEs has created a con- somewhat differently. For example, the distinc- sistent format for reporting, so that over time the tion between Bank performance and the out- reader knows what to expect in each chapter. come of the Bank’s program is unclear in some But this methodological approach also appears CAEs, which can be a consequence of Bank to have several shortcomings: strategies focusing on higher-level objectives and failing to clearly articulate a set of inter- • The structure tends to place more emphasis mediate objectives. Another example is the on the Bank’s instruments (lending, AAA, aid treatment of attribution, which is inherently a coordination) than on the impact or results complex concept in a country program. of the Bank’s interventions. CAEs give a greater emphasis to program inputs than to the pro- The way forward is the adoption of a results- gram’s development impact. Of the 18 CAEs based approach, which clearly links the with clearly similar structures, 13 devoted more Bank’s objectives, instruments, and the than two and a half times as much attention to outcomes achieved. Results should be the or- inputs than to the development impact. ganizing structure of the CAE and the evaluation • Different sections of the CAEs can appear to of the Bank’s instruments (products and ser- lead to different conclusions. As noted in Part vices) should be seen simply as a means to test I, in one-third of the CAEs, satisfactory average whether the end results, which are the Bank’s portfolio ratings did not correlate with a satis- key objectives, have been achieved. This is con- The way forward is the factory outcome rating for sistent with the results-based framework that is the program as a whole. being adopted in Bank strategy formulation and adoption of a results- While the explanation is fairly is also consistent with the methodology being based approach, which straightforward, (that is, an adopted by other donor agencies. The approach clearly links the Bank’s individual project or ESW would be to outline the key objectives and re- objectives, instruments, evaluation tends to focus on sults that the Bank is trying to achieve in each the project’s stated major ob- country, based on the CASs. The strategy dis- and the outcomes jectives without necessarily cussion would be based on a simple logic model achieved. determining how relevant that links inputs with expected results based on 14 PART II: HOW CAN THE CAE INSTRUMENT BE IMPROVED? the information presented in the CAS itself and sults achieved. The Brazil CAEs rate the outcome of related documents. For each objective (and in- CAE and the Bank’s focus on Bank assistance, its tended result), the CAE would evaluate the effi- the Northeast is a good ex- cacy (and relevance) of the instruments used, as ample of the concept of sustainability, and well as the relative contribution of the Bank (as most likely association as a institutional opposed to other contributions) to that objective. way to assess the Bank’s development impact. This approach would be results-based and would contribution. address the Board members’ main recommenda- OED will intensify its efforts to ensure a tion that the CAEs should focus on actual impact more consistent application of the method- at the country level. ology. OED already has a rigorous internal re- A number of recent CAEs have begun to view process that helps ensure consistency across adopt aspects of this new methodology. evaluations. Nevertheless, to improve consistency, For example, the Zambia CAE identified four OED will develop a glossary of terms, similar main themes (objectives) by which the Bank to the “2002 Glossary of Key Terms in Evalua- planned to reduce poverty: (i) promoting a sta- tion and Results Based Management” produced ble macroeconomic environment; (ii) privatizing by the Working Party on Aid Evaluation of the state-owned enterprises; (iii) targeting assistance Development Assistance Committee of the Or- directly to the poor and vulnerable groups, and ganisation for Economic Co-operation and Devel- (iv) improving public sector efficiency and gov- opment (OECD). Finally, for all large-client CAEs, ernance. The CAE then reviewed the Bank’s ac- outside reviewers will continue to be used. These tivities and evaluated the results in each of the external reviewers, who are distinguished retired four areas separately, which it then summarized Bank experts, outside academics, and client into its overall ratings. The Brazil CAE identified country development policy officials, help to and evaluated progress on the key objectives of bring in outside views on the Bank’s programs poverty alleviation and sustainable growth, and and to ensure the validity of CAE findings. also identified and evaluated sub-objectives under each objective. More recently in fiscal 2004, the Ratings China, Tunisia, Bosnia-Herzegovina, and Croatia CAEs rate the outcome of Bank assistance, CAEs all adopted a more objective-based evalua- its sustainability, and institutional develop- tion approach. ment impact. Currently, OED country evalua- OED’s review of CAEs and of the method- tions contain explicit ratings for country programs; ological approach of other donors suggests OED is the only multilateral or bilateral agency that changes are also needed in the attribu- that rates country programs in this way. There tion concept in the CAEs. The Development are three ratings in most CAEs.19 First, CAEs rate Assistance Committee (DAC) definition of attri- the outcome of the Bank’s assistance program. bution involves establishing a causal link between The outcome rating gauges the extent to which a change and a specific intervention. This is a dif- major strategic objectives were relevant and ficult task partly because of the inherent limita- achieved, without any shortcomings. Outcome is tions of proving causality in a naturally occurring rated on a six-point scale, ranging from highly setting. Countries are not laboratories and CAEs satisfactory to highly unsatisfactory. Second, most are not reporting on experiments; controlling for CAEs rate the institutional development impact other influences and isolating influences is ex- (IDI) of the assistance program. The four-point tremely difficult in the real world. The concept of rating of IDI (high, substantial, modest, or negli- “most likely association” captures much better gible) measures the extent to which the program the reality of the link between the Bank or a bolstered the client’s ability to make more effi- donor’s actions and observed outcomes than cient, equitable, and sustainable use of its does “attribution.” This would imply establishing human, financial, and natural resources. Finally, to the degree possible how and to what extent OED rates sustainability or the resilience to risk the Bank’s interventions were related to the re- of the development benefits of the country assis- 15 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION tance program. Sustainability is rated as highly comments on CAEs, that the nature of the out- likely, likely, unlikely, highly unlikely, or, if avail- come rating is not clearly understood. An assis- able information is insufficient, nonevaluable. tance program is evaluated on how well it met its CAE ratings have been the subject of particular objectives, which are typically a subset of much debate and criticism. Underlying this the client’s development objectives. Situations can criticism is a long-standing debate within the eval- arise where the outcome of the Bank’s assistance uation community over which of the two com- program is fully satisfactory yet the country’s econ- plementary goals should be made uppermost: omy is deteriorating, or vice versa. Therefore, establishing accountability for the results, or pro- there is a distinction between the outcome rating moting learning about how to do things better in and the client’s overall development outcomes. the future. Also, as the outcome rating evaluates how well the This historic debate comes out in the Board assistance program met its objectives, it is not syn- interviews, where the majority of participants re- onymous with Bank performance, which CAEs do garded ratings as indispensable, with some argu- not rate. As clearly set out in the CAE method- ing that ratings offer the greatest potential for ological framework, the outcome of the Bank’s as- holding management accountable for its actions sistance program is determined not only by the and ensuring that it pays appropriate attention to Bank’s performance but by the joint impact of the findings. However, some did feel that less at- four agents: (a) the client; (b) the Bank; (c) part- tention should be paid to the ratings, arguing ners and other stakeholders; and (d) exogenous that disputes over ratings create defensiveness forces (e.g., events of nature, international eco- among operational staff and actually impede learn- nomic shocks, etc.). This helps explain the seem- ing from the program experience. Nevertheless, ingly contradictory finding in some CAEs that the most Board interviewees concurred that the rat- Bank’s performance may have been good but the ing scales needed to be clarified, with individual outcome unsatisfactory. For instance, in Bulgaria ratings definitions made more transparent and (1991–97), Haiti (1997–2000), Paraguay (1989–99) mutually exclusive. and Rwanda (1990–93) outcomes of the Bank’s The majority of Bank staff interviewed were assistance strategy were rated unsatisfactory but also critical of the rating system, at times citing in- the CAEs note that the Bank performed well (see consistencies between the ratings and the narra- Box 3.1). tive used to justify them, and a failure in some OED will make changes to help clarify instances to rate different implementation peri- the ratings and make them more transpar- ods individually. Staff also argue that a “one-size- ent and consistent. OED intends to continue to fits-all” rating standard fails to take sufficient rate outcome and sustainability. The revised me- account of crucial differences in client capabili- thodological approach outlined in this retrospec- ties, initial endowments, and exposure to exoge- tive will greatly facilitate an unbundling of the nous risks. Views were divided as to whether outcome rating. In the future, the CAEs will dis- ratings should be ignored, de-emphasized, or im- aggregate the outcome rating, providing separate proved with more rigorous criteria. and distinct sub-ratings of the results achieved for A key issue is that many readers incor- each key objective. These sub-ratings will then rectly perceive the outcome rating as either be aggregated to arrive at a summary outcome a rating of the country’s development rating using weights implied by the strategy. Many readers incorrectly progress or as a rating Modifications are also proposed for the ratings of of Bank performance. sustainability. The concept may be recast as the perceive the outcome CAEs rate the outcomes “likelihood of sustainability,” transforming it into rating as either a rating of of Bank assistance pro- a rating of whether the ongoing benefits of the the country’s development grams, but it is clear from country program are subject to high or low risks. the interviews of Board Since most Bank activities now have institu- progress or as a rating of participants and Bank tional development as an integral part of their Bank performance. staff, as well as written objectives, OED intends to drop the separate IDI 16 PART II: HOW CAN THE CAE INSTRUMENT BE IMPROVED? Unsatisfactory Outcome Ratings Do Not Imply Box 3.1 Poor Bank Performance Bulgaria (1989–97): The objectives of the Bank’s strategy of supporting macroeconomic stabilization and growth, private sector development, and poverty alleviation were not met but the Bank appropriately focused on stabilization and debt reduction. When reforms stalled, the Bank put on hold a major adjustment loan until a more appropriate environment existed for reforms and sup- ported institutional changes and specific sectoral needs through investment lending. Haiti (1986–00): Bank assistance had an unsatisfactory impact on challenges facing Haiti (poor governance, inadequate public sector capacity, lack of donor coordination, poor educational quality and access, and badly deteriorated infrastructure). Until 1997, assistance by the Bank and other donors overwhelmed the government’s absorptive capacity. They resumed traditional types of assistance programs, without first overcoming the enormous governance and institutional barriers to economic recovery and pro- gram implementation. Internal Bank staffing issues also reduced the effectiveness of the program in Haiti. Since 1997, Bank per- formance has improved. Irregularities in the road project were cleaned up. In a situation where it could not lend and where borrower performance continued to be problematic in the ongoing program, the Bank cancelled the remaining balances. It is, ap- propriately, maintaining a watching brief and undertaking reviews of key sectors. Paraguay (1989–99): Bank assistance objectives of private sector development, agricultural development/natural resource man- agement, and poverty alleviation/human resource development were not met. The Bank prepared well-focused ESW. Its invest- ment in dissemination stimulated public debate, particularly on the need for reform. The 1997 CAS focused on short-term objectives that were feasible within Paraguay’s institutional and policy constraints. It recognized that the risks identified in 1993 were still largely in place and that the deteriorating political situation made reform unlikely. It recommended a cautious funding strategy with a maximum of one or two projects approved per year subject to improved implementation of the existing portfolio. Rwanda (1990–93): The objectives of the Country Assistance Strategy of growth with equity, creation of an enabling environment for private sector development, protection of the environment, and improvements in public resource management capacity were not met. There were some shortcomings in Bank performance (e.g., project design did not adequately reflect human and financial constraints confronting the government) but the Bank performed well overall. It understood the fundamental economic difficulties facing Rwanda and tried to persuade the Government, albeit unsuccessfully, to change its long-standing policies and give a greater role to the private sector, reduce public subsidies to failing enterprises, and improve the quality of social services. Source: Bulgaria, Haiti, Paraguay, and Rwanda CAEs. rating and capture it as part of the outcome rat- third of participants in Board interviews credited ing. Finally, as experience is gained with results- counterfactuals with generating useful CODE based CASs and CCRs, which will also contain subcommittee discussions. More than half of the ratings, OED will review once again the rating CODE respondents criticized the counterfactuals system in CAEs and make further modifications for frequently presenting a too rosy or utopian as needed. If the Bank adopts a system of as- picture of the Bank’s sessing CAS performance through ratings in CAS role in the country, com- More than half of the CODE Completion Reports—akin to the ratings for in- plaining that CAEs tend respondents criticized the dividual projects—OED may drop the ratings of to exaggerate the role of the Bank and suggest counterfactuals for CAEs and turn them more into a lesson learning review. that if the Bank had not frequently presenting a too been involved, the coun- rosy or utopian picture of Counterfactuals try would have done CODE and Bank staff views diverge on the much worse. In contrast the Bank’s role in the usefulness of counterfactuals. Less than a to CODE members, country. 17 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION Bank staff in CAE discussions, interviews and pened in the absence of the assistance program. In memos criticized CAEs for not having an effective other words, the CAS itself often provides the counterfactual showing what alternative strategy counterfactual. The quality of such embedded the Bank could have followed, or what would counterfactuals is expected to improve signifi- have happened in the absence of the Bank; or, al- cantly now that the Bank has set uniform guide- ternatively, for not taking into account the often lines for producing results-based CASs. Comparing marginal role of the Bank in the country’s devel- what was expected ex ante in the country strate- opment. A review of Bank staff memos, however, gies with what actually happened and whether the showed that these views were expressed only divergences were reasonable is a useful way of con- when a CAE was critical of the Bank’s assistance. ceptualizing the counterfactual. Where the CAS The counterfactual issue was not raised where does not provide a fully defined counterfactual, the CAE noted that the Bank performed well. other, less attractive options may be utilized, such OED recognizes the importance of coun- as comparing: (i) the client country’s development terfactuals for illustrating what the Bank progress during the implementation period of the could have done differently in specific situ- CAS to that of other countries; (ii) the develop- ations. OED recognizes the need for a counter- ment progress in specific sectors or geographic factual but it is useful to emphasize that this task areas, which logically should have been signifi- has been difficult partly because of the inherent cantly impacted by the program interventions; and difficulties of proving causality in real world situ- (iii) intermediate outcomes logically affected by ations. That is why only 10 of the 25 CAEs re- program interventions. viewed (40 percent) attempt a discussion of the counterfactual, and these generally focus more Timing on what the Bank could have done differently Timing of the CAE is essential to its effec- than on what would have happened in the ab- tiveness, but there is some tension between sence of the Bank. The Russia CAE, for example, Bank staff and the Board about when a CAE notes that it is difficult to construct plausible is perceived as useful. There was widespread counterfactuals in the case of such a complex pro- agreement among all users of CAEs that the tim- gram, where much of what the Bank did was in ing of a CAE was an essential factor in its useful- response to explicit shareholders’ expectations. ness. A CAE that is completed either too far in Instead the CAE describes how the Bank tackled advance of a CAS or after the completion of a CAS difficult challenges and, with the benefit of hind- is viewed as having a limited impact. This is con- sight, identifies cases where it may have been firmed by OED analysis of the usage of CAEs dur- more appropriate to adopt a different stance. ing Board discussions. In effect, the impact of a The proper role of a counterfactual anal- CAE is diminished significantly if it comes either ysis in the CAE is to compare actual out- too early or too late in the CAS review process. comes with the ex ante analysis contained in For example, if the time elapsed between CAE the Bank’ Country Assistance Strategy. The distribution and Board discussion of the CAS is a CAE methodological approach judges a CAS by year or more, the CAE seems to have little influ- what it intended to achieve. Therefore, a counter- ence on the deliberations. On the other hand, a factual already exists against which the CAEs can majority of CODE interviewees also cited cases in Timing of the CAE is assess what actually hap- which CAEs were distributed too close to the pened. CASs typically pro- date of Board CAS review to have any meaning- essential to its vide multiple scenarios, ful impact. Operational staff perceive that the de- effectiveness, but there is matched by a shifting set of livery of CAEs is not sufficiently predictable to be some tension between planned Bank instruments. a useful strategy development tool. Even Bank staff who received their CAEs in a timely fashion Bank staff and the Board So, OED has a ready coun- terfactual, which avoids the expressed skepticism about their future useful- about when a CAE is complexities of trying to ness, because of their awareness of other in- perceived as useful. guess what would have hap- stances in which other CAEs were made available 18 PART II: HOW CAN THE CAE INSTRUMENT BE IMPROVED? too late in the process to inform the CAS. CODE ule of upcoming CAS discussions, as some of interviewees stated they preferred receiving the the timing issues recounted above resulted from CAE one to three months before a CAS discussion. long delays due to rescheduling CAS discussions. Bank staff, on the other hand, indicated that, in CODE can assist in this process by ensuring that order to inform the CAS, at least an early draft of the ‘green sheet’ of the CAE discussion is final- the CAE would have to be available by the up- ized and made available to the Board prior to dis- stream review stage, which in a number of cases cussion of the corresponding CAS. In any event, would be more than six months prior to a Board whether the CAS is delivered on time or not, discussion. OED believes CODE would benefit from sched- A large majority of recent CAEs appear to uling discussion of the underlying CAEs without have been timed appropriately to inform waiting for the CAS. OED will also work to in- CAS preparation and CAS discussions. A key crease interactions with country teams during criterion in OED’s decision to undertake a CAE the CAE preparation process to ensure that evolv- in a particular country is the CAS cycle. Other fac- ing CAE findings are discussed with country teams tors are considered—regional balance, collabo- during the early stages of CAS preparation (this ration with other IFIs, and the long lead times is discussed in the next section). This should required for CAEs in major Bank borrowers—but enable OED to meet both the needs of opera- the CAS cycle is the primary determinant of the tional staff and the Board in terms of timing CAE decision to schedule a CAE. OED’s review of the deliveries. timing of the 25 CAEs completed during fiscal 2001–03 indicates that, on the whole, CAEs have Interaction with Country Teams been timely, but there is scope for improvement. Bank staff voiced a clear preference for Of the 25 CAEs, eight were completed within more interaction with OED staff during the three months of the CAS discussions, four others preparation of the CAE. Bank staff, particularly were completed between four and six months, country directors, who provided a positive as- and another four were completed in the year sessment of CAEs, stressed the value and impor- prior to the CAS discussion. In three cases, no tance of the process through which the CAE was subsequent CAS has yet been prepared and only generated. Participants who assessed the CAEs in six cases were the CAEs completed more than positively were more likely to describe the CAE— one year prior to the CAS discussion. In these and its value for their work—in terms of an on- cases, CAS schedules were changed. Further- going learning process and repeatedly described more, despite the perception of Bank staff that open and regular communications between the CAEs were not timely for CAS preparation, all of country team and the CAE mission team. Con- the CAEs discussed by CODE before Board dis- versely, Bank staff who assessed the CAE nega- cussion of the CAS were mentioned in those tively described a low level of interaction between CASs. Of the 22 CAEs for which CAS discussions the country team and the CAE mission team. have been held, CAE findings and recommenda- CODE interviewees also noted that the CAE tions were explicitly incorporated in 16 CASs, should not only be seen as a stand-alone docu- and the other six CASs either included a box re- ment, but also as a tool for facilitating interactions ferring to the CAE (3) or mentioned the existence between Bank staff and OED in order to improve of the CAE (3). the receptivity of CAE findings and recommen- OED will endeavor to time CAEs to sup- dations with Bank management. port CAS preparations and discussions. OED More interaction with country teams is will continue its long-standing policy of deliver- a vehicle to improve receptivity to CAE ing CAEs to the Board and Bank staff around findings. OED’s review of the CAE process three to nine months ahead of planned CAS dis- did reveal that too many CAEs were prepared cussions, whenever feasible. One very important with little interaction with country teams. In contribution CAE users could make would be to some instances, the country director was asked increase the predictability of the Board sched- to comment on the approach paper, was inter- 19 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION More interaction with viewed by the CAE task tle time in-country, and did most of the work as team and then, only a desk review. They perceive that important country teams is a vehicle months later, was pre- people, who were in key positions at the time to improve receptivity to sented with a draft CAE. when policies were being made, were not ade- CAE findings. While individual mem- quately consulted. bers of the country teams OED’s review does indicate a reliance on were always interviewed individually for each the views of Bank staff and government of- CAE, there have been few instances of system- ficials, but the evidence shows that the ma- atic interactions with the country team and jority of CAEs go well beyond desk reviews. country director during the course of the evalu- A review of CAEs that listed people interviewed ation. OED proposes to involve the country team indicates that the majority of people interviewed in midterm reviews of future CAEs to receive early by CAE teams are either past and present gov- feedback on emerging findings and recommen- ernment officials, from the World Bank Group, dations. This approach would create a vehicle to or other donors (see Annex D). Fewer consulta- assure country teams that their views are valu- tions were held with the private sector, NGOs, able and are being heard; to demonstrate sensiti- country academics, and other client stakehold- vity to the country team’s need for understanding ers. One explanation for this is that project per- and managing the potential impact of a negative formance assessment reports constitute a key evaluation on ongoing operations, and to in- building block of CAEs and they are prepared crease the receptivity of country teams to the after extensive consultation with project benefi- evaluative findings. ciaries, thereby lessening the need for CAE teams to consult with them. The evidence also clearly Interaction with Beneficiaries shows that CAEs do not rely solely on secondary There is a widespread perception that CAEs research and desk reviews. They incorporate rely too heavily on interviews with Bank the views of government officials, both past and staff and government officials and desk present, in the country, and do not omit the views reviews of documents, and place too little of key policy makers from the past who were in- emphasis on the views of in-country bene- volved in influential decisions. ficiaries. Many CODE interviewees expressed OED will make efforts to ensure that CAEs concerns about the excessive reliance on second- reflect more fully the views of civil society ary research conducted exclusively in the Wash- on the Bank’s country programs. OED will ington office rather than working directly with work toward involving in-country beneficiaries government officials and other in-country bene- more in the evaluative process. One approach ficiaries. While most CODE interviewees specifi- that was successful in the Brazil CAE was the use cally acknowledged the inherent difficulties of of in-country experts and academics as part of involv-ing government officials and other coun- the CAE team. This need might also be addressed try beneficiaries in the evaluation process—lim- through wider use of qualitative methods, such ited budget and resources at OED, lack of as small focus groups and informal workshops institutional capacity at the country level, and the with governmental officials and other stakehold- potential to confuse or disrupt existing Bank ers. However, such methods are frequently costly, relationships—the majority insisted that OED and thus, will be used in a manner commensu- should continue to strive for greater country in- rate with OED’s budget. volvement in the CAEs. OED will make efforts to Bank staff also felt that Improving the Presentation of CAEs ensure that CAEs reflect CAE task teams relied ex- CODE interviewees feel that changes in cessively on secondary re- the document design of the CAEs would more fully the views of sources, had inadequate improve their readability. More than half of civil society on the Bank’s consultations with country the CODE interviewees stated that they had dif- country programs. beneficiaries, spent too lit- ficulty reading and finding relevant information 20 PART II: HOW CAN THE CAE INSTRUMENT BE IMPROVED? in CAEs. For the most part, this is less a problem come from the main text. This underlines the im- of thoroughness and length than of organization. portance of the summary-type sections of the One common suggestion was that the section on CAE and of ensuring that discussions of the “Findings and Recommendations,” which is most CAE with the CODE and Subcommittee is effec- likely to be read by management and the Board, tive and available to the full Board for the CAS be strengthened and moved to the front of the discussion. report. OED proposes to experiment with the Reviews of Board discussions reveal that design of the document to enhance read- certain parts of the CAE are more heavily re- ability. One possibility is to give greater pro- ferred to than others. OED’s examination of minence to the findings and recommendations in the issue-oriented CAE references in Board dis- the document’s overall organization and design. cussions suggests very selective reading of the Since most Board members read primarily the CAEs (see Annex D). One third of the references summaries, cross-referencing would assist read- point to the “Recommendations” section of a CAE, ers in finding desired information (which they and another half referred to the CAE’s “Memo- may otherwise wrongly assume is missing from randum to the Executive Directors and the Pres- the document) and in encouraging readers to ex- ident” or to the issues raised at the CODE or plore and familiarize themselves with portions of Subcommittee meetings. Only a few references the document that they may not read thoroughly. 21 ANNEXES 23 ANNEX A: THE CAE METHODOLOGY This methodological annex describes the key el- mance was also unsatisfactory, and vice-versa. ements of OED’s Country Assistance Evaluation This becomes clearer once we consider that the (CAE) methodology.20 Bank’s contribution is only part of the story. The CAEs evaluate the outcomes of Bank as- assistance program’s outcome is determined by sistance programs, not clients’ overall de- the joint impact of four agents: (a) the client; velopment progress, An assistance program (b) the Bank; (c) partners and other stakehold- needs to be assessed on how well it met its par- ers; and (d) exogenous forces (e.g., events of ticular objectives, which are typically a subset of nature, international economic shocks, etc.). the client’s development objectives. If an assis- Under the right circumstances, a negative con- tance program is large in relation to the client’s tribution from any one agent might overwhelm total development effort, the program outcome the positive contributions from the other three will be similar to the client’s overall development and lead to an unsatisfactory outcome. progress. However, most Bank assistance pro- OED measures Bank performance primarily grams provide only a fraction of the total re- on the basis of contributory actions the Bank di- sources devoted to a client’s development by rectly controlled. Judgments regarding Bank per- donors, stakeholders, and the government itself. formance typically consider the relevance and In CAEs, OED evaluates only the outcome of the implementation of the strategies; the design and Bank’s program, not the client’s overall develop- supervision of the Bank’s lending interventions; ment outcome, although the latter is clearly rele- the scope, quality, and follow-up of diagnostic vant for judging the program’s outcome. work and other AAA activities; the consistency of The experience gained in CAEs confirms that the Bank’s lending with its non-lending work and program outcomes sometimes diverge signifi- with its safeguard policies, and the Bank’s part- cantly from the client’s overall development pro- nership activities. gress. CAEs have identified assistance programs that had: Evaluating Assistance Program Outcome In evaluating the outcome (expected develop- • Satisfactory outcomes matched by good client ment impact) of an assistance program, OED development gauges the extent to which major strategic ob- • Unsatisfactory outcomes where clients jectives were relevant and achieved, without any achieved good overall development results, shortcomings. Programs typically express their notwithstanding the weak Bank program goals in terms of higher-order objectives, such as • Satisfactory outcomes where clients did not poverty reduction. The Country Assistance Strat- achieve satisfactory overall results during the egy (CAS) may also establish intermediate goals, period of program implementation. such as improved targeting of social services or promotion of integrated rural development, and Assessments of assistance program out- specify how they are expected to contribute to- come and Bank performance are not the ward achieving the higher-order objective. OED’s same. An unsatisfactory assistance program out- task is then to validate whether the intermediate come does not always mean that Bank perfor- objectives produced satisfactory net benefits, 25 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION and whether the results chain specified in the Highly satisfactory CAS was valid. Where causal linkages were not The assistance program achieved at least acceptable fully specified in the CAS, it is the evaluator’s task progress toward all major relevant objectives and had to reconstruct this causal chain from the avail- best practice development impact on one or more of able evidence and assess relevance, efficacy, and them. No major shortcomings were identified. outcome with reference to the intermediate and higher-order objectives. Satisfactory Evaluators also assess the degree of client The assistance program achieved acceptable progress ownership of international development priori- toward all major relevant objectives. No best practice ties, such as the Millennium Development Goals, achievements or major shortcomings were identified. Bank corporate advocacy priorities, and safe- Moderately satisfactory guards. Ideally, any differences on dealing with these issues would be identified and resolved in The assistance program achieved acceptable progress the CAS, enabling the evaluator to focus on toward most of its major relevant objectives. No major whether the trade-offs adopted were appropriate. shortcomings were identified. However, in other instances, the strategy may be Moderately unsatisfactory found to have glossed over certain conflicts or The assistance program did not make acceptable avoided addressing key client development progress toward most of its major relevant objectives, constraints. The consequences could include a or made acceptable progress on all of them, but either diminution of program relevance, a loss of client (a) did not take into adequate account a key develop- ownership, or unwelcome side effects such as ment constraint or (b) produced a major shortcoming, safeguard violations, all of which must be taken into account in judging program outcome. such as a safeguard violation. To make evaluation less subjective, OED ex- Unsatisfactory amines a number of elements that contribute to The assistance program did not make acceptable assistance program outcomes. The consistency progress toward most of its major relevant objectives, of evaluation elements is further strengthened and either (a) did not take into adequate account a key by examining the country assistance program development constraint or (b) produced a major short- across three dimensions: coming, such as a safeguard violation. • A Products and Services Dimension, involv- Highly unsatisfactory ing a “bottom-up” analysis of major program The assistance program did not make acceptable inputs—loans, AAA, and partnerships progress toward any of its major relevant objectives and • A Development Impact Dimension, involving did not take into adequate account a key development a “top-down” analysis of the principal pro- constraint, while also producing at least one major gram objectives for relevance, efficacy, out- shortcoming, such as a safeguard violation. come, sustainability, and institutional impact • An Attribution Dimension, in which the eval- uator assigns responsibility for the program outcome to the four categories of actors. high, substantial, modest, or negligible. IDI mea- Ratings Scale sures the extent to which the program bolstered OED uses six rating categories for outcome, the client’s ability to make more efficient, equi- ranging from highly satisfactory to highly un- table, and sustainable use of its human, financial, satisfactory. and natural resources. Examples of areas included in judging the institutional development impact Institutional Development Impact of the program are: The institutional development impact (IDI) of a country assistance program can be rated as: • The soundness of economic management 26 ANNEX A: THE CAE METHODOLOGY • The structure of the public sector, and, in par- development benefits of a country assistance pro- ticular, the civil service gram over time, taking into account eight factors: • The institutional soundness of the financial sector • Technical resilience • The soundness of legal, regulatory, and judi- • Financial resilience (including policies on cost cial systems recovery) • The extent of monitoring and evaluation • Economic resilience systems • Social support (including conditions subject • The effectiveness of aid coordination to safeguard policies) • The degree of financial accountability • Environmental resilience • The extent of building NGO capacity • Ownership by governments and other key • The level of social and environmental capital. stakeholders • Institutional support (including a supportive Sustainability legal/regulatory framework and organizatio- Sustainability can be rated as highly likely, nal and management effectiveness) likely, unlikely, highly unlikely, or, if available in- • Resilience to exogenous effects, such as inter- formation is insufficient, nonevaluable. Sustain- national economic shocks or changes in the ability measures the resilience to risk of the political and security environments. 27 ANNEX B: COMPLETED COUNTRY ASSISTANCE EVALUATIONS, FISCAL YEARS 1995–2004 FY95 FY96 FY97 FY98 FY99a FY00 FY01 FY02 FY03 FY04 (1) (2) (2) (10) (12) (10) (8) (9) (8) (8) Ghana Argentina Morocco Albania Azerbaijan Argentina Paraguay West Bank Peru Tunisia & Gaza Zambia Poland Bangladesh Cambodia Burkina Faso Kazakhstan Lesotho Zambia Bhutan Bolivia Croatia Cameroon Morocco Chile Eritrea China Côte d’Ivoire Ecuador Costa Rica India Vietnam Zimbabwe Bosnia- Herzegovina Kenya Ethiopia Egypt Kenya Haiti Lithuania Armenia Malawi Indonesia Ghana Kyrgyz Bulgaria Brazil Moldova Republic Mozambique Jamaica Papua Mexico Mongolia Dominican Croatia New Guinea Republic Philippines Maldives Tanzania El Salvador Russia Jordan Rwanda Thailand Nepal Uganda Guatemala Togo Sri Lanka Uruguay Ukraine Yemen a. A Country Assistance Note for Honduras was prepared in fiscal 1999 but was converted to a Reach following Hurricane Mitch. The grey shaded area represents the 25 CAEs that were reviewed to prepare this CAE retrospective. 29 ANNEX C: SUMMARY OF BOARD/CODE INTERVIEWS Background and Objectives Methodology This annex is based on an independent research To achieve these objectives, the consultant con- study commissioned to assess the quality and ef- ducted in-depth interviews with 30 former and fectiveness of Country Assistance Evaluations current CODE Board members, assistants, and (CAE), as perceived by participants in CODE and advisors. Sixteen of the interviews for the study CODE subcommittee discussions. Primary re- were conducted in person in Washington, D.C., search objectives included: during late June and early July 2003, with the re- maining 14 interviews conducted by telephone • Understanding participants’ perception of the between June and August 2003. Each interview most appropriate audiences, objectives and lasted about 45 minutes. All participants were institutional value for CAEs, including the de- promised confidentiality and were provided gree to which the documents are perceived via e-mail with advanced copies of the interview to fulfill the purposes for which they were guidelines. designed Participants were recruited from a list of cur- • Identifying potential obstacles to the CAE’s rent and former CODE subcommittee attendees overall quality and effectiveness with key provided by OED. The original selection criteria audiences called for the recruitment of current and former • Assessing the role of CAEs in relation to key EDs, assistants and advisors who had attended at objectives (e.g., learning for the future, impact least two CODE subcommittee meetings in the on the development of Country Assistance past. Because of limited availability of Board Strategies, and management accountability) members and their staff during the period when • Identifying the ways in which the CAE’s are the research was conducted, but the selection used by executive directors (EDs), assistants criteria were subsequently expanded to include and advisors, with particular attention to par- a small number of participants who had attended ticipants’ preferences for and evaluations of one CODE subcommittee meeting. This turned specific sections and emphases of the docu- out to be a fortuitous decision, since it was dis- ment (cover memo, ratings system, counter- covered during the course of the interviews that factuals, recommendations, and so on) reviewing CAEs and preparing for subcommittee • Evaluating CAEs in relation to specific criteria discussions and presentations is, in most cases, a (methodological soundness, thoroughness, collaborative process. EDs and other staff re- comprehensiveness, user-friendliness, fairness, sponsible for participating in the CODE discus- objectivity) sions or making presentations often rely heavily • Recommending specific ways to enhance the on summaries and talking points provided by quality and effectiveness of the documents (for assistants and advisors who do not attend the example, timeline for distribution, method- meetings. Consequently, participants with lim- ology, style and organization of the documents, ited subcommittee attendance were in some cases input by and dissemination to domestic stake- more familiar with the content and design of holders, and the like). the source documents than were EDs and other 31 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION more frequent attendees. In this light, the sam- • As a background resource for understanding ple composition provides a balanced represen- and assessing newly developed Country Assis- tation of the professional Board staff who utilize tance Strategies the CAEs in preparing for and participating in • As a background resource on specific sector, CODE subcommittee discussions. The final sam- thematic, or regional issues, to find out how ple included 17 participants who had attended certain issues are treated in specific country three or more meetings (attendance ranging settings. from 3 to 12); seven participants who attended two meetings each, and six participants who at- CAEs are used selectively. For the majority of tended one meeting. participants, CAEs are not documents to be read from cover to cover but resources from which to Findings pick and choose specific areas of interest. Par- Overall Assessment ticipants described a typical routine sequence There was a broad positive view of CAEs. used to review and identify key issues in a CAE: The majority of those interviewed (24 out of 30) (i) reading the cover memo/executive summary; reported that CAEs play an essential role, provid- (ii) looking at the recommendations; (iii) looking ing comprehensive, independent assessments to at the management response; and then (iv) flip- the Board and Bank management that are not ping through the rest of the document to iden- available anywhere else. Most participants de- tify specific areas of interest. scribed the CAE as an indispensable information Many participants reported that CAEs resource of comparatively high substance and have a limited impact on country strategy quality that provides thorough assessments of formulation. While a number of participants Bank policies and activities. Criticisms of the CAEs (8 out of 30) provided firsthand accounts about were frequently accompanied by sympathetic re- the positive influence of CAEs on the CAS pre- marks regarding the inherent difficulties in pro- paration process, the majority (17 out of 30) ex- viding an independent evaluation tool for the pressed doubt that CAEs exert any significant Bank and expressions of confidence in OED’s influence on management decisions or policy de- commitment to providing independent, objec- velopment. Participants pointed to the frequent tive assessments of Bank performance. lack of formal endorsement of CAEs by the Bank’s upper management. This weakened the impact CAE Utilization of the CAE at the management and country level, Participants indicated that from the Board’s per- with the result that operational staff were more spective, accountability and ensuring that Bank likely to be dismissive of the findings and recom- programs are effectively delivered are central con- mendations. Management defensiveness was at- cerns, and a credible, independent voice as pro- tributed both to an excessive emphasis on vided by CAEs was crucial to this process. The formal accountability as well as methodological considerable learning contribution of CAEs was concerns. All interviewees expressed a desire to also noted. Participants are aware of the tension find a mechanism by which CAEs could be given between the accountability and learning func- more influence. tions, and several interviewees stressed the im- There is a lack of clarity about how the portance of finding the right balance. evaluations can and should be used. A ma- Subjects reported that CAEs are routinely jority of participants (16 out of 30) were not clear used to: on how precisely they might use the CAE in in- teractions with management or other possible • Prepare for participation in CODE subcom- audiences (e.g., government beneficiaries, coun- mittee meetings try teams). They felt that some guidance was • Prepare for participation in Board meetings, needed from OED and suggested that there may primarily the conclusion and recommenda- be a need for a formal mechanism for integrating tions section of the CAE CAE recommendations into the CAS. 32 ANNEX C: SUMMARY OF BOARD/CODE INTERVIEWS CAEs were seen as being of limited use dent assessment of the Bank’s performance, and to in-country beneficiaries. Factors thought strive to be fair in their assessments and recom- to limit the CAE’s usefulness to government mendations. Some expressed concern, however, leaders and other key beneficiaries include: that the text had been massaged to make find- ings appear less critical for management. • Lack of time and institutional capacity Thoroughness and comprehensiveness. • Resistance, particularly on the part of govern- The CAEs were generally described as thorough ment officials, to being evaluated by outsiders and comprehensive. A few participants (5 out • Over-reliance on Bank priorities and jargon in of 30), however, thought that essential themes CAE content (political economy, private sector development, • Lack of coordination/standardization of find- cultural issues) are sometimes overlooked. ings (particularly statistical data) with infor- mation provided by other bilateral institutions Areas Where Improvements are Needed • Methodological issues, including lack of input Inadequate structure and presentation. Many from beneficiaries in the evaluation and lack CODE members (16 out of 30) stated that they of up-to-date information that adequately de- had difficulty reading and finding relevant infor- scribes the situation in country. mation in the CAEs. For most, this is less a prob- lem of length than of organization; to the extent CAEs can be a tool for facilitating inter- that length meant thoroughness and compre- action between Bank staff and beneficia- hensiveness, this was considered an asset. It was ries. Some participants felt that the CAE is less a suggested that the chapter on “Findings and Re- stand-alone resource than a tool for facilitating in- commendations,” which is most likely to be read teractions and feedback between Bank staff and by management, be strengthened and moved to beneficiaries. The process of disseminating, inter- the front of the report. preting, and answering questions and concerns Overly academic focus. A majority of par- raised by a CAE provides an excellent opportunity ticipants (17 out of 30) felt that CAEs should not for cultivating relationships with key constituen- be a quantitatively focused, academically credi- cies in country—and for identifying and address- ble research document, with measurable indica- ing specific country needs (e.g., institutional tors of past Bank performance, counterfactuals, capacity, donor coordination, etc.). Several par- and the like. Instead, they said, the evaluations ticipants gave positive examples of their ongoing should strive to provide a broad, richly textured use of CAEs with country beneficiaries. portrait of the Bank’s role in a country’s devel- opment over time with a strong focus on lessons Positive CAE Attributes learned and recommendations that strengthen High-quality analyses. A majority of the par- future Bank strategies. ticipants (23 out of 30) praised the consistently Excessive attention to operational pro- highly quality of the evaluations, with quality de- cedures to the neglect of actual progress on fined in terms such as “rich, substantive analysis” the ground. Many CODE members worried that or “excellent writing and overall presentation.” the documents are sometimes too focused on The most frequently cited caveats were inconsis- Bank performance in following certain proce- tencies in the findings or a too-positive, pro- dures and achieving certain objectives to the ne- Bank style of presentation. Most cautioned that glect of the impact of the Bank’s involvement on the quality of the evaluations depended on the the country itself. They said an increased focus consultants and staff who prepared them and on “the situation on the ground” and tracking that there was some variation in quality from one actual progress would strengthen the CAEs. CAE to the next. Insufficient direct input. Many partici- Objectivity of findings and assessments. pants expressed concerns that CAE authors rely Most participants (21 out of 30) felt that the CAEs too much on secondary research conducted ex- are objective; provide a dispassionate, indepen- clusively in the Washington office rather than 33 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION working directly with country teams, govern- changes in the recipient country, and shifts in ment officials, and other in-country beneficia- strategic objectives and priorities in the Bank? ries. Most participants specifically acknowledged Poor timing of some CAEs. A majority of the inherent difficulties of involving government the participants (17 out of 30) indicated that they officials and other country beneficiaries in the sometimes receive the CAEs too late in the stra- evaluation process. Key issues cited include: lim- tegy development process for them to have a ited budget and resources at OED, lack of insti- meaningful impact. Participants were sympathe- tutional capacity at the country level, potential to tic to the logistical issues that frequently limit the confuse or disrupt existing Bank relationships, evaluations’ availability. However, they would like and potential to limit OED’s independence and to receive the CAEs at least two to four weeks be- objectivity. In spite of these challenges and mis- fore CODE subcommittee meetings, and one to givings, the majority of participants insisted that three months before a CAS discussion. OED should continue to strive for greater country Rating inconsistencies and inaccuracies. involvement in the CAEs. The Bank’s increased The majority of participants regard the ratings as emphasis on country-focused strategies (along a generally reliable supplement to the informa- with related priorities like donor coordination, tion provided in the narrative portion of the CAE capacity enhancement and increased reliance on report. Some described the ratings as the part of local expertise) have created methodological the CAE with the greatest potential for holding challenges that can only be addressed with in- management accountable for its actions and creased participation by country beneficiaries. ensuring that it pays attention to the findings. Long time frames. Questions were raised Others, however, felt that less attention should about the long time period (10 years +) covered be paid to the ratings. Strong concerns were ex- by the evaluations, especially with respect to pressed, moreover, that inconsistencies and in- the issue of Bank and governmental accountabil- accuracies in the rating system weakened their ity. How can such an evaluation deal with staff impact. Most participants indicated that the rating turnover on both the management and govern- scales need to be sharpened or the relative values ment sides, unanticipated political and economic more clearly defined for the ratings to be useful. 34 ANNEX D: BOARD CAS DISCUSSIONS AND CAS DOCUMENTS Background and Objectives There is a certain risk in giving too much im- This review was designed to feed into the CAE portance to how frequently an issue was men- retrospective. Its objective is to focus specifically tioned during a Board discussion, because Board on the utilization of CAEs by the Board of Execu- members seem to adapt and abbreviate their tive Directors. Transcripts of Board discussions, contributions as the discussion proceeds to avoid as well as the relevant CASs and CASPRs, were re- redundancy. In many of the transcripts the last viewed in order to examine the possible impact few speakers’ contributions were much shorter of CAEs on these discussions. The written state- on average, or at least did not contain as many ments of Board members circulated prior to the different issues. This may affect how often OED discussion were also reviewed. The sample in- and the CAE are explicitly referred to. However, cluded 13 CAS discussions on the 12 countries for comments do cluster in certain areas, and this which recent CAEs were available as of April 2003 points to issues of particular interest to chairmen (one country was discussed twice). The countries in the country context or even overall. were: Bulgaria, Chile, El Salvador, Guatemala, As a basis for the review process, a worksheet India, Kazakhstan, Kyrgyz Republic, Mexico, was developed for each transcript, listing CAE rec- Morocco, Peru, Russian Federation, and Vietnam. ommendations and other issues discussed in the relevant CAE. All issues raised by a particular chair- Methodology man were recorded on the list and further catego- As a first step, the review looked for explicit ref- rized, depending on the substance of the comment erences to a CAE in a chairman’s contributions to and whether it contained an explicit reference to the CAS discussion. Substantial references pro- the CAE. If a chairman raised an issue that was not vided the strongest evidence that the CAE has discussed in any detail in the CAE, it was listed in been read and been used to prepare a chair- a separate category and the comment assessed as man’s statement. However, even if a chairman either of superficial or substantial nature. General did not explicitly refer to the CAE, he might still comments on the CAE without reference to par- have read it and been guided in his choice of ticular issues were put into yet another category. issues for the CAS discussion. Also, the level of Moreover, explicit references to the CAE were alignment of issues between the CAE and the recorded in context, categorized by content, and CAS Board discussion is important information in assessed for guidance aimed at the CAS team. itself because serious discrepancies may point to underlying problems of analysis and priority set- Findings ting. It was thus necessary to go beyond counting Overall Assessment and categorizing CAE references and to look also This review suggests that CAEs have an at issues raised during the discussion, comparing impact on CAS/CASPR Board discussions, the subject matter with issues presented in the although the impact varies considerably CAE. All issues raised by chairmen were grouped across countries. Utilization of CAEs by Board by frequency and substance and compared with members overall appears to be modest, with sig- the emphasis they had received in the CAE. nificant variations both between constituencies 35 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION and between CAEs under discussion. Certain other discussion that showed particularly good CAEs receive more attention than others, and alignment of issues with the CAE, but included no certain chairmen seem to rely more on the CAEs explicit reference to the CAE at all. The evidence than others. Overall, there was a good alignment suggests that this issue alignment could result between issues raised in CAEs and subsequent from interaction and cooperation between the CAS/CASPR discussions, but an explicit linkage to CAE and the CAS teams in preparation of the CAS. CAEs often cannot be established. The summary-type sections are referred to most frequently. A closer look at the issue- CAE Utilization oriented CAE references suggests selective read- About one third of chairmen’s statements, ing of the CAEs. One third of the references point either written or oral, explicitly referred to to the “Recommendations” section of a CAE, and the relevant CAE at least once. In total, there another 50 percent referred to CAE Summaries or were 85 references to CAEs in the 13 discussions. to issues raised at a CODE/Subcommittee meet- About a quarter (21) of these were very general ing. Only 17 percent of these references come references while the remaining 75 percent (64) from the main text (figure 1). This underlines were issue-oriented. The latter presented a fairly the importance of the summary-type sections of small fraction (6 percent) of the number of issues the CAE and of ensuring that the discussion of the raised during the Board discussions of these CAE with the CODE/Subcommittee is effective. CASs. On the other hand, there was a very close Some CAEs provoke more discussion than alignment of issues between CAEs and the re- others. A CAE gets more attention when a full spective CAS/CASPR Board discussion. This is CAS is being discussed rather than a CASPR. A true whether or not explicit references were CAE gets less attention when the time period made to a CAE and, hence, no clear connection elapsed between its distribution and the CAS can be established between alignment of issues Board discussion is close to 1 year. To some ex- and use of CAEs by Board members. tent, a CAE with an unsatisfactory outcome rat- The alignment between CAE issues and ing gets more attention. the points raised by chairmen during the Some chairmen make more frequent use Board discussions is fairly good. For exam- of CAEs than others. Ten of the 24 constituen- ple, one CAS Board discussion came almost a year cies represented in the Board referred to the after the relevant CAE was distributed and hardly CAEs with some regularity. The vast majority of anybody referred to it during the discussion. Still, these were either Part I single constituencies or the topics covered in the CAE and the CAS dis- headed by Part I country representatives. Other cussion were overall quite well aligned, and CAE constituencies seldom or never referred to a CAE. recommendations were replicated in several Some issues are raised more often than chairmen’ comments. The same was true for an- others, whether or not they are emphasized in the CAEs under review. Foremost among Distribution of these is the issue of aid coordination. The dis- Figure 1 Related References cussion is by no means dominated by bilateral aspects of donor coordination but rather is sub- Main CAE Text Recommendations 17% 33% stantive, touching on coordination of the Bank’s program with other donor programs, selectivity, collaboration on ESW, harmonization of pro- cedures, and the like. This has implications for future CAEs, as several past CAEs limited their treatment of aid coordination to describing other donors’ activities. Other recurring topics of discussion are public sector management, rural poverty, private sector development, and condi- Memorandum/CODE 50% tionality and lending triggers. 36 ANNEX E: SUMMARY OF BANK STAFF INTERVIEWS Background and Objectives ogy, style and organization of the documents, This annex is based on an independent research input by and dissemination to domestic stake- study commissioned to assess the quality and ef- holders, and so on). fectiveness of Country Assistance Evaluations (CAEs), as perceived by Bank staff involved in Methodology the CAE process. The study focused on the self- To achieve these objectives, OED commissioned reported experiences, perceptions, and expec- an independent consultant to conduct in-depth tations of respondents, and is not intended to interviews with country directors (CDs) and re- provide an objective assessment of the quality, gional vice presidents (RVPs) and focus group presentation, and delivery of CAEs. The aim is to interviews with country team members (econo- identify key attitudes that inform respondents’ mists, managers, coordinators). Research was con- receptivity—or lack of receptivity—to the find- ducted between mid-October and mid-December ings and recommendations included in the Coun- of 2003. Participants were recruited from lists of try Assistance Evaluations. current and former Bank staff provided by OED. Primary research objectives included: The research sample for the study included: four current and former RVPs and one Deputy Vice • Understanding participants’ perceptions of the President; five additional staff members from the most appropriate audiences, objectives and VPs’ offices; 20 current and former country di- institutional value for the CAEs, including the rectors; and 18 additional country team Mem- degree to which the documents are perceived bers, five of whom gave in-depth interviews. to fulfill the purposes for which they were All interview participants were promised con- designed fidentiality and were provided via e-mail with • Identifying potential obstacles to the CAE’s advance copies of the interview guidelines. In- overall quality and effectiveness with key terviews for the study lasted approximately audiences 45 minutes. • Assessing the role of the CAEs in relation to key objectives (e.g., learning for the future, Main Findings impact on the development of Country Assis- Overall Assessment tance Strategies, and accountability) With a few exceptions, Bank management • Identifying the ways in which CAEs are used had broadly negative perceptions of CAEs. by different audiences in their work Only seven of the 20 country directors inter- • Evaluating the CAEs in relation to specific cri- viewed provided a positive assessment of the teria (methodological soundness, thorough- CAEs and the utility of their findings and recom- ness, comprehensiveness, user-friendliness, mendations. Two CDs who were familiar with fairness, objectivity, and the like) more than one CAE offered a positive assessment • Recommending specific ways to enhance the of one or more CAEs and a negative assessment quality and effectiveness of the documents (for of others. Seven CDs had negative views and the example, timeline for distribution, methodol- perceptions of the remaining four were mixed. 37 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION The RVPs interviewed were even less positive any real significance for Bank staff work; did not than the CDs. Among the key findings from these provide any insights or lessons that could change interviews and focus group discussions was that the way the Bank does things; failed to commu- the CAEs generally received a positive assess- nicate the richness and complexity of the coun- ment when it was felt that they had had a direct try situation; and did not contain any new ideas impact on the preparation of a CAS or on other that the country team had not already known ongoing work, and vice versa. about. CD perceptions were not strictly guided by the outcome ratings received for their Positive CAE Attributes countries. While there were a few instances of a Value as a learning tool. Participants who pro- direct correlation between outcome ratings and vided positive assessments of the CAEs focused assessment of the CAE, the overall picture was on the impact of findings on future strategies and different. The seven CDs who provided a positive operations, even when they were describing crit- assessment dealt with three CAEs with positive ical evaluations of the Bank strategy or the coun- outcome ratings and four CAEs with negative out- try team. The learning process was seen as being come ratings. The picture was similar for those derived more from the informal, ongoing process CDs whose assessment of CAEs was negative. On through which the CAE findings are collected, the other hand, the perceptions of RVPs were communicated, and disseminated rather than the primarily determined by their experience with CAE as a freestanding, authoritative document. CAEs that received negative ratings. An associa- tion between ratings and country team member Areas that Need Improvement perceptions in focus groups discussions was dif- Competence and experience of task man- ficult to establish, except that many of the criti- agers. An overwhelming majority (16 out of 20) cisms and suggestions were guided by experiences of the CDs and all of the RVPs reported that the with CAEs with negative outcome ratings. competence and experience of the task manager is a key factor in determining their confidence in CAE Utilization and receptivity to a CAE. The most commonly Some CAEs were seen as having a direct im- cited attributes of a good task manager included: pact on the CAS or the Bank’s approach in previous operational experience; sensitivity to the country. Some participants stated that the the pressures and complexities of making deci- CAE for their country had direct impact on strat- sions and implementing strategies in the field; egy and lending and had a noticeable impact on knowledge of the region and relevant sectors; the way they think about and do their work. In and a lack of bias. one case, it was stated that “in fact, we didn’t lend Degree of interaction with the country to X for a long time after that.” Participants who team. Of the CDs interviewed for the study, six of identified themselves as primary beneficiaries of the eight participants who provided a positive the CAE were more likely to have interacted di- assessment of one or more CAEs stressed the rectly with the mission team, stressed the posi- value and importance of the process through tive impact of the evaluations (with and beyond which the CAE was generated. Participants who the CAS), encouraged broader dissemination of assessed the CAEs positively were more likely to findings to beneficiaries, and reported consulting describe the CAE—and its value for their work— CAEs from other countries in their work. in terms of an ongoing learning process and However, many of the CAEs were seen as repeatedly described open and regular commu- being of little or no value. Many interviewees nications between the country team and the CAE revealed limited appreciation and utilization of mission team that had fruitful discussions regard- the CAEs for their countries. Once the formal pro- ing the key issues to be examined and the appro- cess and CODE meeting were over, these CAEs priate people and resources to be consulted tended to be discarded and ignored. Critics said during the evaluation. Conversely, participants the CAEs: were superficial documents without who assessed the CAE negatively described a low 38 ANNEX E: SUMMARY OF BANK STAFF INTERVIEWS level of interaction between the country team and (and one that summarizes most of the other con- the CAE mission team. cerns identified) was that it diminishes the value Insensitivity to decisionmaking and ope- and usefulness at the strategy development and rational processes. Participants from all groups country team level of the lessons and recom- (including 12 of the country directors, four RVPs, mendations for the future that the evaluations and both of the focus groups) reported that the are meant to provide. Participants from all groups CAEs frequently fail to pay sufficient attention to expressed concern that the current focus on the reasoning behind past strategy development accountability both monopolizes the content, or to adequately acknowledge the commitment methodology, and organization of the CAEs (with and professionalism of management and opera- key recommendations and findings often buried tional staff. The usefulness of even the most at the end of the document) and decreases the positive CAE is sometimes diminished by the receptivity of management and country staff to evaluation’s failure to place decisions within the the recommendations that are provided. context in which they were made or to consider For participants from large, complex coun- the frame of mind of those who developed and tries with extensive Bank investment, as well as implemented the CAS. “I can’t believe that our re- the RVPs for those respective areas, the credibil- sistance to the findings and the counterfactuals is ity of CAEs as accountability measures is further simply because all of us are that parochial and compromised by the failure of the CAE to ad- that resistance to criticism and change,” said one dress the role of the Board and external pressure respondent. “It’s that people’s professionalism is from G7 countries in directing Bank strategies. not sufficiently acknowledged in the process.” This issue—and its impact on the credibility of the Haphazard CAE timing. Regardless of their CAEs for management and the country team— assessment of the CAE, participants from all was mentioned directly by eight of the country groups agreed that the timing of the evaluations directors and three of the RVPs, and was openly is too haphazard and unpredictable in its current discussed in both of the focus groups with coun- practice to be a useful strategy development tool try directors. on an organization-wide basis. Even participants Insensitivity to damaging country rela- who described ideal experiences with the timing tions. More than half of the participants (includ- of a CAE indicated that they were aware of in- ing eight CDs, three RVPs and several participants stances in which a CAE was made available too in the focus groups) spoke of the need for dam- late in the process to inform the CAS—an obser- age control after the public dissemination of a vation which created skepticism about the use- negative CAE. For most, damage to the country fulness of the CAE in their future work. Most team’s reputation and its ongoing relationships participants indicated that, in order to inform the (with governmental and other beneficiaries, other CAS, the CAE would have to be available during donors, and the general public) is regarded as the early stages of the strategy development the key implication of a negative assessment, far process. Participants’ comments regarding the more important than the impact on an individ- timing of the CAE were closely linked, in many in- ual’s career or professional development. Partic- stances, to the degree of interaction with the CAE ipants were much more receptive to negative mission team and the distinction between pro- findings when the CAE team demonstrated sen- cess and document in understanding the CAE. sitivity to the need for understanding and man- Overemphasis on accountability versus aging the potential impact of a negative evaluation learning. Participants who provided positive as- on ongoing operations. In the best cases, partic- sessments of the CAEs rarely used the term “ac- ipants were granted adequate time to assimilate countability.” Participants who provided negative and prepare (through informal conversations with assessments, however, repeatedly complained key beneficiaries) for the distribution of a nega- about the CAE’s overemphasis on accountability. tive CAE to external audiences. The most commonly mentioned problem with Weaknesses in the rating system. Partici- the CAE’s current emphasis on accountability pants were also uniformly critical of the rating 39 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION system used in the CAE. Views were divided be- clear, standardized, consistent, and streamlined tween those who believe that the ratings should be CAE methodology that can be used effectively ignored or de-emphasized and those who call for a anywhere. The focus was seen as too broad to be more rigorous approach. Main concerns included: useful. In place of the CAE’s current comprehen- sive, multi-sector scope, a large number of re- • Inconsistencies between the ratings and the spondents (8 of the country directors and 2 RVPs) narrative section of the CAE suggested that the evaluations identify a limited • Lack of clarity about the real meanings of the number of specific sectors in which a CAE could ratings have the maximum impact. Significant weaknesses • Lack of coordination with the quantitative were also seen in the counterfactuals that were measurements provided by other donor or- perceived as too abstract, over-emphasizing the ganizations role of the Bank in country development, and too • Inherent difficulties in applying uniform crite- utopian and privileged by hindsight to be useful ria from one country to another in making decisions in the future. • General resistance to the concept of graded Weaknesses in research capabilities. Par- performance at the country level ticipants remarked on the excessive reliance on • The inability to measure differences across dis- secondary resources and inadequate consulta- tinct developmental phases (and frustration tions with country beneficiaries and country team with OED’s practice of combining and averag- members in preparing CAEs. OED was seen as ing ratings from distinct periods) adopting a “parachute” approach, dropping in and • Difficulties in attributing country outcomes to out of the country for as brief a period as possible Bank performance and doing most of the work from in Washington. • The creation of undue defensiveness among Many people were seen as left out on all levels Bank staff. (beneficiaries, other donors, country team, policy- makers within the Bank), particularly people who Weaknesses in methodology and approach. were in key positions at the time when the poli- Most participants commented on the lack of a cies were being made. 40 ANNEX F: SUMMARY OF COUNTRY EVALUATION METHODS USED BY OTHER AGENCIES A desk review was undertaken of all publicly avail- out country evaluations and have formats which able country evaluation methods for each of the are similar to OED’s. The evaluations have, for major bilateral and multilateral donor organiza- instance, chapters on country background, strat- tions. This information, along with a short ques- egy, and assistance provided to the country, fol- tionnaire, was sent to the evaluation departments lowed by chapters on lessons/findings, and then of each of the donor organizations to confirm the recommendations. As in CAEs, there is a section findings of the desk review and to request miss- that deals with other donors’ contributions and ing information. All evaluation units provided exogenous factors. The assessments, like CAEs, detailed answers to the questionnaire. The Euro- also follow DAC criteria of relevance, efficiency, pean Bank for Reconstruction and Development effectiveness, and sustainability. Additionally, (EBRD), the International Monetary Fund (IMF), some include coherence, impact, and institutio- and the United States Agency for International De- nal development. velopment (USAID) were dropped from the analy- Donors differ from OED with regard to rat- sis as they had not yet undertaken any substantial ings, attribution, and the use of counterfactuals. work on country evaluations. This summary re- In contrast to OED, other donors do not usually flects information available as of September 2004. explicitly rate outcomes of their assistance pro- The review showed that about half of the or- grams (Table F.1). Because of methodological ganizations have formal guidelines for carrying difficulties in dealing with attribution, only three Table F.1 Use of Key Methodological Tools Organization Ratings Counterfactual Attribution Countries Evaluated Operations Evaluation Department (OED) 70 Multilateral organizations African Development Bank (AfDB) 6 Asian Development Bank (ADB) a 7 Islamic Development Bank (IDB) 16 United Nations Development Program (UNDP) 10 United Nations Children’s Educational Fund (UNICEF) 7 Bilateral organizations Danish Development Agency (DANIDA) 7 (U.K.) Department for International Development (DFID) 3 European Union/European Community (EU/EC) 23 Swiss Development Corporation (SDC) 9 Swedish International Development Cooperation Agency (SIDA) 4 a. In the process of developing a rating system. 41 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION of the organizations are formally attempting to or absent monitoring indicators in country attribute outcomes of their programs at a coun- strategies than about attribution and counter- try level (AfDB, IDB, and DFID, which is just factual analysis. The United Nations Children’s starting to conduct country evaluations). Most of Educational Fund (UNICEF) is attempting to ad- the others do, however, attempt to establish dress this issue by moving to an Integrated Mon- credible association between inputs and the ob- itoring and Evaluation Plan (IMEP) that is built served outcomes and impacts. Only two of the into the country program at the preparation stage. organizations reviewed—AfDB and IDB—use The IMEP focuses UNICEF programs on results, counterfactuals in their evaluations. The view of the use of a logic model to link inputs with re- donors was that both attribution and counter- sults, as well as a monitoring and evaluation (M&E) factuals are difficult to conceptualize for Country system to monitor performance. The European Assistance Evaluations. Union (EU) has also moved towards construct- Donors were more concerned about a lack ing a logic diagram for their Country Assistance of clear and defined goals and inappropriate Strategies. 42 ATTACHMENT 1: DISCUSSION SUMMARY, CODE SUBCOMMITTEE On February 28, 2005, the Informal Subcom- CAE retrospective and the suggested revisions of mittee to CODE met to discuss “Country Assis- the CAE framework. They stressed the important tance Evaluation (CAE) Retrospective—OED role of the CAE instrument for shaping the next Self-Evaluation.” cycle of country strategies, as well as sharpening Comments by OED. OED representatives the Board discussions of CASs. At the same time, noted that the report, which is a self-evaluation, they noted that there is still room for further im- is based on a review of recently completed CAEs, provement and emphasized the importance of extensive interviews with CODE and CODE sub- country ownership, continuous client engage- committee members, their advisers, and Bank re- ment and better understanding of particular coun- gional staff, as well as assessments by external try conditions for producing credible Country evaluators. The retrospective addresses two main Assistance Evaluation reports. issues: lessons learned, and ways to improve the Among the specific issues raised by the Sub- CAE instrument. The report found that a country committee were: level evaluation provides a more complete pic- Lessons learned. Members agreed that CAEs ture than a simple aggregation of project out- provide a more complete picture than project comes. In preparing CASs, the Bank needs to level evaluations and stressed the importance of better understand the country context and pur- country ownership and understanding the com- sue development policy lending only when there plexities of local political economy in this regard. is strong government ownership. The retrospec- Some speakers were concerned that, accord- tive also found that the methodological frame- ing to the report, the Board found the CAE in- work could be strengthened by placing more strument more useful than the Bank staff. OED emphasis on the development impact and the representatives replied that while dialogue is wel- outcomes achieved, rather than Bank instruments come, it is not uncommon for views to differ and compliance issues. on a process of evaluation. They added that, de- The OED evaluation concluded that future spite the differences, there is a growing demand CAEs need to be more results-based, linking from country units for more OED country-level the instruments and the outcomes achieved to evaluations. the Bank’s objectives. As for the CAE ratings and Some members felt that CAEs should also interactions with country teams and in-country take into account the effectiveness of develop- beneficiaries, OED representatives stressed that ment assistance provided by other donors. At the it will unbundle the outcome ratings by objec- same time, they acknowledged the difficulties tive, according to the new framework; and make in evaluating and distinguishing contributions an effort for more consultations with in-country in joint projects. One speaker was interested in beneficiaries. They added that if an appropriate which approach (attribution or contribution) was rating system for CAS Completion Reports is more appropriate for impact evaluation. OED rep- agreed upon with management, the CAE ratings resentatives agreed that attributing individual con- are likely to be phased out. tributions is a challenging task, but noted that an Main conclusions and next steps. Mem- attempt should be made to document an asso- bers broadly endorsed the conclusions of the ciation between outcomes and the contributions 43 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION made by the individual donors. Otherwise, one ing a full-scale CAS. Another speaker noted that could question the rationale for the individual some recommendations of the report could lead donor intervention. In the case of completely joint to a more risk-averse attitude by the country efforts, such as basket funding, attribution could teams, whereas in certain cases, and especially only be determined at the level of the joint effort. in “turnaround situations,” the Bank should be Strengthening the CAE instrument. Mem- ready to take risks to achieve progress. OED rep- bers endorsed the revised framework for country resentatives replied that adequate risk mitiga- evaluations and stressed the crucial importance tion measures would be important to prevent of the timing of their delivery–ideally 6 to 9 the accumulation of unproductive debt, the bur- months ahead of CAS presentation. They noted den of which would be ultimately borne by the the need to ensure that CASs reflect on the pre- borrowers, and in turnaround situations, the ceding CAEs and that both documents are results- Bank’s financing should be calibrated carefully based. to progress on implementing reforms. Some members were eager to see conclusions Donor coordination. Some members felt of other instruments, such as Country Policy and that the report should have covered the issues Institutional Assessments (CPIAs) and business of donor coordination, harmonization and the climate surveys, incorporated into CAEs. OED Bank’s comparative advantage vis-à-vis other representatives replied that while the CPIA is a donors. OED representatives replied that its focus useful tool, it is not necessarily directly related to is on how effective the Bank is in helping the coun- CAE ratings. tries themselves to take leadership, rather than on Members underlined the importance of allo- donor coordination per se, and that OED does not cating adequate resources for CAE preparation, evaluate the assistance provided by other donors. including involvement of staff with requisite OED has addressed donor coordination in its eval- knowledge of country specifics. A member noted uations of the Poverty Reduction Strategy (PRS) that continuous engagement with the client process, the Comprehensive Development Frame- throughout the CAE process will be a significant work (CDF), and a stand-alone evaluation of donor contribution to strengthening local capacity. OED coordination. representatives replied that it uses expert panels Country sample. Some members noted that and peer reviews to strengthen the quality of its the sample of CAEs included few small countries work, and is exploring ways to widen involve- or low-income countries under stress (LICUS) ment of country teams and the clients at various and asked whether it might have affected the ob- stages of evaluation. jectivity of the final conclusions. OED represen- Risks. One speaker stressed the importance tatives replied that the choice of CAE subjects is of carefully analyzing the risks (including a thor- often driven by upcoming CASs, thus leaving out ough look at actual track record and not decla- many “fragile” and LICUS states which are cov- rations of intent) before engaging in preparation ered by other instruments, (TSS and ISN). They of a CAS. That might help to improve cost effi- said OED continues to pay adequate attention to ciency and redirect resources in situations where small states and is also preparing an evaluation country conditions are not adequate for prepar- on LICUS countries. 44 ENDNOTES 1. See Annex A for a brief description of OED’s CAE the implementation of minimum operational stan- methodological approach. (This description is also dards in all schools in Brazil; development of a legal available on OED’s website—http://www.worldbank. and regulatory framework, exploration and develop- org/oed/oed_cae_methodology.html.) ment rights, and environmental regulations in Peru; 2. See Annex B for a list of completed Country introduction of a performance-based private manage- Assistance Evaluations. ment contract for water supply in Jordan; strength- 3. OED has distilled lessons from CAEs in the past. ening the General Directorate of Roads in Guatemala; The Annual Review of Development Effectiveness capacity improvements in the Ministry of Finance and (ARDE) distills lessons from the most recent CAEs the central bank in Mongolia; administrative capacity completed at the time of the ARDE. In 2002, lessons to adjust utilities’ tariffs, enforce cash collections, and for four African countries were discussed in a work- monitor performance in Russia; and strengthening of shop organized by the Africa Region. In the same year, the Ministry of Agriculture in Zambia. OED prepared a note on “Lessons for Low-Income 7. Bank interventions in Brazil in health, education, Countries Under Stress” (LICUS) and LICUS-like coun- and infrastructure were relatively successful because tries for the Committee on Development Effective- the Government defined primary education, health, ness. In 2003, lessons from evaluations in transition and infrastructure as clear development priorities. On economies were synthesized for OED’s transition the other hand, in Morocco the Government had not economies study. yet established clear priorities or an agenda for im- 4. In many of these countries, macroeconomic sta- plementation in education. Two education projects, bilization was undermined by poor revenue collection, which closed in recent years, had unsatisfactory out- a reflection of weaknesses in tax and customs adminis- comes and negligible institutional impact. tration. Financial intermediation suffered because it 8. OED’s IDA review and 2003 ARDE found that the took time to develop an efficient payment system, and link between countries’ policy and institutional perfor- newly privatized banks lacked capacity to make com- mance and lending levels has been strengthened. Ac- mercial credit decisions. Central banks lack capacity to cording to an adjustment lending retrospective, in the regulate banks. Private sector development is hampered last few years, most—but not all—adjustment lending because the judiciary is often incapable of implementing has gone to countries with above-average policy per- key legislation for private sector development, including formance, for sectors where there was a track record of private property rights, creditor rights, bankruptcy leg- progress. islation, anti-monopoly laws, and the like. 9. In Zambia, withholding disbursements until 5. The Bank has developed several new instru- preconditions were being met would likely have ments—programmatic adjustment loans, adjustable forced issues of governance, structural reform, and program loans (APLs), and learning and innovation debt forgiveness to the forefront at an earlier stage. In loans (LILs)—to support reforms involving a long Zimbabwe, a Bank stance in the 1997–2000 period de- learning process, which seem to be well-suited to the manding implementation of reforms rather than mere situation in some economies. expressions of good intentions would have sent a 6. Examples include strengthening of state secre- strong message to the country’s leadership. In Russia, tariats of education, implementation of information while the Bank’s shift to policy-based lending in and evaluation systems for primary education, and 1996–97 was to address systemic reform issues, the 45 COUNTRY ASSISTANCE EVALUATION RETROSPECTIVE: AN OED SELF-EVALUATION message sent to the Russian authorities was that geo- of rewarding the country’s movement toward a more political considerations would keep the international open political system and commitment to reform. The community’s funding window open, regardless of mis- loan was too unfocused to have a major impact on any steps and hesitation in adopting the reform agenda. of the critical reform areas identified in the country 10. For example, the Bank’s 1993 strategy for strategy. Many of the actions taken prior to Board pre- Paraguay was too optimistic about the potential for re- sentation were first steps, sometimes in the form of form and country implementation capacity, given studies or plans, and many others did not show con- Paraguay’s poor track record in these areas. In Zim- crete results. babwe, strategies overestimated government recep- 12. OED’s evaluation of the Heavily-Indebted Poor tiveness and willingness to undertake parastatal and Country (HIPC) Initiative found that unrealistic civil service reforms. In Haiti, the 1996 strategy did not growth projections led to debt problems. adequately recognize the risks posed to achievement 13. For an initial attempt to bring about a conver- of objectives from unresolved political and gover- gence of aid donor methodologies, see “Best Practice in nance issues, which undermined attempts to promote Country Program Evaluation.” OECD DAC Committee economic development. In Lesotho, despite its expe- on Evaluation, 2004. rience with the 1993 elections, the Bank was too opti- 14. See OED 2002, page 28. mistic in assuming that democratization and stability 15. No strategy documents have been prepared for could be accomplished shortly after the May 1998 elec- Lesotho, Zimbabwe, or Eritrea. tions. As a result, the Bank’s assistance strategy did not 16. See Annex D for a complete discussion of this include contingency plans in the event the democrati- review. zation process fell apart, as it did. 17. Annex A contains a full description of the cur- 11. In Kenya, for instance, the appointment of a rent CAE methodology. Change Team in July 1999 and initiation of long-stand- 18. See Annex F. ing economic governance and policy reforms were re- 19. Annex A gives a complete description and defi- warded with an adjustment loan when conditions for nition of CAE ratings. such support as specified in the 1998 country strategy 20. “Assistance program” refers to products and ser- were only partially met. In Zambia, over-optimism led vices generated in support of the economic develop- to less results-oriented or vaguely worded conditions ment of a client country over a specified period of time, in adjustment lending. In Morocco, the Bank provided and “client” refers to the country that receives the bene- a policy reform support loan in the late 1990s as a way fits of that program. 46 REFERENCES OECD (Organisation for Economic Co-operation ———. 2003. Debt Relief for the Poorest: An and Development), DAC Committee on Eval- OED Review of the HIPC Initiative. Wash- uation. 2004. “Best Practice in Country Pro- ington, D.C.: World Bank. gram Evaluation.” Paris. ———. 2002. 2002 Annual Report on Opera- OED. 2004a. 2003 Annual Review of Develop- tions Evaluation. Washington, D.C. ment Effectiveness: The Effectiveness of Bank ———. 1999. 1999 Annual Review of Develop- Support for Policy Reform. Washington, D.C.: ment Effectiveness. Washington, D.C.: World World Bank. Bank. ———. 2004b. Economies in Transition: An OED Evaluation of World Bank Assistance. Washington, D.C.: World Bank. 47 OED PUBLICATIONS Study Series 2003 Annual Review of Development Effectiveness: The Effectiveness of Bank Support for Policy Reform Agricultural Extension: The Kenya Experience Assisting Russia’s Transition: An Unprecedented Challenge Bangladesh: Progress Through Partnership Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy The CGIAR: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research Debt Relief for the Poorest: An OED Review of the HIPC Initiative Developing Towns and Cities: Lessons from Brazil and the Philippines The Drive to Partnership: Aid Coordination and the World Bank Financial Sector Reform: A Review of World Bank Assistance Financing the Global Benefits of Forests: The Bank’s GEF Portfolio and the 1991 Forest Strategy and Its Implementation Fiscal Management in Adjustment Lending IDA’s Partnership for Poverty Reduction Improving the Lives of the Poor Through Investment in Cities India: The Dairy Revolution Information Infrastructure: The World Bank Group’s Experience Investing in Health: Development Effectiveness in the Health, Nutrition, and Population Sector Jordan: Supporting Stable Development in a Challenging Region Lesotho: Development in a Challenging Environment Mainstreaming Gender in World Bank Lending: An Update The Next Ascent: An Evaluation of the Aga Khan Rural Support Program, Pakistan Nongovernmental Organizations in World Bank–Supported Projects: A Review Poland Country Assistance Review: Partnership in a Transition Economy Poverty Reduction in the 1990s: An Evaluation of Strategy and Performance Power for Development: A Review of the World Bank Group’s Experience with Private Participation in the Electricity Sector Promoting Environmental Sustainability in Development Reforming Agriculture: The World Bank Goes to Market Sharing Knowledge: Innovations and Remaining Challenges Social Funds: Assessing Effectiveness Uganda: Policy, Participation, People The World Bank’s Experience with Post-Conflict Reconstruction The World Bank’s Forest Strategy: Striking the Right Balance Zambia Country Assistance Review: Turning an Economy Around Evaluation Country Case Series Bosnia and Herzegovina: Post-Conflict Reconstruction Brazil: Forests in the Balance: Challenges of Conservation with Development Cameroon: Forest Sector Development in a Difficult Political Economy China: From Afforestation to Poverty Alleviation and Natural Forest Management Costa Rica: Forest Strategy and the Evolution of Land Use El Salvador: Post-Conflict Reconstruction India: Alleviating Poverty through Forest Development Indonesia: The Challenges of World Bank Involvement in Forests Uganda: Post-Conflict Reconstruction Proceedings Global Public Policies and Programs: Implications for Financing and Evaluation Lessons of Fiscal Adjustment Lesson from Urban Transport Evaluating the Gender Impact of World Bank Assistance Evaluation and Development: The Institutional Dimension (Transaction Publishers) Evaluation and Poverty Reduction Monitoring & Evaluation Capacity Development in Africa Public Sector Performance—The Critical Role of Evaluation Multilingual Editions Allègement de la dette pour les plus pauvres : Examen OED de l’initiative PPTE Appréciation de l’efficacité du développement : L’évaluation à la Banque mondiale et à la Société financière internationale Determinar la eficacia de las actividades de desarrollo : La evaluación en el Banco Mundial y la Corporación Financiera Internacional Côte d’Ivoire : Revue de l’aide de la Banque mondiale au pays Filipinas: Crisis y oportunidades Reconstruir a Economia de Moçambique Содействие России в переходе к рыночной экономике: беспрецедентная задача http://www.worldbank.org/oed THE WORLD BANK ISBN 0-8213-6316-6