THE WORLD BANK OPERATIONS EVALUATION DEPARTMENT Partnership in Joint Country Assistance Evaluations: A Review of World Bank Experience Fareed M.A. Hassan Senior Evaluation Officer Operations Evaluation Department Director-General, Operations Evaluation: Gregory K. Ingram Director: Ajay Chhibber Manager: R. Kyle Peters 2005 . The World Bank Washington, D.C. ii ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank's Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country's overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. OED Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation. The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Contact: Operations Evaluation Department Knowledge and Evaluation Capacity Development (OEDKE) e-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 http:/www.worldbank.org/oed iii Acronyms AP Approach Paper APPI Aggregate Project Performance Index CAE Country Assistance Evaluation CAS Country Assistance Strategy CODE Committee on Development Effectiveness DEC Development Economics DGO Director-General, Operations Evaluation DMT Department Management Team EBRD European Bank for Reconstruction and Development FIAS Foreign Investment Advisory Service FTB Fast Track Brief HDR Human Development Report HQ Headquarters ICR Implementation Completion Report IADB Inter-American Development Bank IsDB Islamic Development Bank IFC International Finance Corporation IMF International Monetary Fund JOLIS Joint Library Information System LEAP Lending, economy and program (files) MAR Management Action Record MIGA Multilateral Investment Guarantee Agency MIS Management Information System MOP Memorandum and Recommendations of the President NGO Nongovernmental organization OED Operations Evaluation Department OEDCR OED Country Evaluation and Regional Relations OEG Operations Evaluation Group (International Finance Corporation) OEO Operations Evaluation Office (Islamic Development Bank) OEU Operations Evaluation Unit (Multilateral Investment Guarantee Agency) OPS Operations Policy and Strategy Group OVE Office of Evaluation and Oversight (Inter-American Development Bank) PBD Programming and Budgeting Department PIF Project Information Form PPAR Project Performance Assessment Report PREM Poverty Reduction and Economic Management (Network) PRSP Poverty Reduction Strategy Paper SAR Staff Appraisal Report SEC Corporate Secretariat SSP Sector Strategy Papers TM Task Manager TOR Terms of reference UNDP United Nations Development Program USAID United States Agency for International Development WDI World Development Indicator v Contents Preface..............................................................................................................................vii Summary........................................................................................................................... ix I. Introduction........................................................................................................... 1 II. World Bank Experience in Joint Country Assistance Evaluations A. Multilateral Partners.......................................................................................... 2 African Development Bank .............................................................................. 2 Islamic Development Bank .............................................................................. 4 Inter-American Development Bank.................................................................. 6 B. Collaboration within the World Bank Group with input from EBRD.............. 6 C. Partnerships with Government and Local Institutions...................................... 7 Burkina Faso: Partnership with Local Institutions ......................................... 7 Tanzania: partnership: with the Tanzanian Government................................ 8 Eritrea: Partnership with the National Statistics and Evaluation Office........ 8 III. Benefits and Costs of Joint Country Assistance Evaluations A. Benefits of Joint Country Evaluations .............................................................. 8 B. Costs of Joint Country Evaluations................................................................... 9 IV. Lessons Learned.................................................................................................. 10 V. Concluding Remarks .......................................................................................... 10 Annexes 1. OED's Completed and On-going Country Assistance Evaluations (CAEs), FY95­FY04 .......................................................................................................... 13 2. OED-OEO Cooperative Country Evaluations: The Jordan Model....................... 15 3. Guide to OED's Country Evaluation Rating Methodology.................................. 17 4. Steps Required to Conduct an OED Country Assistance Evaluation and CAE Contents ................................................................................................................ 21 Bibliography.................................................................................................................... 25 vii Preface In the last few years, the World Bank Operations Evaluation Department has worked closely with partners in its country evaluations. A number of joint evaluations have been undertaken in collaboration with other multilateral development banks and governments. This paper explores OED's experience in joint evaluations and extracts lessons learned from experience. The evaluation partnership was launched in early 2000 under the guidance of Ruben Lamdany, then Manager, World Bank Operations Evaluation Department. The support and cooperation at all stages of: Gregory K. Ingram, Director General, OED; Ajay Chhibber, Director, OED; Gabriel Kariisa, then Director of the African Development Bank Operations Evaluation Department; and Bader Eddine Nouioua, Adviser, Operations Evaluation and Audit, Islamic Development Bank, is highly appreciated. Special thanks are due to Djelloul Al-Saci, Head Operations Evaluation Office, Islamic Development Bank, and to Oladeji Ojo, African Development Bank, for their assistance. The paper has benefited from constructive suggestions from numerous people. Patrick Grasso, Adviser; R. Kyle Peters, Senior Manager; and Osvaldo Feinstein, Consultant, OED, provided overall guidance with substantial input into the formulation of the paper. Contributions from numerous OED staff are gratefully acknowledged. Janice Joshi provided administrative support. ix Summary This paper reviews World Bank Operations Evaluation Department experience with partnerships in joint country assistance evaluations. The emphasis is on identifying emerging benefits and costs of joint evaluations and on extracting lessons for future partnerships. Joint country assistance evaluations have led to sharing of perspectives, lessons learned, evaluation capacity development, more credibility and a better mutual understanding of differing mandates. They have been effective in identifying key constraints and gaps in donor assistance; thus the African Development Bank's evaluation of the Rwanda program noted, as a stumbling block, the lack of a post-conflict policy, which has since been developed by the AfDB management. Joint evaluations also help reduce evaluation costs to aid recipient countries, and encourage the use of common evaluation standards, consistent with the donors' broader agenda of harmonization. For example, AfDB recently revised its guidelines for country assistance evaluations, closely aligning them with OED's methodology. Joint work also has led to common recommendations, increasing their credibility. However, these benefits are partially offset by transaction costs and added time needed for consultation. Joint evaluations may take more time to prepare than anticipated, delaying their delivery and reducing their value in providing timely lessons to newly designed country assistance strategies. However, as partners gain more experience with joint evaluations these costs decline, as the joint evaluation of the Tunisia program shows. The main lessons are: · Flexibility is needed to accommodate the special circumstances of each evaluation/partner, given differences in mandates, evaluation methods, and work schedules. · When one of the partners is developing experience for the evaluation, upfront evaluation capacity development should be planned and built into the partnership. · Placing priority on partnerships and coordination among donors facilitates opportunities for joint evaluations, as their number has been rather limited despite the importance given by donors to joint work. 1 I. Introduction 1.1 The importance and utility of joint evaluations have increased with the changing context of development assistance approaches. Sector wide approaches and country assistance strategies are emerging and they differ from traditional assistance activities. These new assistance modes emphasize collaborative, multi-donor assistance efforts with shared common objectives and with co-financing arrangements, rather than separate donor projects. Effective evaluations of such new assistance modes call for donor collaboration. Partnerships for global programs and the broad adoption of the Millennium Development Goals also facilitate joint analysis and evaluation work because many donors have an interest in measuring and monitoring the same indicators. Another reason for joint evaluations is the need to lower the costs of evaluations to aid recipients, including the burden of multiple, separate donor evaluation missions on recipient country institutions.1 The World Bank Operations Evaluation Department (OED) has made partnership a cornerstone of its values statement. OED's work program has been aligned with its partnership strategy. This paper reviews OED's experience with partnerships in joint country assistance evaluations.2 Country Assistance Evaluations examine Bank performance in a particular country, usually over the past five to ten years; report on its conformity with relevant Bank Country Assistance Strategies (CAS) and on the overall effectiveness of the strategies; and provide Table 1: Multilateral Partnerships in Country Assistance Evaluations lessons for future Country Assistance Evaluation / (Year Partner strategies. of evaluation completion) 1.2 Despite the Lesotho (2001) African Development Bank importance given by Rwanda (2004) African Development Bank donors to joint evaluations, Jordan (2003) Islamic Development Bank their number has been limited. While OED has Tunisia (2004) Islamic Development Bank conducted 70 Country International Finance Corporation, Assistance Evaluations Multilateral Investment Guarantee Agency, (CAEs)3 since 1995, only Kazakhstan (2001) and European Bank for Reconstruction and six of them have been Development prepared in collaboration Peru* (2003) Inter-American Development Bank with multilateral *In parallel with a Country Program Evaluation by the Office of the Evaluation and Oversight of IADB. development banks, Source: World Bank Operations Evaluation Department data (see annex 1 for details). including the African Development Bank, Islamic Development Bank, European Bank for Reconstruction and Development, and Inter-American Development Bank (table 1). Two of the six partners in various country evaluations are from within the World Bank Group: the Operations 1For a more comprehensive analysis of reasons for joint evaluations see OECD/DAC 2000 publication, Effective Practices in Conducting a Joint Multi-Donor Evaluation, and Osvaldo Feinstein and Gregory K. Ingram's note on "Lessons Learned from World Bank Experience in Joint Evaluations". 2Osvaldo Feinstein and Gregory K. Ingram's note on "Lessons Learned from World Bank Experience in Joint Evaluations" briefly covers all types of joint evaluations, at the project, sector, country, and global level. 3These 70 CAEs cover nearly two-thirds of the World Bank's operational program. 2 Evaluation Group (OEG) of the International Finance Cooperation and the Multilateral Investment Guarantee Agency Operations Evaluation Unit (OEU). 1.3 The Lesotho and Rwanda CAEs were prepared jointly with the African Development Bank Operations Evaluation Department (OPEV) and the Jordan and Tunisia CAEs were prepared jointly with the Islamic Development Bank Operations Evaluation Office (OEO). OED's CAE on Peru was prepared in parallel with a Country Program Evaluation by the Office of Evaluation and Oversight (OVE) of the Inter- American Development Bank. The Kazakhstan evaluation was conducted jointly by OED, OEG, and OEU with input from EBRD (table 1). In addition three CAEs (Burkina Faso, Tanzania and Eritrea) were undertaken in partnership with the corresponding government (aid recipients partnerships). 1.4 So far, partnership models have ranged from cross-referencing findings to part joint, part parallel evaluations. Four types of collaboration have been identified: (i) part joint, part parallel country evaluations with some sections being prepared together but with each multilateral development bank assessing its own program (Lesotho, Jordan, Tunisia, and Rwanda); (ii) parallel evaluations (Peru); (iii) incorporating donor evaluation findings in OED's report (Kazakhstan); and (iv) evaluations undertaken in partnership with governments and local institutions (Burkina Faso, Tanzania, and Eritrea). Joint country evaluations have not reached the stage where one report is prepared for donor assistance because of differences in reporting requirements, coverage, and methodologies. While there are separate evaluation reports, a joint assessment of the partnership and the extent of synergy between the strategies of the partnering institutions has largely been examined. 1.5 This paper is organized as follows. Section II reviews World Bank's OED experience with partnerships in joint country assistance evaluations. This includes collaboration with other multilateral development banks, governments, and partnership within the World Bank Group evaluation units (e.g., OEG and OEU). The emphasis is on identifying emerging benefits and costs of joint evaluations (Section III). Section 1V extracts lessons learned from the experience and Section V offers some concluding remarks. Finally, annex 4 summarizes OED's process of conducting a country assistance evaluation by listing some of the main steps involved and outlines CAE contents. II. World Bank Experience in Joint Country Assistance Evaluations 2.1 This section examines World Bank joint evaluations experience with multilateral partners, collaboration within the World Bank Group evaluation units, and joint evaluations undertaken in partnership with the Government and local institutions. A. Multilateral Partners African Development Bank 2.2 Lesotho. This is the first joint country assistance evaluation. The World Bank OED and the African Development Bank OPEV recently completed joint work on a country 3 assistance evaluation for Lesotho that covered the country's economic and social development in the 1990s. The development challenges Lesotho faces were assessed jointly. These two chapters were jointly prepared and are common to the evaluation reports of both institutions. The roles of each institution over the decade were evaluated in parallel. That is, OED and OPEV prepared separate evaluative chapters for their respective assistance. The last chapter, on conclusions and recommendations, was jointly prepared. The table of contents of the resulting joint report, Lesotho: Development in a Challenging Environment, illustrates how the joint evaluation was conducted (OED-OPEV 2002). 2.3 Although the evaluation focus and methodologies differed between the two institutions, and the evaluation ratings of the two agencies (OED and OPEV) were not always comparable, the joint evaluation made common recommendations for future World Bank and African Development Bank assistance, enhancing their credibility. Both institutions should continue to focus on reducing poverty and inequality by improving the quality of education and human capital development, by combating HIV/AIDS, by strengthening rural institutions, and by enhancing the enabling environment for private sector development. Particular attention should be given to the poor, mountainous region, where the Lesotho High Water Project is located.4 2.4 The partnership contributed to enhancing the evaluation capacity of the African Development Bank and led to further collaboration with OED. The exchange of background papers and draft evaluations, joint mission, and extensive consultation enabled the two institutions to share their experiences. Communications were generally efficient and an understanding on the modus operandi was easily reached. Thus, after this first joint evaluation, the African Development Bank OPEV recently led a joint evaluation with OED of the Rwanda country program. 2.5 Rwanda. The Rwanda CAE represents the second cooperative effort for country evaluations between the two institutions. This country was chosen for joint evaluation because it was of special interest to the AfDB. OED showed flexibility and advanced its evaluation schedule by one fiscal year to allow this evaluation to be conducted jointly. 2.6 Because OED and OPEV had already worked jointly on their country evaluation of Lesotho, under the guidance of OED as it was a first for OPEV, it was decided that the AfDB would take the lead for this evaluation. The leader for the OPEV team sent his approach paper to the OED team in Washington D.C. He then visited Washington and met key staff members working on Rwanda and participated in the approach paper review meeting for OED's CAE.5 He was given project and analytical documents from the World Bank files. The OED team leader also received documents from the AfDB. The two institutions went on a joint mission to Kigali in March 2002, reducing the burden of separate donor evaluation efforts on Rwanda's institutions. Both teams have shared 4 Although the World Bank financed less than 4 percent of the Lesotho Highland Water Project cost of US$3.7 billion, it facilitated an agreement between South Africa and Lesotho, served as a catalyst in securing external financing, and advised on project formulation and implementation. 5The approach paper precedes an OED evaluation and provides information on the key questions, rationale, scope, and evaluation approach. 4 mission terms-of-reference (TOR), background papers, and other relevant information. Thus, cooperation has been good on a professional and on a personal level. 2.7 The two institutions agreed on a part joint, part parallel evaluation. The two Banks have a common chapter on the country context (i.e., political, social and economic development issues). Separate chapters evaluate each Bank's strategy, portfolio, and other services. The two evaluations generally agree on their analysis and conclusions. Both Banks should focus on developing the agricultural sector, improving the quality of heath and education services as well as strengthening of basic infrastructure services. Further OED-OPEV collaboration is envisaged. 2.8 The joint evaluation succeeded in bringing an important policy issue to AfDB management's attention: AfDB did not at the time of the evaluation have policy guidelines as those of the World Bank on post-conflict assistance. The AfDB's CAE for Rwanda noted in its Executive Summary that "the Bank's lack of post-conflict policy has been a stumbling block" and recommended drawing up of a post-conflict operational policy, which has been recently developed by the AfDB management (OED 2004a). 2.9 The relocation of the AfDB to Tunisia contributed to delays in the production of the evaluation, reducing its value. The Rwanda CAE was not available before the World Bank Board of Directors' discussion of the Rwanda country assistance strategy. 2.10 African Development Bank's revised guidelines for CAEs concluded that most of the CAEs produced by OPEV to date have tended to follow the approach adopted by the World Bank OED and recommended that future evaluations should consistently adopt OED's CAE methodology (provided in annex 3). Islamic Development Bank 2.10 Jordan. This is the first time that the Operations Evaluation Department (OED) of the World Bank and Operations Evaluation Office (OEO) of the Islamic Development Bank have collaborated in evaluating their assistance program in any country. The evaluation built on a joint mission and joint analyses. Jordan's socioeconomic development since 1990 and the challenges facing the country were assessed jointly; each institution's assistance program was evaluated in parallel. The Executive Summary of the OEO evaluation was annexed to the OED report and vice versa. OED-OEO cooperative country evaluation model is outlined in Annex 2 as the "Jordan Model" and the table of contents of the resulting joint report: Jordan: Supporting Stable Development in a Challenging Region, shows how the joint evaluation was carried out (OED-OEO 2004). 2.11 The partnership process. Following OED's invitation, a mission led by the former Head of the Operations Evaluation Office, IsDB, visited Washington D.C. during October 29­November 2, 2001, to explore evaluation collaboration with OED. OED led a CAE mission in May 2002 in which OEO did not participate due to other pressing assignments (OEO management had been recently changed and the Office had been recruiting new staff). However, OED mission leader visited the IsDB to inform them about the main findings of the mission and to support IsDB's CAE mission. Subsequent 5 discussions during the joint OED-OEO mission to Jordan (October 20­29, 2002), and meetings in Jeddah (November 1­3, 2002) led to further strengthening of the partnership. 2.12 Evaluation coverage, methodology, rating scales and reporting requirements differ and hence results are not always comparable. OED's evaluation of the Jordan program covers macroeconomic stabilization and growth, poverty reduction and human development, and water resource management. IsDB assistance covers predominantly the mining and industry sector, infrastructure, utilities, and health and education. In terms of lending instruments used, policy-based lending, accounted for 63 percent of total World Bank lending to Jordan, whereas IsDB does not offer policy-based lending. OED's CAE examined the effectiveness of World Bank assistance from three perspectives: (i) its products and services (i.e., lending and analytical and advisory services); (ii) its overall development impact; and (iii) the role of various contributors to achieving the impact of the Bank's assistance strategy (i.e., OED's CAE methodology summarized in Annex 3). However, IsDB's CAE assessed the first two dimensions and to a lesser extent the third dimension. The evaluation scales of both institutions also differ.6 Finally, OED reports to the World Bank Board of Directors, whereas OEO reports to the President of IsDB. 2.13 Because Jordan was the first country level evaluation that the IsDB's OEO had undertaken, the WB's OED team leader provided advice to the IsDB and helped the institution to develop its capacity. The preparation of joint sections, sharing of background papers and draft CAEs contributed to the evaluation capacity development, particularly given that this was the first IsDB's country evaluation. IsDB management acknowledges and appreciates the evaluation capacity development extended by OED. 2.15 Developing IsDB's evaluation capacity however, delayed the completion of the World Bank evaluation beyond planned deadlines. The CAE for Jordan was not available before the World Bank's Board discussion of the country assistance strategy.7 Had upfront evaluation capacity development been planned and supported by specific training (prior to conducting the evaluation), this would have facilitated the joint evaluation work, during which there could still be evaluation capacity development through "learning by doing", without imposing significant delays in the production of the evaluation and reducing its value. 2.16 Tunisia. This is OED's second cooperative effort with IsDB. The Tunisia country assistance evaluation was a difficult task, as it had both a tight time line due to an acceleration of the World Bank's country assistance strategy preparation and it was a joint evaluation with the IsDB. However, the evaluation capacity developed during the partnership, including the Jordan CAE, facilitated the completion of the Tunisia CAE on 6Both WB's OED and IsDB's OEO use relevance, efficacy and efficiency to evaluate outcome. But, while OEO uses a three-category rating scale (successful, partially or marginally successful, and unsuccessful), OED uses a six-category rating scale for outcome (highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory). 7Despite the fact that OED's evaluation was not presented to the Bank's Board before the Board discussion of the Jordan country assistance strategy, the latter benefited significantly from the availability of the draft CAE and the background papers. 6 time, enhancing its value. World Bank management agrees with the CAE analysis and recommendations, which are well reflected in the new assistance strategy for Tunisia. The Tunisia CAE is well received by the World Bank Board. Board members welcomed OED's cooperation with IsDB, consistent with the Bank's broader agenda of harmonization and reducing the cost of assistance to the clients, and urged OED to continue that practice in the future (OED 2004b). Further collaboration with IsDB is envisaged. Inter-American Development Bank 2.17 OED's CAE on Peru was prepared in parallel with a Country Program Evaluation (CPE) by the Office of Evaluation and Oversight (OVE) of the IADB. This partnership was particularly important because IADB's lending commitments in Peru during FY92- 00 were slightly larger than those of the World Bank. The two evaluations covered the respective Banks' assistance programs over the decade of the 1990s, and were prepared in close cooperation. An OVE staff member accompanied the OED mission to prepare the CAE, while the Executive Summary of the CPE is included in the CAE. 2.18 The findings of both evaluations are highly similar. While recognizing progress made over the 1990 decade, both evaluations find that outcomes expected from the policies supported by both Banks were significantly less positive than hoped for, and many key challenges identified in 1990, such as poverty reduction, remained unresolved by the end of the decade. There are methodological differences relating to evaluation ratings: OVE evaluation does not rate outcomes explicitly while OED does. B. Collaboration within the World Bank Group with input from EBRD 2.19 Kazakhstan. The Kazakhstan evaluation was conducted jointly by OED, OEG and OEU of World Bank, IFC and MIGA activities in Kazakhstan. The European Bank for Reconstruction and Development (EBRD) also participated in the evaluation. This collaboration was particularly appropriate because one of the key focus areas of the evaluation was how the three institutions in the World Bank Group (IBRD, IFC and MIGA) have helped to facilitate the development of the private sector in Kazakhstan. 2.20 The joint evaluation process led to fielding of a joint mission to Kazakhstan thereby reducing the burden of separate donor evaluation efforts on Kazakhstan's institutions. The task managers conducted interviews jointly and separately allowing a much larger coverage of individuals, civil society, the Government and the private sector. OED met MIGA and IFC clients in the private sector. This led to a greater appreciation by OED of constraints at the micro level faced by businesses. OEU and OEG met a number of Government officials who they normally don't meet in project level evaluations. They were able to learn about the role that the Government can play in the development of the private sector. They also got an opportunity to communicate to top officials the constraints IFC and MIGA face in their activities in Kazakhstan and their catalytic role in the development of the private sector. In turn the officials told them of areas of IFC and MIGA's comparative advantage. 2.21 The collaboration led to in-depth learning about the roles of each institution in the development of the private sector, and the synergies that have or should have been exploited. These were highlighted in a joint back-to-office report, and the preparation of 7 joint comments on the World Bank Group's draft country assistance strategy. The joint input and the preparation of joint recommendations in the CAE increased their credibility. OEG and OEU for the first time prepared a list of monitorable actions by management in parallel with OED to report at a later date to the Bank's Board of Directors. 2.22 Full EBRD participation was sought in the evaluation but could not materialize. This was due to differences in disclosure requirements, the imperatives imposed by the World Bank Group CAE timetable which EBRD was unable to meet, methodological issues in CAEs for EBRD and difficulties for EBRD in adding new evaluation work mid- stream in work programs that had already been agreed with the EBRD Board. However, despite these constraints, EBRD provided valuable insights into donor programs in Kazakhstan, via telephone, personal interviews in London and agreed to field a joint mission with OEG in evaluating a large project financed jointly by EBRD and IFC. EBRD evaluation staff also prepared a summary of evaluation findings from EBRD projects as a formal input into the evaluation document (OED 2001). C. Partnerships with Governments and Local Institutions Burkina Faso: Partnership with Local Institutions 2.23 The OED Burkina Faso CAE team made substantial use of local consultants to increase the capacity of the team to collect information and to provide a national perspective on the issues. Seven local consultants were hired in advance of the mission to prepare background papers, using OED's CAE methodology. Five of the consultants were professors from the University of Ouagadougou, who analyzed different aspects of the adjustment program. The remaining two consultants examined Bank projects in the education, health and urban sectors. The background papers were based on information available in Burkina Faso, and on the personal knowledge of the consultants; the latter were asked not to conduct interviews.8 Upon arrival in Ouagadougou, the two international consultants dealing with these areas worked closely with the local consultants to revise the papers and incorporate their findings into the CAE (OED 2000a). They also carried out all necessary interviews. 2.24 Finding the appropriate consultants, writing and translating the terms of reference, and arranging the contracts proved to be time consuming, and the team also spent a significant amount of time in the field helping the consultants revise their papers. The papers, however, provided very useful information, as well as insights into the Bank's program from the Burkinabe perspective. They contributed to a better understanding by the mission of the socio-political causes of the slow structural adjustment process and of the ways in which Bank assistance affected the results in the education, health and urban sectors. 2.25 The use of local consultants provided a significant amount of local evaluation capacity development. The approach was particularly appropriate in a country that places a high value on consultation and participation. Also it gave the mission added credibility and contributed to a franker dialogue than might otherwise have been the case. 8OED did not want to burden the authorities with multiple interviews. 8 Tanzania: Partnership with the Tanzanian Government 2.26 A staff member of the Ministry of Finance participated in the mission, produced background notes for the CAE, and came to Washington to participate in the OED management review of the draft evaluation. Government's input contributed to the substance of the evaluation (OED 2000b). 2.27 The participation of a government official in the evaluation facilitated its preparation and enhanced its analysis. The Region, however, expressed the view that involvement of government officials in the evaluation raises the issue of balance. Eritrea: Partnership with the National Statistics and Evaluation Office 2.28 Eritrea. The Government of Eritrea had been approached to partner with OED in the CAE for Eritrea. Eritrea was a good candidate for such an exercise, as they had assumed strong leadership in all areas of development assistance with donors. They had expressed interest in this type of evaluation and designated a partner agency; the National Statistics and Evaluation Office, in the President's Office. This is the body used for all program and project evaluations. However, the office depended heavily on the University of Asmara, sub-contracting most evaluations. The Government used the CAE process to develop evaluation capacity in government agencies and the university. The partnership led to Government ownership of the CAE conclusions (OED 2004c).9 III. Benefits and Costs of Joint Country Assistance Evaluations 3.1 OED's experience with partnership in joint country evaluations suggests the following benefits and costs. Given OED's experience with a relatively small number of joint country evaluations, these findings should be viewed as preliminary. A. Benefits of Joint Country Evaluations 3.2 Partnerships have led to significant evaluation capacity development--especially if the partnering institution is developing experience. Joint work has led to sharing of perspectives, lessons learned, methodologies, and better understanding of differing mandates of institutions. This facilitated evaluation capacity development among partnering multilateral development banks and aid recipients. · The World Bank's positive experience with the African Development Bank in conducting a joint CAE for Lesotho led to significant capacity development, and further collaboration: AfDB led a joint evaluation with OED of the Rwanda program. Prior to Lesotho, AfDB had not conducted a country evaluation. · Similarly, the World Bank and Islamic Development Bank partnership in assessing their assistance programs to Jordan has led to sharing of perspectives, 9In particular, the conclusion on how the Bank should have dealt with a new country emerging from years of conflict and isolation, lacking knowledge of Bank's internal procedures and was skeptical of donors intentions is basically sound (Governments comments on the CAE, (OED 2004c, page 59)). 9 lessons learned, and evaluation capacity development. The sharing of OED background papers with OEO and the preparation of joint chapters and draft CAEs provided the basis for capacity development, particularly given that this was the first IsDB's country assistance evaluation. IsDB management acknowledges and appreciates the evaluation assistance extended by OED. T he evaluation capacity developed facilitated the completion of another joint evaluation on Tunisia. · The substantial use of local consultants (e.g., the evaluation of the Burkina Faso program in particular) provided a significant amount of local evaluation capacity development and led to government ownership of the evaluations findings. 3.3 Despite differences in mandates, evaluation focus, and methodologies, joint evaluations have promoted discussions of evaluation methods and have encouraged the use of some common evaluation standards, consistent with the Bank's and other donors' broader agenda of harmonization. · For example, AfDB recently revised guidelines for CAEs recommended following OED's CAE methodology. 3.4 Joint work has been effective in identifying key constraints and gaps in donor assistance. The joint evaluation of the Rwanda program highlighted the fact that during the course of the evaluation, the AfDB did not have policy guidelines as detailed as those of the World Bank on post-conflict assistance. 3.5 Partnerships have also been effective in lowering the transaction costs to recipients of development assistance as the burden of multiple, separate evaluation efforts on recipient country institutions has been reduced. B. Costs of Joint Country Evaluations 3.6 On the other hand, partnerships can impose the following costs that should be taken into account in undertaking joint evaluations. 3.7 Experience with partnerships shows that joint country evaluations may take more time to prepare than anticipated and may cost more. This is due to greater time devoted to coordination, exchange of work programs and plans, joint drafting of common chapters, as well as time taken for comment and review of documents by joint evaluation stakeholders. Exchanges with partners reveal that differences in organizational cultures, mandates, and methodologies can impose significant constraints/delays to joint evaluations. 3.8 Delays in the completion of joint evaluations adversely affect their timeliness and reduce their value in providing timely lessons to newly designed country assistance strategies. However, as partners gain more experience with joint evaluations these costs (both time and financial) decline. The Tunisia CAE provides an example where the 10 experience gained earlier during the preparation of the Jordan joint evaluation facilitated the completion of the Tunisia CAE on time. IV. Lessons Learned 4.1 Flexibility is needed to accommodate the special circumstances of each evaluation/partner, given differences in mandates, organizational cultures, evaluation methods, and work programs. Experience has shown that some differences that loomed large in the beginning did get resolved during the evaluation because of commitment to work jointly and flexibly. 4.2 When evaluation capacity development is part of the joint evaluation, it should be recognized as a separate objective and planned for in terms of time and cost. This would facilitate the joint evaluation work, during which there could still be evaluation capacity development through learning by doing, without imposing delays in the delivery of the evaluation, which could reduce its value. 4.3 Despite the importance given to joint evaluations, their actual number has been rather limited. Joint evaluations continue to be largely coordinated by one institution (usually OED) and there is a need to actively involve management at donor institutions so that protocols for partnership can be established in advance at the highest levels. For example, the Multilateral Development Banks Evaluation Cooperation Group (ECG) can facilitate the identification of opportunities for partnerships among donors, as the major purposes of ECG are to strengthen cooperation among evaluators and harmonize evaluation methodology in its member multilateral institutions. V. Concluding Remarks 5.1 OED's experience to date suggests that joint country assistance evaluations strengthen evaluation capacity and harmonization. They promote discussions of evaluation methods and encourage the use of common evaluation standards, consistent with the donors' broader agenda of harmonization. For example, AfDB recently revised its guidelines for country assistance evaluations, closely aligning them with OED's methodology. Joint country evaluations have also been effective in identifying key constraints and gaps in donor assistance; thus the AfDB evaluation of the Rwanda program noted as a stumbling block the lack of a post-conflict policy, which has since been developed by the AfDB management. Joint evaluations also help contain or reduce the costs of evaluation to recipient countries. But these benefits are partially offset by transaction costs and added time needed for consultation. 5.2 The following lessons emerged from OED's experience with joint country assistance evaluations. · Flexibility is needed to accommodate the special circumstances of each evaluation/partner, given differences in mandates, evaluation methods, and work schedules. 11 · When one of the partners is developing experience for the evaluation, upfront evaluation capacity development should be planned and built into the partnership. · Placing priority on partnerships and coordination among donors may facilitate opportunities for joint evaluations, as their number has been rather limited despite the importance given by donors to joint work. 13 Annex 1: OED's Completed and Ongoing Country Assistance Evaluations (CAEs), FY95­FY04 FY95 (1) FY96 (2) FY97 (2) FY98 (10) FY99* (12) FY00 (10) FY01 (8) FY02 (5) FY03 (8) FY04 (9) Ghana ArgentinaMorocco Albania Azerbaijan Argentina Paraguay West Bank & Gaza Peru Rwanda Zambia Poland Bangladesh Cambodia Burkina Faso Kazakhstan Lesotho Zambia China Bolivia Croatia Cameroon Morocco Chile Eritrea Armenia Côte d'Ivoire Ecuador Costa Rica India Vietnam Zimbabwe Croatia Kenya Ethiopia Egypt Kenya Haiti Lithuania Bosnia & Her. Malawi Indonesia Ghana Kyrgyz Dom. Rep. Moldova Mozambique Jamaica Papua New Guinea Mexico Brazil Bhutan Philippines Maldives Tanzania El Salvador Jordan Tunisia Thailand Nepal Uganda Pacific Islands Togo Sri Lanka Uruguay Ukraine Yemen *A Country Assistance Note for Honduras was prepared in FY99 but was converted to an FTB following Hurricane Mitch. Bold denotes joint CAEs with multilateral development Banks. Italic denotes CAEs undertaken in collaboration with aid recipient countries. 15 Annex 2: OED-OEO Cooperative Country Evaluation The "Jordan Model" A. World Bank Operations Evaluation Department Jordan Country Assistance Evaluation Outline Preface Executive Summary Chapter 1. Jordan's Socioeconomic Development During the 1990s (jointly with the Islamic Development Bank Operations Evaluation Office) · Background · The Economy · Poverty and Social Conditions · The Role of Aid Chapter 2. Development Challenges/Constraints Facing Jordan (jointly with the Islamic Development Bank Operations Evaluation Office) · Identify four major development challenges Chapter 3. The World Bank Products and Services to Jordan · Strategy and Policy Dialogue · Bank Lending - Lending Priorities - Completed Project Performance - Ongoing Assistance · Nonlending Services - Economic and Sector Work - Aid Coordination and Resource Mobilization Chapter 4. The Development Effectiveness/Impact of World Bank Assistance · Macroeconomic Outcomes · Human Development · Water Resource Management Chapter 5. Contributors' Performance · World Bank Performance · Client Country Performance · Aid Partner Performance Issues · Impact of Exogenous Factors Chapter 6. Conclusions and Recommendations Annexes · OEO Executive Summary annexed to OED report Annex 2 (continued) 16 OED-OEO Cooperative Country Evaluation The "Jordan Model" B. Islamic Development Bank Operations Evaluation Office Jordan Country Assistance Evaluation Outline Preface Executive Summary Chapter 1. Jordan's Socioeconomic Development During the 1990s (jointly with the World Bank Operations Evaluation Department) · Background · The Economy · Poverty and Social Conditions · The Role of Aid Chapter 2. Development Challenges/Constraints Facing Jordan (jointly with the World Bank Operations Evaluation Department) · Identify four major development challenges Chapter 3. Islamic Development Bank Assistance to Jordan · Mining/Industry Sector - Sectoral Strategy and its Relevance - Sustainability of the Outcomes - Institutional Development - Performance of the Bank - Performance of the Borrower - Rating and Conclusion · Infrastructure and Social Sector - Sectoral Strategy and its Relevance - Sustainability of the Outcomes - Institutional Development - Performance of the Bank - Performance of the Borrower - Rating and Conclusion · Non-project Assistance: Outcomes and Sustainability Chapter 4. Overall Assessment Chapter 5. Conclusions Annexes · OED Executive Summary annexed to OEO report 17 Annex 3: Guide to OED's Country Evaluation Rating Methodology 1. This methodological note describes the key elements of OED's country assistance evaluation (CAE) methodology.10 CAEs rate the outcomes of Bank assistance programs, not Clients' overall development progress 2. An assistance program needs to be assessed on how well it met its particular objectives, which are typically a sub-set of the Client's development objectives. If an assistance program is large in relation to the Client's total development effort, the program outcome will be similar to the Client's overall development progress. However, most Bank assistance programs provide only a fraction of the total resources devoted to a Client's development by donors, stakeholders, and the government itself. In CAEs, OED rates only the outcome of the Bank's program, not the Client's overall development outcome, although the latter is clearly relevant for judging the program's outcome. 3. The experience gained in CAEs confirms that program outcomes sometimes diverge significantly from the Client's overall development progress. CAEs have identified assistance programs which had: · satisfactory outcomes matched by good Client development; · unsatisfactory outcomes in Clients which achieved good overall development results, notwithstanding the weak Bank program; and, · satisfactory outcomes in Clients which did not achieve satisfactory overall results during the period of program implementation. Assessments of assistance program outcome and Bank performance are not the same 4. By the same token, an unsatisfactory assistance program outcome does not always mean that Bank performance was also unsatisfactory, and vice-versa. This becomes clearer once we consider that the Bank's contribution to the outcome of its assistance program is only part of the story. The assistance program's outcome is determined by the joint impact of four agents: (a) the Client; (b) the Bank; (c) partners and other stakeholders; and (d) exogenous forces (e.g., events of nature, international economic shocks, etc.). Under the right circumstances, a negative contribution from any one agent might overwhelm the positive contributions from the other three, and lead to an unsatisfactory outcome. 5. OED measures Bank performance primarily on the basis of contributory actions the Bank directly controlled. Judgments regarding Bank performance typically consider the relevance and implementation of the strategy, the design and supervision of the Bank's lending interventions, the scope, quality and follow-up of diagnostic work and 10In this note, assistance program refers to products and services generated in support of the economic development of a Client country over a specified period of time, and client refers to the country that receives the benefits of that program. Annex 3 (continued) 18 other AAA activities, the consistency of Bank's lending with its non-lending work and with its safeguard policies, and the Bank's partnership activities. Evaluation in Three Dimensions 6. As a check upon the inherent subjectivity of ratings, OED examines a number of elements that contribute to assistance program outcomes. The consistency of ratings is further tested by examining the country assistance program across three dimensions: (a) a Products and Services Dimension, involving a "bottom-up" analysis of major program inputs -- loans, AAA, and aid coordination; (b) a Development Impact Dimension, involving a "top-down" analysis of the principal program objectives for relevance, efficacy, outcome, sustainability, and institutional impact; and, (c) an Attribution Dimension, in which the evaluator assigns responsibility for the program outcome to the four categories of actors (see paragraph 4. above). Rating Assistance Program Outcome 7. In evaluating the outcome (expected development impact) of an assistance program, OED gauges the extent to which major strategic objectives were relevant and achieved, without any shortcomings. Programs typically express their goals in terms of higher-order objectives, such as poverty reduction. The country assistance strategy (CAS) may also establish intermediate goals, such as improved targeting of social services or promotion of integrated rural development, and specify how they are expected to contribute toward achieving the higher-order objective. OED's task is then to validate whether the intermediate objectives produced satisfactory net benefits, and whether the results chain specified in the CAS was valid. Where causal linkages were not fully specified in the CAS, it is the evaluator's task to reconstruct this causal chain from the available evidence, and assess relevance, efficacy, and outcome with reference to the intermediate and higher-order objectives. 8. Evaluators also assess the degree of Client ownership of international development priorities, such as the Millennium Development Goals, and Bank corporate advocacy priorities, such as safeguards. Ideally, any differences on dealing with these issues would be identified and resolved by the CAS, enabling the evaluator to focus on whether the trade-offs adopted were appropriate. However, in other instances, the strategy may be found to have glossed over certain conflicts, or avoided addressing key Client development constraints. In either case, the consequences could include a diminution of program relevance, a loss of Client ownership, and/or unwelcome side- effects, such as safeguard violations, all of which must be taken into account in judging program outcome. 19 Annex 3 (continued) Ratings Scale 9. OED utilizes six rating categories for outcome, ranging from highly satisfactory to highly unsatisfactory: Highly Satisfactory: The assistance program achieved at least acceptable progress toward all major relevant objectives, and had best practice development impact on one or more of them. No major shortcomings were identified. Satisfactory: The assistance program achieved acceptable progress toward all major relevant objectives. No best practice achievements or major shortcomings were identified. Moderately Satisfactory: The assistance program achieved acceptable progress toward most of its major relevant objectives. No major shortcomings were identified. Moderately Unsatisfactory: The assistance program did not make acceptable progress toward most of its major relevant objectives, or made acceptable progress on all of them, but either (a) did not take into adequate account a key development constraint or (b) produced a major shortcoming, such as a safeguard violation. Unsatisfactory: The assistance program did not make acceptable progress toward most of its major relevant objectives, and either (a) did not take into adequate account a key development constraint or (b) produced a major shortcoming, such as a safeguard violation. Highly Unsatisfactory: The assistance program did not make acceptable progress toward any of its major relevant objectives and did not take into adequate account a key development constraint, while also producing at least one major shortcoming, such as a safeguard violation. 10. The institutional development impact (IDI) can be rated as: high, substantial, modest, or negligible. IDI measures the extent to which the program bolstered the Client's ability to make more efficient, equitable and sustainable use of its human, financial, and natural resources. Examples of areas included in judging the institutional development impact of the program are: · the soundness of economic management; · the structure of the public sector, and, in particular, the civil service; · the institutional soundness of the financial sector; · the soundness of legal, regulatory, and judicial systems; · the extent of monitoring and evaluation systems; · the effectiveness of aid coordination; · the degree of financial accountability; · the extent of building NGO capacity; and, · the level of social and environmental capital. Annex 3 (continued) 20 11. Sustainability can be rated as highly likely, likely, unlikely, highly unlikely, or, if available information is insufficient, non-evaluable. Sustainability measures the resilience to risk of the development benefits of the country assistance program over time, taking into account eight factors: · technical resilience; · financial resilience (including policies on cost recovery); · economic resilience; · social support (including conditions subject to safeguard policies); · environmental resilience; · ownership by governments and other key stakeholders; · institutional support (including a supportive legal/regulatory framework, and organizational and management effectiveness); and, · resilience to exogenous effects, such as international economic shocks or changes in the political and security environments. 21 Annex 4. Steps Required to Conduct an OED Country Assistance Evaluation and CAE Contents 1. This section describes some of the main steps involved in conducting an OED's country assistance evaluation. These steps constitute best/common practice and should not be interpreted rigidly. In initiating a joint evaluation, flexibility is needed to accommodate the special circumstances of each evaluation/partner, particularly in cases where one of the partner is developing experience and expertise for the evaluation. CAE Production Steps 1. Country Assistance Evaluation (CAE) initiated. Country selected/Task Manager (TM) appointed. 2. Memo from TM to OEDCR Group Manager requesting budget. 3. Formal memo from OEDCR Group Manager to Country Director informing of intent to evaluate. 4. Consult with the Executive Director for the country selected for evaluation. 5. Memo to all OED staff requesting inputs (i.e., possible interview candidates, suggested literature, insights based on prior work in the country). 6. Appointment of two peer reviewers: one internal, one external to OEDCR. 7. Request collaboration/input from IFC (OEG) and MIGA (evaluation staff). · In standard CAEs, request a box or annex, as appropriate, describing IFC/MIGA activities in Client Country. · In joint CAEs, request collaboration in establishing joint terms of reference, mission, and final evaluation, as appropriate. 8. Consultant recruitment/TORs (big budget CAEs). · Check for Consultant Trust Fund availability. 9. Request relevant documents · Bank documents covering past ten years: CASs, economic reports, sector reports, LEAP general files, SARs/ICRs/PPARs (per project), sector/policy notes, internal audit reports, Consultative Group meeting minutes, etc. · Other donor resources: UNDP Annual Country Cooperation Framework, USAID Annual Presentation to Congress, Regional Development Bank country strategy papers, multi/bilateral donor evaluations and reports, FIAS reports, IFC/MIGA country strategies, IMF Recent Economic Developments and Staff reports, NGO country profiles. Annex 4 (continued) 22 · External documents: Central Bank annual reviews, other government publications, local newspapers and magazines (web sites), Foreign Investment Promotion agency publications/web site, Economist Country Profile, JOLIS search for journal articles and external literature, and so on. 10. Compile relevant data · Including: OED project ratings, APPI Index, lending project data (MIS), PIF ratings, comparative PBD cost data, Bank staff and management over past ten years, economic and social data (WDI) for target country and comparator countries, cross-check with IMF data, UNDP HDR data, other data from specialized UN agencies, donor disbursements (UNDP), PREM/DEC indicators, Freedom Index, Transparency International index, etc. · Create standard tables and other relevant tables. 11. Circulate a draft Approach Paper, describing the focus and methodology of the evaluation, to OEDCR, peer reviewers, and DMT. 12. Consult with Region about above. 13. One-stop review meeting to discuss the draft Approach Paper (attended by task manager, OEDCR manager, peer reviewers, and DMT). 14. Issue Approach Paper to CODE. 15. Investigation/writing (concurrent with steps 14). · Read, think, form, and test hypotheses. 16. Conduct HQ interviews. · Interview present and past country managers, country economists, country officers, sector managers, sector specialists, project advisors, Bank staff nationals of target country, IFC, IMF, and MIGA staff. 17. Conduct field interviews. · Prepare and send questionnaire for key stakeholders to be interviewed. · Field visit to interview past and present government officials in the various sectors, Resident Mission staff, private sector representatives, other donors, political opposition, NGOs, community groups, academics, local think tanks/research institutions, journalists, etc. · Design and conduct surveys (if appropriate). · Prepare Back-to-Office Report within three weeks of return. 18. Draft CAE to peer reviewers; incorporate comments. 19. Informal draft CAE to Country Team; incorporate comments. 23 Annex 4 (continued) 20. Draft CAE to OEDCR Group Manager; incorporate comments. · Include draft memo to Executive Directors, draft Management Action Record. 21. Draft to OED DMT at least one week in advance of one-stop review meeting. · Include draft memo to Executive Directors, draft Management Action Record, comments from peer reviewers. · TM to prepare minutes of one-stop review meeting, recording tone and main points of meeting and listing follow-up actions agreed upon. Comments incorporated into report. 22. Formal draft to Country Department and OPS/IFC/IMF over Group Manager's signature. · Request response to draft MAR in cover memo. · Request clearance to send CAE to client country in cover memo. · Share CAE informally with IMF Mission Chief. 23. If Country Department clearance is denied, send formal draft CAE to Regional Vice President with Region comments incorporated over DGO signature. 24. Formal draft CAE to Government; incorporate comments. 25. Final draft CAE to Regional Vice President and IMF area Department Director with cover memo from DGO. 26. Fast Track Brief (FTB) to President. · Obtain clearance from OEDCR Group Manager. · Obtain clearance/signature from DGO. 27. Final Report to SEC at least one month prior to CODE discussion. 28. Preparation of oral OED presentation to the Committee on Development Effectiveness (CODE) (1 ½ pages, double-spaced, maximum). · Obtain OEDCR Group Manager and DGO approval. 29. CODE discussion. 30. Publish and disseminate CAE. · The copy that is sent to the print shop for gray cover should include CODE minutes (green sheet), government letter/s (if received), completed MAR. But it should not include the PREM indicators table. · Undertake other dissemination activities (i.e., workshops). 31. Receive congratulations. Annex 4 (continued) 24 Suggested Minimum CAE Contents 32. Memorandum to the Executive Directors ­ summary of key findings and recommendations (1­2 pages). 33. Outline of CAE. a. Introductory section: (two chapters on socioeconomic development and challenges/constraints facing the country) · Government development policy directions and outcomes over the past ten years. · Basic opportunities and constraints in country. · Description of key economic and social trends/outcomes (with comparators). · Description of major policy initiatives over the past decade. · Medium-term prospects and challenges facing the country. Optional Box: Overview of key country strategy documents (CASs, PRSPs, SSPs, etc.) operative during relevant evaluation period, including key objectives of country assistance program, proposed interventions and instruments to achieve them). b. Bottom-up evaluation of Bank products and services for relevance, efficacy, and efficiency in meeting stated objectives with minimal shortcomings. · Bank loans, credits, and grants. · Nonlending services (strategic and policy advice, aid coordination and resource mobilization, analytical and advisory services). c. Top-down assessment of development impact of country assistance program for relevance, efficacy, and efficiency in (i) alleviating key development constraints, while (ii) meeting priority Bank and client country development objectives. · Outcome · Sustainability · Institutional development d. Attribution of development impact. · Bank · Client country · Aid partners · Exogenous factors e. Conclusions and lessons of experience. 25 Bibliography World Bank Operations Evaluation Department and Islamic Development Bank Operations Evaluation office (2004). Jordan: Supporting Stable Development in a Challenging Region: A Joint World Bank-Islamic Development Bank Evaluation, OED, Washington D.C.: World Bank and OEO, Jeddah, Saudi Arabia: Islamic Development Bank. World Bank Operations Evaluation Department (2004a). Rwanda: Country Assistance Evaluation, Report No. 27568, OED, Washington D.C.: World Bank. ------. (2004b). Republic of Tunisia: Country Assistance Evaluation, Report No. 29669, OED, Washington D.C.: World Bank. ------. (2004c). Eritrea: Country Assistance Evaluation, Report No. 28778-ER, OED, Washington D.C.: World Bank. ------. (2003). Peru: Country Assistance Evaluation, Report No. 24898-PE, OED, Washington D.C.: World Bank. World Bank Operations Evaluation Department and African Development Bank Operations Evaluation Department (2002). Lesotho: Development in a Challenging Environment: A Joint World Bank-African Development Bank Evaluation, OED, Washington D.C.: World Bank and OPEV, Abidjan: African Development Bank. World Bank Operations Evaluation Department (2001). Kazakhstan: Country Assistance Evaluation of Bank Assistance, Report No. 21862, OED, Washington D.C.: World Bank. ------. (2000a). Burkina Faso: Country Assistance Evaluation, Report No. 20704, OED, Washington D.C.: World Bank. ------. (2000b). Tanzania: Country Assistance Evaluation, Report No. 20902, OED, Washington D.C.: World Bank.