THE WORLD BANK Financial Sector Assessment Program IEG Review of the Joint World Bank and IMF Initiative INDEPENDENT EVALUATION GROUP ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Independent Evaluation Group (IEG) reports directly to the Bank's Board of Executive Directors. IEG assess- es what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contri- bution of the Bank to a country's overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. W O R L D B A N K I N D E P E N D E N T E V A L U A T I O N G R O U P Financial Sector Assessment Program IEG Review of the Joint World Bank and IMF Initiative 2006 The World Bank http://www.worldbank.org/ieg Washington, D.C. © 2006 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 Telephone 202-473-1000 Internet www.worldbank.org E-mail feedback@worldbank.org All rights reserved 1 2 3 4 5 09 08 07 06 This volume is a product of the staff of the International Bank for Reconstruction and Development / The World Bank. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750- 4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. Photo credit: World currencies. © Royalty-free/CORBIS. ISBN-10: 0-8213-6652-1 ISBN-13: 978-0-8213-6652-3 e-ISBN: 0-8213-6653-X DOI: 10.1596/978-0-8213-6652-3 Library of Congress Cataloging-in-Publication Data has been applied for. Ainsworth, Martha, 1955­ Committing to results : improving the effectiveness of HIV/AIDS assistance: an OED evaluation of the World Bank's assistance for HIV/AIDS control / Martha Ainsworth, Denise A. Vaillancourt, Judith Hahn Gaubatz. p. cm. -- (Operations evaluation studies) Includes bibliographical references. ISBN-13: 978-0-8213-6388-1 ISBN-10: 0-8213-6388-3 1. Economic assistance--Developing countries--Evaluation. 2. AIDS (Disease)--Economic aspects--Developing countries. 3. HIV infections--Economic aspects--Developing countries. 4. AIDS (Diseasse)--Developing countries--Prevention. 5. HIV infections--Developing countries--Prevention. 6. World Bank. I. Vaillancourt, Denise. II. Hahn Gaubatz, Judith. III. Title. IV. World Bank operations evaluation study. HC60.A4575 World Bank InfoShop 2005 Independent Evaluation Group E-mail: pic@worldbank.org Knowledge Programs and Evaluation Capacity Telephone: 202-458-5454 Development (IEGKE) Facsimile: 202-522-1500 E-mail: eline@worldbank.org 362.196'9792'0091726--dc22 Telephone: 202-458-4497 2005052329 Facsimile: 202-522-3125 Printed on Recycled Paper Contents v Acknowledgments vii Foreword xi Preface xiii Executive Summary xix Acronyms and Abbreviations 1 1 Introduction 1 Main Objectives of FSAP 2 Design and Implementation of the FSAP 2 Scope of the Evaluation 5 2 Relevance, Program Design, and Inputs 5 Relevance 5 Country Selection 8 Scope of Assessments 9 Updates 9 Costs 11 Advance Preparation 11 Review Stage and Staffing 13 Joint Program between the IMF and World Bank 15 3 Outputs 15 Quality of Analysis 20 Articulation of Findings 21 Informing the Executive Boards of the World Bank and IMF 25 4 Impact of the FSAP 26 Policy Debate 26 Implementation of Recommendations 27 Integration with Bank Country Programs 31 Knowledge Sharing at the Bank FINANCIAL SECTOR ASSESSMENT PROGRAM 31 Influence on the IMF 32 Influence on other Donors, IFIs, and Agencies 33 Standard Setters 33 Influence on Financial Markets 33 Disclosure 35 5 Key Findings and Recommendations 38 Recommendations 41 Appendixes 43 A: Assessment and Updates 45 B: FSAP Framework 47 C: Methodology for FSAP Evaluation 51 D: Survey of Stakeholders 53 E: Management Response 59 F: Chairman's Summary, Committee on Development Effectiveness (CODE) 63 Endnotes 69 References Boxes 17 3.1 Stress Testing 19 3.2 Different Approaches to Nonbanking Issues 20 3.3 Assessment of FSAP Updates 29 4.1 Link between the FSAP and Bank Programs 32 4.2 Donor Concerns About the FSAP Figures 7 2.1 Percentage of Countries with FSAPs 9 2.2 Recommended Frequency of FSAP Updates 10 2.3 Average FSAP Cost by Income Level 12 2.4 Satisfaction with Comments from Bank Departments 12 2.5 Satisfaction with Comments from IMF Departments 16 3.1 Most Useful Analytical Components 21 3.2 Reasons for Noncandid FSAP Recommendations 26 4.1 Reasons Why Authorities Requested an FSAP 28 4.2 Contributions to the Policy Debate 28 4.3 Degree of Implementation of FSAP Recommendations Tables 6 2.1 Select Countries with Financial Crises: Changes in Poverty Rates and GDP 7 2.2 Countries with FSAPs, by Income Level 7 2.3 Percentage of Countries with FSAPs per Bank Region 8 2.4 Frequency of Assessment Done in Fiscal Years 2000­04 8 2.5 ROSCs per Category of Country 11 2.6 Completion Times of FSAP Documents 16 3.1 Summary of Results of Detailed Reviews 22 3.2 Completed and Disclosed FSAP Documents 27 4.1 Usefulness of FSAP Areas iv Acknowledgments This evaluation was written by Lily Chu, with the edited the report for publication. Roziah Baba assistance of Khalid Siraj, Manuel Hinds, Marcelo (IEGCR) provided administrative assistance. In Bueno, Ilka Funke, Jane Hwang, and Gulmira addition, IEG would like to thank the IEO team Karaguisheva (IEGCR). This evaluation has also members: David Goldsbrough, Martin Kaufman, benefited from the comments of two peer Marko Skreb, Mariano Cortes, Heinz Rudolph, reviewers: Laurie Effron (IEGCR) and Gloria and Tim de Vaan for their excellent insights and Grandolini (BCFBD). Helen Chin (IEGKE) assistance. Director-General, Evaluation: Vinod Thomas Director, Independent Evaluation Group, World Bank: Ajay Chhibber Senior Manager, Country Evaluation and Regional Relations: R. Kyle Peters Task Manager: Lily L. Chu v Foreword T he Financial Sector Assessment Program (FSAP) is a major initiative, un- dertaken jointly by the World Bank and the IMF, in response to the fi- nancial crises of the late 1990s. The ultimate objectives of the program are (i) Country Selection. The FSAP has not covered all to identify and resolve financial sector vulnera- "systemically important" or vulnerable bilities and their macroeconomic stability countries, or focused on countries where implications; and (ii) to foster financial sector assessments can be most effectively used for development and its contributions to economic development. The voluntary nature of the growth. In addition, the FSAP is expected to program limits the program's overall effective- help the Bank, IMF, and other institutions in the ness in identifying systemic risks; however, design of appropriate assistance to address there is a strong consensus among authorities issues identified by the FSAP. and staff that the voluntary nature of the This evaluation focuses on reviewing aspects program should be maintained. of the FSAP that affect the attainment of the While the overall quality of diagnostics was above objectives, specifically, the relevance of generally good, the quality and appropriate- the program, quality, and effectiveness of ness of coverage of specific sectors was uneven. inputs and outputs, and the impact of the Assessments of the nonbanking sectors were assessments on different audiences. An evalua- not as consistent as those of the banking sector. tion of the ultimate impact of the program The Bank also needs to (i) develop better would be premature, given the limited time approaches toward analyzing missing market since the program was expanded. and access issues; (ii) devise creative solutions The evaluation found that the FSAP is a to improve those areas; and (iii) improve good-quality diagnostic tool. Joint Bank and prioritization, sequencing, and analysis of IMF cooperation has allowed an integrated implementation capacity. approach toward identifying financial sector vulnerabilities and development needs, and Candor. The candor of the reports was generally has expanded the depth and quality of the skills satisfactory, although there have been some base. The assessments, however, fall short in instances where the government or manage- prioritizing recommendations and integrating ment has pressured staff to soften the written the findings and recommendations of the messages, or where the staff has chosen to assessments into overall country programs. convey key messages through presentations vii FINANCIAL SECTOR ASSESSMENT PROGRAM and discussions, rather than through written The program has improved IMF surveil- reports. Also, the current practice of informing lance, but has not had as strong an impact on the Executive Boards of the Bank and IMF is other donors' and institutions' programs. not effective. Financial Sector Assessments While other donors support the initiative, and (FSAs) are delivered long after mission work is some have been able to use the FSAP to help completed, so the Boards are not kept shape their own programs, improved donor informed on a timely basis. In addition, coordination would allow better leverage of because FSAs are summaries of the full assess- donor resources to support reforms ments, sometimes the full context and nuances of the report have been inadequately Efficiency. The FSAP is a costly program, conveyed. although the Bank's share of costs are commen- surate with the costs of other major financial Impact. While the program is still too new to sector studies. Bank resources could be used have had measurable effects on ultimate more effectively if more time were spent on outcomes (reduction of vulnerabilities and appropriate country selection and better tailor- improved financial sector development), the ing of coverage. Efficiency and quality of the assessments have been positively cited by assessments would also improve if the Bank country authorities for (i) providing an made full use of existing data and staff with independent evaluation of the system; (ii) country experience, and used the FSAP reports expanding their knowledge of the financial to internally leverage its knowledge base. sector; (iii) improving technical abilities; and The relevance and efficiency of the program (iv) contributing to policy dialogue within the could be enhanced through improved country country. "Results on the ground" have been selection and better tailoring of assessments. more difficult to assess; authorities have stated Selection criteria would strengthen the that most recommendations have been program's mandates to decrease systemic risk implemented, whereas Bank and IMF staff, as and support financial sector development well as Independent Evaluation Group (IEG) through focusing on systemically important reviews of country programs, have not yet seen countries, while realistically assessing the likeli- as much evidence of implementation of critical hood of impact (client commitment, preexist- reforms. ing conditions, etc.). While the program has raised awareness Better integration of the findings and among government authorities and within the recommendations of the FSAP into the overall Bank on the importance of the financial sector country program is needed. This would include on stability and development, IEG finds that better coordination with donors. In the cases only 42 percent of the assessments have had a where a decision is made not to support FSAP significant impact on the Bank's country recommendations through the Bank's programs or dialogue. In another 34 percent, program, the Country Assistance Strategy the assessments had a minor influence, such as (CAS) needs to make the rationale clear through reinforcing previous sector knowledge (limited resources and other higher priority or contributing to the ongoing dialogue. needs, availability of support through other The FSAP does appear to have influenced sources, need to work on preconditions, etc.). economic and sector work (ESW) at the Bank; Improvements must be made in the process countries that have undertaken assessments of keeping the Executive Board informed. The appear to follow up with smaller, more targeted current process, on average, takes almost a year ESW, whereas countries with no assessments after the completion of mission work to deliver are more likely to carry out major pieces of documents to the Board. In addition, there is ESW. This indicated that assessments are not a clear venue for discussing critical issues effectively leveraged. and findings of the assessments. viii FOREWORD Finally, the Bank must do a better job of programs within countries, across regions, and knowledge sharing, both within the Bank and with partners. In addition, more effective with partners. The assessments have resulted in partnering with other sectors would improve the a large amount of information that could assist knowledge base and insights of the assessments. Vinod Thomas Director-General Evaluation ix Preface his evaluation presents an independent assessment of the joint World T Bank-IMF Financial Sector Assessment Program (FSAP). This is the third in a series of reviews of financial sector work conducted by the Independent Evaluation Group (IEG). The first evaluation was of World Bank Lending This evaluation was circulated to the Bank for Lines of Credit; the second was an IEG Review Management involved in financial sector of World Bank Assistance for Financial Sector support, the Financial Sector Board of the Reform, not including lines of credit or financial Bank, the FSAP Unit (FSEFS), and the Financial sector assessments. Sector Operations and Policy Department The FSAP is conducted as a joint program (OPD). The Management response is attached between the Bank and the IMF, and the IMF's as appendix E. Independent Evaluation Office (IEO) carried This evaluation was discussed by the out a parallel assessment of the program. The Committee of Development Effectiveness two teams worked in close collaboration, (CODE) on February 6, 2006. The CODE conducting joint interviews and surveys, and Chairman's summary is included as appendix F. sharing data and interim work products. xi Main Messages · The quality of the diagnostics has been good, although quality and the appropriateness of specific sectors is uneven. · Country authorities have found the assessment to be useful. · FSAP recommendations need to be better integrated into Bank pro- grams. · Country selection needs to better reflect surveillance priorities and the likelihood of financial sector reform. · The scope of assessments must be more tailored to the specific needs of the country. · The program must do a better job of keeping the Executive Board informed in a timely manner. Executive Summary his evaluation is one of three IEG evaluations of the Bank's financial sec- T tor work. The first was a review of World Bank Lending for Lines of Credit; the second was an IEG Review of World Bank Assistance for Fi- nancial Sector Reform, which did not include lines of credit or financial sec- tor assessments. This evaluation focuses on the Financial Sector resolution of financial sector vulnerabilities and Assessment Program (FSAP). The IMF's their macroeconomic stability implications, Independent Evaluation Office (IEO) which would then lead to a reduction in the undertook a parallel evaluation of the FSAP. likelihood of crisis and improved global The IEO and IEG have collaborated closely on financial stability, and (ii) fostering financial all aspects of their respective evaluations. sector development and its contribution to The Financial Sector Assessment Program economic growth. In addition, the FSAP was (FSAP) is a major initiative, undertaken jointly expected to help the Bank, IMF, and other by the World Bank and the IMF, in response to institutions design appropriate assistance to the financial crises of the late 1990s. The FSAP address issues identified by the assessments. was set up in May 1999, initially as a 12-country Ideally, this FSAP evaluation would measure pilot exercise consisting of diagnostic studies whether the ultimate objectives have been designed to facilitate early detection of achieved. But, given the limited time period financial sector vulnerabilities and identifica- since the program started and the voluntary tion of financial sector development needs, as nature of the program, such an analysis is well as support an improved and coordinated premature. This evaluation instead focuses on dialogue among the national authorities, the reviewing aspects of the FSAP that affect the Bank, and the IMF. As of October 2005, over attainment of those objectives, specifically the 109 country assessments and 18 updates have relevance of the program, the quality and been completed or are ongoing, and the effectiveness of inputs and outputs, and the program has involved a significant deployment impact of the assessments on different of resources at the Bank. audiences. Various policy documents and review papers The FSAP is predicated on the assumption have indicated that the ultimate objectives of that the objectives of financial sector stability the FSAP comprise: (i) the identification and and development are essential and relevant to xiii FINANCIAL SECTOR ASSESSMENT PROGRAM the Bank's and the IMF's missions. Reviews of of some Bank staff that the FSAP is only a economic data and literature have confirmed diagnostic tool, although the mandate of the that financial sector crises often lead to sharply FSAP envisioned the use of the assessments as increased poverty and reduced growth, and platforms for future work, policy dialogue, and that there is a link between financial sector as catalysts for financial sector reform. The development and poverty reduction and assessment could be better oriented toward improved growth. Hence, this evaluation follow-up, with improved prioritization and confirms that the program's objectives are sequencing. relevant to the Bank's mission. This evaluation has found that the FSAP is a Teams. The quality of teams was rated quite good-quality diagnostic tool and the overall highly; 93 percent of country authorities concept for the program is sound. Joint Bank responding to a survey on the FSAP expressed and IMF cooperation has allowed an integrated satisfaction with the FSAP team's skills. approach toward financial sector vulnerabilities However, authorities have raised concerns and development needs, while expanding the about continuity of staff and the need for depth and quality of analytical expertise. strong country experience. In addition, senior- However, the assessments are weak in prioriti- ity of staff involved in the program has been zation and the Bank falls short in integrating dropping off; if the quality of the assessments the findings and recommendations of the is to be maintained, management will need to assessments into its country programs, and monitor staffing closely. could also take other steps (discussed below) to sharpen the program's relevance, quality, Candor. The candor of the reports was generally impact, and efficiency. satisfactory, although there have been some instances where the governments or manage- Quality. The overall diagnostics were generally ment have pressured the staff to soften the good, but the quality and appropriateness of written reports, or where the staff have chosen coverage of specific sectors was uneven. While to convey key messages through presentation banking sector coverage was satisfactory, the and discussions, rather than through the coverage in the nonbank financial sectors was written reports. not consistently of the same quality. The Bank also needs to develop better approaches Informing the Executive Board. The current toward analyzing missing markets and access practice of informing the Board is not effective. issues, and devise creative solutions to improve Financial Sector Assessments (FSAs) are those areas. This would include: (i) improving delivered a substantial time (both in absolute the use of indicators and datasets; (ii) deepen- terms and compared with the IMF's delivery of ing the analysis of political economy, culture, the Financial Sector Stability Assessments and other country-specific aspects that would (FSSAs) after the mission work is completed, allow the team to help authorities design so the Board is not informed on a timely basis. tailored solutions; and (iii) focusing more on Also, given that the FSAs are summaries of the interlinkages between sectors, which would full assessments, sometimes the full context help authorities to develop better priorities and nuances of the report have been and sequencing. inadequately conveyed. Prioritization of recommendations was weak, Impact. While it is still too early to judge the full which adversely affected the impact of the impact of the program, the assessments have overall program. There were often too many been well received, and country authorities "priority" recommendations, or sequencing have cited the assessments for (i) providing an and implementation capacity were not well "independent evaluation" of the system, (ii) addressed. This may partly be due to the view expanding their knowledge of financial sector xiv EXECUTIVE SUMMARY vulnerabilities, (iii) improving technical abili- government authorities in the context of ties, and (iv) contributing to the policy dia- surveillance work); (iii) the country units are logue within the country. "Results on the not always fully involved in planning the scope ground" have been difficult to assess; authori- of, or participating in, the assessment, which ties have stated that most recommendations may in turn affect both the efficiency of the have been implemented, whereas Bank and assessment, and the effectiveness of follow-up IMF staff, as well as IEG reviews of the country activities. programs, have not seen as much evidence of The FSAP does appear to have influenced implementation of critical reforms. For ESW at the Bank, as countries that have had example, authorities have noted post-FSAP assessments appear to have undertaken changes, especially in legal, regulatory, and smaller, more targeted ESW in subsequent supervisory reforms, but the attribution is years. Countries without assessments are more difficult, as many of these reforms have been likely to carry out major pieces of ESW. This part of previous Bank or IMF policy discus- indicates that assessments can be usefully and sions. In addition, the more difficult reforms effectively leveraged. will take more time, and greater impact may be seen in the future. Impact on the IMF. One impact of the program has been an improvement in IMF surveillance. Impact within the Bank. While the program has IEO concluded in its evaluation that the raised awareness with authorities and within program had significantly deepened the IMF's the Bank on the importance of the financial understanding of the financial sector in specific sector on stability and development, IEG has countries, improved the articulation of policy found that only 42 percent of the assessments recommendations and discussions with author- had a significant impact on the Bank's country ities, and helped support policy and institu- programs or dialogue, as manifested by tional changes in the countries. important changes or strengthening of the Bank's program or dialogue. In another 34 Partners. However, the FSAP has not had as percent of countries, the assessments had a strong an impact on other donors' and institu- minor influence; for example, the assessment tions' programs. While other donors support might have reinforced previous sector the initiative, and some have been able to use knowledge and contributed to the ongoing the FSAP to help shape their own programs, dialogue, and may have helped spur minor the flow of information is less timely and changes in the financial sector program, but complete than they would wish. Improvements there was no evidence that the direction or in donor consultation would allow better pace of the program had changed due to the leverage of donor resources to support assessment. The remaining 24 percent seemed reforms. to have had little influence on the Bank's program. Factors that could affect the degree Country selection. The FSAP has not covered all of impact include the following: (i) country "systemically important" or vulnerable selection does not always factor in the useful- countries, or selected countries where financial ness of a development assessment (countries sector development assessments can be most that are more advanced may not need Bank effectively used. The voluntary nature of the assistance; countries that lack the precondi- program limits the program's overall effective- tions for a well-functioning financial sector, or ness in identifying systemic risks; however, have no interest in reform, might not be able there is a strong consensus among authorities use a development assessment effectively or in and staff surveyed for this review that the a timely fashion); (ii) there is no clear voluntary nature of the program should be mechanism for Bank follow-up (such as the maintained. Without the full cooperation of IMF's Article IV consultations (discussions with authorities, access to detailed information and xv FINANCIAL SECTOR ASSESSMENT PROGRAM key staff would be difficult, and would reduce that a systemically important, vulnerable the effectiveness of the exercise significantly. country may require a full set of Reports on Observance of Standards and Codes (ROSCs), Efficiency. The FSAP is an expensive program, and minimal discussion on missing markets or although the Bank's share of costs is commen- access issues, whereas another country may surate with the costs of other major financial not require any formal ROSCs, but instead sector studies. Bank resources could be more focus on access issues. Also, this process would efficiently used if more time were spent on be more efficient if more time was spent in appropriate country selection and better tailor- advance, working with country units and ing of coverage. Efficiency and quality of the authorities on developing prioritized terms of assessments would also improve if the Bank references (TORs). made full use of existing data and staff with country experience, and used the FSAP reports Integrate the FSAP into a full reform program. The to internally leverage its knowledge base (and Bank, in consultation with the IMF and other that of the IMF and other institutions). technical assistance providers, must establish a clearer framework for coordinating follow-up Recommendations programs, including technical assistance activi- ties, based on the country's own action plans. Sharpening country selection and the scope of While details will differ by country, building assessment have major impacts on the rele- upon the FSAP by having follow-up discussions vance of the program, both for the reduction of with country authorities on technical assistance vulnerabilities, and the impact on the develop- and other development needs is essential; such ment agendas of the countries. Given limited discussions should include Bank technical and budgets, there are difficult tradeoffs to be country staff, as well as the IMF's Monetary and made, particularly when balancing the need to Financial Systems Department and area staff. improve surveillance with development priori- Once development and other assistance priori- ties. Management staff of the Bank and the IMF ties are determined, this information should be need to select countries for assessments and quickly shared with the donor community. In updates according to the priorities identified addition, to improve donor effectiveness, the by the Boards, as well as the likely impact of the following recommendations might be consid- FSAP in a particular country, and adjust timing ered: (i) inform donors in advance about the to suit the situation. In some cases, such as timing of the FSAP, so that donors can adjust systemically important and vulnerable their program timetables accordingly; (ii) countries, this may mean that updates are improve consultations with donors who are carried out every two or three years. In other active in the financial sector during the FSAP countries that are less systemically important mission, including presentations of key or that are unlikely to treat financial sector findings; and (iii) provide donors with better development as a priority, initial assessments and more timely access to reports. or updates could be delayed by several years. A Country units and the financial sector teams priority list, as well as the rationale for must work more closely to better integrate the selection, should be presented to the Board. In FSAP findings and recommendations into the cases where countries have not volunteered, Bank's country strategies. This would include management needs to inform the Board of greater participation of the country units in the those gaps, and seek other ways to work with design of TORs, mission teams, and follow-up the country in addressing critical issues in the discussions. If a strategic decision is made that financial sector. Bank follow-up to the FSAP will be limited, the The scope of the initial assessments and CAS needs to briefly explain the reasons why updates must be more tailored to address the (limited resources which need to be allocated specific needs of the country. This may mean to other priorities in the county, availability of xvi EXECUTIVE SUMMARY other donor programs, need to sequence other information to provide recommendations for reforms first, lack of interest or commitment of future work (within countries, across regions, the client in those sectors, etc.). and for future assessments). Informing the Executive Board. The process of Knowledge sharing. While remaining mindful of informing the Board is currently not working the needs of safeguarding sensitive and well. The FSAs take too long to produce. A confidential information, the Bank must more deadline needs to be placed on the production actively leverage the knowledge gained from time of the FSA. One option to speed up this extensive exercise. This would include production is to leverage the work by the IMF actively disseminating "best practice" analysis, more effectively. The IMF releases the FSSA an more discussion on concerns and issues that average of 16 days after the final report is sent arise while conducting assessments, and to authorities. While cognizant that the differ- improved sharing of data and insights gained ent foci, timetables, and processes make it through the program. The financial sector difficult to quickly produce a joint paper, the teams also need to partner more effectively Bank could use the FSSA as a platform to with other sector teams to maximize collection summarize the main findings of the assess- and sharing of information. ment, and then add chapters or appendixes on In summary, the technical quality of the development or other issues, on which the IMF diagnostics has generally been good, although may not have focused. there are inconsistencies in the coverage and Unlike the IMF, which discusses the FSSAs quality of some sectors. The assessments, with its Board, there is no discussion of FSA however, often have weak prioritization of documents at the Bank's Board, so the import, recommendations, and do not adequately context, and nuances of the reports are often consider sequencing or institutional capacity not clear. Critical information must be brought constraints. Furthermore, the Bank does not quickly to the Board or a subcommittee for do enough to integrate the assessments' discussion. In addition, discussions of key FSAP findings and recommendations into its country findings and recommendations, and their programs. While the IEO has found that the integration into the country program (or, as assessments have improved the IMF's surveil- noted above, the decision not to include them lance work and significantly deepened the into the country program), would ideally be IMF's understanding of the financial sector in part of the Country Assistance Strategy (CAS) some countries, the parallel impact on the report and discussions. Bank management Bank and its financial sector development work could also consider periodic technical briefings has not yet occurred to the same degree. The summarizing major findings from recent full potential of the FSAP's usefulness to the assessments, as well as synthesizing the Bank, therefore, has not yet been attained. xvii ACRONYMS AND ABBREVIATIONS BCP Basel Core Principles (for Effective Banking Supervision) CAS Country Assistance Strategy CODE Committee on Development Effectiveness CPSS Committee on Payment Systems and Settlements DFID Department for International Development ESW Economic and sector work FIRST Financial Sector Reform and Strengthening Initiative FSA Financial Sector Assessment FSAP Financial Sector Assessment Program FSEFS FSAP Unit FSI Financial Soundness Indicators FSSA Financial System Stability Assessment FY Fiscal year IADB Inter-American Development Bank IAIS International Organization of Insurance Supervisors IEG Independent Evaluation Group (formally OED) IEO Independent Evaluation Office (of the IMF) IFI International financial institution IMF International Monetary Fund IOSCO International Organization of Securities Commissions OED Operations Evaluation Department (changed to IEG) PRSP Poverty Reduction and Strategy Paper ROSC Report on Observance of Standards and Codes SECO Secretariat d'Etat a l'Economie TMF Code of Good Practices on Transparency in Monetary and Financial Policies TOR Terms of reference xix 1 Introduction he Financial Sector Assessment Program (FSAP) is a major initiative, un- T dertaken jointly by the World Bank and the IMF, in response to the fi- nancial crises of the late 1990s. The FSAP was set up in May 1999, initially as a 12-country pilot exercise to provide diagnoses to client countries to facilitate early detection of financial sector vulnerabilities and identification of financial sector development needs, as well as to support an improved and coordinated dialogue among the national authorities, World Bank, and IMF. Each assessment would include written reports institutions design appropriate assistance to for government authorities and the Executive address financial sector issues identified by the Boards of the Bank and the IMF.1 As of October FSAPs. 2005, over 109 country assessments and 18 The rationale for placing development and updates have been completed or are ongoing, stability objectives under the same program and the program has involved a significant (and thus for having a joint Bank-IMF initiative) deployment of resources at the Bank. (See has been elaborated over time. Some reasons appendix A for a list of assessments and were operational--to optimize the use of the updates). limited pool of expert resources, to avoid duplication of efforts, to promote consistency Main Objectives of FSAP of the two institutions' analyses and advice on Various policy documents and review papers financial sector issues. Other reasons reflected have indicated that the ultimate objectives of the recognition--reinforced by the early pilot the FSAP comprise: (i) the identification and experience--that most countries face both resolution of financial sector vulnerabilities and vulnerability and development issues, and their macroeconomic stability implications, considerable synergies might be achieved by which would then lead to a reduction in the addressing them jointly (e.g., institutional likelihood of crisis and improved global development aspects that may affect financial financial stability, and (ii) fostering financial stability). These factors argued for an integrated sector development and its contribution to approach to financial sector assessment. In economic growth.2 In addition, the FSAP was terms of responsibilities, based on policy expected to help the Bank, IMF, and other documents and their respective institutional 1 FINANCIAL SECTOR ASSESSMENT PROGRAM expertise and mandates, the IMF would take the its regular Article IV consultations. While the aide- lead on stability issues, while the Bank would mémoire is a confidential document, the FSA and lead on development issues. However, because FSSA are provided to the Board. Governments development aspects may have stability implica- then have the option to publicly disclose these tions, and vice-versa, coordination between the assessments or not. two institutions is key.3 The FSAP was initially conceived of as a Scope of the Evaluation diagnostic and policy advice tool that would This evaluation is one of three IEG evalua- provide: (i) confidential advice to country tions of the Bank's financial sector work. The authorities, (ii) information for the Bank and first evaluation was of World Bank Lending for the IMF on development and stability issues, Lines of Credit; the second was an IEG Review and (iii) in some cases, information to the of World Bank Assistance for Financial Sector international community. Reform, not including lines of credit; and this evaluation, the third, is of the Financial Sector Design and Implementation of the FSAP Assessment Program. The IMF's Independent In September 1998, the Financial Sector Liaison Evaluation Office (IEO) has undertaken a Committee was set up, comprising senior staff parallel evaluation of the FSAP. The IEO and IEG from the Bank and IMF, to enhance operational have collaborated closely on all aspects of their coordination between the two institutions on respective evaluations. financial sector issues, including policy advice Ideally, this FSAP evaluation would measure and support. Subsequently, the committee took whether the ultimate objectives of increased managerial responsibility over the FSAP, includ- financial stability and the reduction in the likeli- ing country selection, assigning lead responsi- hood of crises, and fostering financial sector bility between the IMF and the Bank in each development and its contribution to economic country, and resolving contentious issues in growth have been achieved. But, given the specific cases. limited time period since the program started The FSAP provides and the voluntary nature of the program, such In response to the findings and recommen- an analysis is premature.4 This FSAP evaluation financial crises of the late dationstocountryauthor- instead focuses on reviewing aspects of the ities both verbally-- FSAP that would affect the attainment of those 1990s, the FSAP was in a concluding session objectives, specifically the relevance of the created to facilitate early with senior national program, the quality and effectiveness of inputs detection of authorities--and in writ- and outputs (see appendix B for an illustrative tendocuments.Thedocu- framework), and impact of the assessments on vulnerabilities and ments include (i) an aide- different audiences. financial sector mémoire, previously re- To this end, IEG reviewed all FSAPs completed development needs. ferred to as an FSAP by fiscal year 2004, as well as related documents report; (ii) detailed assess- such as CASs and other Bank documents, and ments of compliance with selected standards and considered the articulation of recommendations codes (and associated Reports on the Observance and linkages to technical assistance (TA), policy of Standards and Codes (ROSCs)); and (iii) and institutional change, and integration with supporting technical notes. In addition, drawing other Bank work.5 In addition, together with IEO, on the FSAP findings for a country, Bank staff detailed country reviews were completed for 19 prepare a Financial Sector Assessment (FSA), countries (IEO conducted an additional six summarizing major findings of the FSAP mission; detailed reviews of FSAPs in developed the IMF staff prepare a Financial System Stability economies),6 where the review of core docu- Assessment (FSSA), which summarizes the ments were supplemented with detailed desk findings of relevance to IMF surveillance as part of reviews of additional documents including 2 INTRODUCTION project documents, relevant economic and tative of the group of countries that had partici- sector work (ESW) (both financial sector and pated in the FSAP. (See appendix C for a detailed multisector, such as Poverty Reduction Strategy discussion of methodology; appendix D provides Papers and Country Economic Memorandums), more details on the survey). IEG and IEO also and IMF documents. The detailed country conducted additional interviews with donors and reviews also included interviews with team other outside parties. leaders, mission chiefs, and country directors; This paper is structured as follows: chapter 2 and in all but two cases, interviews with authori- focuses on relevance, program design and ties were conducted.7 Following the detailed inputs; chapter 3 discusses outputs; chapter 4 reviews, IEG and IEO carried out a survey to discusses the impact of the program outside ascertain to what extent information gleaned the Bank; and chapter 5 summarizes key from interviews and desk reviews were represen- findings and recommendations. 3 2 Relevance, Program Design, and Inputs Relevance he FSAP is predicated on the assumption that the objectives of finan- T cial sector stability and development are essential and relevant to the Bank's and the IMF's missions. The initial documents setting out the rationale for the program note that crises in the financial sectors of emerging markets "highlighted the devastating effects of financial system failures on macroeconomic stability, growth, and welfare"1 and "a well-functioning fi- nancial services sector is essential for sustained economic development and poverty reduction."2 The last decade provides ample evidence of the mission of fostering economic growth and devastating impact that financial crises can have reducing poverty. on countries. Most of the countries that have had such crises have seen their poverty rates Country Selection jump dramatically and growth rates turn sharply The first key design element that will affect negative (table 2.1), and some of the countries relevance is country selection. If the objectives have taken years to recover their precrisis per of the FSAP are to be met, then the FSAP should capita income levels. focus on assessing countries that are systemi- Research has also shown a causal link cally important or are considered more vulnera- between financial sector development and ble, and ones where the FSAP can have an growth,3 and there is a new body of literature influence on financial sector development. showing the link between financial sector The Boards noted these priorities in their development and poverty, including Beck, early discussions of country selection, which Demirguc-Kunt, and Levine (2004), which finds stated that the FSAP arose as a response to that financial sector development leads to less "episodes of financial crisis and cross-border income inequality, and Honohan (2004), which contagion"4 in the 1990s. Given the resource shows that financial depth is correlated with the constraints of the FSAP program, "within any level of poverty. The objectives of the FSAP are one year, giving a higher priority to systemically thus highly relevant to the Bank's overall important countries would be warranted, while 5 FINANCIAL SECTOR ASSESSMENT PROGRAM Table 2.1. Select Countries with Financial Crises: Changes in Poverty Rates and GDP Poverty rates Real change in GDP Countries Crisis year Precrisis Postcrisis (%, year following crisis) Mexico 1995 23.32 28.60 ­6.2 (1994) (1996) Indonesia 1998 25.7 37.0 ­15.4 (1996) (1999) Korea, Rep of 1998 11.4 23.2 ­10.6 (1997) (1998) Thailand 1998 12.5 15.7 ­13.7 (1996) (1999) Malaysia 1998 .04 0.11 ­12.7 (1997) (2000) Argentina 2002 37 58 ­10.9 (2001) (2002) Source: Argentina data are from the World Bank 2003a. For other countries, "crisis year" is the peak crisis year, as determined in Caprio and Klingebiel 2003; Poverty rates are from World Bank 2000a, annex 1 and "Mexico: Country Assistance Strategy (late 1990s), and use the poverty levels as defined in those papers. GDP growth numbers are from Claessens, Kingebiel, and Laeven 2001. Different intervals are used for pre- and postcrisis poverty rates since data are not available for each year. reiterating the importance of including voluntary participation, but it was envisioned countries at various levels of development in that over time, the full membership of the Bank order to give a full range of countries access to and the IMF would be covered,7 with reassess- benefits from the program's support for ments being carried out as needed. strengthening their financial policies and These factors, particularly the voluntary nature institutions in line with international of the program, are not always consonant with practices."5 Specific factors for country each other, which has led to important omissions. selection included: (i) A number of countries which are generally viewed The objectives of the FSAP interest of authorities in as "systemically important" or vulnerable have not financial sector assess- been assessed yet, nor have some countries with are highly relevant to the ment and willingness, if large financial sector development needs. Across a Bank's mission, as necessary, to undertake range of definitions, a large proportion of systemi- financial sector crises reforms, address vulner- cally important countries have had an FSAP, but abilities and otherwise about 20-30 percent have not (including China, and depth have strong strengthen the financial Indonesia, Thailand, Turkey, the United States, and links with poverty and sector; (ii) size, com- Venezuela)(seefigure2.1),andaboutone-thirdof growth. plexity, and relative countries with significant indications of vulnerabil- efficiency of the financial ity have also not yet undergone an FSAP.8 system; (iii) size of the The Financial Sector Liaison Committee has country and economy and economic and attempted to balance the selection of countries financial links with other countries in the by carrying out assessments of countries at all region; and/or (iv) scope and timing of levels of development. To date, low-income structurally related work under way in the countries are underrepresented (table 2.2), as country (i.e., existing Bank and IMF programs).6 are countries in the East Asia and Pacific Region Although not explicitly listed, regional distribu- and Sub-Saharan Africa Region (table 2.3). This tion and income levels of countries have also reflects the tradeoff between emphasizing been factors in selection. The program relies on systemic importance versus depth of financial 6 RELEVANCE, PROGRAM DESIGN, AND INPUTS Figure 2.1. Percentage of Countries with FSAPs 73% of all Systemically systemically importanta 66.7% 76.2% important xountries have FSAPs Not 54% of the not systemtically 53.8% 47.0% systemically important important countries have FSAPs Below median Above median M2/GDP 55% of countries with low M2/GDP 54% of countries with high M2/GDP have FSAPs have FSAPs Source: World Bank databases, IEO and IEG analyses. a. Defined as the G-7 plus countries monitored by Bank for International Settlements. sector development, as well as the voluntary nature of the program. Table 2.2. Countries with FSAPs, by Income Level Voluntary nature of program. The voluntary nature FSAPs completed of the program limits the program's overall Income Level Number % of countries effectiveness in identifying systemic risks. Low 21 36 Nevertheless, interviews and surveys indicate Lower Middle 32 59 that the majority of authorities and staff support Upper Middle 22 55 the maintenance of the voluntary nature of the High Income 31 56 process.9 The FSAP is a time- and labor- intensive exercise for staff and authorities Source: World Bank data. alike.10 Without the full cooperation of country authorities, access to detailed information and key staff would be difficult, and would reduce Table 2.3. Percentage of Countries with FSAPs the effectiveness of the exercise significantly. per Bank Region This tradeoff between effectiveness of the individual assessments and the effectiveness of Percentage of Countries the program's overall surveillance role has not Bank Regions with FSAPs per Region been resolved. Although most systemically East Asia and Pacific 8.3 important countries acknowledge the Europe and Central Asia 81.5 importance and support participation in the Latin America and the Caribbean 59.4 program, these statements of support have not Middle East and North Africa 64.3 always been translated into action.11 Given the South Asia 50.0 practical realities of the situation, and the Sub-Saharan Africa 35.4 limited resources available for the program, resources should be directed at those countries BANK Regional Total 47.7 that are willing to provide the data, access to Advanced/High-income economies 77.4 key personnel and other support needed to BANK MEMBER TOTAL 52.7 make the assessments effective. Source: FSAP database, World Bank country group websites. 7 FINANCIAL SECTOR ASSESSMENT PROGRAM Scope of Assessments IEG found that since the 2003 Board update, The FSAP program sought to "develop an which proposed more streamlining, the approach to financial system assessment that is frequency of conducting the five most broadly consistent across countries, while frequently used ROSCs has begun to drop off allowing for difference in emphasis to reflect (table 2.4), and that there are fewer ROSCs in different country circumstances." Staff would countries with less developed economies (table begin from a standard template, but would 2.5). Based on the review of individual FSAPs, develop "a narrower country-specific focus decreasing the number of formal ROSCs has not depending on the structure of the country's impaired the quality of the FSAPs.13 financial system and an evaluation of priorities Beyond the streamlining of ROSCs, IEG's for both system stability and sector develop- review of FSAPs did not find consistent ment."12 The recent progress reports from Bank evidence of tailoring of topics to the country and IMF Management to the Boards on the circumstances.14 While many FSAPs are clearly FSAP stated that over time, there has been tailored to reflect the circumstances of the greater selectivity of assessments, with better country, others appear to be more heavily streamlining and tailoring to country needs. template driven. A review of the terms of Table 2.4. Frequency of Assessment Done in Fiscal Years 2000­04 Average FY BCP TMF CPSS IOSCO IAIS Sum per country 2000 13 11 3 8 7 42 3.2 2001 21 14 16 15 12 78 3.7 2002 20 18 19 17 14 88 4.2 2003 13 10 10 6 4 43 3.3 2004 17 11 9 10 5 52 3.1 Total 84 64 57 56 42 303 3.6 Source: IEG analysis based on reviews of FSAPs. Full assessments only; excludes updates. Reports reviewed were: Core Principles for Effective Banking Supervision (BCP), Code of Good Practices on Transparency in Monetary and Financial Policies (TMF), Committee on Payment Systems and Settlements (CPSS), International Organization of Securities Commissions (IOSCO), and International Association of Insurance Supervisors (IAIS). Table 2.5. ROSCs per Category of Country Average Category of Total number number of country of countries BCP TMF CPSS IOSCO IAIS Total ROSCs Advanced in percent of countries 19 19 17 15 19 15 85 4.5 100% 89% 79% 100% 79% Emerging in percent of countries 31 31 22 21 25 21 120 3.9 100% 71% 68% 81% 68% Developing in percent of countries 34 34 25 21 12 6 98 2.9 100% 74% 62% 35% 18% Total 84 84 64 57 56 42 303 3.6 Source: IEG analysis based on reviews of FSAPs. 8 RELEVANCE, PROGRAM DESIGN, AND INPUTS reference (TORs) and the main reports of input to development plans. To date, resource detailed country reviews found that 63 percent constraints have limited the number of assess- of the TORs and 47 percent of the main reports ments and updates to about 17­19 per year; at either did not prioritize sectors or did not this rate, coverage of the full Bank/IMF adequately discuss the reasons for selecting membership would take approximately 10 particular sectors. Failing to tailor did not years. Having an update every 10 years would necessarily result in a poor FSAP; 75 percent of not support either surveillance or develop- FSAPs reviewed were rated satisfactory overall, ment. Indeed, most survey respondents stated although with some weaknesses in specific that two to three years is the appropriate time areas (see table 2.6 in the section on output frame for updates (figure 2.2), with 90 percent quality). supporting a time frame of four years or less. Updates Costs Questions of scope and priorities are more IEG and IEO reviewed cost data for the FSAPs17 marked for updates. Whereas the original of joint mission teams. Joint FSAPs averaged assessments were envisioned to be comprehen- US$685,000, with an average Bank contribution sive, updates were to be more selective and of US$244,000. Bank and IMF management had focused, with flexibility on the scope. All expected that increased tailoring (especially in updates were to include progress in implement- lower-income countries), as outlined in the ing FSAP recommendations, a financial stability fiscal year 2003 progress report on the FSAP, analysis, updates of standards and codes would result in lower included in the initial assessment, and a costs. While fiscal year A significant number of reassessment of key development issues 2004 costs are lower (an systemically important discussed in the first assessment.15 Updates average of US$652,000), countries have not yet could also address issues not covered in the there is not yet sufficient previous assessment, as well as carry out a evidence to determine had FSAPs. deeper review of areas that were covered in the whether this is due to initial assessment.16 tailoring, selection of countries, or other Ideally, the FSAP and the updates would be factors18 (figure 2.3). Data also show that the used to help support ongoing and timely Bank's share of costs is higher in fiscal year 2004 surveillance of vulnerabilities, as well as provide than in previous years but the IMF still is Figure 2.2. Recommended Frequency of FSAP Updates 100 80 60 40 percentage In 20 0 2­3 years 4­5 years Over 5 Flexible Country authorities IMF chiefs Bank country directors Mission leaders Source: IEG/IEO survey results; government authorities; IMF mission chiefs; team leaders; World Bank country directors. 9 FINANCIAL SECTOR ASSESSMENT PROGRAM Figure 2.3. Average FSAP Cost by Income Level 1,400 1,200 1,000 800 (thousands) 600 US$ 400 200 0 FY01 FY02 FY03 FY04 Fiscal years Low income Lower-middle income Upper-middle income High income Overall FSAP average FY01 FY02 FY03 FY04 Average FSAP Costs $561,443 $859,059 $775,748 $652,019 World Bank Portion 37% 35% 31% 40% IMF Portion 63% 65% 69% 60% Source: World Bank budget data; IMF budget data. responsible for about 60 percent of costs (figure portion of the assessment is, on average, 2.3). US$244,000 per assessment, which is slightly Complete cost data were only available for lower than the average cost for financial sector four updates. These updates were substantially studies.19 From fiscal year 2000 to fiscal year less costly than the full assessments, averaging 2004, 45 major pieces of financial sector ESW US$266,000, of which the Bank paid approxi- were carried out, at an average cost of mately US$159,000, or 60 percent of the costs. US$251,000. This is consistent with the content of the The FSAP has not appreciably changed the updates, which were stability assessments, but amount of other financial sector ESW carried often additional work on development topics out by the Bank; 44 major pieces of financial was involved. Although initial costs submitted sector ESW were carried out from fiscal year for other updates indicate that the average cost 1995 to fiscal year 1999. However, the composi- will be more expensive than the four completed tion of the ESW has changed. Of the major thus far, data available at country-specific ESW pieces since fiscal year For timely surveillance this time indicate that 2000, three-quarters were carried out in updates will still be countries that had not yet carried out an FSAP. and input into substantially less costly Smaller, targeted pieces of ESW are more development plans, than full assessments. common in counties that already have had updates should be Compared with most FSAPs. The indications are that the Regions are pieces of ESW, the full using FSAPs as background pieces, and are conducted every two to FSAPs are very costly. subsequently carrying out smaller studies to three years. However, the Bank's supplement those assessments. 10 RELEVANCE, PROGRAM DESIGN, AND INPUTS The FSAP does deliver more information than quality control. While both the country and standard ESW, usually in the form of analyses sector directors are generally involved to some such as the ROSCs and stress tests, which are extent, and a peer review process is carried out, more critical for surveillance than financial sector major changes are rarely made at the review development. Much of the cost of that additional stage. Because the draft of the main document information and analysis is borne by the IMF. The is left in the field, and the major points have IEO report indicates that the IMF has been already been discussed with authorities, it is satisfied with the improvements in surveillance. difficult to alter core messages in the review Another aspect to be considered is the link stage. In interviews, the few examples of between cost and country selection. The FSAP messages being changed were cases where the can be cost-effective for the Bank, when it is country authorities or management pushed for carried out in countries where a development a softening of a critical report. The review stage assessment can be of use. However, in countries does, however, seem to increase production where financial sector development is still time considerably (table 2.6). nascent, carrying out a full assessment does not One indicator of the efficacy of the review appear to be a good use of Bank funds. stage can be seen by the value placed by the Cost-effectiveness can be expected to team on resulting comments received. In the improve over time, as more updates are carried survey, team leaders and out in lieu of full assessments. In addition, team members were Economic and sector increased streamlining should decrease overall asked about the useful- work is better targeted in costs. However, a more discriminating selection ness of the comments countries that have of countries to be assessed, and more up-front received from the IMF work to tailor the scope of the assessments, and Bank (figures 2.4 received FSAP especially the inclusion of ROSCs, would further and 2.5). A few things assessments. improve the cost effectiveness of the program. that can be noted from the survey: (i) most respondents were satisfied Advance Preparation with the comments; (ii) IMF comments were One way of reducing costs is to maximize viewed more favorably than Bank comments advance preparation. As noted above, 63 percent (by both Bank and IMF staff); (iii) 18 percent of of TORs did not prioritize sectors or adequately team leaders and 15 percent of team members discuss the reasons for inclusion of specific were actively dissatisfied with comments from topics. In interviews, the authorities noted that the Bank departments; and (iv) more than one- the effectiveness of the program could be third of mission members responded "don't improved if there had been more consultation at know," when asked about Bank comments, as the TOR stage, resulting in greater selectivity of compared with 15 percent that responded the topics. Others commented that, even in same to IMF comments. countries where extensive work had been previously carried out in the financial sector by the Bank and the IMF, the FSAP team often used Table 2.6. Completion Times of FSAP Documents a standard questionnaire rather than data previously obtained in the context of other missions. Other suggestions for improvement Number of days since start of first mission were to: (i) increase the lead time for data Joint Bank/ IMF-only IMF FSAP FSAP requests; and (ii) increase continuity of staff from previous financial work and for follow-up work. Draft aide-mémoire left in field 68 75 Delivery of final FSAP report to authorities 297 289 Review Stage and Staffing Completion of FSSA 311 293 The review stage for the FSAPs may be too late Completion of FSA 394 N.A. in the process to provide input or add much Source: Financial Sector Liaison Committee data, as of February 14, 2006. 11 FINANCIAL SECTOR ASSESSMENT PROGRAM Figure 2.4. Satisfaction with Comments from Bank Departments Team leaders 40.5 24.2 17.8 17.6 Team members 28.7 19.3 15.3 36.7 0 10 20 30 40 50 60 70 80 90 100 In percentage Satisfied Neutral Dissatisfied Don't know Source: IEG/IEO survey; team leaders and team members. Figure 2.5. Satisfaction with Comments from IMF Departments p Team leaders 57.4 27.8 13.1 1.7 Team members 49.5 24.7 10.9 14.8 0 10 20 30 40 50 60 70 80 90 100 In percentage Satisfied Neutral Dissatisfied Don't know Source: IEG/IEO survey, team leaders and team members. Staff skills. There is high praise for the technical agreeing or strongly agreeing that team skills quality of the teams, especially those undertaking were "completely adequate." However, as noted detailed standards assessments. Survey results above, continuity of staff was a problem raised in indicate that the teams, as a whole, were well interviews by several authorities, who stated that thought of, with 93 percent of authorities not enough use was made of knowledge gained 12 RELEVANCE, PROGRAM DESIGN, AND INPUTS by previous teams and that key staff from the detailed country reviews. As they are FSAP teams were not available for follow-up work. completed, however, it will be important for There is a divergence of views on the Management to monitor them to ascertain appropriate level of country experience of whether the decline in senior staff involvement staff. In some interviews, managers felt that has had any adverse effect on quality. team members should not have recent country experience, because those who had worked Joint Program between the IMF and recently in the country or who had a current World Bank work program could be biased in their assess- The joint nature of the The review process ments, as they would be, in part, critiquing their FSAP program has both lengthens processing time own work. However, the majority of country positive and negative authorities interviewed felt that some team features, but based on considerably, yet it is too members must have deep country experience, interviews and surveys late to provide much as that would allow for better tailoring of FSAPs, of team leaders and team input into key messages. a better understanding of realities in the members, the consensus country, and better continuity of dialogue. is that the advantages of While the issue of country knowledge might the joint nature of the program outweigh the also be assuaged with more time spent in disadvantages. Among the positive aspects are: preparation or in the field, it would be more (i) a deeper pool of analytical expertise; (ii) efficient to maximize the use of staff with pooled knowledge of the countries; (iii) more country knowledge. As discussed in chapter 4, focused attention on the program by the author- the inclusion in the FSAP process of more staff ities; and (iv) better coordination between the with country experience, including country unit Bank and the IMF on recommendations and staff, might also have the effect of improving the policy advice. Both Bank and IMF staff felt that, integration of the assessments into the Bank's overall, the FSAP was improved by the participa- country programs. tion of both institutions. Given that most of the key messages are Coordination between the two institutions discussed in the field by the team, prior to the may generate inefficiencies though, as formal review, the quality of staff is critical. The evidenced by the longer production times for team leader and deputy team leader have final documents when both institutions are crucial roles in the selection of priorities of the involved (table 2.6).20 Interviews also indicated assessment, integrating the results from many that friction between the teams sometimes sector reports into an overall assessment, and existed. Most staff, The inclusion of more developing, clear, prioritized recommenda- however, acknowledged tions. This task requires considerable technical that these inefficiencies staff with country expertise and policy judgment. Concern has might be even greater if knowledge would allow been raised in interviews with both Bank and the program did not for better tailoring and IMF management that as the FSAP has become exist, for example, more "routine," less experienced team leaders disagreements on policy continuity of dialogue. and mission members have been selected, with advice, if not discussed a potential adverse effect on quality. IEG's within the framework of the FSAP, could cause review of the composition of FSAP teams does problems if authorities do not receive coordi- show that the percentage of senior technical nated advice. On balance, both IEG and IEO staff on the teams (H-level and above) has fallen agree that there are strong synergies from the from 33 percent in fiscal years 2001­02 to 13 joint program and it should be continued. percent in fiscal year 2005 and year-to-date in A related question was raised at earlier Board fiscal year 2006. Because these recent assess- discussions of the FSAP on whether the ments are still being revised and completed, program should be formally expanded to they were not included in this evaluation's include other institutions. The program has 13 FINANCIAL SECTOR ASSESSMENT PROGRAM The advantages of a joint successfully used ex- 85 percent of surveyed mission members felt Bank-IMF program perts from a number of that outside experts worked well with the team. outweigh the other agencies and However, given the difficulties of coordination institutions, and some within the Bank/IMF arrangement, it seems disadvantages. institutions have indi- impractical to expand the program to formally cated an interest in include more institutions on a regular basis. being more actively involved with the program. Chapter 5 discusses a number of recommenda- Indeed, 91 percent of surveyed team leaders and tions on how to improve the role of donors. 14 3 Outputs his chapter presents findings on the scope and quality of the assessments, T as well as the usefulness and candor of the findings and recommen- dations. The primary sources were detailed reviews of all the FSAP re- ports for the 19 detailed countries' studies;1 supplemented by survey results, and more limited reviews of other FSAP documents. Quality of Analysis receiving a problem rating from reviewers. There The overall ratings for the FSAPs from the was greater variability in the coverage and quality detailed country studies were generally satisfac- of the analysis of the other sectors, ranging from tory, although problems were identified, 19 to 58 percent of the analyses rated moderately depending on the area rated, in 7 to 58 percent unsatisfactory or unsatisfactory. of the FSAPs reviewed (table 3.1). Interview and The emphasis on the banking sector was survey respondents also indicated that they often appropriate, because in many countries, were generally satisfied with the FSAPs, with the banking sector dominated the financial almost 90 percent of country authorities sector;2 hence, the other sectors might have a responding that they were satisfied with the minimal effect on overall coverage and more than 70 percent were financial sector stability. Overall ratings for the satisfied with the depth of analysis. IMF mission In a number of cases, FSAPs were generally chiefs and Bank country directors had similarly the FSAP clearly noted satisfactory. high evaluations of the reports. Authorities that the small size of were also positive, on balance, between stability those sectors led the and development, and the usefulness of the authorities and team to limit assessment of assessment in delivering new insights about the specific areas. However, some of the omissions financial sector. appeared inconsistent with country circum- IEG's detailed country studies also found that stances; for example, over one-third of assess- most sectors were adequately covered, but there ments in low-income countries had little was variability in quality. The quality and discussion on access issues thoroughness of the banking sector analyses For the nonbanking sectors, table 3.1 shows were the most consistent, with 100 percent of that while the average ratings were satisfactory, the FSAPs covering the banking sector in detail, there were higher percentages of "problem" and only 7 percent of the banking sector analyses assessments. The weaker analyses reflect, in 15 FINANCIAL SECTOR ASSESSMENT PROGRAM Table 3.1. Summary of Results of Detailed Reviews Mean score Percentage of ratings indicating Criteria (on scale of 1-4) some problems (i.e., ratings of 3 or 4) Coverage of overall financial sector 2.38 26 Balance of development and stability issues 2.02 16 Banking 1.76 7 Insurance 1.73 29 Capital Markets 1.78 19 Asset Management/Pensions 2.29 58 Market Infrastructure 1.98 31 Clarity and candor of findings 2.16 16 Importance and consequence well explained 2.25 26 Clarity of recommendations 1.93 11 Usability of recommendations 2.08 21 Prioritization of recommendations 2.62 53 Source: IEG evaluations. "1" is the highest rating, "4" is the lowest rating. See endnote 1 for more detailed information on the ratings. part, a weaker set of analytical tools and poorer frequently pointed to stress tests and the data availability. In addition, few FSAPs fully ROSCs as extremely useful tools, and asked for analyze the linkages between sectors and follow-up TA or training. Survey results were discussed the effects on sequencing; for consistent with the interviews, with analytical example, only about a third of detailed FSAP tools being highly rated by country authorities reviews had an integrated discussion of (figure 3.1). insurance issues with capital markets and While these tools can be helpful in identify- investment issues. ing areas of vulnerability, and thereby assist authorities in prioritizing supervision efforts Analytical tools. Analytical tools were well and identifying needed reforms, each of these received; in interviews, country authorities tools must be carefully interpreted as they also Figure 3.1. Most Useful Analytical Components Stress testing Financial soundness indicators Assessment of financial infrastructure Othera 0 10 20 30 40 50 60 70 80 In percentage Mission leaders Country authorities Source: IEG/IEO survey, multiple responses allowed. a. Includes analysis of vulnerabilities and development needs. 16 OUTPUTS have a number of shortcomings. For example, commodity risks); in other cases, the selection the quality of stress tests varies greatly, owing to of issues did not seem to be as tailored to the a number of factors including: (i) quality and countries. Finally, stress tests are not always an availability of data; (ii) complexity of models; appropriate tool; they are best used to look at and (iii) selection of factors including the ability of individual banks to withstand interlinked factors, which might cause a shock stress, not entire systems. to the system (box 3.1). The type of shocks Bank and IMF staff are well aware of these considered by stress tests varied by country. In risks and stressed in their documents the limita- some cases, the variability seemed to be a tions of these tools, which should be considered choice (such as exposure to specific interpreted only in the context of a comprehen- Box 3.1. Stress Testing Stress testing is a method of quantifying the impact of future ex- Selection of shocks. The stress tests should select for plausi- treme (but plausible) shocks on a financial system. The degree ble shocks for each country. For example, countries with heavy de- of sophistication of approaches used varies substantially across pendence on exports of a specific commodity would include price FSAPs, depending largely on the availability of data, sophisti- changes in their stress tests. However, some assessments omit- cation of the financial system, cooperation from the authori- ted important possible shocks, particularly if they were politically ties, available time for the analysis, and judgment of the FSAP sensitive. For example, the banking sector in one Latin American team. The issues that will affect the quality and validity of stress country holds one-quarter of its assets in government bonds, yet tests are summarized here. the stress test does not include a scenario for public debt default. Data quality. The quality of data, and its implications, for any re- One G-20 country's stress tests fail to test for the risks of its large sultsshouldbedescribedcandidly;manyFSAPsareweakinthisre- net open position. There was also often inadequate consideration spect. There are some cases where the FSAPs stress that the data of regional or global effects. For example, the impact of a general are extremely poor; however, the results of the stress tests are re- lowering of international interest rates on bank profitability in an ported with a great deal of precision, without an adequate discus- ECA country was not modeled, even though 40 percent of the sionoftheextremelylimitedvalueofastresstestdonewithpoordata. banks' assets are invested abroad. In another example, contagion Notconductingstresstestingshouldalwaysbeanalternativeinsuch effects were omitted from a G-7 country's analysis. A positive ex- cases (e.g., the Costa Rica and Nigeria FSAPs appropriately did not ample is Kazakhstan, which does consider the contagion effects undertakeanyformalstresstests),becausethereisahighriskofpro- from shocks in Russia. ducingspuriouslyconcreteresultsthatmaskanunknownsituation. Multifactor analysis. Most stress tests have included single- Calibration of shocks. The challenge is to develop a common factor sensitivity analysis. The most recent vintages (e.g., Jordan understanding for what shocks could be considered as exceptional and NewZealand and many European countries) have also included but plausible. Where feasible, calibration could use models to the use of scenarios that involve simultaneous movements in var- characterize the relationships among macroeconomic risk factors ious macroeconomic risk factors. This is a positive trend as such in the context of different scenarios and/or cases in which single scenarios could help better analyze the vulnerabilities of the fi- variables are shocked (by using statistical or historical approaches). nancial system. Some recent FSAPs (Germany and Chile) derived a consistent set Interpretation of the results. More attention needs to be given of shocks to macroeconomic variables from a macro model. In one to the interpretation of stress test results, not only in light of the OECD country, however, the stress scenarios did not reflect ex- methodological caveats but also in terms of the relative importance ceptional shocks; they included growth rates of 0 percent and ­1.7 of different shocks (e.g., avoid overemphasizing market risks when percent, which were milder shocks than what the country had al- credit risks are more relevant from a vulnerability perspective). This ready experienced (in 1998 and 2002, actual growth rates were ­0.9 is an area where many FSAPs are weak, but Korea and Cameroon percent and ­2.4 percent, respectively). are examples of good practices. Source: IEO 2006, box 1; IEG staff analysis. 17 FINANCIAL SECTOR ASSESSMENT PROGRAM The nonbanking sectors sive and ongoing surveil- there may be insufficient time to carry out had a higher percentage lance program.3 Yet, assessments under the umbrella of the FSAP. these caveats are often For example, one assessor from an outside of "problem" assessments. absent in the FSAP agency commented that he had two weeks to discussions. In addition do one assessment (including field time), to the general caveats, IEG's and IEO's detailed whereas he might normally do the same assess- country reviews show that individual stress tests ment over six months, with multiple trips and sometimes have other flaws, including: (i) lack exchanges of information in between trips. of a clear explanation for their choice of shocks Financial Soundness Indicators (FSIs) are and the appropriate magnitudes to be used; (ii) also used as a regular part of the analysis. avoidance of modeling politically sensitive Although most FSAPs presented data on core shocks; and (iii) inadequate consideration of banking FSIs, there was often insufficient analysis regional or global effects. In some cases, such as to interpret the data, and insufficient explanation when data are poor, the best approach may be on the limitations of the data. In addition, using to omit a formal stress test, as was done in aggregate data often obscures vulnerabilities of several FSAPs. individual institutions or subgroups of institu- Bank and IMF staff have also noted that tions. Financial sector indicators for the nonbank- freestanding ROSCs are not that useful in ing sector were more sporadically presented, and reducing vulnerabilities, and should also be had even less interpretive information or analysis. carried out in the context of an overall surveil- This reflects an analytical gap in the amount of lance program.4 In interviews with Bank and development work done in the banking versus IMF staff, as well as representatives of standard- nonbanking sectors. One benefit of the FSAP setting bodies, the following concerns about program is that it has helped highlight the need ROSCs were raised: for improved analytical tools, and has led to additional work in improving these tools (such as · Authorities and other readers often focused the recently released Financial Sector Assess- only on ratings; team members noted that dis- ment: A Handbook, World Bank and IMF 2005).8 cussion with authorities often centered on However, there have been neither consistent whether there was agreement on the ratings approaches in areas (such as use of similar tools rather than on the underlying substance that was or benchmarks in countries at similar stages of being discussed. In some cases, the staff agreed development), nor has there been a consistent on a "compliant" rating, relying on the text in progression in these areas (i.e., a deepening the FSAP to spell out their ongoing concerns.5 analytical approach over time in specific sectors); · There is an uneven approach to assessments. some of the latest FSAPs appear not to have Some assessors took the view that the same drawn on lessons from previous FSAPs, perhaps standards should be applied to all countries (an owing to weaknesses in knowledge sharing in the approach which many country authorities de- program (see chapter 4).9 scribed as "failing to take into account coun- try circumstances"), whereas other assessors Data. While survey results show that teams rated stated that the ratings should reflect the coun- access to data as adequate,10 IEG's detailed try's stage of financial development.6 One country reviews found that 63 percent of the downside of lowering the assessment stan- aide-mémoires either noted there were dards to reflect the level of development is weaknesses in the data, or were missing that it can lead to com- important data that would normally be included There is too much placencyaboutsystemde- in the analysis. All the FSAPs reviewed have emphasis on ROSC ficiencies.7 basic banking sector data (nonperforming loans, provisioning ratios, capital adequacy, ratings, instead of the Feedback from asses- etc). In contrast, data for other sectors were content. sors also indicated that more sporadic. Most FSAPs report statistics on 18 OUTPUTS size and structure of the nonbank financial medium enterprises, number of clients served institution markets, and liquidity and composi- by microcredit institutions, interest rates for tion of securities markets, but do not consis- microcredit, repayment rates, etc.). While the tently report on other areas which would reflect FSAPs do present the limited available data, the sector's vulnerability, term transformation, they sometimes fail to fully analyze the informa- or profitability, in part because of data availabil- tion, factors such as culture and political ity problems. For example, many assessments economy, and benchmark data available from exclude discussions of reinsurance, underwrit- other FSAPs. To make this part of the FSAP more ing skills, an analysis of loss ratios, or a discus- effective, better approaches are needed for sion of the matching of investment composition analyzing development issues (see box 3.2 for with projected liabilities. Data availability also examples of different approaches to analyzing hampers the analysis of access to financial nonbanking financial issues). The Financial services; usually only a handful of statistics is Sector Assessment handbook lists a number of available (the percentage of deposits held by suggested indicators in each sector; however, households, percentage of loans to small and these indicators are not consistently used. Box 3.2. Different Approaches to Nonbanking Issues FSAP coverage of nonbanking financial sectors have varied the capital, insurance, and pension markets are all extremely substantially. Countries face many different conditions, with small, not systemically important, and suffer from weak regulatory varying amounts of data available, thus requiring FSAP teams to oversight. The FSAP then suggested that because these sectors take different approaches and devise different solutions. Here will not be significant in the short term, work in these sectors are some examples of the different approaches: should be limited. Interlinkages and sequencing of sectoral development. A Access to finance. In many middle- and higher-income coun- number of FSAPs discussed the need to expand the insurance and tries, access to finance was not a priority, and justifiably was not pension sectors, and to diversify asset holdings, but failed to dis- discussed in depth. In many lower-income countries, where ac- cuss the lack of available investment instruments. In contrast, the cess is an issue, the discussion was surprisingly cursory. Some Tanzania FSAP noted that investment funds were highly liquid, good examples of discussions do emerge, however. (i) In Nigeria, and in order to maximize returns, the investment funds should be the discussion of access was placed in the context of the overall able to invest more assets in equities. However, because equity need for rural finance, given that the rural economy employs about markets are not well developed in Tanzania, the assessment ad- 90 percent of the country's workforce and accounts for about 40 vises that the funds be allowed to invest a portion of assets in eq- percent of GDP. (ii) In Sri Lanka, microfinance is relatively well de- uities of highly rated foreign corporations. veloped, with an estimated 3 million loans outstanding in the for- Insurance. Most FSAPs presented the basic industry structure, mal sector, with a value of 22 billion rupees (approximately US$270 data on premiums and assets, and focus on the need to improve million) as of 2000. The FSAP report leveraged information taken regulation, supervision, and depth. The Nigeria FSAP also dis- from a poverty survey to provide data not only on formal mi- cussed the link between insurance and development, and dis- crolending institutions, but also informal loans, and an analysis of cussed vulnerabilities that could be inferred from the data, including borrowers, savings patterns, etc. Given the relatively extensive ac- the treatment of unearned premiums and the implications of un- cess, the focus of the FSAP is on improving the safety and efficiency characteristically low claims ratios. of the microfinance system. (iii) In Gabon, microfinance is tiny, and Capital markets. Many of the FSAPs focused on providing little data are available; as of June 2000, there were only 13 mi- guidance on strengthening and deepening a vulnerable and shal- crofinance institutions with only 2,781 clients. The Gabon FSAP thus low domestic market. The Costa Rica FSAP, instead, recommended focused more on analyzing the cultural reasons for the lack of mi- that the country should integrate with a regional stock exchange crocredit, and avoided assuming that regulatory or legal stric- rather than develop its own. The Kyrgyz Republic FSAP noted that tures were constraining the sector. Source: IEG analysis. 19 FINANCIAL SECTOR ASSESSMENT PROGRAM While FSAPs are generally While data limitations Although some data are market sensitive and are a major problem, should remain confidential, industrywide data candid, there have been several actions could be such as market structure and industrywide asset cases where there was taken to strengthen the composition, can and should be shared across pressure to soften the analyses. First, more FSAP teams. coordination between written reports. the financial sector staff Quality of Updates. IEG also reviewed the and other units in the updates; the quality was not substantially differ- Bank would help. For example, extensive data ent from that of full assessments, although the on household finance for Sri Lanka came from a foci of the updates were intentionally more household survey undertaken for work on narrow (box 3.3). poverty; the extensive work being done in the Private Sector Development Unit on "doing Articulation of Findings business" can be a useful source of data on small IEG's detailed country reviews indicated that and medium enterprises and larger corporate most of the FSAPs had clearly identified key borrowers. Second, even when there is limited issues (table 3.1), and most of the survey data availability, an understanding of the respondents felt that the FSAPs were generally country environment and culture can make up candid. The most common reason given for lack for data shortfalls (see discussion on Gabon in of candor (in cases where it was an issue) was box 3.2). Finally, there should be wider dissemi- concern that negative findings could adversely nation of data within the financial sector.11 affect the financial sector (figure 3.2). In Box 3.3. Assessment of FSAP Updates Each of the 10 FSAP updates completed in the post-pilot phase, might have concluded that updating such tests was not a high as of June 2005, was reviewed by following a streamlined tem- priority use of resources. plate, based on the one used for a detailed country review.a · The limitations on what can realistically be expected from up- The key messages are the following: dates of different scope and depth (e.g., that narrowly focused updates cannot be expected to provide an in-depth assess- · While the scope of the various updates has been implemented ment of progress in sectors that fall outside of its scope) are flexibly, in line with the Board's guidance, there was limited dis- not adequately signaled to the reader. Stronger warnings of cussion of the rationale for the scope of updates. Thus, it is dif- these limitations and the necessary qualifications to any ficult to see how each of the FSAP activities fits into an overall conclusions are still needed. strategy for the financial sector in each country. · Inadequate prioritization of recommendations remains a · As intended, updates reviewed the status of implementa- problem in most cases. The updates do not provide an ac- tion of recommendations. tion plan or strategy for developments. Hence, if effective fi- · While most updates conducted a new round of stability as- nancial sector development is the priority and objective of the sessments, including stress tests, in most cases there was updates, they must either provide the additional guidance for little improvement in the methodological approach; thus, in strategy or actions, or they must be supplemented by other a number of instances, data limitations still forced a highly instruments. simplistic approach. This raises questions about whether a · The degree of integration of findings into Bank programs ap- greater ex-ante assessment of changes in data availability pears to be broadly similar for updates as well as for full FSAPs. Source: IEO 2006, box 4; IEG staff analysis. a. The 10 updates completed in the post-pilot phase were: Armenia, Colombia, El Salvador, Ghana, Hungary, Kazakhstan, Peru, Senegal, Slovenia, and Uganda. IEO also reviewed Iceland, which IEG omitted from its sample because it was an IMF-only update. 20 OUTPUTS Figure 3.2. Reasons for Noncandid FSAP Recommendations Pressure from country authorities Pressure from IMF management Pressure from World Bank management Pressure from my department Concern over adverse country consequences Not applicable Don't Know Other 0 10 20 30 40 50 In percentage Source: IEG/IEO survey, question 8, includes multiple responses from mission leaders. interviews, staff explained that concerns about country studies gave low ratings to the prioriti- potential leaks led them to tone down the zation in the assessments (table 3.1). FSAPs language in the text of aide-mémoires, while often had too many "priority" recommenda- delivering tougher messages in wrap-up tions; sequencing and implementation capacity meetings; differences in wrap-up presentations also were often not adequately addressed. This and aide-mémoires support these statements. may, in part, be because of the view of some Of greater concern are the cases where Bank staff that the FSAP should be a diagnostic pressure was put on the team to soften the tool only, although the mandate of the FSAP written reports. Fifty-three percent of team envisioned the use of the assessments as leaders who responded to the survey, stated platforms for future work as well. Without the that there had been factors influencing the development of a strategic vision, concrete candor of results; of which, more than two out action plan, or an analysis of the political of five said they that they had felt pressure from realities, the FSAP risks becoming a sterile and country authorities to change the FSAP, and irrelevant exercise.13 more than one in five stated that management had pressured the team to change the FSAP. Informing the Executive Boards of the Interviews with team leaders confirmed this World Bank and IMF perception. Disguising problems, even in a few The Bank's FSA and the IMF's FSSA documents cases, defeats the purpose of the FSAP are intended to keep the Executive Boards program.12 informed of vulnerability and stability issues, particularly those that Presentation of FSAP recommendations. While the might have spillover Prioritization of FSAPs usually provide clear diagnoses and effects into other coun- recommendations is often recommendations, prioritization is often tries, as well as impor- lacking; indeed, more than half of the detailed tant development con- lacking. 21 FINANCIAL SECTOR ASSESSMENT PROGRAM cerns. The FSSAs and The usefulness of the FSAP as a means of The FSA documents FSAs (usually 15­25 keeping the Board informed is also diminished convey key messages to pages) are substantially by the time it takes to complete the summary the Board, but its shorter shorter than the main documents. As seen in table 2.6, on average, it format causes nuances to FSAP reports, which can takes almost a year to turn the draft aide- run into hundreds of mémoire into an FSA delivered to the Board. be lost. pages. IEG's detailed From the pilot phase in May 1999 until October review of FSAP docu- 2005, 97 additional assessments (including both ments indicated that although FSAs do provide full assessments and updates) were completed, a summary of the key issues and messages, including 18 developed-country assessments nuances or caveats were often lost, which can and updates. The completion rate for post-pilot sometimes present a misleading picture. A FSAs is 72 percent, versus 81 percent for FSSAs common flaw is for the main report to describe (table 3.2). Moreover, as of February 14, 2006, weaknesses and inconsistencies in data or the remaining uncompleted FSAs have been findings, while the FSA presents the main outstanding for a median of 646 days after the findings without drawing the reader's attention initial mission was launched. (See chapter 4 for to the limitations of the analysis. related discussion on disclosure.) The clarity of messages from the FSAP would The longest part of the delay in producing be enhanced by greater discussion of FSAP final documents is the review stage, which, on findings by the Board. IEO's evaluation noted average, takes almost eight months. As that the discussions of the FSAPs, which take discussed in chapter 2, because the draft aide- place in conjunction with the IMF's Article IV mémoire has already been delivered and consultations, do not always pick up on discussed with authorities by the time of the important messages. review, the key messages have already been The Bank takes too long However, whatever their conveyed to authorities. Given this fact, the to deliver FSAs to the limitations, the IMF eight-month lag before finalization is too long. Board. discussions are an Similarly, while the IMF releases the FSSA an improvement over the average of 16 days after the final report is sent process in the Bank, where there is no formal to authorities, the Bank takes an average of mechanism for discussing the findings from the more almost three months longer to release the FSAPs. FSA. Overall, the FSSA is completed before the FSA in over 80 percent of the joint FSAPs. If the Board is to be kept informed in a timely Table 3.2. Completed and Disclosed manner, Bank Management should set FSAP Documents deadlines for the delivery of the FSAs. One option for improving delivery times is to Developing All leverage the work of the IMF. Currently, the country FSAPs FSAPs Bank and the IMF draft two separate Number of FSAPs 79 97 documents; the rationale was because the Bank Number of FSAs 57 -- places a greater emphasis on development Number of FSSAs 62 79 issues and the IMF places a greater emphasis on Number of disclosed 30 (FSA) -- stability issues, the foci of the reports would be 44 (FSSA) 62 (FSSA) different. IEG has found that, while it is true that Completed FSA/FSSA as percent of FSAPs 72 (FSA) -- there is more discussion of financial sector 79 (FSSA) 81 (FSSA) development issues in the FSAs, and more Percent disclosed 53 (FSA) -- discussion of stability in the FSSAs, there is not 71 (FSSA) 79 (FSSA) a substantial difference in content between the two papers. There is, however, strong resistance Source: Financial Sector Liaison Committee data, as of February 14, 2006. These numbers exclude pilot assessments. from staff to combining the two documents. 22 OUTPUTS IMF staff are particularly concerned about there are resource constraints, the Bank could delays caused by differences in process and speed delivery of the FSA by using the FSSA as a timing, as they are expected to produce the platform to summarize the main findings of the FSSA to meet the Article IV consultation assessment, adding chapters or appendixes on timetable, while Bank staff have no similar development or other issues on which the IMF deadline. While keeping separate documents, if may not have focused. 23 4 Impact of the FSAP M easuring the impact of the FSAP is difficult. As discussed in the first chapter, the program is still too new to have measurable effects on ultimate outcomes (such as the reduction of vulnerability and im- proved financial sector development). This section considers the extent to which the FSAP has: (i) had an impact on the reform process in the country; (ii) in- fluenced the programs of other donors, agencies, and other IFIs; and (iii) in- formed markets. Authorities rated the impact of the FSAP as Onesideeffectofthis"independentevaluation" high--72 percent of surveyed authorities is some countries used the FSAP as incentive to agreed the FSAP had contributed to positive improve their financial sectors prior to the formal changes in the financial sector and 83 percent assessment. Authorities from India, for example, were satisfied with the FSAP recommendations. noted that the assessment did not find anything Interviews and survey results indicated that the new, but the process of preparing for the FSAP was authorities agreed that the FSAP expanded their useful. Interviews with authorities of the European country's technical abilities, their understand- Union/European Community, and countries that ing of issues, and their awareness of financial had been seeking to join, viewed the assessment sector vulnerabilities. as an important validation of progress made in the One significant survey result shows that the financial sectors of those countries. Bank staff primary reason most authorities requested an working with some of the Country authorities rate FSAP was to obtain "an independent evaluation" Asian countries that have of the financial sector (figure 4.1). This is consis- not yet participated, have the impact of the FSAP as tent with the authorities' responses that the been told by authorities high. most useful output from the FSAP process was that the countries are an objective evaluation of the financial sector working on improving their financial systems, and (table 4.1). The "evaluation" aspect of the will subsequently participate. Hence, preparing for program was rated much higher than use of the an assessment, even if that assessment does not program for identifying development priorities add more to the countries' knowledge base, may or TA needs, which were rated as two of the serve as an instrument for countries to improve least useful aspects of the program. their financial sectors. 25 FINANCIAL SECTOR ASSESSMENT PROGRAM Figure 4.1. Reasons Why Authorities Requested an FSAP An independent assessment of my country's financial system Recommended by IMF/World Bank Concern over economic vulnerabilities To learn more about my country's financial system Expected that every country should have an FSAP Signal to international capital markets Other countries within my region have had FSAPs Other Facilitate lending by international financial institutions/donors Don't know 0 10 20 30 40 50 60 70 80 90 In percentage Country authorities Source: IEG/IEO survey. A related effect is some countries that have main FSAP recommendations had been mostly undergone FSAPs have used the assessment as implemented; however, Bank and IMF managers a template for their own self-assessments. rated the incidence of implementation much Pakistan, for example, developed a series of lower than did the authorities. Figure 4.3 shows indicators and other tools, which it is using to the survey responses for degree of implementa- monitor its financial sector. tion; these responses are consistent with interview responses. The difference in impact Policy Debate may reflect differences in opinion on priority Fifty-six percent of surveyed authorities felt that actions and on the depth of implementation. For the FSAP contributed to the policy debate in the example, in some cases, a new regulation may country, with the greatest impact on debate have been passed, but enforcement may be lax, within the government, and in supporting the resulting in no meaningful change. IEG's detailed authorities' discussions with the legislature1 country reviews were in line with staff percep- (figure 4.3). The FSAP was most useful in cases tions; there were a number of cases where where the government already had a high numerous technical improvements were made, commitment to financial sector reform; as but key recommendations were not imple- mentioned above, the FSAP's status as an mented. This may also be a function of the independent, external assessment sometimes limited amount of time since the program began; aided in pushing re- major reforms (such as restructuring of weak The FSAP contributed to forms. banks) will take much longer than administrative policy debate, but reforms (such as strengthening regulations), and primarily within Implementation of may yet take place with ongoing support from Recommendations the government and donors. government or with the Country authorities gen- Attribution is also difficult to ascertain. In a legislature. erally responded that the number of cases where recommendations were 26 IMPACT OF THE FSAP Table 4.1. Usefulness of FSAP Areas Authorities IMF Mission Chiefs Country Directors Team Leaders Team Members Three Most Useful: Objective evaluation Objective evaluation Objective evaluation Objective evaluation Objective evaluation of fin. sec. (58 percent) of fin. sec. (62 percent) of fin. sec. (40 percent) of fin. sec. (30 percent) of fin. sec. (38 percent) Integrated view of Identification of new Integrated view of Integrated view of fin. sec., Integrated view of fin. sec. fin. sec. (16 percent) fin. sec. risks and fin. sec. (31 percent) identification of new fin. (15 percent) learning new analytical sec. dev. needs and priori- tech. (13 percent each) tization of fin. sec. ref. (13 percent each) Learning best inter- Enabling reforms by Identification of new fin. Identification of new fin. Identification of new fin. national practices contributing to public sec. development needs, sec. risks (12 percent) sec. risks (12 percent) (11 percent) debate (8 percent). prioritization of fin. sec. reforms, learning new analytical techniques, learning best int. practices and identification of TA needs (8 percent each) Three Least Useful: Identification of TA Improved coordination Enabling reforms by Enabling reforms by Enabling reforms by needs (58 percent) among regulators contributing to public contributing to public contributing to public (52 percent) debate (69 percent) debate (48 percent) debate (46 percent) Enabling reforms by Enabling reforms by Improved coordination Improved coordination Improved coordination contributing to public contributing to public among regulators and among regulators among regulators debate (50 percent) debate (48 percent) prioritization of fin. sec. (41 percent) (40 percent) reforms (39 percent each) Identification of new Learning new Identification of new Learning new analytical Learning new analytical fin. sec. dev. needs analytical tech. fin. sec. risks (31 percent) tech. (34 percent) tech. (28 percent) (33 percent) (35 percent) Source: IEG/IEO survey, government authorities; mission chiefs; country directors; team leaders; and team members. After initial interviews with authorities and staff members on the usefulness of FSAPs, the evaluation team compiled a full list of outputs named by the interviewees from the FSAPs. Surveys were sent to the full set of authorities and staff members, and respondents were then asked to select the three most important outputs, and then select the three least important outputs. implemented, the recommendation was already have contributed to reform in a number of part of a proposed program (supported through countries. a loan, or through ongoing dialogue); hence, it is difficult to determine whether the reform Integration with Bank Country Programs would have been made in the absence of an This section reviews the extent to which FSAP FSAP. There were, however, cases where the findings are integrated into country programs, authorities specifically pointed to the FSAP as a as well as follow-up by the Bank. IEG supple- factor, or where the FSAP seemed to be the mented survey and interview data with detailed catalyst for a new or increased financial sector document reviews. IEG reviewed 50 FSAPs,2 reform program (for more discussion and and the associated CASs, proposed and actual examples, see chapter 5). Overall, despite the lending and nonlending programs, and other difficulties of attribution, the FSAP appears to relevant Bank documents. In addition, IEG also 27 FINANCIAL SECTOR ASSESSMENT PROGRAM Figure 4.2. Contributions to the Policy Debate ...within government agencies ...with FS stakeholders ...with legislators ...with public at large Don't know Other 0 20 40 60 80 100 In percentage Country authorities Bank country directors Mission leaders Source: IEG/IEO survey, government authorities; World Bank country directors; mission leaders. Multiple responses allowed. Percentages are based on a number of respondents who had previously agreed that the FSAP had contributed to policy debate. Figure 4.3. Degree of Implementation of FSAP Recommendations 70 60 50 40 30 percentage 20 In 10 0 Completely 4 3 2 Not at all Don't know implemented implemented Country authorities IMF chiefs Bank country directors Source: IEG/IEO survey; IMF surveillance mission chiefs; World Bank country directors; government authorities. reviewed information about other donor or the FSAP or its findings, ignored the FSAP, or programs, as discussed in public documents inaccurately presented findings. such as IMF reports or donor databases. The CAS documents, however, are a much less The IEG review of Bank documents indicates significant indicator of the impact of FSAPs than that there is a highly divergent range of integra- the extent to which the FSAP recommendations tion with, and follow-up through the country were reflected in actual Bank programs and programs. The integration of the FSAPs with the dialogue. Although attribution is difficult, in 42 CAS process is not consistent. Of the 34 cases percent of the cases, the FSAP appears to have where a CAS has been written after an FSAP, 68 influenced the Bank program in a significant way. percent had a discussion of the FSAP and its In 34 percent of the cases, the FSAP seems to have primary findings and recommendations. The had only a minor influence on dialogue or remaining 32 percent had only a brief mention ongoing programs, and the remaining 24 percent 28 IMPACT OF THE FSAP appeared to have no influence on the program, or Estonia, Poland, etc.) Although authorities had components that were inconsistent with FSAP indicated they wanted generally stated that the recommendations. (See box 4.1 for illustrations of the FSAP primarily for its levels of impact of the FSAP on Bank programs). evaluative purposes, and main FSAP IEG's findings from country reviews are did not want further recommendations had consistent with information received during assistance from the been mostly interviews and from the survey. During Bank or other donors interviews, when asked an open-ended and IFIs, except for tar- implemented, Bank and question about follow-up, authorities almost geted advice in specific IMF managers rated the always cited some form of IMF follow-up or areas. These countries incidence of assistance; Bank follow-up was often only generally are character- implementation to be mentioned when asked about in a direct ized by having good question. In the survey, only 34 percent of the overall economic man- lower. authorities recalled that the Bank had followed agement, sound finan- up in some way, whereas 80 percent recalled cial sector development, and a favorable that the IMF had followed up (primarily business environment. through their Article IV consultations). At the other end of the spectrum are The factors that lead to follow-up, both countries that lack preconditions for develop- within the country and within the Bank are ing an extensive financial sector (i.e., overall complex. Most of the countries that have minor economic management and business environ- or negligible follow-up have small Bank ments are still weak). Given other priorities and programs overall. Some countries (Hungary, the availability of financial sector TA from other Box 4.1. Link between the FSAP and Bank Programs Although attribution is difficult to measure, IEG reviewed FSAPs, a Financial Sector TA loan. Four key laws were passed, capital ad- CASs, and other Bank documents to assess the impact of the equacy averages 15 percent, and the number of banks has been FSAP on the country program; three categories were used: (i) reduced. significant impact, (ii) minor impact; and (iii) negligible impact. Minor impact. This category includes cases where (i) the FSAP Examples of these assessments are below: served to confirm findings and recommendations from previous Significant impact. In some countries, the FSAP contributed work, so the impact of the FSAP was minor; (ii) minor recommen- to an important change in the Bank's program, or strengthened an dations were supported by the work programs; (iii) important rec- ongoing program. In Guatemala, prior to the FSAP, attempts by the ommendations have been supported in minor ways. In Jordan, Bank to support financial sector reform had been unsuccessful. policy dialogue on key issues, such as restructuring of insolvent An IEG country assistance evaluation noted, "Opponents of reforms banks and improving portfolios have not advanced much. The in the financial sector succeeded in their goal of thwarting efforts World Bank Group has continued to provide support in areas such of making financial institutions accountable and stronger," and that as housing, finance, and business law. the sector should be a priority area for the Bank's assistance. The Negligible impact. In Costa Rica, the CAS was much more pos- FSAP coincided with a period of instability in the banking sector, itive about the financial sector than the FSAP had been. In the con- which helped break down resistance to reform. The FSAP laid out text of a highly selective lending program, support was limited to a series of urgent reforms, which the Bank used as one of the key a few pieces of Analytic and Advisory Activities and donor-sup- components in its CAS progress report of fiscal year 2002. The Bank, ported TA. In Gabon, the FSAP found the banking sector to be rel- IMF, and IADB used the FSAP as a basis for analytic work to help atively sound, but shallow. The majority of the population lacks the government's reform program; the Bank also supported the pro- access to any type of financial services. In the context of a limited gram with two loans, the Financial Sector Adjustment Credit and Bank program, limited follow-up is planned. Source: IEG analysis. 29 FINANCIAL SECTOR ASSESSMENT PROGRAM sources, it may have been a considered decision low level of financial sector development, not to focus on the financial sector. For cases carrying out formal ROSCs would not be the such as these, it may have been appropriate for best use of resources. The assessments should limited discussions in the CASs; however, the focus on development, and depending on the CASs should be explicit in explaining the scope for Bank assistance, should try to identify reasons for not providing support to help a few priority areas where development work implement FSAP recommendations. could be focused. The Bank's follow-up role For the remaining could also focus on donor coordination. Despite difficulties of countries, there is a In countries such as Costa Rica and Algeria, attribution, the FSAP does wide range of responses where there is a fairly deep understanding of to the FSAP. There are the major issues facing the financial sector, but appear to have some countries (Costa with a poor reform record, and no apparent contributed to reform in Rica) where the Bank change in commitment, the costs of carrying a number of countries. previously identified the out an FSAP may outweigh the benefits, as the priority areas of vulnera- knowledge base is not substantially increased, bility and development, and the resultant reform dialogue does not but did not find a responsive counterpart; change appreciably. Given limited resources, carrying out the assessment did not change the these FSAPs (and updates) could be deferred in response. In other cases (India), the authorities favor of higher-priority countries. These were already working on a program of reform, countries might be better served with and the FSAP did not alter that program traditional Bank Analytic and Advisory Activities, markedly, but instead the FSAP used primarily targeted to specific topics. to confirm and consolidate previous findings. For countries that have been engaged in In some cases, even though there had been financial sector reform, a full FSAP can be extensive pre-FSAP work and an ongoing catalytic in helping to shape priorities and program, the FSAP seemed to refocus the strengthen the hand of reformers within program (Tunisia), or generate more support government. Countries that have not been for reform (Guatemala, Bangladesh). engaged in reform can also benefit from an All of these factors tie in with one of the initial FSAP, but only if there are factors that indicate questions of the evaluation: What should be the reform would be supported (such as a change priority for selecting countries for full assess- in government, improved economic manage- ments and updates? For countries such as ment, or a favorable business climate). Hungary and Estonia, conducting assessments or updates for development purposes are of lower Process factors. While interest by the govern- priority than for other countries; the inclusion of ment and suitable economic preconditions are these countries in the program should be based critical factors in follow-up, there are design on stability and vulnerability issues. If assess- elements of the program that also affect follow- ments are deemed necessary for surveillance up. From inception, the FSAP was conceived of purposes, the FSAP as part of the Article IV surveillance program. Two-thirds of CASs should be focused on The timing of the FSAP missions and outputs discussed the FSAP and ROSCs and identification were coordinated with the timing of Article IV its primary findings, but of vulnerabilities, with a consultations, and discussions with authorities few focused topics, of the key FSAP findings are held in the context the FSAP appears to have which the authorities of the Article IV discussions. The Bank has no influenced the Bank have identified in premis- analogous follow-up process. In addition to the program significantly in sion discussions. lack of an instrument to ensure that follow-up In countries with few discussions are held with the authorities, there onlt two-fifths of the systemic risks from their is no official instrument to discuss the findings programs. financial sectors and a with the Board (as there is with the IMF's Article 30 IMPACT OF THE FSAP IV discussions). The FSA is usually sent to the same country); (ii) task managers of financial Board for information only, and is not sector projects in those countries; and (iii) team discussed. leaders of other FSAPs, including those working A related issue is that whereas the program on neighboring or similar countries. Further- documents for the FSAP state "Fund area more, whereas other studies at the Bank are department staff will normally participate in the often widely disseminated and discussed, with FSAP missions,"3 no analogous statement exists "best practice" and "bad practice" examples for country department staff; therefore, there is circulated to help future teams, this is not done often no one in the core country unit who has for the FSAP. As noted in chapter 3, the lack of participated in the FSAP discussions and is fully knowledge sharing has meant that even cognizant of the findings.4 nonconfidential data, such as benchmarking Another problem may be the lack of data collected for one assessment, is not passed "ownership" by the country departments. on for use in other assessments.7 Because the FSAP is an externally initiated "free The Bank has started to draw lessons from good," with no mandatory follow-up, and the the FSAP, as evidenced by the recent publica- assessments and FSAs do not count as regional tion of the Financial Sector Assessment "outputs," there is little incentive for Regions 5 handbook, which is to support the FSAP. The country departments designed to help author- Only 34 percent of control the budgets for follow-up work but have ities conduct self-assess- authorities recalled that little ownership in the program or incentive to ments. In addition, the the Bank had followed up follow-up. For these reasons, follow-up work is Bank is working to sometimes not supported with appropriate develop a better set of on the FSAP, whereas 80 funding (fewer than one in four country financial sector indica- percent recalled that the directors indicated that the FSAP findings had tors including those for IMF had followed up on it. led to a budget increase for financial sector stability, depth and work).6 Given that there is always more demand vulnerability of nonbank for FSAPs in any given year than can be financial institutions, and access issues. accommodated, one element to be considered However, the limited access to the data and in country selection might be the willingness of analyses gleaned from the assessments has the country department to commit to funding meant that there has been little work on to support follow-on work (in countries with drawing regional and global lessons from small programs, this might simply be financing groups of FSAPs.8 the costs of donor coordination and dialogue). Influence on the IMF Knowledge Sharing at the Bank This evaluation did not review the impact of the Poor knowledge sharing affects both the quality FSAP on the IMF because the IEO's parallel of FSAPs and follow-up. The effect on donors is evaluation reviewed this in great detail. Key discussed below. The effect within the Bank is findings from IEO evaluation include: also significant. Confidentiality policies limiting dissemination of the FSAP aide-mémoire, while The FSAP represents a distinct improve- they are officially similar at the Bank and the ment in the Fund's ability to conduct IMF, they are, in practice, significantly more financial sector surveillance and in restrictive at the Bank than at the IMF, as well as understanding the important interlink- more restrictive than for most other confiden- ages between financial sector vulnerabil- tial Bank documents. While FSAP reports do ities and macroeconomic stability. While contain sensitive information and should be an overall judgment on the cost-benefit handled carefully, in practice, the confidential- tradeoff will always be difficult for such ity policy has led to access being denied to: (i) activities because of the problems in managers from other sectors (but covering the quantifying the benefits, the evaluation 31 FINANCIAL SECTOR ASSESSMENT PROGRAM Countries with a history of concludes that the uing coverage of the bulk of countries FSAP has signifi- where strong financial sector surveillance limited reform would have cantly deepened the is most needed.9 lower priority for FSAPs. IMF's understanding of the financial sector More detail can be found in the IEO evalua- in specific countries, helped articulate tion (2006). policy recommendations, prompted better discussions with authorities, and Influence on other Donors, IFIs, helped support policy and institutional and Agencies changes. The evaluation also suggests that Feedback from other donors and agencies was the joint IMF-World Bank nature of the generally positive toward the overall program. exercise has been beneficial. Almost all of the donors noted that they, themselves, do not have the resources to carry However, the evaluation also notes, out research and analysis on the scale of the FSAP. The donors, therefore, expressed great Despite these achievements, the initiative is interest in being able to access the information at an important crossroads and there is a from the FSAP, as it could help them to identify danger that some of the gains already funding priorities. The use of the FSAP varied by achieved could be eroded without some donor, and sometimes varied within a donor significant modifications. The evaluation agency, in part, because much of the access to indicates two interlinked sets of problems. FSAP findings was dependent on informal First, financial stability assessments have dialogue between individual staff members at a not yet been fully "mainstreamed" as a donor agency and the corresponding Bank/IMF regular part of IMF surveillance. Second, staff, rather than any formal process of inform- looking beyond the stage of initial FSAPs, ing donors. Most interviewees cited the FSAPs as there are doubts that current incentives for a useful source of background information; participation and priority-setting pro- when formal documents could not be shared, cedures will be sufficient to ensure contin- donors received parts of reports, which allowed Box 4.2. Donor Concerns About the FSAP Access. The donors generally only have access to published ticularly given the lead times that the donors need to plan their own FSSAs and FSAs. In some cases, donors were given access to programs. a small part of the FSAP report, which was directly pertinent to Relevance. The sharpest comments were on the focus of the the assistance requested of them, but because the donors can- FSAPs. Although Bank staff stated that the FSAPs are tailored to not read the full report, it is hard for them to gain perspective on the needs of the country, a number of donors stated that not the overall strategy or the relative importance of that request. enough emphasis was placed on development in developing coun- Even the Financial Sector Reform and Strengthening Initiative tries, and that too much time was spent on ROSCs or other issues (FIRST), which was set up specifically to provide follow-up to the that were not as relevant. The donors also felt that the FSAPs did FSAP, has had a difficult time accessing information needed to not always reflect the realities on the ground, including a failure design programs with significant impact in the country. to address political economy issues. Timeliness. Given that FSSAs and FSAs only become available Lack of strategy. The donors stated frustration that the FSAPs to donors when they are published, and these documents some- rarely led to the development of an overall strategy for financial times do not appear until more than two years after the start of the development, with a clear action plan that could be implemented initial mission, the information is not available on a timely basis, par- by the authorities with donor assistance. Source: IEG interviews with donor agencies. 32 IMPACT OF THE FSAP them to help develop or improve their financial noting that they would The Country Departments sector programs. Examples include SECO's like to be viewed as have little ownership of (Secretariat d'Etat a l'Economie, Switzerland) impartial setters of work program in Azerbaijan, a financial sector standards, not "judges" the FSAP. program in Kenya cofinanced by DFID (Depart- of what is good and what ment for International Development, U.K.), is not. Standard setters noted that they would, Agence Francaise de Development, the World however, welcome: (i) franker language, Bank and other donors, and IADB's financial especially when problems are detected; and (ii) sector program in Mexico. In addition, FIRST greater use of the FSAP results to draw cross- (Financial Sector Reform and Strengthening country analysis. Initiative) was established to provide assistance to help convert recommendations from the Influence on Financial Markets FSAP and ROSCs into projects. As of September A number of authorities identified the "signal- 30, 2005, FIRST had developed 48 FSAP-related ing role" of the FSAPs as one of their motiva- projects. tions for participating in the exercise.11 Many donors, however, stated concerns However, the impact of the FSAPs on financial about access, timeliness, relevance, and lack of markets appears to be Many donors expressed strategy (box 4.2) of the FSAPs, and noted that minimal. There is some they could be more active and effective in use of published FSSAs, concern about access, follow-up if these issues were resolved. FSAs, and ROSCs by timeliness, relevance, and Feedback from country authorities institutions such as lack of strategy. confirmed the difficulty of donors in crafting credit-rating agencies appropriate follow-up; only 13 percent of (which appear to be authorities who responded to the survey agreed primarily interested in the ROSCs and vulnera- with the statement that they had "received bility assessments). support from other international financial institutions/donors to implement the FSAP Disclosure recommendations." During the pilot phase of the FSAP, it was agreed that the FSSAs and FSAs for the 12 pilot Standard Setters countries would not be disclosed. After the pilot Interviews with representatives of standard- phase, both Boards agreed that FSSAs and FSAs setting bodies10 indicated that they found both could be made public if the country agreed. The formal and informal feedback from the FSAP to disclosure rate for post-pilot FSAs is 56 percent, be useful. Most do not have the resources to versus 77 percent of FSSAs (table 3.2). It is not conduct the assessments on the scale that the clear what the reasons are for the significant Bank and IMF have under the FSAP. Some differences in the disclosure rates of the FSAs interviewees also prefer not to do assessments, and FSSAs. 33 5 Key Findings and Recommendations his evaluation has found that the FSAP is a good-quality diagnostic tool T and the overall concept for the program is sound. Joint Bank and IMF cooperation has allowed an integrated approach toward financial sec- tor vulnerabilities and development needs, while expanding the depth and qual- ity of analytical expertise. However, the assessments are weak in prioritization, and the Bank falls short in integrating the findings and recommendations of the assessments into its country programs. In order to sharpen the program's relevance, quality, impact, and efficiency, a number of steps could be taken. Country selection. The FSAP has not covered all addition, while current analytical tools, such as "systemically important" or vulnerable stress tests and the reports on standards and countries, or selected countries where financial codes, are highly valued by country authorities sector development assessments can be most and other constituencies and can be very useful effectively used. The voluntary nature of the as a core framework for vulnerability assess- program limits the program's overall effective- ments and development priorities, they can be ness in identifying systemic risks; however, improved by: (i) strengthening the methodol- there is a strong consensus among authorities ogy of stress tests to better tailor the tests to and staff surveyed for this review that the reflect country circumstances and risks; (ii) de- voluntary nature of the program should be emphasizing ratings when reporting ROSCs, as maintained. Without the full cooperation of the discussion of ROSCs is too focused on authorities, access to detailed information and ratings, and not enough on the content of the key staff would be difficult, and would reduce assessments or the development of an action the effectiveness of the exercise significantly. plan to address weaknesses; and (iii) ensuring that data limitations are clearly flagged. In some Quality. The overall diagnostics were generally cases, it may be appropriate to drop formal good, but the quality and the appropriateness stress tests, ROSCs, or other analyses, rather of coverage of specific sectors were uneven. than lend a false sense of precision or security While banking sector coverage was satisfactory, to the assessment. the coverage in the nonbank financial sectors The quality of teams was rated quite highly; was not consistently of the same quality. In 93 percent of country authorities responding to 35 FINANCIAL SECTOR ASSESSMENT PROGRAM a survey on the FSAP expressed satisfaction with Informing the Board. The current practice of the FSAP team skills. However, authorities have informing the Board is not effective. FSAs are raised concerns about the continuity of staff and delivered a substantial time after the mission the need for strong country experience. work is completed and the final report is sent to Moreover, the seniority of staff involved in the authorities, so the Board is not kept informed program has been dropping off; if the quality of on a timely basis, whereas FSSAs are completed the assessments is to be maintained, Manage- within 16 days of the report finalization. Also, ment will need to monitor staffing closely. given that the FSAs are summaries of the full The Bank also needs to develop better assessments, sometimes the full context and approaches toward analyzing missing markets nuances of the report have been inadequately and access issues, and devise creative solutions conveyed. to improve those areas. This would include: (i) improving the use of indicators and data sets; Impact. While it is still too early to judge the full (ii) deepening the analysis of political economy, impact of the program, the assessments have culture, and other country-specific aspects that been well received, and country authorities would allow the team to help authorities design have cited the assessments for: (i) providing an tailored solutions; and (iii) focusing more on "independent evaluation" of the system, (ii) interlinkages between sectors, which would expanding their knowledge of financial sector help authorities to develop better priorities and vulnerabilities; (iii) improving technical abi- sequencing. lities; and (iv) contributing to the policy dia- The candor of the reports was generally logue within the country. "Results on the satisfactory, although there have been some ground" have been difficult to assess; authori- instances where the governments or manage- ties have stated that most recommendations ment have pressured the staff to soften the have been implemented, whereas Bank and IMF written reports, or where the staff have chosen staff, as well as IEG reviews of the country to convey key messages through presentations programs, have not seen as much evidence of and discussions, rather than through the implementation of critical reforms. For written reports. Another weakness of the example, authorities have noted post-FSAP reports was that many of the caveats and changes, especially in legal, regulatory and nuances of the analyses that were raised with supervisory reforms, but the attribution is government officials were "lost in translation" difficult, as many of these reforms have been as the documents were shortened for the part of previous Bank or IMF policy discussions. Board or public review. Warnings on the limita- In addition, the more difficult reforms will take tions of the reports, as well as the limits on more time, and greater impact may be seen in individual tools and data, should be strongly the future. conveyed. While the program has raised awareness with Prioritization of recommendations was authorities and within the Bank on the weak, which adversely affected the impact of importance of the financial sector on stability the overall program. There were often too many and development, IEG has found that only 42 "priority" recommendations, or sequencing and percent of the assessments have an impact on implementation capacity were not well the Bank's country programs, as manifested by addressed. This may partly be because some important changes or strengthening of the Bank staff view the FSAP as only a diagnostic Bank's program and dialogue. In another 34 tool, even though the mandate of the FSAP percent of countries, the assessment had a envisioned the use of the assessments as minor influence; for example, it might reinforce platforms for future work, policy dialogue, and previous sector knowledge and contribute to as catalysts for financial sector reform. The the ongoing dialogue, and may help spur minor assessments should be better oriented toward changes in the financial sector program, but follow-up. there was no evidence that the direction or pace 36 KEY FINDINGS AND RECOMMENDATIONS of the program had changed due to the assess- The FSAP has not had as strong an impact, ment. The remaining 24 percent appears to have however, on other institutions' programs. While had little evidence on the Bank's program. This other donors support the initiative, and some is consistent with survey results which show that have been able to use the FSAP to help shape only 34 percent of country authorities recall their own programs, the flow of information is follow-up on the FSAP from the Bank (as less timely and complete than they would wish. opposed to 80 percent recalling follow-up from Improvements in donor consultation would the IMF). Factors which could affect the degree allow better leverage of donor resources to of impact include: (i) country selection support reforms. (countries that are more advanced may not need Bank assistance; countries that lack the precon- Efficiency. The FSAP is an expensive program, ditions for a well-functioning financial sector, or although the Bank's share of costs is commen- have no interest in reform, might not be able to surate with the costs of other major financial use a development assessment effectively) does sector studies. Bank resources could be used not always factor in the usefulness of a develop- more efficiently if more time is spent on ment; (ii) there is no clear mechanism for Bank appropriate country selection and better tailor- follow-up (such as the IMF's Article IV discus- ing of coverage. The efficiency and quality of sions); and (iii) the country units are not always the assessments would also improve if the Bank fully involved in planning the scope of, or partic- made full use of the existing data and staff with ipating in, the assessment, which in turn may country experience, and used the FSAP reports affect both the efficiency of the assessment and internally to leverage its knowledge base (as the effectiveness of follow-up activities. well as that of the IMF and other institutions). The FSAP does appear to have influenced Best-practice examples should be shared, and ESW at the Bank, as countries that have had data should be synthesized to draw lessons assessments appear to undertake smaller, more across countries. This would also feed into the targeted ESW in subsequent years. Countries development of better analytical tools and without assessments are more likely to carry out approaches. Cross-border and regional issues major pieces of ESW. This indicates that assess- must be better factored into the assessments; ments can serve as useful background pieces regional and international solutions should also which are effectively leveraged. be more actively considered. This evaluation and the IEO's evaluation Integration. The FSAP alone cannot spark major have found that the joint participation of the reforms or development in the financial sector. Bank and the IMF in the program has been As an assessment tool, it is most valuable in the beneficial; it has contributed significantly to the context of a full program, as an instrument to depth of analytical expertise, credibility of assist in surveillance and to serve as input into findings, and coordination of recommendations a complete development strategy. Its compre- and policy advice. Although there are some hensive overview can also be an important tool frictions and inefficiencies, on balance, the joint for coordinating the Bank, IMF, and donor nature of the program is a positive feature and programs. should be continued. One impact of the program has been an In summary, the technical quality of the improvement in IMF surveillance. The IEO's diagnostics has generally been good, although evaluation concluded that the FSAP had signifi- there are weaknesses in the coverage and cantly deepened the IMF's understanding of the quality of some sectors. The assessments, financial sector in specific countries, improved however, often have poor prioritization, and do the articulation of policy recommendation and not adequately consider sequencing or institu- discussions with authorities, and helped tional capacity constraints. Furthermore, the support policy and institutional changes in the Bank does not do enough to integrate the countries. assessments' findings and recommendations 37 FINANCIAL SECTOR ASSESSMENT PROGRAM into its programs. While the IEO has found that focus on access issues. Also, this process would the assessments have improved the IMF's be more efficient if more time were spent in surveillance and significantly deepened the advance, working with country units and IMF's understanding of the financial sector in authorities on developing prioritized TORs. some countries, the analogous impact on the Bank and its work on financial sector develop- Integrate the FSAP into a full reform program. The ment has not yet occurred to the same degree. Bank, in consultation with the IMF and other The full potential of the FSAP's usefulness to technical assistance providers, should establish the Bank, therefore, has not yet been attained. a clearer framework for coordinating follow-up programs, including technical assistance activi- Recommendations ties, based on the country's own action plans. While details will differ by country, FSAP follow- Country selection and the scope of assessment have up discussions with country authorities on major impacts on the relevance of the program, technical assistance and other development both for the reduction of vulnerabilities in needs are essential. Such discussions should countries' financial sectors as well as the impact include Bank technical and country staff, as well on the development agendas of the countries. as the IMF's Monetary and Financial Systems Given the limited budget envelope, there will Department and area staff. Once development be difficult tradeoffs to be made, particularly and other assistance priorities are determined, when balancing the desire to improve surveil- this information should be quickly shared with lance with development priorities. The Manage- the donor community. In addition, to improve ments of the Bank and the IMF should select donor effectiveness, the following recommen- countries for assessments and updates accord- dations should be considered: (i) inform ing to the priorities identified by the Boards, as donors in advance about the timing of the FSAP, well as consider the likely impact of an FSAP in so that donors can adjust their program timeta- a particular country, and adjust the timing to bles accordingly; (ii) improve consultation with suit the situation. In some cases, such as donors who are active in the financial sector systemically important and vulnerable during the FSAP mission, including presenta- countries, this may mean that updates are tions of key findings; and (iii) provide donors carried out every two or three years, while in with better and more timely access to reports. other countries that are not systemically Country units and the financial sector teams important or that are not likely to treat financial must work more closely to improve integration sector development as a priority, initial assess- of the FSAP findings and recommendations into ments or updates may be delayed by several the Bank's country strategies. This would years. A priority list, as well as the rationale for include greater participation of the country selection, should be presented to the Boards. units in the design of TORs, mission teams, and In cases where countries have not volunteered, follow-up discussions. If a strategic decision is Management needs to inform the Boards of made that Bank follow-up to the FSAP will be those gaps, and should seek other ways to work limited, the CAS should briefly explain the with the country on addressing critical issues in reasons why (limited resources which need to the financial sector. be allocated to other priorities in the country, The scope of the initial assessments and the availability of other donor programs, the updates should be more tailored to address the need to sequence other reforms first, lack of specific needs of the country. This may mean interest or commitment of the client in those that a systemically important, vulnerable sectors, etc.). country may require a full set of ROSCs, and minimal discussion on missing markets or Informing the Board. The process of informing the access issues, whereas another country may not Board is currently not working well. Financial require any formal ROSCs but may, instead, Sector Assessments (FSAs) take too long to 38 KEY FINDINGS AND RECOMMENDATIONS produce. A deadline should be placed on the the decision not to include them), should be production time of the FSA. One option to part of the CAS report and discussions. Bank speed production is to leverage the work of the management should also consider periodic IMF. The IMF FSSAs are usually completed technical briefings summarizing major findings within 16 days of the delivery of the final report. from recent assessments, as well as synthesize For future assessments, knowing that the differ- the information to provide recommendations ent foci, timetables, and processes of the two for future work (within countries and across institutions make it difficult to quickly produce regions). a joint paper, the Bank could use the FSSA for countries and regions as a platform to Knowledge sharing. While remaining mindful of summarize the main findings of the assessment, the needs of safeguarding sensitive and and then add chapter or appendixes on confidential information, the Bank should development or other issues on which the IMF leverage more actively the knowledge gained may not have focused. from this extensive exercise. This would include Unlike the IMF, there is no discussion of the active dissemination of "best practice" analyses, documents at the Bank's Board, so the import, more discussion of concerns and issues that context, and nuances of the reports are often arise while conducting assessments, and better not clear. Critical information should be sharing of data and insights gained through the brought quickly to the Board or a subcommit- program. The financial sector teams should also tee for discussion. In addition, discussions of work to partner more effectively with other key FSAP findings and recommendations, and sector teams to maximize the collection and their integration into the country program (or sharing of information. 39 APPENDIXES APPENDIX A: ASSESSMENT AND UPDATES FY00 FY01 FY02 FY03 FY04 FY05 FY06 (Planned) Assessments Cameroona Armenia Barbados Algeria Austria Albania Australia Canada Croatia Brazila Bangladesh Azerbaijan Bahrain Bosnia & Colombia Czech Republic Bulgariaa Bolivia Chilea Belarus Herzegovina El Salvador Dominican Rep.a Costa Ricaa Germanyb ECCUc Italy Brunei Estonia Finland Côte d'Ivoire Honduras Ecuador Madagascar Denmark Ghanaa Gabon Egypta Hong Kong France Mauritania Fiji Hungary Georgia Japanb Macedonia Jamaica Norway Guyana Indiaa Guatemala Koreab Malta Jordana Paraguay Namibia Iran Iceland Kyrgyz Rep. Mauritius Kenya Rwanda Portugal Irelandb Israel Lithuania Mozambique Kuwait Serbia Qatar Kazakhstana Latvia Luxembourg Oman Moldova Sudan San Marino Lebanon Mexicoa Morocco Romaniaa Netherlands Trinidad and Serbia-Montenegro South Africaa Peru Nigeria Tanzania New Zealandb Tobago Turkey Poland Philippinesa Nicaragua Uruguay Senegal Russiaa Pakistan Sloveniaa Slovak Rep. Saudi Arabia Switzerland Sri Lankaa Singapore Tunisiaa Sweden Uganda Ukraine United Arab United Emirates Kingdom Zambia Total: 13 Total: 21 Total: 21 Total: 13 Total: 17 Total: 12 Total: 13 Updates El Salvador Armenia Georgia Ghana Colombia Guatemala Kazakhstan Hungary Ireland Slovenia Peru Kyrgyz Rep. Senegal Mexico Uganda Morocco Philippines Poland Tunisia Total: 4 Total: 6 Total: 9 Source: FSAP Database. Note: The FSAP for Argentina (FY01) and Uruguay (FY02) were not completed. a. Denotes joint desk study with IEO;. b. Denotes desk study by IEO. c. ECCU is the abbreviation for the Eastern Caribbean Currency Union. 43 APPENDIX B: FSAP FRAMEWORK Inputs Diagnosis · Assessment of financial sector risks and vulnerabilities; analysis of institutional, structural, and market features; stress testing; FSI; standards and codes. · Reassessments Outputs Findings, recommendations, and prioritization · FSAP reports to country authorities · FSA to Executive Board Follow-up on recommendations · Bank and other donor programs for lending / Analytic and Advisory Activities · Government actions: policy, legal and institutional changes Outcomes · Reduction of financial vulnerabilities · Improved financial sector depth and efficiency Impact · Crisis prevention · Financial sector development leading to economic growth 45 APPENDIX C: METHODOLOGY FOR FSAP EVALUATION IEG and IEO worked together to develop a The evaluators began with a detailed common methodology for the evaluation of document review for each country, including: FSAPS. The core components of the evaluation (i) the full set of FSAP documents, including the were based on: aide-memoire, all technical annexes, FSA, FSSA, TORs, and any other FSAP documents on file; i. Detailed country studies; (ii) CASs and CAS progress reports both prior to ii. Survey of major stakeholders; and following the FSAP; (iii) Article IV surveil- iii. Feedback from other important stakeholders; lance reports and other IMF documents; (iv) iv. Review of country assistance strategies, Article World Bank project documents for financial IV surveillance documents, and other relevant sector projects or multisector projects with Bank and IMF documents. financial sector components; (v) relevant ESW and other Analytic and Advisory Activities Detailed Country Studies documents (both financial sector and multisec- Nineteen detailed country studies were tor, such as PRSPs or CEMs); and (vi) informa- undertaken (the table below lists the countries).1 tion from other donors. All of these countries had completed FSAP After the document review, the evaluators processes and, as a group, represented one-third interviewed country authorities, using a of the total number of completed FSAPs at the structured interview template to ensure consis- time.2 Countries were selected with the aim of tency in the questions. Eleven out of 19 cases having a representative sample of the total involved a visit to the country to interview universe of completed FSAPs, giving due weight major stakeholders, including authorities to the following distinguishing factors: (government and the central bank), regulators, selected major banks/financial institutions, and a. Country income levels, knowledgeable think-tanks. In an additional six b. Regional distribution, cases, telephone interviews with authorities c. Fiscal year of FSAP (to cover pilot FSAPs as were carried out (see table below). well as those done prior to and after the 2003 The evaluators then interviewed staff from Board review of the program, which resulted the IMF and Bank who were involved with the in a major policy guidance for future FSAP), and FSAP. Typically, the group was comprised of the d. Type of FSAP (initial or update). (i) team leader and co/deputy leader; (ii) country director; (iii) sector director/manager; To maintain consistency of approach, a and (iv) Article IV / area department mission common methodology comprising standard chief. Sometimes the team was also referred to sets of questionnaires and an evaluation matrix3 other staff (such as the country economist or a was used for each detailed country study. particular team member). Moreover, a two-person team, one each from For each case study, a standard evaluation IEO and IEG, was formed to work jointly on matrix model was used, which provided a rating each case study. system. After completion of all case studies, an 47 FINANCIAL SECTOR ASSESSMENT PROGRAM Countries in 19-Country In-Depth Sample Country Type of contact with country authoritiesa Brazil Conference call Bulgaria Country visit Cameroon Country visit Chile Conference call Costa Rica Country visit Dominican Republic None Egypt Country visit Ghana Country visit India Country visit Jordan Country visit Kazakhstan Conference call Mexico Interviews in Washington, DC Philippines Conference call Romania Country visit Russia Country visit Slovenia Country visit South Africa Country visit (IEO only) Sri Lanka Conference call Tunisia None a. All country authorities were offered an opportunity to provide feedback to the evaluation on their experience with the FSAP process. aggregate country evaluation matrix was survey was quite high (53 percent of the net prepared showing rating trends. deliverable sample).6 Different response rates were obtained from different groups, but the Survey of Major Stakeholders rate for the authorities was 59 percent. (See The IEO and IEG also undertook jointly a survey appendix D for more detail on the survey.) of five major stakeholder groups: (i) authorities in the country (the primary FSAP counterpart in Feedback from Other Important Stakeholders the country, usually the central bank governor); Interviews were also held with other important (ii) IMF Article IV mission chiefs and area stakeholders not covered by the survey, to department division chiefs for countries that assess the extent of FSAP's impact on their work had had an assessment; (iii) Bank country program and their views on the program. These directors for countries that had had an assess- included: ment; (iv) all FSAP team leaders as well as deputies and coleaders; and (v) all FSAP team a. bilateral donors, e.g., KfW (Kreditanstalt für members from the Bank and IMF.4 The services Wiederaufbau), DFID, etc.; of an external company were used to ensure the b. regional or multilateral banks, e.g., African De- confidentiality of the survey and its results. velopment Bank, Asian Development Bank, They conducted the survey in spring of 2005, European Bank for Reconstruction and De- largely online,5 and provided aggregate results velopment, European Central Bank, European to the IEG and IEO (individual responses were Investment Bank, Inter-American Develop- not revealed). The overall response to the ment Bank, etc.; 48 APPENDIX C: METHODOLOGY FOR FSAP EVALUATION c. standard setters, e.g., BIS, IOSCO, Financial reviewed, along with CAS documents and other Stability Institute, CPSS, IAIS, etc.; and Bank documents. These reviews were used to d. sources of technical assistance, e.g., SECO, supplement, confirm, or clarify findings from GTZ (Gesellschaft für Technische Zusamme- the detailed country studies and the surveys. narbeit), US Agency for International Devel- opment, European Community, Directorate- Review of FSAP Background Information and General for Development Cooperation, FIRST, Data etc. In order to build a solid knowledge base on FSAP, the IEG team collected all relevant Reviews of Other FSAPs and Updates background documents. In addition, it also FSAPs not covered in the detailed country collected and analyzed the relevant operational studies, as well as the updates, were also and budget data. 49 APPENDIX D: SURVEY OF STAKEHOLDERS As part of the evaluation, IEO and IEG undertook to all team members from the IMF and World jointly a survey of the key stakeholders involved Bank staffs. External experts were not included. in the FSAP. This appendix presents the method- ology used for surveying the views of partici- Main Features of the Questionnaires5 pants. The main findings from the surveys have been incorporated in the report. To ensure the · The outline of each questionnaire followed confidentiality of survey responses, an external broadly the outline of the evaluation ques- company was hired to administer the implemen- tions in the IEO and IEG issues/approach pa- tation and collection of results.1 The surveys pers. The main components of each were conducted in the spring of 2005, to a large questionnaire related to inputs, outputs, out- degree through an online modality.2 comes, and process issues. · There were about 30 questions for each group Population Surveyed of stakeholders. Where applicable, the same Survey questionnaires were sent to five groups questions were posed to different groups; a of stakeholders, consisting of different users number of questions applied only to specific and producers of the FSAP:3 groups. · Survey questions were mostly of the closed-end 1. Authorities. A single survey was sent to the type. Many consisted of specific statements authorities of all countries that had completed where respondents were asked to identify their an FSAP by the first quarter of 2005. Every ef- views on a 5-point scale (ranging from "strongly fort was made to send the survey directly to the agree" to "strongly disagree"). Some questions authorities in the country most directly in- had multiple choices, and others sought volved with the FSAP. "yes/no" answers. Where applicable, the re- 2. IMF Article IV mission chiefs and area spondents were given the opportunity to chose department division chiefs. The survey a "don't know" option and to write in their re- was sent to the relevant staff that worked on sponse ("other, please specify"). At the end of countries that had an FSAP. the survey, all respondents were given the op- 3. World Bank country directors. The sur- portunity to provide comments on the FSAP. vey was sent to the relevant directors that worked on countries with an FSAP. Survey Response 4. FSAP team leaders as well as deputies The overall stakeholder response to the survey and coleaders. Team leaders and co/deputy was quite high (53 percent of the net deliver- leaders are typically drawn one each from the able sample).6 Significantly different response IMF and World Bank.4 FSAP updates were rates were obtained across groups; those from treated as a separate assessment from the orig- the authorities and FSAP leaders and members inal FSAP. were the highest at around 60 percent (see 5. FSAP team members. The survey was sent table below). 51 FINANCIAL SECTOR ASSESSMENT PROGRAM Survey Samples and Response Rates Original Total Net unique nonqualifying deliverable Responses Response ratea Survey sample sample sample received (in percent) Authorities 81 5 76 45 59 IMF Article IV mission chiefs 83 9 74 27 36 World Bank directors 57 3 54 14 26 FSAP team leaders 79 8 71 45 63 FSAP team members 289 41 248 148 60 Total 589 66 523 279 53 Source: IEO report, January 2006. a. The response rate is the number of responses received as a percentage of the net deliverable sample. 52 APPENDIX E: MANAGEMENT RESPONSE Introduction · "The quality of the diagnostics has been good." Management welcomes IEG's Review of the Joint Bank-IMF cooperation has allowed an Financial Sector Assessment Program (FSAP), integrated approach toward financial sector which centers on (a) the relevance of the vulnerabilities and development needs, while program in assessing the stability and develop- expanding the depth and quality of analytic ex- ment needs of national financial systems, (b) pertise. Within this positive assessment, cov- cooperation between the IMF and the Bank as erage of specific sectors of the financial systems well as with other partners, and (c) coordina- has been of uneven quality, particularly where tion within the Bank. Management notes the nonbank financial institutions are concerned. Review's clear acknowledgment that "[t]he Also, "the scope of assessments should be objectives of the FSAP are thus highly relevant more tailored to the specific needs of the to the Bank's overall mission of fostering country." economic growth and reducing poverty." · "Country authorities have found the assess- ment to be useful" and have cited them for (a) General Alignment of Views. Management concurs providing an "independent evaluation" of the with most of the Review's analyses and system, (b) expanding their knowledge of fi- recommendations. They are generally in line nancial sector vulnerabilities; (c) improving with staff views and ongoing endeavors to adapt technical abilities; and (d) contributing to the the program to the evolving needs of our clients policy dialogue within the country. and to make it more effective and relevant. In · "FSAP recommendations need to be better in- this context, Management believes that the tegrated into Bank programs." IEG has found need for more systematic follow-up is the key that 42 percent of the FSAPs have a significant concern. Management, however, believes the impact on the Bank's country programs or di- issues raised by the Review in respect of country alogue. In another 34 percent of cases, the selection need to be understood in the context FSAP had a minor influence, while in the re- of the Bank's development mandate and the maining 24 percent it seemed to have had lit- needs of its clients, including low-income tle influence on the Bank's program. FSAPs countries. can serve as useful background pieces which are effectively leveraged. Overview. This response summarizes the main · "Country selection needs to better reflect sur- findings and conclusions of IEG's Review and veillance priorities and the likelihood of fi- sets forth Management's comments. The nancial sector reform." It involves difficult Management Action Record is attached. tradeoffs, notably in balancing the need to im- prove surveillance with development priorities. Summary of IEG's Findings and The voluntary nature of the program limits Recommendations the program's overall effectiveness in identi- The IEG Review's main messages in the fying systemic risks; however, there is a strong Executive Summary are quoted below with its consensus among authorities and staff sur- summary of associated findings: veyed for the IEG Review that the voluntary na- 53 FINANCIAL SECTOR ASSESSMENT PROGRAM ture of the program and associated country supervised with a longer tradition of informa- "ownership" should be maintained. tion collection and with longer-tested analytical · "The program must do a better job of keeping tools. Nonbank financial intermediation is of the Board informed in a timely manner," and more recent derivation when it exists, is Financial Sector Assessments (FSAs) should generally less systematically monitored, and is have a quicker turnaround. subjected to fewer analyses. Moreover, banking systems may exercise different functions across The recommendations for Management are countries, reflecting different legal and regula- in the attached Management Action Record tory structures. This can blur the distinction under the following headings: between banking and nonbanking activities and requires addressing issues of financial interme- · Sharpen country selection and scope of as- diaries in a functional rather than strictly institu- sessment, tional context. Notwithstanding the foregoing, · Integrate the FSAP into a full reform program, additional efforts are required to develop · Informing the Board, and relevant indicators and analytical tools for · Knowledge sharing. assessing development needs outside the banking system. The FSA Handbook1 provides a Management Comments benchmark on approaches and methodologies. An ongoing project to develop financial Coverage and Quality development indicators will improve monitor- Management welcomes the Review's finding on ing capacity. A review of methodologies aimed at the relevance of the FSAP to the Bank's mission nonbank financial institutions (NBFIs) to guide and the generally good quality of the assess- FSAP assessments will be pursued with due ments. It is gratified with the appreciation attention to resource trade-offs. expressed by country authorities on the useful- ness of the program. Reports on the Observance of Standards and Codes. While the IEG Review does not emphasize the Coverage. Management notes the recommenda- Bank Reports on the Observance of Standards tion to tailor the scope of the FSAP assessments and Codes (B-ROSCs), they can and should to the specific needs of the country. Given the contribute additional information to FSAP objectives of development, stability, and efficient assessments.2 B-ROSC and FSAP staffs are use of resources, Management agrees that there working to improve the coordination of assess- is an inherent need for judgment in resolving ments and follow-up work. The initiative is the trade-off between tailoring and comprehen- consistent with the call for more effective siveness. Early consultations across the Bank, partnership with other sector teams. with the authorities and the IMF, as well as awareness of other partners' programs should Staffing. Finally, the Review's observation on help guide the judgment. The current practice quality and coverage clearly suggests the need of involving country management in reviewing to ensure the consistent availability of staff with the scoping note should be helpful, although it sufficient experience and financial and political is unlikely to eliminate conflicting views. economy skills. Management will continue to work to ensure that it has the right staff and that Quality. Management notes the Review's observa- it assigns them effectively. The Financial Sector tion that assessments of the nonbanking sector Network plays a key role in this process. were not as consistent as those of the banking sector. This is inherent to the nature of nonbank Integration of FSAP Recommendations in financial intermediation, which is generally at a Country Programs much earlier development stage than that of Management concurs with the message that banking. The latter is broadly established and FSAP recommendations should be better 54 APPENDIX E: MANAGEMENT RESPONSE integrated in country programs. The nature of Country Selection Poverty Reduction Strategy Papers (PRSPs) and The Review acknowledges the difficult tradeoffs CASs and their multiyear cycle, as well as the the Bank must make in allocating limited FSAP joint World Bank/IMF nature of the FSAP, may resources, but highlights that "Country compound the coordination challenge. Notwith- Selection needs to better reflect surveillance standing, program adjustments are being made priorities and the likelihood of financial sector to involve country units more closely at the reform."3 Covering systemically important or initiation of the process and to keep them vulnerable countries is a priority inasmuch as involved throughout the assessments. For financial instability of such a country can have example, the practice of early involvement of serious economic and social costs, nationally country management should provide an and internationally. The Bank will maintain its opportunity to organize the handover from readiness to respond promptly to a request for FSAP diagnosis to follow-up. Participation of an assessment from a systemically important country unit staff in FSAP policy discussions may country. However, as the report notes, the Bank facilitate more prompt dialogue on follow-up. must factor in its focus on development and poverty reduction, including in smaller Bank Program versus Country Program. The Bank countries that may not be systemically need not be the sole source of financing and important from the perspective of global technical assistance in implementing the financial stability. recommendations of the FSAP. Often, reform needs identified by the FSAP can be Systemically Important Countries. Management has implemented by the country without external encouraged systemically important countries to TA, for example as follow-up to country-owned participate in the program, with overall good Poverty Reduction Strategies (PRSs) in low- success. Consequently, among countries still income countries. The full drawing down of due for initial assessments, there are only a Financial Sector Reform and Strengthening limited number of systemically important Initiative (FIRST) resources allocated to middle- countries (with signs of growing interest), and a income countries is a good reflection of follow- group of smaller and low-income countries, up that is not part of Bank country programs. including in Africa. The real test is not the Bank country program, but the country's reform program. Process of Country Selection. Currently, the process of country selection involves Regional Further Integration into Country Programs. Manage- and central units, both in the Bank and the ment wants, however, to increase the integra- IMF, while the Financial Sector Liaison tion of FSAP recommendations within country Committee acts as a forum to reach consensus programs in order to further enhance FSAP's on a feasible program. Both Boards have value to member countries. The recent practice confirmed that the program should be of early involvement of country management is voluntary and have already stated their criteria expected to provide an opportunity to organize for selection in the context of keeping the the handover from FSAP diagnosis to follow-up. program voluntary. That said, Management would ask the Bank Board to encourage partic- Donor Coordination. Management concurs that ipation as a form of good citizenship in the greater donor cooperation can increase international community. effectiveness of FSAP follow-up. The involve- ment of donors raises the issue of confidential- Informing the Board ity and the scope of information to be shared. Management concurs that the Board should Arrangements will continue to be explored in receive FSAs in a timelier manner and it will consultation with country authorities and the take action to ensure this. However, what is IMF to involve interested donors. more important is informing the Board on the 55 FINANCIAL SECTOR ASSESSMENT PROGRAM follow-up to these findings in country UseofFinancialSystemStabilityAssessments. Manage- programs. Whenever the main objective is mentdoesnotshareIEG'sviewontheeffectiveness related to a specific country assistance of using Financial System Stability Assessments program, discussions with the Board should (FSSAs) to expedite the preparation of FSA. Both take place in the context of discussions of the the FSA and FSSA are based on the FSAP main CAS, programming documents, and related report, with the former emphasizing developmen- lending operations. In addition, Management talissuesandthelatteremphasizingstability.Enforc- will organize periodic technical briefings to ing current deadline policy should improve the FSA highlight the cross-country financial sector timeliness. IEG's suggestion to leverage the FSSA issues that have an impact in shaping the downplays the relevance of the Bank's focus on Bank's financial sector work. Management has development and poverty reduction. In addition, it submitted a request for scheduling a first isunlikelytoacceleratetheprocessasitstillrequires technical briefing this spring. substantive additions to the document. 56 APPENDIX E: MANAGEMENT RESPONSE Management Action Record Major Monitorable IEG Recommendations Requiring a Response Management Response Sharpen country selection and scope of assessment. Management Management will continue to respond promptly to requests from system- of the Bank and the IMF should select countries for assessments and up- ically important and vulnerable countries for initial assessments and up- dates according to the priorities identified by the Board and adjust timing dates. A different phasing for less systemically important countries would to suit the situation. In some cases, such as systemically important and vul- be considered with due consideration to regard for the Bank's objectives nerable countries, this may mean that updates are carried out every two of development and poverty reduction and the lagging coverage of certain or three years, while in other countries, that are not systemically impor- countries, notably in Africa. tant or that are not likely to treat financial sector development as a prior- ity, initial assessments or updates may be delayed by several years. This Management will inform the Board on the confirmed list of countries due priority list, as well as the rationale for selection, should be presented to for an assessment through the current vehicles of the periodic reporting to the Board. In cases where countries have not volunteered, management the Boards of the IMF and the Bank on the program, country programming needs to inform the Board of those gaps, and should seek other ways to documents and discussions, and semiannual technical briefings. Man- work with the country on addressing critical issues in the financial sector. agement will continue to encourage countries to volunteer and will use al- The scope of the initial assessments and updates should be more tailored ternative vehicles for financial sector work when needed. to address the specific needs of the country. This may mean that a sys- temically important, vulnerable country may require a full set of ROSCs, and Management will continue to seek mechanisms to improve the tailoring of minimal discussion on missing markets, whereas another country may not the scope of initial assessments and updates to the specific needs of the require any formal ROSCs, but instead focus on access issues. Also, this country, with due consideration for the efficient use of resources. Early scop- process would be more efficient if more time was spent in advance, work- ing notes, benefiting from consultation with authorities, will permit a bet- ing with country units and authorities on developing prioritized TORs. ter understanding of country needs. They will permit more informed cross-unit discussions within the Bank and with the IMF and an improved decision-making process on scope and focus. Periodic Board updates on the FSAP program will review experience with increased tailoring, notably scoping notes. Integrate the FSAP into a full reform program. The Bank, in consulta- Management concurs with the need for a clearer framework for coordinat- tion with the IMF and other technical assistance providers, should seek to ing follow-up programs based on country ownership and the country's own establish a clearer framework for coordinating follow-up programs, in- action plan. Early consultation at the initiation of the FSAP and subsequently cluding technical assistance activities, based on the country's own action among the FSAP team, the IMF, country management, and country authori- plans. While details will differ by country, management should consider ties will be organized to facilitate the process and ensure a smooth handover building upon the FSAP by having follow-up discussions with country au- from the FSAP team to the Bank's country operations and dialogue. The ac- thorities on technical assistance and other development needs; such dis- tual process will vary according to country circumstances and partners. cussions should include Bank technical and country staff, as well as the IMF's Monetary and Financial Systems Department and area staff. Cognizant of country ownership, the Bank will encourage and support the Once development and other assistance priorities are determined, this in- sharing of information with the donor community by the country authori- formation should be quickly shared with the donor community. In addition, ties, consistent with confidentiality policies. While FSAP teams can hold in order to improve donor effectiveness, the following recommendations consultations with interested donors, within the framework of confiden- should be considered: (i) inform donors in advance about the timing of the tiality commitments, it would be premature to share key findings before their FSAP, so that donors can adjust their program timetables accordingly; (ii) confirmation through internal reviews within the IMF and the Bank, and the provide donors with better and more timely access to reports; (iii) improve information's formal conveyance to the authorities. Country-led consulta- consultation with donors who are active in the financial sector during the tions with interested donors to organize support for follow-up will be en- FSAP mission, including presentations of key findings. couraged. 57 FINANCIAL SECTOR ASSESSMENT PROGRAM Major Monitorable IEG Recommendations Requiring a Response Management Response The financial sector teams must also work to improve coordination with Coordination between country and sector teams will benefit from consul- the country units to ensure integration of the FSAP findings and recom- tation meetings at the time of country selection and again at the time of mendations into the Bank's country strategies. This would include greater the initial review of issues to be addressed in the assessment. Meetings participation of the country units in the design of TORs, mission teams, and chaired by the country director will provide guidance to the team leader and follow-up discussions. helps set the stage for later follow-up discussions. Management will re- port on the implementation of the above actions in its proposed periodic Board updates. Informing the Board. A deadline should be placed on the production time Currently, both the FSA and FSSA are based on the FSAP main report ,with of the FSA. One possibility to speed production is to leverage off the work the former emphasizing developmental issues and the latter emphasizing done by the IMF. The IMF FSSAs are usually completed within 16 days of stability, with due attention to their inherent linkage. The IEG's suggestion the delivery of the final report. While cognizant that the different foci, timeta- to leverage the FSSA downplays the relevance of the Bank's focus on de- bles, and processes make it difficult to quickly produce a joint paper, the velopment and poverty reduction. Moreover, it is unlikely to accelerate the Bank should consider using the FSSA as a platform to summarize the main process as it would require additions of substance to the FSSA. Pursuing findings of the assessment, and then add chapters and annexes to address timely delivery of an FSA, responding to well-identified information needs, issues upon which the IMF may not have focuses. directly based on the FSAP main report is deemed more appropriate. Critical information should be brought quickly to the Board or subcommit- A timely FSA, incorporating essential messages, will be a vehicle for bring- tee for discussion. In addition, discussions of key FSAP findings and rec- ing critical information to the Board. CAS discussions will provide an op- ommendations, and their integration into the country program (or the portunity to discuss integration of FSAP in country programs and follow-up. decision not to include them in the country program) should be part of the A request was submitted to schedule a first semiannual technical briefing CAS report and discussions. Bank management should also consider pe- in the spring. The briefings will include information on progress in improv- riodic technical briefings summarizing major findings from recent FSAPs, ing the timeliness of FSAs. as well as synthesizing the information to provide recommendations for fu- ture work (within countries, regions, and for future FSAPs). Knowledge sharing. While remaining mindful of the needs of safe- Management concurs with the objective of a more active leveraging of knowl- guarding sensitive and confidential information, the Bank should more ac- edge gained from the assessments. Initiatives in this direction have been tively leverage the knowledge gained from this extensive exercise. This taken. They include the completed FSA handbook, reviews of treatment of would include actively disseminating "best practice" analysis, more dis- specific topics across FSAPs (including International Organization of Securities cussion on concerns and issues that arise while conducting FSAPs, and im- Commissions issues and microfinance), the development of templates to proved sharing of data and insights gained through the program. The share accepted practices, periodic team leaders meetings as well as gath- financial sector teams should also work to partner more effectively with erings of assessors of standards to exchange experience. A more system- other sector teams to maximize collection and sharing of information. atic effort will be pursued with an annual program of knowledge sharing, within the overall objective of efficient resource use. Progress will be re- ported in the planned periodic briefings for Executive Directors. 58 APPENDIX F: CHAIRMAN'S SUMMARY, COMMITTEE ON DEVELOPMENT EFFECTIVENESS (CODE) The Informal Subcommittee (SC) of the uneven quality and appropriateness of coverage Committee on Development Effectiveness of specific sectors; (iii) candor of the reports (CODE) met on February 6, 2006 to discuss the was generally satisfactory, but there were also IEG Review of the Financial Sector Assessment instances of pressure to soften the written Program (FSAP), prepared by the Independent messages; (iv) the assessments often had weak Evaluation Group-Bank (IEG) and the Draft prioritization of recommendations, and did not Management Response to the IEG review. adequately consider sequencing or institutional Written statements were issued by capacity constraints; (v) the Bank needs to Messrs/Mmes Dorn, Hermann, and Ketsela. better integrate the assessments' findings and recommendations into its programs; and (vi) Background. The Financial Sector Assessment the impact of the FSAP on the Bank and its work Program (FSAP) is a joint World Bank-IMF initia- on financial sector development has not been tive, undertaken in response to the financial consistent, indicating that the full potential of crises of the late 1990s. The ultimate objectives the FSAP usefulness to the Bank has not yet of the program are: (i) the identification and been attained. mitigation of financial sector vulnerabilities and The report's recommendations focused on: their macroeconomic stability implications; and (a) the need to select the countries and scope (ii) fostering development of the financial for assessments based on the institutional sector and its contribution to economic growth. priorities, systemic importance of countries, The IEG evaluation focused on reviewing the likely country impact, country needs, and relevance of the program, quality and effective- timing; (b) integrating the FSAP into a country ness of inputs and outputs, and impact of the reform program and establishing a clearer assessments on different audiences. The IMF's framework for coordinating follow-up Independent Evaluation Office (IEO) has programs, based on the countries' own action undertaken a parallel evaluation of the FSAP. plans and the Bank's country strategies; (c) IEO and IEG have collaborated closely on all improving the process of informing the Board aspects of their respective evaluations. about the main findings; and (d) more actively The IEG evaluation concluded that the FSAP leveraging the knowledge gained from the FSAP is a good-quality analytical tool with overall through dissemination, information sharing, sound program framework, where joint Bank- and partnerships. IMF cooperation allowed for an integrated In its written response to the review, manage- approach toward financial sector vulnerabilities ment welcomed the IEG evaluation of the FSAP and developmental needs. Among specific and concurred with most of its analyses and lessons and findings identified by the evaluation recommendations. Management noted, are: (i) country selection ­ not all "systemically however, that the issue of country selection important" or vulnerable countries were needs to be viewed in the context of: the Bank's covered, nor were developing countries development mandate; clients' needs-- selected to reflect the potential impact the FSAP especially the small and low-income countries, could have on financial sector development; (ii) and the voluntary nature of the program. 59 FINANCIAL SECTOR ASSESSMENT PROGRAM Management agreed with the need to inform benefits both developed and underdeveloped the Board in a more timely manner and noted financial systems. Members acknowledged that that measures are being undertaken in this it would be premature to measure the actual regard (e.g., technical briefings). At the same impact of the program started in late 1990s, but time, management disagreed with IEG's sugges- also expressed interest in seeing some empiri- tion that the FSSA could be used as a platform cal evidence of implementation of reforms in to summarize the main findings of the assess- countries that have been assessed in the early ment in the preparation of the Bank's FSA, and years, and the impact of adopting the FSAP emphasized that the FSAP aide-mémoire is the recommendations on: country financial stabil- platform for both FSSA and the FSA and that ity, occurrence of crises (or absence thereof), these reports have different foci (for the FSSA, credit availability, investment and growth rates, stability, and for the FSA, development). etc. IEG noted that it had looked at the mentioned issues and found it difficult to Main Conclusions and Next Steps. The Subcom- construct a statistically valid dataset and make mittee commended the high quality of the IEG attributions owing to multiple lags between review of FSAP and the Management Response reforms, impact and data, and sample selection and broadly agreed with the review's analysis bias (due to the voluntary nature of the and recommendations. Members concurred program). IEG added that doing an FSAP does that the FSAP is highly relevant to the Bank's not guarantee avoiding crisis, and it is not mission and, therefore, warrants additional possible to prove that a crisis has been averted Board involvement through timelier reports to in any particular case. A member was interested the Board. Members were strongly in favor of in more information on the viability of the stress maintaining the voluntary nature of the test, and reliability of the underlying data. IEG program, which is equally relevant for systemi- noted that the stress test is just one of the tools cally important countries and those interested to evaluate the financial system and its limita- in implementing the reforms. Members tions should be properly communicated to and welcomed the joint Bank-IMF approach to acknowledged by the authorities. Management conducting FSAPs, and the efficient division of added that although stress testing is a useful labor based on expertise, mandates, and tool, it has its limitations when applied to comparative advantages of two institutions. detecting systemic vulnerabilities of the entire Issues raised included: the need to keep the system and is probably more valuable when scope of FSAPs manageable; employing applied to individual institutions. country-specific approach; importance of nonbanking sectors, including access to Integration into Country Programs. Some members finance; more extensive use of the FSAP in noted with concern the IEG observation that terms of donor coordination and identification less than half (42 percent) of the assessments of best practices; centrality of the authorities as had a significant impact on the Bank's country the primary audience of the reports and the programs, and only 34 percent of country importance of country ownership. It was authorities recalled the Bank engaging in suggested to proceed with disclosure of the follow-up to the FSAP (as opposed to an 80 evaluation following the CODE Subcommittee percent follow-up rate at the IMF), and asked discussion and to brief the Board on the IEG for more detailed explanation in this regard. FSAP review at the time of Management's They supported the IEG recommendation to semiannual Board briefing on FSAP. The follow- establish a follow-up mechanism, and make the ing points were raised. best use of the expertise of the FSAP team even in the absence of a relevant follow-up Bank Relevance and Impact of the FSAP. Members program or lending operation. Management broadly agreed with the relevance of the FSAP noted that the differences in the rate and time for poverty reduction and growth, which of follow-up actions are stipulated by the differ- 60 APPENDIX F: CHAIRMAN'S SUMMARY, COMMITTEE ON DEVELOPMENT EFFECTIVENESS (CODE) ent business models, organizational structures, institutions--a point that was also confirmed by and priorities of the two institutions. Unlike in management. A view was expressed regarding the IMF, financial sector work in the Bank is the need to examine the relation of the FSAP highly decentralized, with over 20 units with the Review of Standards and Codes involved in providing policy-based TA and program and how they can be better coordi- advice on financial sector-related matters to the nated to increase efficiency of the process. clients. Management recognized the importance of working within the existing Country Selection. Members noted that the main institutional constraints and will make criteria for country selection and allocation of additional efforts to involve country units in the necessary resources should be their FSAP process. Several members agreed with the demonstrated interest in reforms, existing Management view that financial sector reform preconditions for financial sector development, needs identified by the FSAP may not necessar- and systemic importance or vulnerability. It was ily be supported by the Bank only, but include broadly agreed that participation in the other donors who may choose to support some program should remain voluntary. Several recommendations. They urged Management to members emphasized that giving priority to implement specific measures to increase systemically important countries within the communication and collaboration within the allocated resource envelope should not exclude Bank Group, other donor partners and the the poor or smaller countries, whose financial clients to assist countries interested in incorpo- systems do not play significant role in the rating FSAP findings into their PRSPs and to international financial system. facilitate inclusion of FSAP findings in CASs. In response to some members' request to honor Coverage. Some members and participants felt the confidentiality of country assessments and that expanding FSAP coverage beyond its close dialogue with country authorities, IEG present scope (including topics such as access concurred that no confidential information can to financial services, missing markets, political be disclosed and shared with other donors economy, etc.) would undermine the consis- without the consent of the client, but added tency of the exercise and would require that, usually, there is plenty of publicly available additional skills and resources. Others noted information that can be brought to the that nonbanking sectors often carry equal attention of the donor partners. Some members importance and were interested in why analyz- highlighted the importance of the FSAPs both ing them might be problematic. A member to inform and draw on other pieces of the suggested that future FSAPs include the issues Bank's analytical work, but cautioned not to of cross-border linkages of financial systems duplicate the work done in the context of other more systematically. Some members felt that sectoral ESW. the issue of access to finance should have been given more prominence in the FSAPs, and Bank-IMF Cooperation. Members broadly especially those done in poorer countries. welcomed the joint Bank-IMF execution of the Management agreed with the importance of the FSAPs. They stressed the importance of balanc- FSAPs looking at access to finance, but also ing the objectives of the two institutions underscored that they should not be viewed in through the Financial Sector Liaison Committee a dichotomy vis-à-vis stability. (FSLC) and making the best use of their respec- tive comparative advantages: the IMF is well- Audience and Country Approach. Several members equipped to deal with macroeconomic and stressed that the main audience of the FSAP is, financial stability issues, and the Bank has an and should remain the country authorities, advantage in handling other sectoral issues. IEG which should benefit from an independent noted that its review of the FSLC indicated a evaluation of the financial sector vulnerabilities. good level of cooperation between the two In this context, they also underlined the crucial 61 FINANCIAL SECTOR ASSESSMENT PROGRAM role of country ownership for achieving the the Bank much longer to issue its summary of maximum positive impact of the program. the FSAP (FSA), as compared with the IMF's Some members urged taking opportunities to FSSA and asked for explanations in this regard. standardize the assessment framework, Management agreed with the need to keep the especially for the banking sector and its linkages Board better informed and added that it is to the other segments of the financial sector. A planning to provide the Board periodic reports member noted that a "handbook" coming out on the FSAPs and ROSCs, together with a of the FSAP experience could be a useful and semiannual technical briefing to the Board on cost-saving tool. Others were in favor of tailor- the FSAP this spring. Management also agreed ing FSAPs to the country needs, commensurate that delays in the circulation of the FSAs should with their levels of development. IEG replied be kept to the minimum and will undertake that although a certain degree of standardiza- necessary measures in this regard. tion is possible in selected sectors, the differ- ences in the levels of development of financial Discussion at the Board and the Report's Dissemina- systems in the countries under review do not tion. Some participants noted that the IEG allow applying a common-template approach. evaluation of the FSAP should have been discussed by the full Board, taking into account Informing the Board. Some members supported that the parallel IMF IEO review was discussed the IEG recommendation to have a deadline on at the IMF's Board. IEG noted that it is normal the FSA production and indicated that there practice for its evaluations (with few was a need to have a Board discussion of the exceptions) to be discussed at either CODE or main findings of the FSAP. Others noted that, CODE Subcommittee. A suggestion was made given the technical nature of the FSAP, a to brief the Board about the IEG review of the summary of the main findings should be a FSAP in tandem with the upcoming semiannual sufficient vehicle for informing the Board and Management briefing on the FSAP implementa- should normally be discussed in the context of tion. In the absence of objections, the report a respective CAS. Some members and partici- will be disclosed following the CODE Subcom- pants expressed disappointment that it takes mittee discussion. Pietro Veglio, Chairman 62 ENDNOTES Chapter 1 a short timeframe. With only 12 FSAPs completed in 1. A typical report would include reviews of all the 2000, and 20 in 2001, and given the differing charac- major financial sectors, including banking, insurance, teristics of the countries, it is not possible to find a sta- capital markets, pensions, and asset management, tistically significant connection between the FSAP and and would cover legal, supervisory, and regulatory changes in indicators. The problem is compounded systems, the infrastructure supporting the sectors, by the fact that the countries volunteering for the and a discussion of market structure, competition, and program are not necessarily representative of the vulnerabilities. The reviews usually include several overall population. formal Reports on the Observance of Standards and 5. FSAPs conducted since fiscal year 2004 were Codes (ROSCs), which assess the degree to which a also reviewed; however, because the most recent system is compliant with recommended standards FSAPs often have not yet completed their outputs set by different standard-setting bodies. Technical (FSAs and FSSAs), and are too recent to have had an notes usually include more detailed reviews and dis- impact on with Bank programs or country programs, cussions on specific financial sector topics, such as they were not included in the surveys or detailed re- stress tests, rural finance, access to credit, etc. Nor- views. mally, the draft aide-memoire is provided to the au- 6. While IEG participated in interviews on the ad- thorities at the end of the mission, and after review vanced economies, it did not assess those FSAPs as part and revision at Bank and IMF headquarters, the report of this evaluation. See appendix C for a list of FSAP is finalized and sent to the authorities. countries and detailed review countries. 2. "The objective of the FSAP is to benefit countries 7. Authorities from those two countries were also by alerting them to likely vulnerabilities within their contacted for their views on the FSAP, but discus- financial sectors and to assist the Bank and the IMF sions were not held. (See appendix C). and the international community more broadly, in designing appropriate assistance. The FSAP will ensure Chapter 2 that financial sector analysis is brought further into the 1. Internal Bank document. core of economic policy discussions. It is expected that 2. Internal Bank document. the program will better help countries reduce finan- 3. See Caprio 2001: Rajan and Zingales 1998; Levine, cial sector vulnerabilities, and therefore the likeli- Loayza, and Beck 2000. hood of crisis. It will also provide national authorities 4. Internal Bank document. with a strategic framework within which to under- 5. Internal Bank document. take the strengthening of the financial systems." (In- 6. Internal Bank document. ternal Bank document.) 7. World Bank and IMF 2000. 3. In the case of most industrialized countries, the 8. This section relies heavily on an analysis by exercise was led by the IMF. The Bank contributed staff IEO(see IEO 2006, pp. 19­22, and annex 4) for addi- with specific development expertise. tional detail. Note that the FSAP documents do not 4. While intermediate actions (such as passage of provide definitions of "systemically important" or a new banking law, or improved regulations) may "vulnerable" countries. This analysis used a number occur reasonably quickly, the strengthening and deep- of different definitions for systemic importance, in- ening of the financial sector cannot be measured over cluding G-20, G-7 plus markets covered by the Emerg- 63 FINANCIAL SECTOR ASSESSMENT PROGRAM ing Markets Bond Index, G-7 plus countries covered creased through 2003, and then decreased in 2004, but by the IMF's International Capital Markets depart- had no overall pattern (i.e., consistent increase or ment, or countries monitored by the Bank for Inter- decrease over time). More significant than the num- national Settlements. For measures of "vulnerability," ber of topics was the matching of topics to the indi- internal IMF Monetary and Financial Systems De- vidual countries' priorities. partment reports and Standard and Poor's global fi- 15. World Bank and IMF 2005a, p. 33. nancial ratings were used. The findings did not vary 16. Management believes the scope of tailoring significantly under the different definitions. initial assessments depends on available background 9. Eighty percent of authorities, 69 percent of work on the country's financial sector and country con- country directors, 52 percent of IMF mission chiefs, ditions. The scope for tailoring is likely to be greater and 59 percent of team leaders surveyed supported for updates, given the availability of initial assess- a voluntary program. ments and subsequent work. IEG believes that even 10. Fifty-eight percent of authorities surveyed given the varying level of current knowledge, there is thought too much data was required, and 41 percent a wide scope for tailoring in the initial assessments (for thought the exercise was too time consuming. example, in a low-income country under stress, the 11. From the Communique of the Meeting of G-20 initial assessment might focus mainly on stability and Finance Ministers and Central Bank Governors, access issues, and then, as the country's economic en- 1999: "G-20 Finance Ministers and Central Bank Gov- vironment improves, the update might actually be ernors welcomed the important work that has been broader, as it reviews additional subsectors). done by the Bretton Woods institutions and other 17. Data from all completed FSAPs from fiscal years bodies toward the establishment of international 2001­04 were used; the first set of pilot FSAPs were codes and standards in key areas, including trans- omitted, as inspection of figures indicated that budget parency, data dissemination, and financial sector pol- codes had not been consistently used in the first year. icy. They agreed that the more widespread Fiscal year 2005 FSAPs and some fiscal year 2004 FSAPs implementation of such codes and standards would were also omitted because not all final documents contribute to more prosperous domestic economies had been delivered; therefore costs to date were only and a more stable financial system. To demonstrate partial. Costs include direct labor costs and travel, but leadership in this area, Ministers and Governors agreed exclude overhead calculations. "Fiscal year" refers to to undertake the completion of Reports on Obser- that of the first mission. This evaluation only includes vance of Standards and Codes ("Transparency Re- costs from FSAPs with joint participation; FSAPs car- ports") and Financial Sector Assessments, within the ried out by IMF staff-only were omitted from the data context of continuing efforts by the IMF and the World set. Note that because the IMF and World Bank use dif- Bank to improve these mechanisms." ferent budget systems, the two sets of costs may not 12. Internal Bank document. be directly comparable; however, based on discus- 13. Management has commented that FSAPs would sions with budgets officers in both institutions, the fig- benefit from the inclusion of ROSCs on corporate ures used are generally comparable; the remaining governance, insolvency and creditor rights, and ac- differences are not sufficiently great to affect the over- counting and auditing. IEG does not disagree with the all conclusions. importance of those ROSCs. However, IEG's review 18. Initial indications are that the mix of countries of FSAPs does not provide evidence supporting or dis- affects costs, with FSAPs for systemically important puting the hypothesis that conducting formal ROSCs countries being more expensive. However, because in those areas in the framework of the FSAP has been there is a weighting toward the early years for those more or less effective than carrying out tailored reviews countries, there is not yet enough data to distinguish outside the ROSC or FSAP framework, nor is there suf- the country effects versus time effects. ficient evidence to determine whether these ROSCs 19. The definitions of major and minor sector are more critical than other ROSCs. work are not firm. "Major" work or "formal" sector 14. IEG reviewed all the FSAPs carried out for Bank work are usually pieces of sector work which were client countries. The number of major topics reviewed "delivered" to the Board, in addition to the govern- (i.e., significant discussion or analysis in the paper) in- ment, such as Financial Sector Reviews, Financial 64 ENDNOTES Sector Strategies, etc. In addition, regions often pro- ately Unsatisfactory (3); and Unsatisfactory (4). Note duced "informal" sector work, which were usually that the figures in table 3.1 are not the same as the fig- shorter policy pieces sent to the government, but not ures in the IEO's evaluation, reflecting differences in formally published. IEG has attempted to categorize the sample (IEO's evaluation includes six advanced the ESW by scope and depth. IEG has omitted from economies), as well as IEG's greater emphasis on the its count ESW that is funded under special programs development aspects of the assessments. (such as Anti-Money Laundering), and also has not 2. In the low-income countries where FSAPs have counted general research on financial sector topics, been completed, banking sector assets represented ap- but instead has focused on ESW written for client proximately 85 percent of total financial sector assets. countries (such as policy notes or FSRs), or groups 3. An internal Bank document states: of client countries (such as "Central American Capi- tal Markets" or "Bank Consolidation in the Europe and The experience to date suggests that there is a Central Asia Region" (ECA). role in the FSAP for stress tests, in part because 20. It has been suggested that the shorter pro- they highlight the link to macroeconomic de- cessing times for "IMF-only" FSAPs reflect that those velopments and can be a useful tool to help FSAPs are less complex. There is no clear evidence to missions form an overall view of a financial support this hypothesis; review of the complexity of system's robustness. However, it is also clear the FSAPs (scope, depth of coverage, time in field) in- that stress tests are subject to limitations and dicate that the IMF-only assessments are as complex, their value erodes over time. They can pro- and in some cases, even more complex than many of vide only a static analysis at one point in time the joint assessments. of financial system health based on specific (and often crude) assumptions on the evolu- Chapter 3 tion of key variables as well as on the reliability 1. In addition to the FSAP documents, the re- of balance sheets. Further, sufficiently detailed viewers read CASs, Article IV reports, relevant ESW, and data on individual institutions may not exist projects reports to evaluate whether the FSAP had fully or may not be provided to FSAP teams . . . reviewed, prioritized, and analyzed the issues. Re- viewers rated the FSAPs along a number of dimensions, . . . Stress test analysis is likely to be more use- including coverage, depth of analysis, clarity and can- ful in cases where there are few banks, so that dor of findings, clarity, and prioritization of recom- the stress tests can contribute to a picture of mendations. IEO assessors carried out ratings in the health of the banking system as a whole. In parallel. Both sets of reviewers used the same set of any event, stress testing should be viewed as guidelines, in order to ensure consistency of ratings. one of the tools available to help form views on For example, a question on coverage of a sector might potential risks and vulnerabilities, but the out- have the following rating definitions: come of the tests should be carefully evaluated, taking into account the quality of input data as 1: Comprehensive, including (where appropri- well as the underlying models and their as- ate) market structure, compliance with stan- sumptions. dards and codes stress testing, governance issues, legal and institutional aspects, etc. 4. An internal Bank document states: 2: Some aspects not adequately covered (with- out a clear explanation of why) or only partially While standards assessment may be a useful discussed. first step, they have not been found by FSAP mis- 3: Several aspects not covered, partially discussed. sions to be particularly helpful in isolation in 4: Focus only on a few aspects (without proper ex- identifying short-term risks, vulnerabilities, planation), cursory discussion across the board. or key development weaknesses. Moreover, an analysis of the Basel Core Principles (BCP) as- The ratings for questions generally correspond to sessments done by Bank and IMF staff both in Satisfactory (1); Moderately Satisfactory (2); Moder- FSAP mission and as free-standing exercises in 65 FINANCIAL SECTOR ASSESSMENT PROGRAM the context of technical assistance, confirms FSAPs have benchmark data on insurance, but Lebanon that for effective surveillance, BCP assessments and Yemen do not. should be conducted in a broader institutional 12. The Dominican Republic is an interesting case and macroeconomic context, rather than on study of the limitations of FSAPs. The FSAPs did diag- a stand-alone basis. nose severe and widespread vulnerabilities in the banking system, including an undercapitalized bank- 5. For example, the Basel Core Principles ratings for ing system, inadequate provisioning, overall weak one ECA country showed that it was compliant with 17 compliance with BCP standards, and weak institu- principles, largely compliant with seven, and materially tional capacity and judicial enforcement. Despite pres- noncompliant with only one, even though the FSAP text sures from the government and IMF area department, stated "remaining gaps are mainly in the regulation the key messages were conveyed in the FSSA and FSA and supervision of country, market, and operational risk, (although the language was toned down from the aide formalizing information sharing arrangements, and mémoire and a presentation to the authorities). The strengthening corporate governance in banks." Infor- FSAP did not detect the immediate cause of a subse- mation on ownership is considered largely compliant quent crisis, which involved fraudulent accounting. and connected lending limits are rated "in compli- The FSAPs cannot substitute for effective accounting ance," although the text notes that "lack of knowledge and supervision. (See IEO 2006, box 3 for a more de- of the ultimate shareholders also makes effective mon- tailed discussion of the Dominican Republic case). itoring of prudential standards impossible." 13. In some cases, authorities indicated they were 6. The authorities for a low-income African coun- only interested in the assessment, and did not seek try pressed management to raise the ratings to take assistance for follow-up work. into account their level of development; the ratings were lowered back to the original ratings when the Chapter 4 team leader pointed out that increased ratings were 1. IMF mission chiefs and Bank country directors higher than those of Canada. were less sanguine about the impact on debate. Only 7. For example, authorities and donors working in about one-third of Bank country directors and IMF mis- one Middle East and North Africa Region country sion chiefs thought that the FSAP had contributed to stated that the positive ratings of the FSAP made it a policy debate. harder to push reforms through, as those opposed to 2. IEG omitted FSAPs for countries that do not reform pointed to the ratings as evidence that re- have an active Bank program, or for which the FSAP form was not needed. In another case, an ECA coun- was only recently completed and would therefore try used the FSAP ratings as justification for not moving not have been factored into country programs yet. forward with reform. 3. Internal Bank document. 8. World Bank and International Monetary Fund 4. This is not only owing to lack of interest from 2005b. the country units. Some team leaders did not want to 9. Management has noted that the analytical tools take country unit staff on mission, preferring to take have been improved during the FSAP program, point- only technical staff. ing in particular to the treatment of annuities in pen- 5. There were numerous examples of countries sion reform. IEG review of FSAPs indicates that where a CAS was prepared shortly before an FSAP was individual analytical tools have not been systematically launched, or was already under way, but the CAS applied (or increasingly applied over time), including made no mention of the FSAP. This is another illus- the treatment of annuities. tration of how little ownership the country depart- 10. On a scale from 1 to 5 (with 5 being "strongly ments have in the FSAP. agree") 91 percent of team members gave a rating of 6. Reasons for not increasing funding included, 4 or 5 on the question of whether authorities coop- inter alia: (i) competing priorities for funds, (ii) erated with the process, and 88 percent felt that the lack of commitment or political support for reforms; access to data was sufficient. (iii) current funding levels that were deemed ade- 11. Benchmarking of data does not appear regu- quate; and (iv) support that was available from other larly. For example, the Tunisia, Jordan, and Morocco sources. 66 ENDNOTES 7. One example is the Bank's work in pensions. 6. Net deliverable sample means total target pop- IEG's evaluation, Pension Reform and the Develop- ulation minus those who could not be contacted for ment of Pension Systems (2006), found that in many various reasons. countries, access of Bank staff working on pension re- form to FSAP documents was limited owing to confi- Appendix D dentiality issues. 1. The external company was Fusion Analytics LLC, 8. Management is broadly in agreement with these Washington, DC. goals, and has noted that it is already working on as- 2. Participants were also given an opportunity to sessing and designing policies, costs, and modalities send their responses directly to Fusion Analytics by fac- for improving knowledge sharing, including the ex- simile. Only a handful of authorities replied by fax. pansion of current initiatives. 3. Stakeholders other than the authorities that had 9. IEO 2006, p.59. been involved with more than one FSAP were invited 10. The majority of these interviews were con- to submit a survey response for each country (up to ducted by IEO staff. a maximum of three). 11. The downside of this perception of "signal- 4. In advanced economies, the IMF has responsi- ing" is that several countries that have delayed bility for the FSAP. FSAPs have cited the desire to do their own review 5. For those readers who are interested in seeing and cleanup prior to the FSAP, rather than risk hav- details of the specific questions and responses, a full ing the FSAP review problems or make negative as- version of each questionnaire (and a summary of the sessments. responses) will be made available on the IEO website at www.imf.org/ieo. Appendix C 6. Net deliverable sample is defined as the total tar- 1. IEO carried out 25 detailed country studies; in get population minus those who could not be con- addition to the 19 reviewed by IEG, IEO also reviewed tacted for various reasons. FSAPs for six advanced economies. 2. At the initiation of the FSAP evaluation, 70 FSAPs Appendix E had been completed, of which 57 were of developing 1. World Bank and IMF 2005b. countries. 2. The three B-ROSCs are corporate governance, 3. The questionnaires and evaluation matrix mod- insolvency and creditor rights, and accounting and au- els were developed jointly by IEO and IEG. diting. Accounting and auditing involve separate stan- 4. A sample of external team members were in- dards, but they are typically assessed as a single terviewed directly. ROSC. 5. Only a handful of authorities replied by fax. 3. See main messages box, Executive Summary. 67 REFERENCES Beck, Thorsten, Asli Demirguc-Kunt, and Ross Independent Evaluation Group (IEG). 2006. Pen- Levine. 2004. "Finance, Inequality and sion Reform and the Development of Pen- Poverty: Cross-Country Evidence." World sion Systems. Washington, DC: World Bank. Bank Policy Research Working Paper 3338. Klingebiel, Daniela, and Luc Laeven. 2002. "Man- Washington, DC: World Bank. aging the Real and Fiscal Effects of Banking ------. 2005. "SMEs, Growth, and Poverty: Cross- Crises." World Bank Discussion Paper No. Country Evidence." National Bureau of Eco- 428, World Bank, Washington, DC. nomic Research Working Paper No. 1124, Levine, Ross, Norman Loayza, and Thorsten NBER, Cambridge, MA. Beck. 2000. "Financial Intermediation and Caprio, Gerard. 2001. "Finance for Growth: Pol- Growth: Causality and Causes." Journal of icy Choices in a Volatile World." World Bank Monetary Economics 46 (1): 31­77. Policy Research Report. Washington, DC: Nabi, Ijaz, and Jayasankar Shivakumar. 2001. Back World Bank. from the Brink: Thailand's Response to the Caprio, Gerard, and Daniela Klingebiel. 2003. 1997 Economic Crisis. Directions in Devel- "Episodes of Systemic and Borderline Finan- opment Series. Washington, DC: World Bank. cial Crises." Washington, DC: World Bank. National Bureau of Economic Research (NBER). Claessens, Stijn, and Daniela Klingebiel, and Luc 2004. "Finance and Growth: Theory and Ev- Laeven. 2001. "Financial Restructuring in Bank- idence." NBER Working Paper No. 10766, ing and Corporate Sector Crises: What Policies NBER, Cambridge, MA. to Pursue?" National Bureau of Economic Re- Rajan, Raghuram G., and Luigi Zingales. 1998. search Working Paper No. 8386, NBER, Cam- "Financial Dependence and Growth." Amer- bridge, MA. ican Economic Review 88 (3): 559­86. Claessens, Stijn, and Enrico Perotti. 2005. "The World Bank. 2000a. "East Asia Brief," East Asia and Links between Finance and Inequality: Chan- Pacific Region, World Bank, Washington, DC. nels and Evidence." Background paper pre- http://wbln0018. worldbank.org/eap/eap.nsf. pared for the World Development Report ------. 2000b. "The Evolution of Poverty dur- 2006, World Bank, Washington, DC. ing the Crisis in Indonesia, 1996­99." World G-20 Finance Ministers and Central Bank Gov- Bank Policy Research Working Paper No. 2435. ernors. 1999. "Communique of the Meeting East Asia and Pacific Region, World Bank, of G-20 Finance Ministers and Central Bank Washington, DC. Governors," Berlin, Germany, December ------. 2003a. "Argentina-Crisis and Poverty 15­16. 2003: A Poverty Assessment." Report No. Honohan, Patrick. 2004. "Financial Sector Pol- 26127. Washington, DC: World Bank. icy and the Poor: Selected Findings and Is- ------. 2005. Financial Sector Assessment Pro- sues." World Bank Working Paper No. 43, gram (FSAP) ­ Review, Lessons, and Issues World Bank, Washington, DC. Going Forward. February 22, 2005. Washing- Independence Evaluation Office (IEO). 2006. Re- ton, DC: World Bank. port on the Evaluation of the Financial Sec- World Bank and IMF. 2000. Financial Sector As- tor Assessment Program. Washington, DC: IMF. sessment Program (FSAP): A Review--Lessons 69 FINANCIAL SECTOR ASSESSMENT PROGRAM from the Pilot and Issues Going Forward. ------. 2005a. Financial Sector Assessment Pro- November 27, 2000. Washington, DC: World gram (FSAP)--Review, Lessons, and Issues Bank.http://www.worldbank.org/finance/html/ Going Forward. February 22, 2005. Washing- policy_fsap.html. ton, DC: IMF. http://www.imf.org/external/np/ ------. 2003. Financial Sector Assessment Pro- fsap/2005/022205.htm. gram (FSAP)--Review, Lessons, and Issues ------. 2005b. Financial Sector Assessment: A Going Forward. February 24, 2003. Washing- Handbook. Washington, DC: World Bank and ton, DC: IMF. http://www.imf.org/external/ IMF. http://www.imf.org/external/pubs/ft/fsa/ np/fsap/2003/review.htm. eng/index.htm. 70 IEG PUBLICATIONS Study Series 2004 Annual Review of Development Effectiveness: The Bank's Contributions to Poverty Reduction Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank's Approach to Global Programs Agricultural Extension: The Kenya Experience Assisting Russia's Transition: An Unprecedented Challenge Bangladesh: Progress Through Partnership Brazil: Forging a Strategic Partnership for Results--An OED Evaluation of World Bank Assistance Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy Capacity Building in Africa: An OED Evaluation of World Bank Support The CIGAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research Country Assistance Evaluation Retrospective: OED Self-Evaluation Debt Relief for the Poorest: An OED Review of the HIPC Initiative Developing Towns and Cities: Lessons from Brazil and the Philippines The Drive to Partnership: Aid Coordination and the World Bank Economies in Transition: An OED Evaluation of World Bank Assistance The Effectiveness of World Bank Support for Community-Based and ­Driven Development: An OED Evaluation Evaluating a Decade of World Bank Gender Policy: 1990­99 Evaluation of World Bank Assistance to Pacific Member Countries, 1992­2002 Financial Sector Reform: A Review of World Bank Assistance Financing the Global Benefits of Forests: The Bank's GEF Portfolio and the 1991 Forest Strategy and Its Implementation Fiscal Management in Adjustment Lending IDA's Partnership for Poverty Reduction Improving the Lives of the Poor Through Investment in Cities India: The Dairy Revolution Information Infrastructure: The World Bank Group's Experience Investing in Health: Development Effectiveness in the Health, Nutrition, and Population Sector Jordan: Supporting Stable Development in a Challenging Region Lesotho: Development in a Challenging Environment Mainstreaming Gender in World Bank Lending: An Update Maintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh The Next Ascent: An Evaluation of the Aga Khan Rural Support Program, Pakistan Nongovernmental Organizations in World Bank­Supported Projects: A Review Poland Country Assistance Review: Partnership in a Transition Economy Poverty Reduction in the 1990s: An Evaluation of Strategy and Performance The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank's Support Through 2003 Power for Development: A Review of the World Bank Group's Experience with Private Participation in the Electricity Sector Promoting Environmental Sustainability in Development Putting Social Development to Work for the Poor: An OED Review of World Bank Activities Reforming Agriculture: The World Bank Goes to Market Sharing Knowledge: Innovations and Remaining Challenges Social Funds: Assessing Effectiveness Tunisia: Understanding Successful Socioeconomic Development Uganda: Policy, Participation, People The World Bank's Experience with Post-Conflict Reconstruction The World Bank's Forest Strategy: Striking the Right Balance Zambia Country Assistance Review: Turning an Economy Around Evaluation Country Case Series Bosnia and Herzegovina: Post-Conflict Reconstruction Brazil: Forests in the Balance: Challenges of Conservation with Development Cameroon: Forest Sector Development in a Difficult Political Economy China: From Afforestation to Poverty Alleviation and Natural Forest Management Costa Rica: Forest Strategy and the Evolution of Land Use El Salvador: Post-Conflict Reconstruction India: Alleviating Poverty through Forest Development Indonesia: The Challenges of World Bank Involvement in Forests The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support Uganda: Post-Conflict Reconstruction Proceedings Global Public Policies and Programs: Implications for Financing and Evaluation Lessons of Fiscal Adjustment Lesson from Urban Transport Evaluating the Gender Impact of World Bank Assistance Evaluation and Development: The Institutional Dimension (Transaction Publishers) Evaluation and Poverty Reduction Monitoring & Evaluation Capacity Development in Africa Public Sector Performance--The Critical Role of Evaluation All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual selection, please visit http://www.worldbank.org/ieg TMxHSKIMBy366523zv":':':;:- THE WORLD BANK ISBN 0-8213-6652-1