55567 Analyzing the Effects of Policy Reforms on the Poor An Evaluation of the Effectiveness of World Bank Support to Poverty and Social Impact Analyses THE WORLD BANK GROUP WORKING FOR A WORLD FREE OF POVERTY The World Bank Group consists of five institutions--the International Bank for Reconstruction and Development (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Investment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their environment by providing resources, sharing knowledge, building capacity, and forging partnerships in the public and private sectors. THE INDEPENDENT EVALUATION GROUP IMPROVING DEVELOPMENT RESULTS THROUGH EXCELLENCE IN EVALUATION The Independent Evaluation Group (IEG) is an independent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses on assessment of IFC's work toward private sector development, and IEG-MIGA evaluates the contributions of MIGA guarantee projects and services. IEG reports directly to the Bank's Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group's work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. Analyzing the Effects of Policy Reforms on the Poor An Evaluation of the Effectiveness of World Bank Support to Poverty and Social Impact Analyses 2010 The World Bank http://www.worldbank.org/ieg Washington, D.C. ©2010 The International Bank for Reconstruction and Development/The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved 1 2 3 4 13 12 11 10 This volume is a product of the staff of the International Bank for Reconstruction and Development/The World Bank. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other informa- tion shown on any map in this work do not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development/The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 , H Street NW Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. Cover: A student at the public primary school Alvaro Contreras in Tegucigalpa, Honduras. Photo by Alfredo Srur, courtesy of the World Bank Photo Library. ISBN-13: 978-0-8213-8293-6 e-ISBN-13: 978-0-8213-8294-6 DOI: 10.1596/978-0-8213-8293-6 Library of Congress Cataloging-in-Publication data have been applied for. World Bank InfoShop Independent Evaluation Group E-mail: pic@worldbank.org Communications, Learning and Strategy Telephone: 202-458-5454 E-mail: ieg@worldbank.org Facsimile: 202-522-1500 Telephone: 202-458-4497 Facsimile: 202-522-3125 Printed on Recycled Paper Contents v Abbreviations vii Acknowledgments ix Foreword xi Executive Summary xvii Management Response xxi Chairperson's Summary: Committee on Development Effectiveness 1 1 Introduction 4 What Is a PSIA? 4 Recent Status 6 Evaluation Objectives and Rationale 7 Evaluation Scope, Questions, and Building Blocks 9 2 Overview of PSIAs 11 Numbers and Costs 11 Objectives 12 PSIA Practice versus Initial Concept 15 3 PSIA Effect at the Country Level 17 Effect on Country Policies 24 Contribution to Country Capacity for Policy Analysis 27 4 PSIA Effect within the Bank 29 Effect on Bank Operations 30 Effect on Thinking and Practice across the Bank 33 Recent Developments 35 5 Lessons and Recommendations 37 Lessons 39 Recommendations 41 Appendixes 43 A PSIA Timeline 45 B Universe of Bank-Funded PSIAs 51 C Methodology 57 D Description of PSIA Effects--Country Case Reviews 69 E Semistructured Stakeholder Interviews--Summary of Results 71 F Portfolio Review Tables 77 Endnotes 81 Bibliography iii A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Boxes xii ES.1 What Is a PSIA? 5 1.1 World Bank Guidance on PSIA 6 1.2 Institutional Requirement Contained in Operational Policy 8.60 Relating to Assessing Poverty and Social Consequences of Reforms 18 3.1 Criteria to Assess PSIA Effect on Country Policies 19 3.2 Factors in the Success of the Cambodia Land Reform PSIA 22 3.3 Republic of Yemen Water PSIA: Participation by Government, Other Local Stakeholders, the World Bank, and Other Donors 25 3.4 Criteria to Assess PSIA Contribution to Country Capacity 30 4.1 Criteria to Assess PSIA Effect on Bank Operations 31 4.2 Corporate Benefits Identified by PSIA Proponents Figure 12 2.1 Regional and Sectoral Breakdown of PSIAs iv Abbreviations ADMARC Agricultural Development and Marketing Corporation (Malawi) CAS Country Assistance Strategy CPA Chittagong Port Authority (Bangladesh) DFID Department for International Development (United Kingdom) DPL Development Policy Loan/Lending ESW Economic and sector work GTZ German Agency for Technical Cooperation IEG Independent Evaluation Group IMF International Monetary Fund LASED Land Allocation for Social and Economic Development (Cambodia) NGO Nongovernmental organization OP Operational Policy OPCS Operations Policy and Country Services PREM Poverty Reduction and Economic Management Network PRSP Poverty Reduction Strategy Paper PSIA Poverty and Social Impact Analysis SAM Social Accounting Matrix SDN Sustainable Development Network v Children with their mothers in Bhutan. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Acknowledgments This report was prepared by a team led by Soniya background work. Pedro Alba, Jeffery Hammer, Carvalho. The core team included Joan Nelson, and Howard White were peer reviewers. Victoria Manuel Penalver, and Morgan Rota. Catherine Elliott provided comments. William Hurlbut Gwin conducted interviews with senior Bank edited the report and provided document staff and managers, and Shonar Lala interviewed production support. Heather Dittbrenner edited in-country stakeholders and Bank task managers the report for publication. The team was assisted and task team members. Country case reviews by Romayne Pereira. Juicy Zareen Qureishi-Huq were undertaken by Jeffrey Alwang, Jaimie Bleck, provided administrative and production support. Anthony Killick, Shonar Lala, Anne Pitcher, Sylvia Financial support from the Norwegian Agency for Saborio, and Jan Kees van Donge. Jabesh Amissah- International Development (Norad) is gratefully Arthur, Happy Kayuni, and David Korboe contrib- acknowledged. uted to the country case reviews. Domenico Lombardi reviewed donor collaboration in The report was initiated under the direction of Poverty and Social Impact Analyses. Danielle Alain Barbu, former Manager, Sector Evaluations, Resnik and Morgan Rota undertook the portfolio and completed under the direction of Monika review, and Melvin Vaz contributed to it. Antine Huppi, Manager, Sector Evaluations, Indepen- Legrand and Shampa Sinha contributed to other dent Evaluation Group. Director-General, Evaluation: Vinod Thomas Director, Independent Evaluation Group ­ World Bank: Cheryl Gray Manager, Independent Evaluation Group, Sector Evaluations: Monika Huppi Task Manager: Soniya Carvalho vii Boys in Nigeria. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Foreword The current global financial and economic crises country policies, their contribution to country are likely to put enormous pressure on govern- analytic capacity, and their effect on Bank ments to respond with immediate measures operations. This study finds that PSIAs have had and to undertake far-reaching reforms in the a moderate effect on country policies and Bank medium term, requiring a substantial increase in operations and a negligible effect on country donor support. To protect the poor and enhance analytic capacity, on average, although there have benefits to them, key policy reforms will need to been some outstanding examples of success. be underpinned by systematic analysis of their expected poverty and social impacts. The World Overall, implementation of the PSIA approach Bank's experience to date with the Poverty and has had considerable limitations. There have Social Impact Analysis (PSIA) approach provides been tensions between the various operational useful lessons for addressing these issues. objectives assigned to PSIAs. The tensions concern inconsistencies between informing Introduced in fiscal 2002, the PSIA approach country and Bank policy decisions in a timely aimed to help the Bank and its client countries way and building country analytic capacity. understand the distributional impacts of policy PSIAs have had limited ownership by Bank reforms and design reform processes that took staff and managers and have often not been account of beneficiaries and those adversely effectively integrated into country assistance affected. The PSIA approach was also expected to programs. Quality assurance, monitoring, and provide an understanding of the institutional and evaluation of the overall effectiveness of PSIAs political constraints to development and help have been weak. build domestic ownership of policy reforms. To improve PSIAs' effectiveness, this evalua- The World Bank had already been conducting tion recommends that the Bank take measures distributional analyses before the introduction to ensure that staff fully understand what the of the PSIA approach. However, this approach PSIA approach is and when to use it, clarify the was distinctive in emphasizing the use of social operational objectives of each PSIA, and ensure and economic analytical tools and techniques that the approach and timeline adopted are to conduct ex ante analysis of the impacts of aligned with those objectives. Quality assurance economywide policy reforms and in support- mechanisms should be strengthened to ensure ing a more systematic use of that analysis to that PSIAs are designed to achieve the intended inform policy advice and design. By fiscal 2007 effects. The evaluation also recommends that the Bank had undertaken 156 pieces of analyti- the Bank shift significant decision-making and cal work using one or more elements of the PSIA funding authority for PSIAs to the Regional Vice approach in 75 countries across several sectors. Presidencies and ensure that PSIAs are grounded Total Bank and other donor support to PSIAs was in country assistance programs. $15 million from fiscal 2004 to fiscal 2006. Consistent with the roles envisaged by the Bank for PSIAs, this study distils three operational objectives for evaluation: the effect of PSIAs on ix Student at the public primary school in Tegucigalpa, Honduras. Photo by Alfredo Srur, courtesy of the World Bank Photo Library. Executive Summary T he World Bank introduced the Poverty and Social Impact Analysis (PSIA) approach in fiscal 2002 to help governments and the Bank anticipate and address the possible consequences of proposed policy reforms, especially on the poor and vulnerable, and to contribute to country capacity for policy analysis. By fiscal 2007 the Bank had undertaken 156 pieces of analytical work using one or more elements of the PSIA approach (hereafter called PSIAs) in 75 countries and 14 sectors. Total donor support to PSIAs over fiscal 2004­06 was $15 million, which came from the Bank's earmarked Incremental Fund for PSIAs ($5.8 million), earmarked PSIA Trust Funds contributed by various bilateral donors, and non-earmarked Bank budget and other donor funding. The Bank had been conducting distributional considerable limitations, and there are tensions analysis in some of its economic and sector work between the various operational objectives even before the introduction of PSIAs. As early as assigned to PSIAs. The tensions concern inconsis- 1987 the Bank's Operational Guidelines required tencies between informing country and Bank analysis of the short-term impact of adjustment policy decisions in a timely way and building programs on the urban and rural poor. The country analytic capacity. Furthermore, PSIAs PSIA approach was distinctive in emphasizing do not always explicitly state the operational the use of social and economic analytical tools objectives related to their intended effect, and techniques to conduct ex ante analysis of which reduces the chance that there will be a the impacts of economywide policy reforms and well-conceived strategy to achieve this effect. more systematic use of that analysis to inform PSIAs have had limited ownership by Bank staff policy advice and design. and managers and have often not been effectively integrated into country assistance programs. Development literature has emphasized the Quality assurance, monitoring, and evaluation importance of understanding the institutional of the overall effectiveness of PSIAs have been and political constraints to development, weak. building domestic ownership of policy reforms, and assessing the distributional impacts of policy To improve PSIA effectiveness, the evalua- actions. The PSIA approach correctly emphasizes tion recommends that the Bank take measures these aspects. The Bank has produced a substan- to ensure that staff fully understand what the tial body of guidance on how to address these PSIA approach is and when to use it, clarify the aspects, and that guidance has been refined over operational objectives of each PSIA, and ensure time to incorporate lessons learned. that the approach and timeline adopted are aligned with those objectives. Quality assurance This evaluation finds that although there have mechanisms should be strengthened to ensure been some highly effective individual PSIAs, that PSIAs are designed to achieve the intended overall implementation of the approach has had effects. The evaluation also recommends that xi A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R the Bank shift significant decision-making and also aimed to inform Country Assistance Strate- funding authority for PSIAs to the Regional Vice gies, investment loans, and analytical work. Presidencies and ensure that PSIAs are grounded in country assistance programs. Evaluation Background Although the Bank has submitted progress Introduction reports to donors regarding the implemen- The development community first became tation of PSIAs, it has not yet completed concerned about the impacts of economic a comprehensive self-evaluation of the policy reforms on the poor during the structural PSIA experience. This evaluation by the adjustment period of the 1980s. In the 1980s Independent Evaluation Group, requested and '90s, external groups continuously criticized by the Bank's Board of Executive Directors, the International Monetary Fund and the World represents the first independent evaluation of Bank for failing to properly assess the impacts of the PSIA experience. the policy reforms they supported. In response, the Bank in fiscal 2002 introduced PSIA. Accord- The PSIA experience is highly relevant today. ing to the revised Operational Policy (OP 8.60) The recent financial crisis and global slowdown on development policy lending issued in 2004, are likely to put pressure on governments and Development Policy Loans with likely and donors to undertake far-reaching reforms. To significant poverty and social consequences protect the poor and enhance benefits to them, were required to summarize in their program key reforms will need to be underpinned by documents relevant analytic knowledge of systematic analysis of their expected poverty and these consequences. But the Operational Policy social impacts. does not mandate that the PSIA approach be used to undertake an assessment of these The Bank has envisaged several roles for PSIAs, consequences. mainly in the elaboration and implementa- tion of poverty reduction strategies in develop- The Bank supported 156 PSIAs over fiscal ing countries, supporting in-country capacity 2002­07, partly driven by the availability of building, and informing Bank operations. These earmarked funds both from within the Bank roles can be distilled into three operational and from the United Kingdom's Department for objectives that form the basis for the questions International Development, the German Agency addressed in this evaluation: for Technical Cooperation, Norway, Italy, Belgium, and Finland. The PSIAs were done mainly in the 1. What effect have PSIAs had on country policies context of Development Policy Loans, but some (including policy debate)? Box ES.1: What Is a PSIA? The Bank defines PSIA as "analysis of the distributional impact before those reforms are carried out (ex ante analysis), and of policy reforms on the well-being of different stakeholder more systematic use of that analysis to inform policy advice groups, with particular focus on the poor and vulnerable." and policy design." The Bank sees the innovative aspects of PSIA as "the ap- The Bank has identified two key elements of PSIA: "First, an plication of the tools and techniques of social and economic analysis to determine the distributional impacts, and second, a pro- analysis to analyze impacts of economywide policy reforms cess that engages appropriate stakeholders in policy making." Sources: World Bank (2003h, 2006n, 2008b). xii EXECUTIVE SUMMARY 2. What contribution have PSIAs made to the with the approach required to build country development of country capacity for policy analytic capacity. analysis? 3. What effect have PSIAs had on Bank operations Content and approach (including strategy and analytical work)? PSIA practice has frequently departed from the initial guidance on how PSIAs should be This evaluation covers analytical work identified conducted. The portfolio review shows the as PSIAs by the Bank's Poverty Reduction and following: Economic Management (PREM) and Sustainable Development Network (SDN) Anchors. Analyses · Even though PSIAs originated out of concern not identified as PSIAs by these anchors are not about the impact of reform programs, about covered. one-third of the PSIAs in the portfolio review did not examine well-specified reforms but The two anchors identified 156 analyses as were more general sector or macroeconomic PSIAs over the fiscal 2002­07 period. This analyses. evaluation draws its findings from the following · About one-third of the PSIAs explicitly identi- resources: a portfolio review of a statistically fied beneficiaries or those adversely affected, representative sample of 58 out of the universe and about half did not, although they included of 156 PSIAs; in-depth country case reviews of disaggregated data or results. 12 PSIAs in 8 countries with Regional, sectoral, · About 60 percent of the sampled PSIAs iden- and fiscal year coverage; and interviews with tified the institutions responsible for imple- key informants. This last resource includes menting the reform. interviews with country stakeholders and Bank · More than half of the sampled PSIAs included staff associated with an additional 11 PSIAs in 10 some sort of stakeholder participation, but countries and interviews with senior Bank staff no consultations were mentioned for about and managers. two-fifths of the PSIAs. Main Findings PSIAs have had widely varying characteris- tics partly because Bank staff have an unclear Objectives understanding of them. At the same time, analyti- The portfolio review found that about one-fifth cal work possessing PSIA-like characteristics has of the PSIAs have not explicitly identified which not always been classified by the Bank as "PSIA." of the three operational objectives they intend These errors of omission and commission are to pursue. Among those that have identified likely to have inhibited quality assurance and operational objectives, informing country policies lesson learning. has been the most frequently stated, followed by informing Bank operations and increasing Effect on country policies country capacity for policy analysis. Most PSIAs Tracing links between PSIA analyses and country have had more than one operational objective. decisions is often difficult, especially where the About a third of the task manager survey respon- PSIA is only one of many possible influences. dents indicated that their PSIA pursued all three PSIA effects may also be diffuse and occur over objectives. time--a report may have no discernable impact on immediate action yet may affect ideas and Pursuit of the multiple operational objectives debate that shape future policy choices. This of PSIAs can create tension and raise unrealis- evaluation focuses only on near-term effects of tic expectations of what a PSIA can achieve. For PSIAs. example, PSIAs seeking to inform government policy decisions must adjust to the timing of the The PSIAs reviewed in this evaluation had a decision process, but often that is inconsistent moderate effect on country policies, on average, xiii A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R although there have been some outstand- Effect on Bank operations ing examples of success. Informing a policy The PSIAs reviewed suggest a moderate effect process is not easy and requires the conver- on Bank operations, on average, with some gence of a number of factors. These include the outstanding examples of success. Key factors that operational focus of the PSIA, a match between have inhibited PSIA effect on Bank operations PSIA topic and country priorities, govern- include the ambiguity of the PSIA concept; insuffi- ment ownership, engagement of parts of the cient country director buy-in, resulting in lack of government that have policy jurisdiction over grounding of the PSIA in the country assistance the areas covered by the PSIA, engagement program; and weak engagement between PSIA with appropriate nongovernmental stakehold- teams and other operational staff. ers, timeliness in relation to country decision- making processes, sensitivity to the politics of Interviews with senior Bank staff and managers reform, and active dissemination beyond the indicate that the uptake of PSIAs by country distribution of reports. One or a few of these directors and operational teams remains factors have been present in most PSIAs, but the dependent on individual inclinations more than presence of a majority of these factors--which it reflects established practice. There has been has occurred only in some PSIAs--is important only modest acceptance so far of the PSIA as a for substantial effect. robust practice across the Bank, although staff directly involved with PSIAs see a number of Contribution to country capacity corporate benefits from the experience, notably The PSIAs reviewed suggest a negligible contri- the creation of an important body of knowledge bution to country analytic capacity, on average, through PSIA guidance and an appreciation of with a few positive examples. The main reason the importance of process issues in addition for this finding is that most PSIAs have treated to analytics. However, operational staff lack a this objective as a by-product rather than a core common understanding about the objectives and concern, and there has been a tension between processes of the PSIA approach. PREM Anchor the need to provide timely inputs to policy staff generally tend to focus on economic analysis, decisions and the longer-term sustained engage- and SDN staff tend to emphasize mixed methods, ment needed to build capacity. Where capacity including social and institutional analysis and a building has been an explicit objective, it has not participatory process. always been backed by an appropriate strategy to achieve it. Many PSIAs have involved local The 2008 PSIA Good Practice Note (World Bank ministries, consulting firms, nongovernmental 2008b) is an improvement over the 2004 PSIA organizations, academics, or local consultants, Good Practice Note (World Bank 2004f) in that and others have held training workshops to it helps to lighten PSIA guidance, which was transfer skills. However, the time allocated to previously seen by staff as overly demanding. The either of these approaches has generally been 2008 Note provides Bank staff with the flexibil- insufficient to build lasting capacity for policy ity to determine, based on country context, the analysis. The few PSIAs reviewed that have balance among economic, social, institutional, successfully helped build capacity have taken a and political analyses (and between quantitative more deliberate approach. Given that in many-- and qualitative techniques) and between analyt- perhaps most--situations, the time frame ics and such process issues as stakeholder partic- and approach required for capacity building ipation and disclosure. The 2008 Note does not, are inconsistent with the approach needed to however, require staff to provide a rationale for inform timely government decision making, the particular choices made. Providing a rationale greater selectivity in operational objectives is especially important to ensure that Bank staff is desirable; more effective approaches to from the different networks do not continue building country analytic capacity will also be to place undue emphasis on their respective needed. disciplinary approaches in undertaking PSIAs. xiv EXECUTIVE SUMMARY The PSIA approach has correctly emphasized the tives, ensuring that the concept note-- importance of understanding the institutional · Contains a clear statement of the opera- and political constraints to development and tional objectives of the PSIA with respect the need to build domestic ownership of policy to the intended effect (not just the topics/ reforms in addition to assessing the distributional issues to be analyzed) impact of policy actions. But implementation of · Indicates how its approach--in particular, the approach has had considerable limitations. stakeholder engagement, team composi- Some notable successes have modeled what tion, partner institutions, budget, and time PSIAs can accomplish when done right. The frame--has been tailored to meet the op- recommendations below are intended to help erational objectives and provides the ratio- improve PSIA implementation and realize its nale for the choices made potential. · Shows how any tensions and trade-offs among the operational objectives will be Recommendations reconciled This evaluation makes four recommendations · Discusses if the intended dissemination au- to strengthen the Bank's work using the PSIA dience and strategy are consistent with the approach, whether done as freestanding analysis stated operational objectives. or embedded in other analytical work. · Improve integration of the PSIA into the Bank's country assistance program by-- · Ensure that staff understand what the · Shifting significant decision-making and PSIA approach is and when to use it. funding authority to the Regional Vice Bank management can do this by pro- Presidencies to ensure that the PSIA top- viding clear guidance (perhaps through ics, scope, and approach are consistent updating of the 2008 PSIA Good Prac- with the country assistance program and tice Note) and actively disseminating this that PSIAs ask questions that are relevant guidance, particularly on-- to policy · Whether and how the PSIA approach differs · Requiring that all earmarked funding for from other distributional analyses, includ- PSIAs be matched by a substantial contribu- ing whether the inclusion of the word "so- tion from the country unit budget. cial" in Poverty and Social Impact Analysis · Strengthen PSIA effectiveness through en- suggests the need to include a different hanced quality assurance, including-- type of analysis · Subjecting PSIAs to systematic review by · Whether or not PSIAs should be linked Regional management at concept and com- to specific reforms and identify beneficia- pletion stages to ensure relevance and fit of ries and those adversely affected by the the PSIA to the country assistance program reform as well as consistency of the proposed ap- · What criteria should be used to determine proach with operational objectives, in ad- when the PSIA approach is appropriate dition to ensuring technical quality for a particular operation in a country · Ensuring that the Bank establishes a moni- program. toring and self-evaluation system designed · Clarify the operational objectives of each to assess whether PSIAs are being under- PSIA with regard to its intended effect, taken where appropriate and are achieving and tailor the approach to those objec- their stated operational objectives. xv Two women with grain in Ghana. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Management Response Introduction some of the finding ambiguous. It also notes an Management regards the Independent Evalua- issue that is often difficult in evaluation--the tion Group (IEG) review of the World Bank's fact that approaches like PSIA evolve continually Poverty and Social Impact Analysis (PSIA) as a based on experience. That said, management comprehensive attempt to reflect lessons from is in broad agreement with the thrust of the a wide range of PSIA done by the Bank. The recommendations. findings of the review reflect work conducted through a mix of portfolio review and interviews A Flexible Approach. Some of the key findings and highlight important areas where the PSIA of the evaluation appear to be linked to its approach can be strengthened. treatment of PSIA as more of a standardized approach adhering to somewhat rigid guidelines, Support for Strengthening the PSIA Process. rather than one that incorporates a range of tools The emphasis of the report in the following areas and methods to be flexibly applied to specific is especially welcome: the need to specify clear situations.1 As reflected in the initial PSIA User's operational objectives; the importance of politi- Guide, the revised Good Practice Note, and the cal economy in doing poverty and distributional forthcoming revised PREM-SDV Web page, PSIA analysis; the need for strong linkages between comprises a range of tools that can be applied to the country unit and the PSIA team; the potential analyze distributional and social impacts in widely for more effective use of PSIA findings to inform varying situations, depending on the specific monitoring and evaluation systems; and the contexts of policy reform and country circum- importance of strengthening quality assurance. stances. The updated Good Practice Note of 2008 Although robust progress has been made on reinforces the flexible nature of the approach as many of these crucial elements, IEG's observa- necessary for PSIA to be effective in informing tions will assist future actions in these areas. policy dialogue and operations. In particular, the evaluation of PSIA against the three specific A Differing View on Some Elements. That criteria selected by IEG (see below) appears to said, management is of the view that some flow from IEG's treatment of PSIA as a standard- of the findings of the report are based on a ized approach. partial understanding of the PSIA approach, including its objectives and limitations. The Criteria for Evaluating PSIA Work. Some of comments below on the analysis in the review the findings of the review appear to be based reflect this differing view. Management broadly on criteria drawn from an overly broad charac- concurs with IEG's recommendations on the terization of the objectives of PSIA. Based on way forward to enhance the effectiveness of the Good Practice Note, the institution defines PSIA and notes a number of actions already the main objective of the PSIA approach to be taken or planned in the directions suggested to analyze the distributional impact of policy by the recommendations. reforms on the well-being or welfare of different groups, with particular focus on the poor and Management's Views on the Findings vulnerable. The review, on the other hand, uses Management concurs with much of the analysis what IEG considers as PSIA's three goals from but believes that the analytical framework makes its reading of PSIA documents as the criteria for xvii A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R evaluation--effect on country policies, support- Better Integration into Bank Programs. ing in-country capacity building, and informing Management concurs with IEG's recommenda- Bank operations--without making the distinc- tions to ensure better integration of PSIA into tion between the main objective and potential the Bank's program and enhance the quality benefits. Informing country policies and building assurance process of PSIA. Conclusions along capacity in countries should be considered as similar lines are emerging from internal reviews of desirable second-order impacts that may come PSIA by PREM and SDV anchor units. A new Multi- from a process of engagement over time, but Donor Trust Fund to support PSIA work, to be rarely from a single PSIA exercise. operational in fiscal 2010, incorporates a number of features consistent with these recommenda- Evolution of PSIA over Time. The review tions. The Trust Fund assigns responsibility to the takes a somewhat static view of PSIA, which in Regions, in terms of allocating funds, monitoring, management's view does not adequately reflect and quality review of the analytical work funded the significant progress made over the last seven by the Trust Fund. years. The learning and evolution over time has resulted in the refinement of the institutional Guidance. Management concurs with the understanding and guidance concerning best recommendation to provide clear guidance practice of PSIA, which has been reflected in to Bank staff about what the PSIA approach the Good Practice Note of 2008. The static view involves and when it can be used. Over the years, adopted by the review also gives little indication a number of knowledge products on PSIA have of whether some of the problems highlighted in been developed and updated, including compre- the evaluation were mitigated over time.2 This, hensive user guides, Good Practice Notes, in management's view, constitutes a missed methodological notes, and edited volumes. The opportunity to draw lessons for the future, by outreach of such materials will be enhanced identifying and analyzing areas where progress further through an improved Web site and has been made and where it has been harder to more frequent and accessible learning events, achieve. drawing on additional resources provided by the aforementioned Multi-Donor Trust Fund. Recommendations Management, however, notes an important caveat As noted above, management concurs with the in addressing this recommendation. Efforts to broad thrust of IEG's recommendations and expand awareness about the PSIA approach must will undertake actions to address the issues be careful to not create perceptions of minimum raised. Management's specific responses to IEG or universal standards for PSIA, and instead stress recommendations are given in the Management the flexibility of the approach to adapt to specific Action Record matrix. country and policy contexts. xviii MANAGEMENT RESPONSE Management Action Record IEG recommendation Management response Ensure that staff understand what the PSIA Agreed. Management agrees that it should disseminate more widely what is meant approach is and when to use it by providing by the PSIA approach as outlined in the 2008 Good Practice Note. clear guidance (perhaps through updating Management, however, notes an important caveat in addressing this recommendation. of the 2008 PSIA Good Practice Note) Efforts to expand awareness about the PSIA approach must be careful to not create disseminated to staff and disseminating this perceptions of minimum or universal standards for PSIA, and instead stress the guidance, particularly on-- flexibility of the approach to adapt to specific country and policy contexts. The revised Good Practice Note (GPN) discusses key elements of the PSIA approach: first, an · Whether and how the PSIA approach differs analysis to determine the distributional impacts and, second, a process that engages from other distributional analyses, including appropriate stakeholders in policy making.3 The GPN also makes it clear that an ideal whether the inclusion of the word "social" in PSIA will vary based on country and reform-specific conditions. In particular, the range Poverty and Social Impact Analysis suggests the and extent of stakeholder engagement in the PSIA process will vary according to the need to include a different type of analysis political context of the reform and the related opportunities for promoting the public · Whether or not PSIAs should be linked to debate on policy options. The GPN clarifies when it could be a priority for a country specific reforms and identify beneficiaries and team to carry out detailed PSIA: when there are potential negative poverty and social those adversely affected by the reform impacts on different groups, particularly poor and vulnerable groups; when there is · What criteria should be used to determine when potential to significantly improve the benefits of a reform for poor and vulnerable the PSIA approach is appropriate for a particular groups; the prominence and urgency of the report in the government's policy agenda; operation in a country program. and the level of debate surrounding the reform. With regard to the methods and tools adopted for PSIA, the GPN stresses the need for flexibility as well. While multidisciplinary or mixed methods of analysis enhance the understanding of the poverty and social impacts of a reform, the design of the methodology and the selection of tools will depend on the nature of the reform problem being addressed, the availability and quality of data, the time available for analysis, and the available in-country capacity. PREM and SDV are already incorporating the revised GPN into their PSIA-related learning programs (Fundamentals of Bank Operations, PSIA e-learning, PSIA course). Management will further enhance outreach using these materials through an improved Web site that more clearly frames the main objective and potential benefits, as well as the need for a flexible approach, and more frequent and accessible learning events, drawing on additional resources provided by the aforementioned Multi-Donor Trust Fund. Management will consider its agreed action complete with the full roll-out of these activities in fiscal 2010. Clarify the operational objectives of each PSIA Agreed. Management agrees that poverty, social, and distributional impact analysis with regard to its intended effect and tailor the should have a clear operational objective and a methodology and strategy for stakeholder approach to those objectives, ensuring that engagement and dissemination that is consistent with the operational objective. the concept note-- Almost all PSIA type analysis is done in the context of economic and sector work (ESW), where Bank guidelines require that the operational objective, scope, and · Contains a clear statement of the operational participatory approaches be clearly specified in the concept note. The revised GPN objectives of the PSIA with respect to the indicates that poverty, social, and distributional analysis should be an integral part of intended effect (not just the topics/issues to be the ESW cycle. All dissemination activities need to be fully consistent with the Bank's analyzed) Disclosure Policy. · Indicates how its approach--in particular PSIA can also be done in the context of technical assistance, where the objective stakeholder engagement, team composition, is to strengthen client institutions and capacity to influence reforms or as a factual partner institutions, budget, and time frame--has technical document analyzing a specific policy reform supported by a Development been tailored to meet the operational objectives, Policy Operation. and provides the rationale for the choices made · Shows how any tensions and trade-offs among the Once operational, the new PSIA trust fund will require that the Regions detail in their operational objectives will be reconciled concept notes how they will ensure that poverty, social, and distributional analysis · Discusses if the intended dissemination (supported by the Trust Fund) is treated as an ESW, technical assistance, or a project- audience and strategy are consistent with the related factual technical document. Management will consider its agreed action stated operational objectives. complete with the introduction of these requirements. (continued on next page) xix A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Management Action Record IEG recommendation Management response Improve integration of the PSIA into the Bank's Agreed. Poverty, social, and distributional impact analysis that is part of ESW, country assistance program by-- technical assistance, or a project-related factual technical document is currently managed and funded by the Regions. The revised GPN explicitly advises that poverty, · Shifting decision-making and funding authority social, and distributional impact analysis be anchored in the Country Assistance to the Regional Vice Presidencies to ensure Strategy (CAS). Under the revised policy for poverty reduction (OP 1.00), the CAS that the PSIA topics, scope, and approach are summarizes existing knowledge on poverty, identifies analytical gaps, and presents the consistent with the country assistance program work program by the Bank and others to fill these gaps. The CAS can usefully lay out and that PSIAs ask policy-relevant questions key reform areas that the Bank will support and indicate whether there are any plans · Requiring that all earmarked funding for PSIAs for poverty, social, and distributional impact analysis. be matched by a substantial contribution from the country unit budget. The new PSIA trust fund will decentralize the management of resources for poverty, social, and distributional impact analysis carried out as ESW, technical assistance, and factual technical documents to the Regions. The Regions will be required to provide matching resources from the country unit budget. The annual Trust Fund Regional monitoring reports will indicate the share of matching resources provided to complement Trust Fund support. Management will consider its agreed actions complete with the roll-out of the trust fund. Strengthen PSIA effectiveness through Agreed. PSIA work that is part of ESW is subject to Regional quality assurance enhanced quality assurance, including-- procedures and the preparation of an activity completion report. As noted in the Management Response to IEG's evaluation of ESW and technical assistance, · Subjecting PSIAs to systematic review by management will undertake a review of analytic and advisory activities in fiscal 2010 Regional management at the concept and that will address institutional arrangements, notably quality assurance. completion stages to ensure relevance and fit of The quality and effectiveness of PSIA informing the Bank's Development Policy the PSIA to the country assistance program as Operations has been monitored through the retrospective assessment of the Bank's well as consistency of the proposed approach Development Policy Lending portfolio that has been carried out every two years by with operational objectives, in addition to Operations Policy and Country Services. A Development Policy Operation retrospective ensuring technical quality is under preparation and will be available early in fiscal 2010. · Ensuring that the Bank establishes a monitoring The proposed PSIA trust fund annual Regional monitoring reports as well as the final and self-evaluation system designed to assess independent Trust Fund evaluation will assess the extent to which poverty, social, and whether PSIAs are being undertaken where distributional analysis (supported by the Trust Fund) has met its operational objectives appropriate and are achieving their stated and has had an impact on the ground. operational objectives. Management will consider its agreed actions complete with the completion of the fiscal 2010 analytic and advisory activities review and the implementation of changes coming out of that process, the issuance of the Development Policy Operation retrospective, and the implementation of the trust fund reporting system. xx Chairperson's Summary: Committee on Development Effectiveness (CODE) On June 3, 2009, the Committee on Develop- The following key issues were raised at the ment Effectiveness (CODE) met to consider meeting: the document How Effective Have Poverty and Social Impact Analyses Been? An IEG Study of PSIA Operational Objectives. Many members World Bank Support to PSIAs, prepared by the stressed the importance of PSIA in emphasiz- Independent Evaluation Group (IEG), together ing social and economic analysis ex ante on with the draft management response. the distributional impact of policy reform, with particular focus on the poor. They agreed that Overall Conclusions. The Committee the operational objectives of such analysis should welcomed the timely discussion of IEG's study be clarified and adapted to each specific country of Poverty and Social Impact Analysis (PSIA), policy context. Some members underlined that particularly in light of the ongoing discussion to PSIA should not be an item within a checklist establish a Multi-Donor Trust Fund to support to be complied with once an intervention has PSIA work, to be operational in fiscal 2010. already been decided, and suggested that poverty Members recommended enhancing the use of distribution analysis ideally should be conducted PSIA but clarifying the operational objectives of upstream so as to be an input to policy design. each PSIA exercise. They proposed establishing There were comments on the relevance of the a hierarchy of objectives and following up on objectives of informing the country policies PSIA findings with appropriate dissemination and building capacity in client countries to and disclosure, including in several languages. conduct their own PSIA. However, there were They stressed the importance of linkage with also views that perhaps both should be second- country assistance programs and flexibility in order objectives of PSIA. One member felt that PSIA, albeit with clear accountability, rather building capacity should not be an objective at than having a standardized approach adhering all because it requires separate activities. One to rigid guidelines. There were comments speaker queried whether local institutions could on the importance of capacity building but be more involved in the preparation of PSIA. perhaps as a second-order objective of PSIA. Members felt that important lessons could be Questions were raised on integration and drawn from the past experience in conducting compliance with Operational Policy 8.60, which PSIA. In this regard, they felt the decentral- requires that the Bank determine whether ized management of the new PSIA Trust Fund specific country policies supported by Develop- to the Regions was appropriate. The desirabil- ment Policy Loans (DPLs) are likely to have ity for the Bank to share the PSIA report with significant poverty and social impact. Support- other donors or development partners was ing assessment can be done through PSIA or mentioned. embedded in other analytic work. In addition, xxi A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R there were questions on the rationale and member felt that in addition to an analysis criteria for country selection, and who initiates ex ante on the distributional impact of policy the request or proposal for conducting a PSIA. reform, ex post evaluations would be valuable One member asked to what extent gender contributions to knowledge. Some questions issues have been adequately mainstreamed were raised on PSIA's relationship with in the new Good Practice Note. Management economic and sector work, including Poverty agreed with IEG on the need to ensure that Assessments. Management stated that PSIA is staff understand what the PSIA approach is a set of tools that are frequently conducted in and stated that it would disseminate widely the context of DPLs and often used in Poverty the revised Good Practice Note, and enhance Assessments, Public Expenditure Review, and the quality assurance process. other core economic and sector work or in the preparation of investment projects. It Country Program. Speakers underscored also clarified that regional PSIA teams draw the need for PSIA to become an integral on staff from all networks and Development part of country assistance programs and Economics and the Chief Economist and are Bank operations, and to consider the politi- further supported by the Poverty Reduction cal economy dimension in doing poverty and and Economic Management and Sustainable distribution analysis. The need for stronger Development Anchors. It also noted that it is linkages between country units and PSIA teams working to enhance the outreach of knowledge was emphasized. Some speakers noted the in PSIA and to improve Web-based learning. need for flexibility in applying a range of tools IEG pointed out that there is still insufficient and methods to specific situations rather than ownership of PSIA findings by operational treating PSIA as a standardized approach. It was staff and managers within the Bank. also suggested that the topic and timing of the PSIA should be aligned with the country's priori- Trust Fund. Some members felt that the ties and the objectives of the Country Assistance main purpose of the Trust Fund should not be Strategy (CAS). Relatedly, there were comments to save resources from regular budget. They on the need for engaging country stakehold- also proposed the new Trust Fund should be ers to build ownership, especially among policy designed to finance activities in which PSIA was makers. There were also questions on the extent not regularly used. A question was raised on to which PSIAs are used in the design of DPLs. possible lack of sustainability of PSIA funded In addition, some members sought further through earmarked Multi-Donor Trust Funds. information on countries' demand for PSIA. There was also a request for the Bank to share Management agreed with IEG's recommen- the findings of PSIA with other donors. Relatedly, dation on the need to ensure better integra- a comment was raised on the need to understand tion of PSIA into the Bank's country program. whether important donor-funded activities are Management clarified that a number of issues, not bought by country directors, who therefore including compliance with Operational Policy do not integrate them in CASs. The recommenda- 8.60 and consultations with stakeholders, will tion on shifting the decision-making and funding be addressed in the DPL Retrospective report mandate of the new Trust Fund to Regional vice currently under preparation. presidential units was welcomed. Manage- ment clarified that the bulk of resources used Knowledge. Some members felt that in the to support PSIA comes from Bank resources, context of the knowledge agenda there was and that the financial contributions from the extensive experience that should be used to new Trust Fund will be very small. However, it better implement the PSIA approach in the indicated that it hopes to leverage the new Trust future. In this regard, they encouraged more Fund so as to make the Regional vice presiden- dissemination, including translation into local tial units more responsible and accountable languages and follow-up of PSIA findings. One for carrying out this analysis. xxii C H A I R P E R S O N ' S S U M M A R Y: C O M M I T T E E O N D E V E L O P M E N T E F F E C T I V E N E S S ( C O D E ) Monitoring and Evaluation (M&E). Some underscored the potential for more effective use members took note that about 75 percent of of PSIA findings to inform the M&E systems. the sample PSIA did not include M&E indica- tors or data collection methods necessary for monitoring the impact of reform policy. They Giovanni Majnoni, Chairperson xxiii Chapter 1 Evaluation Highlights · The World Bank has defined Poverty and Social Impact Analysis (PSIA) as "the analysis of the distributional impact of policy reforms on the well-being of different stakeholder groups, with particular focus on the poor and vulnerable." · PSIAs were a donor response to the concerns of external groups--not client countries--about the impact of Bank-supported programs and policies. · A Bank Operational Policy requires Development Policy Loans with likely significant poverty and social impacts to address those impacts, but there is no specific requirement to do so using the PSIA approach. · The Bank has supported 156 PSIAs, partly driven by the availability of earmarked funds both from within the Bank and from other donors. · Consistent with the roles envisaged by the World Bank for PSIAs, this evaluation assesses their effect on country policies and Bank op- erations and their contribution to country analytic capacity. 1 Hanoi, Vietnam, bicycle and scooter riders. Photo by Simone D. McCourtie, courtesy of the World Bank Photo Library. Introduction C ountries--both their governments and the public--need to under- stand the probable distributional impact of policy reforms and pro- grams. Equipped with that information, they need to design reform processes that take account of beneficiaries and those adversely affected, if they are going to be effective in achieving objectives of growth and poverty reduction. Experience has shown that understanding and defining effective reform processes often require careful economic and social analyses of the potential impacts of the reforms within specific country contexts. In fiscal 2002, the World Bank introduced a distinc- This process was designed to ensure that reforms tive analytical approach--the Poverty and Social were aligned with the domestic priorities of client Impact Analysis (PSIA)--to help countries achieve countries (Wolfenson 1997). As stated in a World this kind of evidence-based policy making and to Bank publication, "Following the adoption of the ensure that its own operations take distributional [poverty reduction strategy] approach, the World impacts into account. This evaluation assesses Bank and International Monetary Fund commit- the effect of Bank-supported analytical work that ted, in 2001, to a systematic assessment of the uses one or more elements of the PSIA approach poverty and social impacts of policy reforms" (hereafter called PSIAs) both on the countries and (World Bank 2006n). on the Bank itself from fiscal 2002 to 2007. The Bank had been conducting distributional The impetus to assess the poverty and social analysis in some of its economic and sector work impacts of World Bank operations can be traced (ESW) even before the introduction of PSIAs. back to the mid-1990s, when external groups In fact, as early as 1987 the Bank's Operational criticized the International Monetary Fund (IMF) Guidelines required an analysis of the short-term and the World Bank for not properly assessing impact of adjustment programs on the urban the impacts of reforms before supporting them. and rural poor (World Bank 1987). Then, over In the late 1990s, several steps were taken to the 1987­92 period, the World Bank, the African improve the poverty reduction focus of Bank Development Bank, and the United Nations operations; among these steps was the develop- Development Programme jointly supported ment of the poverty reduction strategy process. the Social Dimensions of Adjustment in Africa 3 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Three main roles Program, which aimed to help partici- The Bank has envisaged three main roles for were envisaged for pating African countries integrate PSIAs: supporting the elaboration and implemen- PSIAs--the elaboration poverty and social concerns in the tation of poverty reduction strategies in develop- and implementation design and implementation of their ing countries, supporting in-country capacity of poverty reduction adjustment programs. And in 1996, building, and informing Bank operations (World strategies in developing in conjunction with national govern- Bank 2003h, 2004f, 2008e). These roles have been countries, supporting ments and civil society organiza- distilled into three operational objectives in this in-country capacity tions, the World Bank launched evaluation: informing country policies, contrib- building, and informing the Structural Adjustment Partici- uting to country analytic capacity, and informing Bank operations. patory Review Initiative to assess Bank operations. the economic and social impact of structural adjustment policies on various social Literature on development over the past groups in borrowing countries.1 In 2000, a group decade has emphasized the importance of of nongovernmental organizations (NGOs), led understanding the institutional and political by Oxfam, reminded the Bank of its commitment constraints to development and the need to to assess proposed reforms in Bank programs build domestic ownership of policy reforms in (Oxfam International 2000). The Bank responded addition to assessing the distributional impact by introducing the PSIA approach in fiscal 2002 as of policy actions. The PSIA approach correctly a formal way to conduct distributional analysis. emphasizes these aspects. As discussed in The approach was based on collaborative work chapters 2­4, however, implementation of the with the IMF, the United Kingdom's Depart- approach has faced considerable limitations. ment for International Development (DFID), These include, for example, limited success the German Agency for Technical Cooperation in fostering a common understanding among (GTZ), and other donors. Appendix A shows the Bank staff about what a PSIA is and if and how timeline of PSIA history. it differs from other distributional analyses and tensions between being able to inform country What Is a PSIA? and Bank policy decisions in a timely manner The Bank defines PSIA as "the analysis of the and building a country's analytic capacity. distributional impact of policy reforms on the Although informing a policy process is not easy well-being of different stakeholder groups, with and requires several factors to come together particular focus on the poor and vulnerable" (chapter 3), limitations in implementing the (World Bank 2003g). The Bank laid out 10 key PSIA approach have reduced the effectiveness elements of a PSIA in 2003 (see box 1.1) (World of the approach. Bank 2003h). The Bank sees the distinctive aspects of PSIA as follows: "What is new is the Recent Status application of the tools and techniques Through fiscal 2007 the Bank funded 156 PSIAs The PSIA approach of social and economic analysis to (not including the initial pilot PSIAs), as iden- emphasizes the analyze impacts of economy-wide tified by the Bank's Poverty Reduction and Eco- importance of policy reforms before those reforms nomic Management (PREM) and Sustainable De- understanding the are carried out (ex ante analysis), and velopment Network (SDN) Anchors. Appendix B institutional and more systematic use of that analysis provides a list of the 156 PSIAs.2 Total donor sup- political constraints to inform policy advice and policy port to PSIAs over fiscal 2004­06 was $15 million, to development and design" (World Bank 2006n). Further- and this funding was partly a driver of PSIAs. It came the need to build more, the Bank has identified two key from the Bank's earmarked Incremental Fund for domestic ownership elements of the PSIA: "first, an analysis PSIAs ($5.8 million), earmarked PSIA Trust Funds of policy reforms in to determine the distributional contributed by various bilateral donors (including addition to assessing the impacts, and second, a process that DFID, GTZ, Norway, Italy, Belgium, and Finland), distributional impact engages appropriate stakeholders in and non-earmarked Bank budget and other donor of policy actions. policy-making" (World Bank 2008e). funding. After the exhaustion of the Incremental 4 INTRODUCTION Box 1.1: World Bank Guidance on PSIA The Bank's 2003 PSIA User's Guide (World Bank 2003h) does not accounting matrix, or computable general equilibrium) will specify minimum standards for PSIA; instead it provides sugges- depend on the expected nature of the impact and the avail- tions on how to approach the analysis based on 10 key elements. ability of data. It points out that although there is a logical sequence to these ele- 7. Contemplating enhancement and compensation measures: ments, that does not imply that they need to be undertaken in strict To the extent that there are those who may not benefit from the order or that all the steps will be feasible in every country. reform, PSIA can inform policy design, leading to choices that 1. Asking the right questions: Asking questions such as the minimize the number of people who are affected negatively or prominence of the issue on the government's agenda, the timing the extent of adverse impacts. of the reform, and the nature of the national debate on the issue 8. Assessing risks: Risk analysis addresses the issue of what will help orient the analysis correctly. could go wrong to prevent policy reform from delivering the in- 2. Identifying stakeholders: An early identification of relevant tended poverty or social impacts. By addressing these questions stakeholders is important. Not only can policy choices affect explicitly, adjustments can be made to the reform to mitigate different stakeholders or economic agents in different ways, the risks. but these stakeholders can also influence whether a policy is 9. Monitoring and evaluating impacts: Good PSIA calls for adopted and how it is implemented. monitoring and evaluation, both to validate ex ante analyses 3. Understanding transmission channels: Once stakeholders and to influence the reformulation of policy. have been identified, an important step in the PSIA process is to 10. Fostering policy debate and feeding back into policy choice: delineate the channels by which the analyst expects a particular Fostering and drawing on public discussion of policy can be policy change to impact various stakeholder groups. useful at various points of the PSIA process--for example, to 4. Assessing institutions: An analysis of the market structure help identify stakeholders and their positions, to understand and implementing agencies can elucidate what impact policies transmission channels, to validate technical impact analysis, will have on the welfare of different households or groups. or to leverage social accountability. It is critical for PSIA to en- sure that the lessons learned from impact analysis, monitoring 5. Gathering data and information: Assessing data needs and and evaluation, social accountability, and public policy debate available data and planning the phasing of future data collection actually inform and affect policy. efforts are an important part of PSIA. 6. Analyzing impacts: The choice of tools (such as participa- The Bank reinforced these 10 key elements in its 2004 Good tory poverty assessment, benefit incidence analysis, social Practice Note on PSIA. Sources: World Bank (2003h, 2004f). Fund for PSIAs in fiscal 2006, PSIAs were funded have significant poverty and social The Bank and other from the remaining PSIA Trust Fund resources consequences (box 1.2). However, the donors funded 156 PSIAs and Bank budget. The amount of the Bank bud- extent and nature of this analysis is not over fiscal 2002­07. get spent on PSIAs after exhaustion of the Incre- specified in the OP itself, although an mental Fund is not available. endnote in the policy refers staff to the accompa- nying Good Practice Note on Poverty and Social With the publication of the Bank's revised Impact Analysis and Development Policy Lending Operational Policy (OP) on Development Policy for guidance on conducting PSIA (World Bank Lending (OP 8.60) in 2004, the analysis of poverty 2004f). Whereas OP 8.60 applies only to DPLs, and social impacts of development policy lending the Bank has undertaken PSIAs in a range of was made a required part of Bank Develop- other contexts, including informing investment ment Policy Loans (DPLs) that were likely to projects, Country Assistance Strategies, and 5 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Donor support for PSIAs other analytical work. Most of the sourcebook (World Bank 2003h, 2007g), and, has amounted to $15 PSIAs have, however, been intended in consultation with the Operational Policy and million over fiscal to inform DPLs. Country Services (OPCS) Vice Presidential Unit, 2004­06 and has included the PSIA Good Practice Notes (World Bank 2004f, a mix of earmarked Because no Bank OP governs PSIAs, 2008b). Bank funds, earmarked the main sources of guidance for donor funds, and non- conducting them are the Bank's PSIA Evaluation Objectives and Rationale earmarked Bank funds. User's Guide (World Bank 2003h); This evaluation responds to a request from the some other guidance such as the World Bank's Board of Executive Directors for toolkit for evaluating the poverty and distribu- an independent assessment of PSIAs, against the tional impact of economic policies (World Bank backdrop of the concerns expressed by Oxfam and 2003g); a lessons and examples book (World other NGOs3 about the effectiveness of PSIA to date Bank 2006n); practitioners' guides (World Bank and the interest of IDA-15 deputies in strengthen- 2005a, 2006b); a sourcebook of tools for institu- ing poverty and social impact assessments.4 tional, political, and social analysis of policy reform (World Bank 2007g); the Good Practice Though the Bank has prepared several progress Notes on PSIA (World Bank 2004f, 2008b); reports on PSIAs, it has yet to complete a compre- and The Impact of Macro-Economic Policies hensive self-evaluation of the PSIA experience. on Poverty and Income Distribution: Macro- Progress reports have been undertaken mainly Micro Evaluation Techniques and Tools (World in the context of reporting back to donors on Bank 2008c). These documents indicate what implementation. There are currently two ongoing the Bank expected PSIAs to look like around reviews of poverty and social impact analysis in the time of their initiation and the the Bank's DPLs--a stocktaking by PREM and OP 8.60 requires that subsequent evolution in thinking a DPL retrospective by OPCS. An ongoing joint the poverty and social about PSIAs. Overseas Development Institute and World Bank impacts of DPLs that report on PSIAs (ODI and World Bank 2009) aims are likely to have Responsibility for the promotion of to improve the practice and enhance the use of significant poverty and PSIA within the World Bank lies princi- poverty and social impact analysis in the Bank social consequences be pally with the PREM and SDN Anchors. and in partner countries.5 assessed, but it does They have administered the dedicated not require that this be Incremental Fund for PSIAs and Interest in the PSIA experience is likely to done through PSIAs. produced the PSIA User's Guide and increase worldwide. The recent financial crisis Box 1.2: Institutional Requirement Contained in Operational Policy 8.60 Relating to Assessing Poverty and Social Consequences of Reforms The Bank determines whether specific country policies sup- gaps in the analysis or shortcomings in the borrower's systems, ported by a DPL are likely to have significant poverty and so- the Bank's program document describes how such gaps or cial consequences, especially for poor people and vulnerable shortcomings would be addressed before or during program groups. For country policies with likely significant effects, the implementation, as appropriate. OP 8.60 refers staff to a 2004 Bank's program document summarizes relevant analytic knowl- Good Practice Note on Poverty and Social Impact Analysis edge of these effects and of the borrower's systems for reducing (World Bank 2004f) for applying this guidance, including possible adverse effects and enhancing positive effects associated with criteria for the selection of policies for analysis. the specific policies being supported. If there are significant Sources: World Bank (2004f, 2004j). 6 INTRODUCTION and global slowdown are likely to put pressure come from other sources. It is then The main sources of on governments to undertake far-reaching difficult to assess the PSIA's contribu- guidance on conducting reforms in the medium term. To protect the tion. Similar problems of attribution PSIAs are the User's Guide; poor and enhance benefits to them, key reforms apply to decisions or actions within the sourcebook on tools will need to be underpinned by systematic the Bank. The findings of this evalua- for institutional, political, analysis of their expected poverty and social tion should be interpreted bearing and social analysis of impacts. these caveats in mind. policy reform; and the Good Practice Notes. Evaluation Scope, Questions, and Evaluation questions Building Blocks Consistent with the three roles the Bank envisaged for PSIAs, this evaluation distils three Scope operational objectives that form the basis for the This evaluation covers analytical work identified as questions addressed: PSIAs by the Bank's PREM and the SDN Anchors. The evaluation does not cover Bank analyses of Effect at the country level: The questions this poverty and social impacts that the anchors did · What effect have PSIAs had on evaluation aims to not identify as PSIAs. The anchors identified 156 country policies (including policy answer derive from the analyses that incorporated one or more elements debate)? three roles the Bank has of the PSIA approach over the fiscal 2002­07 · What contribution have PSIAs made envisioned for PSIAs, period. Most of these are freestanding analyses, to the development of country ca- also referred to as but some are embedded in the Bank's other ESW . pacity for policy analysis?6 operational objectives Accordingly, "PSIA" is used in this evaluation to in this evaluation. cover analytical work that incorporates one or Effect within the Bank: more elements of the PSIA approach, regardless · What effect have PSIAs had on Bank opera- of whether this analytical work is freestanding tions (including strategy and analytical work)? or embedded. During the course of this evalua- tion, some Bank staff pointed to the existence Building blocks of PSIA-type work that was not on the PREM/ Evidence from various sources has been used SDN list but that should be, and others pointed collectively to formulate the findings: to PSIAs currently on the list that they believed should not be there. · A portfolio review of a statistically representa- tive sample of 58 PSIAs chosen randomly from This evaluation does not aim to assess the the universe of 156 PSIAs. downstream poverty reduction impacts of PSIAs. · Case reviews for 12 PSIAs in 8 countries The attribution problems involved in doing so (Bangladesh, Cambodia, Ghana, Malawi, Mali, would be prohibitive. Nor does the evaluation Mozambique, Nicaragua, and Zambia) chosen aim to assess compliance with OP 8.60, which is purposively for diverse regional, sectoral, and being done by PREM and by OPCS. fiscal year coverage. · Semistructured telephone interviews with 47 Moreover, this evaluation focuses on near-term stakeholders (including government officials, effects of PSIAs. Longer-term effects can be private sector representatives, NGO staff, aca- difficult to trace and are unlikely to be measur- demics and researchers, officials from donor able yet for recent PSIAs. However, the evalua- agencies, and Bank staff) chosen purposively tion recognizes that a seemingly ineffective PSIA to include stakeholders familiar can be powerfully effective in the future. with the PSIA process. These in- To date, the Bank terviews covered 11 PSIAs in 10 has not completed a Other measurement problems also exist. A govern- countries (and were additional to comprehensive self- ment decision may accord with recommendations the ones covered in the country evaluation of the in a PSIA, but similar recommendations may have case reviews). PSIA experience. 7 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R · Interviews with 30 senior Bank staff and man- · A literature review, including relevant material agers (including country directors, advisors, from several recent evaluations by the Inde- and so forth) chosen purposively to include pendent Evaluation Group (IEG). staff who were knowledgeable about PSIA strat- egy or who were responsible for overseeing Appendix C provides a detailed description of PSIAs. each of these sources. Chapter 2 provides an · A thematic review of donor involvement in overview of PSIAs. PSIAs. 8 Chapter 2 Evaluation Highlights · Bank-supported PSIAs over the fis- cal 2002­07 period have had widely varying characteristics. · In many cases, practice has de- parted from the initial concept of how PSIA should be conducted. · A desired characteristic that PSIAs have most frequently adopted has been analysis of risks. · About one-fifth of the sampled PSIAs did not explicitly state any of the three operational objectives relating to the PSIAs' intended ef- fect, which reduced the chance that there would be a well-conceived strategy to achieve or monitor in- tended effects. · Practice has not been institutional- ized in a way that ensures monitor- ing and evaluation of PSIAs, which detracts from lesson learning. 9 Woman in China carrying a crop in her basket. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Overview of PSIAs T his chapter focuses on a statistically significant sample of 58 PSIAs from a universe of 156 Bank-identified PSIAs completed between fis- cal 2002 and 2007. It presents an overview of this sample in terms of their number, costs, and operational objectives. It also examines how PSIA practice compares with the Bank's initial concept of how such analysis should be conducted. Numbers and Costs they are analyzing, they do not always PSIAs do not always The 156 PSIAs covered 75 countries and 14 state their operational objectives, that state their operational sectors. As shown in figure 2.1, they were is, objectives pertaining to the effect objectives. concentrated in the Sub-Saharan Africa Region that the PSIA intends to have. Explic- and in the energy and mining, economic policy, itly stating operational objectives is important and rural sectors. because it enables the PSIA to be specifically designed to achieve those objectives. Over fiscal 2002­07, the average cost of PSIAs was $148,230--somewhat lower than the average of Among the 58 sampled PSIAs, of those that do $178,000 for all Bank ESW over fiscal 2002­06 explicitly state their operational objectives, the (IEG 2008; World Bank 2009b). The wide range in most frequently cited operational objective (in PSIAs' costs--from $4,000 to $921, 000--attests more than two-thirds of the cases) has been to to the variety of topics and types of products that inform country policies and/or debate, followed are classed as PSIA (as discussed below). by to inform Bank operations and then to contribute to country capacity (appendix table About half of the PSIAs over the period fiscal F.1). About one-fifth of the PSIAs did not explic- 2002­07 took less than a year to complete, and itly state any of the three operational objectives. about a fifth took more than two years. By compar- Most PSIAs identified more than one objective. Of ison, 54 percent of Bank ESW over fiscal 2002­06 the 37 responses received from PSIA took less than a year, and 12 percent took more task managers, about one-third had About one-third of the than two years (IEG 2008; World Bank 2009b). all three objectives. Appendix table PSIAs did not examine F.1 shows that there are some differ- a well-specified reform Objectives ences between what task managers but were more general Although PSIAs typically cite their objectives consider PSIA objectives and what sector or macro- relating to the content of the reform or sector the PSIA documentation states as the economic analyses. 11 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Figure 2.1: Regional and Sectoral Breakdown of PSIAs A. Regional B. Sectorala MNA Social protection 4% SA Otherb 9% 7% 9% Public sector governance EAP 12% 12% AFR 42% LCR Rural 16% 14% Energy & ECA mining Economic policy 19% 9% 19% Source: Social Analysis Team, Social Development Department, World Bank, 2007. Note: N =1 56. AFR = Sub-Saharan Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SA = South Asia. a. Sector is classified by network/sector board. b. Other includes water; health, nutrition, and population; education; multisectoral; and so forth. PSIA's objectives; these differences indicate a lack most country contexts. The discussion in this of clarity of purpose. chapter is based on a portfolio review of the statistically representative sample of 58 PSIAs, PSIA Practice versus Initial Concept although a few references are also made to the PSIAs can incorporate any or all of the following country case reviews. characteristics noted in the Bank's PSIA User's Guide (World Bank 2003h) and the PSIA Good About two-thirds of the sampled PSIAs Practice Note (World Bank 2004f): assessed the impact of a well-specified reform.1 Although PSIAs originated with concern · Assessing the impact of a well-specified about the impact of reform programs, about reform one-third of the sampled PSIAs did not examine · Using a combination of quantitative and quali- well-specified reforms; they were more general tative methods sector or macroeconomic analyses.2 As the PSIA · Identifying stakeholders--beneficiaries or User's Guide notes (World Bank 2003h), robust those adversely affected by the reform as well distributional analysis will generally be easier for as proponents and opponents well-specified reforms. Preparing PSIAs further · Identifying the implementing institutions upstream in the process of developing reforms · Analyzing risks to reform implementation may allow for the consideration of a broader range · Setting up systems for monitoring and of policy options, but unless the PSIA is linked to evaluation well-specified reforms, it may be impossible to · Ensuring stakeholder participation undertake meaningful distributional analysis.3 · Undertaking dissemination. An example of a PSIA undertaken to analyze Although not every PSIA is expected to reflect all a well-specified reform is the 2007 Montene- dimensions of good practice, many of the charac- gro electricity price PSIA (World Bank 2007e). teristics discussed in this chapter are desirable in The PSIA took place while the government was 12 OVERVIEW OF PSIAs preparing an electricity tariff reform that would and opponents and proponents of a reform. It significantly increase electricity tariffs for residen- spelled out who stood to benefit as well as who tial consumers. Through an ex ante investigation would likely lose from the proposed reform and of the welfare impact of this price increase on detailed which stakeholders would likely oppose households in Montenegro, the PSIA showed the reform. It went on to suggest ways to lessen that the anticipated price increase would result the negative impact on those who might be in a significant increase in households' energy adversely affected and to gain the acceptance of expenditures; it included a simulation of alterna- potential opponents. tive policy measures. About half of the sampled PSIAs analyzed data In comparison, contrary to the original intent specific to women or discussed effects on of PSIAs, a 2002 paper on the impact of the them. Among the PSIAs covered by the country Indonesian financial crisis (World Bank 2002) was case reviews, for example, the Cambodia social categorized as a PSIA but did not examine any concessions of land reform PSIA (World Bank reform or offer any policy recommendations. It 2004a), the Mozambique education PSIA (World was more of a general look at the poverty effects Bank 2005h), and the Zambia land, fertilizer, and of the financial crisis. infrastructure PSIA (World Bank 2005l) discussed gender-specific issues. As with other About half of the sampled PSIAs used a stakeholder groups, the importance About half of the sampled combination of quantitative and qualita- of specifically analyzing impacts on PSIAs used a combination tive techniques.4 Slightly fewer than half of the women in the PSIA will depend on the of quantitative and PSIAs relied on purely quantitative economic type of policy reform in question. qualitative techniques. analyses (appendix table F.2) such as descrip- tive statistics or regression analysis of household About 60 percent of the sampled PSIAs survey data, benefit incidence analysis, social identified the institutions responsible for accounting matrices, growth-incidence curves, implementing the reform (appendix table microsimulation, or computable general equilib- F.4).5 Institutions are the primary agents through rium models. Qualitative methods included which reforms are implemented. In many cases, stakeholder analysis, institutional analysis, social identifying which institutions will be responsible impact assessment, focus groups, and targeted for implementation and assessing their capacity interviews. In Guyana, for example, the sugar is crucial for ensuring the effectiveness of the sector PSIA (World Bank 2006f) used semistruc- reform and understanding what the distributional tured stakeholder interviews and pair-ranking impacts might be. The Sierra Leone mining sector methods to explain and expand on the quantita- reform PSIA (World Bank 2007f) contained such tive findings from the household survey. analysis, including details about which govern- ment institution was responsible for oversee- Stakeholder identification in PSIAs has been ing the implementation of specific policies. It variable. About one-third of the PSIAs explic- described the capacity constraints of itly identified beneficiaries or those adversely key government institutions, coordi- About 60 percent of affected, and about half--although not explic- nation problems between relevant the PSIAs identified the itly identifying beneficiaries or those adversely ministries, and the role of local leaders, institutions responsible affected--did include disaggregated data or including chiefs, in the mining sector. for implementing reform. results (appendix table F.3a). Furthermore, about half identified proponents or opponents of the About two-thirds of the sampled PSIAs reform (appendix table F.3b). analyzed the risks of implementing the reform program. The risks included institu- The Mauritania mining sector PSIA (World Bank tional capacity risks, political economy risks, 2006l) is an example of a PSIA that identified market risks, weather risks, and so forth. For both beneficiaries and those adversely affected instance, the Egypt social policy reform PSIA 13 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Stakeholder participation (World Bank 2005c) noted that the in all cases where stakeholder buy-in is critical to and the extent and nature implementation of its recommended the success of the reforms. of dissemination varied. reforms would encounter political risks. It also incorporated a sensitiv- The extent and nature of dissemination of ity analysis that involved simulations of different PSIAs have varied. Dissemination of PSIAs has energy price and subsidy scenarios. been more frequent at the back end (draft or final report stages) than at the front end (concept paper About three-quarters of the sampled PSIAs or background work stages) and has been most did not suggest key indicators or a data common among government officials (appendix collection method for monitoring and tables F.8 and F.9). Where English is not widely evaluating the reform, policy, or sector spoken, only 60 percent of the sampled PSIAs in question. Appendix table F.6 shows that were translated into a local language (appendix monitoring and evaluation has been a weak table F.10). Almost 60 percent of the sampled aspect of PSIAs. For example, although the 2004 PSIAs have been made available on the Internet Bolivia PSIA (World Bank 2004o) on the impact (appendix table F.11). of hydrocarbon price increases thoroughly addressed various scenarios for increased fuel Dissemination can help sell the analysis and prices, it did not formulate a system by which the improve the uptake of recommendations for PSIAs Bank or the Bolivian government could follow the intending to inform country policies. However, effects of a price increase. This would have been wide public dissemination may not always be useful not only to validate the accuracy of the possible. For example, if a government asks for the predicted effects, but also to provide feedback Bank's advice on a highly sensitive reform decision to tailor the subsidy mechanism or other reform on a confidential basis, dissemination must be at feature aimed at protecting the poor. the discretion of that government. Nevertheless, it is helpful to disseminate where possible. There was wide variability in stakeholder participation.6 The PSIA User's Guide (World This chapter has revealed that PSIAs have widely Bank 2003h) points out that the process of policy varying characteristics and that practice has often debate, including among stakeholders, can be just departed from the initial concept. Some variation as important as the analysis. The wider develop- in the characteristics of PSIAs is to be expected, ment literature also points to the importance of given the varying country contexts in which they broad ownership and of stakeholder participa- are undertaken. However, many of the charac- tion and consultation to achieve that ownership. teristics discussed in this chapter are desirable in More than half of the sampled PSIAs included most country contexts. some sort of stakeholder participation, but for about two-fifths of the PSIAs no consultations The characteristic that PSIAs have most frequently were mentioned (appendix table F.7). adopted has been analysis of risks. The most neglected dimensions have been explicitly stating The exact degree and nature of participation operational objectives, clearly identifying benefi- needed in a specific case will depend, among ciaries and those adversely affected by the reform, other things, on the type of policy under consid- and setting up monitoring and evaluation systems. eration, counterpart capacity, political sensitiv- Overall, the sampled PSIAs had some weaknesses ity of the policy, and operational objectives. But with regard to factors that are important for some stakeholder consultation will be necessary country ownership and impact on policy making. 14 Chapter 3 Evaluation Highlights · This evaluation suggests that on average the PSIAs evaluated had a moderate effect on country poli- cies, with some outstanding exam- ples, and a negligible contribution to country analytic capacity, with a few positive examples. · A number of factors need to be met if PSIAs are to be effective at the country level: relevance of PSIA topic to country priorities, timeli- ness, and treatment of the politics of reform. Engaging the appropriate stakeholders can contribute posi- tively toward these factors. · There is an inherent tension in ex- erting a timely effect on a country's policy process while also building its analytic capacity. · Where the impetus for addressing poverty and social impacts comes from external groups and not client governments, client government demand for capacity building can- not be taken for granted. 15 Indian woman standing in doorway. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. PSIA Effect at the Country Level T his chapter assesses the in-country effect of PSIAs on two aspects: country policies (including policy debate) and country capacity for policy analysis. The chapter draws on the country case reviews of 12 PSIAs in 8 countries, semistructured stakeholder interviews for an additional 11 PSIAs in 10 countries, the portfolio review, and Independent Evaluation Group (IEG) and non-IEG literature. Tracing links between PSIA analyses and country used to assess the effect of PSIAs on country decisions is often difficult, especially if the PSIA is policies are presented in box 3.1. Appendix D only one of many sources of possible influence. provides details of these effects for the 12 PSIAs PSIA effects may also be diffuse and may occur in included in the country case reviews. The 11 the future, as a report may have no discernable additional PSIAs that IEG assessed through 48 impact on immediate action yet may affect ideas semistructured stakeholder interviews (with 47 and debate that shape future policy choices. This stakeholders, including one stakeholder who evaluation focuses only on near-term effects of was interviewed on two PSIAs), which did not PSIAs. include independent field verification, yielded somewhat more positive but still moderate It is also important to note that strong or weak results on average (see appendix E). The portfo- performance with regard to one operational lio review discussed in chapter 2 showed that objective is not the same as high or low effective- the sampled PSIAs had some weaknesses with ness of a PSIA overall. A PSIA may be effective regard to factors that are important for country for one goal and ineffective for others. Indeed, ownership and effect on policy making. Identifi- certain goals may be inconsistent with others. cation of relevant stakeholders--that is, benefi- ciaries and those adversely affected by the Effect on Country Policies reform, as well as stakeholder participation--has been variable. These findings also point toward Extent a moderate rating for PSIA effect on country The PSIAs reviewed in this evaluation suggest policies on average. a moderate effect on country policies, on average, although there are some outstand- Cambodia offers a good example of positive ing examples of substantial effect. The criteria PSIA impact. Land reform was a key poverty 17 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Box 3.1: Criteria to Assess PSIA Effect on Country Policies Substantial: The PSIA contributed to the design and/or adoption policy or program recommendations (for example, the govern- of government action (such as legislation, regulation, or decree) ment implements some efficiency improvements proposed in or to launching or altering a policy (or set of policies) or program the PSIA but not the overall findings on the need for broad (for example, the government/parliament issues a decree or institutional reform). passes law addressing major points of PSIA). Negligible: The PSIA had little or no discernable impact Moderate: The PSIA contributed to policy deliberation/debate on policy debate or actions undertaken by the government but without affecting follow-on policy action; there was follow- (for example, the PSIA was devoid of policy recommenda- on policy action, but it was not clearly attributable to the PSIA; tions and did not feed into a policy debate or decision in the and/or the PSIA affected only a peripheral aspect of the PSIA's country). Source: World Bank. reduction strategy paper objective, and results policies. Qualitative interviews conducted as from the PSIA on social land concessions part of the PSIA revealed to government officials provided critical analytical inputs that helped the numerous additional expenses parents pay implement this national priority (World Bank when they send their children to school and 2004a). The state land management subdecree the variation across districts. In addition, as a of 2007, which addressed many of the deficien- result of the PSIA research, the government cies of the previous land laws, was fostered by gained a greater understanding of the role of the knowledge and consensus gained during the gender in access to education as well as the PSIA. This subdecree provided the necessary legal particular challenges that orphans face regard- framework to identify and manage land conces- ing education. The extensive information on sions at the local level. The PSIA identified clear supply and demand factors that influence access problems with the existing legal framework and to education, as discussed in the PSIA, shaped suggested movement toward resolution of these government thinking on the sector. Many of the problems. The PSIA also helped build support concerns highlighted in the PSIA regarding access within the government for a smallholder-based to education by the most vulnerable members of agricultural development scheme. society are echoed in the government's second strategic plan of education. Before 2003, many in government felt that efficiency in agriculture and investments in the Explanatory factors for weak effect sector would best be stimulated by supporting There were a number of reasons PSIAs had a large-scale agricultural development. The PSIA, moderate or negligible effect on country policies. by focusing on potential impacts of smallholder They suffered, for example, from bad timing, development and identifying how smallholder poor choice of topic, or lack of attention to the agriculture could succeed, helped change this politics of reform. perception and built support for the govern- ment's subsequent smallholder-based project, The following factors emerge as explanations Land Allocation for Social and Economic Develop- for the extent or lack of PSIA effect on country ment. Factors explaining this effect are discussed policies: in box 3.2. · Operational focus of the PSIA The Mozambique education PSIA (World Bank · Match between PSIA topic and country pri- 2005h) also had a substantial effect on country orities 18 P S I A E F F E C T AT T H E C O U N T R Y L E V E L Box 3.2: Factors in the Success of the Cambodia Land Reform PSIA The central importance of land reform to the government of tion in subsequent dialogue with donors about projects needed Cambodia contributed to the relevance of the PSIA and enabled to support the social land concession program. The availability its direct insertion into the national policy dialogue. The ex ante of complementary services, such as technical assistance, input nature of the analysis and its timing relative to key policy deci- supplies, marketing services, and so forth, represents an essential sions also helped increase its relevance. component of reform sustainability. The land reform PSIA was frank about institutional weak- The inclusiveness of the process (with government, NGOs, nesses, particularly at the level of technical support units for local and civil society participating) and the transparent, qualitative land use allocation committees. This information helped frame a nature of the analysis won the confidence of the government. clear path for successful implementation of the program. The PSIA Stakeholder participation was broad and active. The primary also identified key weaknesses in enabling legislation and a lack of government partner--the Ministry of Land Management, Urban clear guidance in the legal framework for local decision makers. Planning, and Construction--was directly responsible for plan- These institutions were strengthened in subsequent years partly ning and implementing land policies. Its participation built a as a result of the high profile the PSIA afforded these institutional strong institutional base of support within the government. The weaknesses. same ministry was also engaged in the analysis of available The focus on institutions necessary for reform success helped land. stimulate further policy changes: prior to 2003, the government From this experience it was able to understand the multiple knew that additional legislation was needed (such as the 2007 problems associated with identifying and quantifying such land. subdecree), and the PSIA provided concrete guidance about el- First-hand experience was invaluable. The ministry was engaged ements of the subdecree. The analysis also produced important from the very start of the PSIA and continued its involvement information about complementary services and support needed through the end (and beyond). The analysis did not rely on abstract to ensure that smallholder households would benefit after they models or methods, and the simplicity of the analysis stimulated received concession land. The government used this informa- acceptance. Sources: Country case review (appendix D), World Bank (2004a). · Government ownership , impact of ESW notably IEG's recent evaluation of · Engagement with parts of the government that the effectiveness of ESW and technical assistance have policy jurisdiction over the area covered (IEG 2008). by the PSIA · Engagement with appropriate nongovernmen- Operational focus of the PSIA. PSIAs that tal stakeholders lacked an operational focus failed to adequately · Consistency of PSIA timing with regard to the inform country policies, regardless of the country's decision-making process soundness of their economic analyses. For · Political economy issues being addressed example, the Mali cotton PSIA work provided · Dissemination beyond the distribution of descriptive information and poverty estimates reports. based on simulations and models that could not easily be transformed Achieving policy influence Informing a policy process is not easy and requires into policy advice. The economic is not easy--many factors that several of these factors come together-- analysis in the PSIA was of high need to come together where most of the factors were addressed, the quality and provided useful data on for a substantial effect PSIAs had a greater effect. The examples cited the cotton sector. However, because on country policies, below under each factor provide guidance it did not identify what specific although not all pertaining to that particular factor. The explana- policies the analysis would inform factors apply equally tory factors that emerge from this evaluation are or explicitly specify concrete policy and unambiguously similar to those identified in other analyses of the actions, and because it was not widely in all cases. 19 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R disseminated, its ability to inform country regardless of the technical quality of the policies remained low. analysis. The cases reviewed for this evaluation revealed several PSIAs with insufficient govern- In semistructured stakeholder interviews, ment buy-in. interviewees noted that although a PSIA in Egypt (World Bank 2005c) helped the Ministry In Nicaragua, the fiscal PSIA (World Bank 2003a) of Social Solidarity build an "environment for was conducted when the country was about change," it could have been more influential to become eligible for debt relief under the had it elaborated on how the safety net could enhanced Heavily Indebted Poor Countries actually be designed and developed. In Nepal, Initiative. In this context, the government was stakeholders pointed out that it is better to likely to go along with suggestions from the conduct PSIAs along with ongoing lending and Bank. Indeed, several PSIAs were carried out in analytical products, particularly development Nicaragua at the prompting of IMF and World policy lending, which are the focus of the policy Bank staff, with the acquiescence--but not the makers' attention. full buy-in--of the government of Nicaragua. Similarly, the water PSIA (World Bank 2005j) in Match between PSIA topic and country that country was carried out under considerable priorities. Close alignment with country priori- prodding from the Bank, the Inter-American ties was a feature of the more influential PSIAs. Development Bank, and others, and a reform Notably, in Cambodia the government had was launched to privatize water and sanitation demonstrated its concern with the topic of the services, but without full government ownership. land reform by beginning to clarify the 2001 Land In both these cases, the effect of the PSIAs on Law, issuing its first subdecree on the subject in country policies was negligible. March 2003. The PSIA began in September of the same year, feeding directly into the govern- In semistructured stakeholder interviews ment's interest and ongoing policy discussions conducted for this evaluation, interviewees said with other stakeholders around the role of land that the 2006 social protection PSIA work in and its importance to agricultural growth and Indonesia (World Bank 2006k) was successful development. The Cambodia PSIA (World Bank because the Bank was dealing with a new govern- 2004a) also exhibited a number of success factors ment that was committed to poverty reduction beyond the relevance of its topic (box 3.2). In and job creation. The main lesson to be learned Mongolia, the stakeholders interviewed through from the semistructured interviews in Sri Lanka the semistructured interviews believed that the was that there has to be a sense of ownership and choice of the PSIA topic--the cashmere sector-- urgency from the counterpart, without which the was appropriate given Mongolia's dependence Bank should not push PSIA. If the Bank does so, on cashmere and the vulnerability of cashmere the effort may enhance the Bank's knowledge herders (World Bank 2003c). In contrast, the base but not the government's. Ghana Energy PSIA addressed issues of lifeline tariff and consumption inefficiencies that had A stakeholder from Sri Lanka suggested includ- support from some parts of government, but ing a milestone in the PSIA process that would there was no clear consensus within government require that a high-level government panel about the importance of these issues (World be established before the concept note stage Bank 2004e). to ensure ownership of the process. Another stakeholder noted that the way the Bank Government ownership. Sufficient support sometimes conducts PSIAs--hiring a private for the PSIA among government officials in sector agency or NGO to do the consultations relevant agencies and in decision-making and analytical work--implies that the agency positions is essential from early on; otherwise, works more closely with the Bank than with the the chances of having an impact are slight, government. 20 P S I A E F F E C T AT T H E C O U N T R Y L E V E L A PSIA can be designed to have immediate impact port reform, including the Chittagong Timing of the PSIA should if government buy-in exists. In circumstances Port Authority, but neglected to capitalize on a policy with little initial buy-in, there could still be a case substantively consult relevant parts opening, although there for conducting PSIA if the Bank ascertains there of the government on the design, may sometimes be a is a high probability that there may be a future analysis, or drafting of the study, even role for anticipatory opening, but in this case up-front clarity about though they were major stakeholders analytical work. the time horizon and strategy for impact will be in the port. The PSIA acknowledged necessary, with the scope and timing of the PSIA this: "We appreciate there are many gaps in the designed to maximize the prospects for winning text below and the annexes, not least because the attention of decision makers over time. In the opinions of the main decision makers--the situations that the Bank anticipates a policy Ministry of Shipping, National Revenue Board issue that is not yet high on the government's and Ministry of Finance--were not canvassed for agenda, it can try to call attention to that issue their opinions" (World Bank 2005e, p. 8). through policy dialogue and a PSIA. It can also consider separate initiatives to build awareness Engagement with the appropriate nongov- and sensitivity to distributive issues or options ernmental stakeholders. Identifying benefi- for institutional reform, such as study tours ciaries and those adversely affected, as well as or training for key officials or politicians. Such proponents and opponents of policy anticipatory effort may or may not succeed, but reform, and putting measures in place The Bank has sometimes in any case, it should focus on informing country to address them, ideally incorporating engaged that has parts stakeholders, not imposing views on country their input, can increase the chances of the government stakeholders. of the PSIA informing country policies. that has no policy The Zambia land, fertilizer, and rural jurisdiction over the area Engagement with parts of the government infrastructure PSIA (World Bank covered by the PSIA. that have policy jurisdiction over the area 2005l) exemplifies good practice in covered by the PSIA. To have governments nongovernmental stakeholder engagement. "own" the PSIA findings and implement them The PSIA team commissioned participatory requires that the Bank engage with parts of the poverty research in 10 communities and engaged government that have policy and implementa- government officials, business leaders, farmers' tion jurisdiction over the area covered by the unions, NGOs, and other donors throughout PSIA. The PSIA analyzing Mozambique's labor the study. This enriched the analysis greatly. policies (Ministry of Planning and Develop- The PSIA, drawing on the insights thus gained, ment, Mozambique, and World Bank 2006) was uncovered several risks in the proposed land law conducted with the Ministry of Planning and and recommended against going forward with Development, but the PSIA team had little contact that law. The government has indeed not moved with the Ministry of Labor. This prevented the forward, and though it does not attribute this to PSIA from thoroughly evaluating alternatives to the PSIA, another stakeholder does. the reform and obtaining full support for them. The Republic of Yemen water PSIA also exempli- In Zambia, the Ministry of Agriculture and Cooper- fies good practice with regard to stakeholder atives was not included in the research, writing, involvement (see box 3.3). or discussion of the PSIA (World Bank 2005l), even though the issues were under its jurisdic- In contrast, stakeholders interviewed through tion (which some staff attributed to the PSIA semistructured interviews believed that in Arme- being carried out in association with the Country nia the low levels of consultations with NGOs Economic Memorandum, which had a different led to a lack of focus on the need to accom- country counterpart). The PSIA team in Bangla- pany privatization with building awareness of desh conducted stakeholder consultations with consumers, which they saw as being necessary 11 different groups who would be affected by in a market economy. Stakeholders in Benin were 21 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Box 3.3: Republic of Yemen Water PSIA: Participation by Government, Other Local Stakeholders, the World Bank, and Other Donors The Republic of Yemen water PSIA, carried out jointly by the Bank, Early in the process, donors and stakeholders worked together Germany, and the Netherlands, took advantage of GTZ's activity in to determine the methodology and the relevant subsectors. the sector and involved its resident advisor in the Yemeni govern- They then commented on a preliminary draft and were again ment in the effort. Strong partnership among these donors, the gov- involved in a third stage when their feedback was sought on ernment, and other stakeholders--such as consultative councils, how to use the findings to update the national water sector water associations, water corporations, civil society organizations, strategy and how to use the findings in the design of the sub- and donors--ensured that the findings of the PSIA would feed into sequent multidonor sector-wide approach, which was funded the update of the national water sector strategy. by the same donors that supported the PSIA, with the addition This strong partnership was achieved through broad en- of DFID. gagement of stakeholders that started at the design stage. Sources: World Bank (2007j) and IEG interviews with Bank staff. adamant that unless the Bank worked with civil The PSIA directly and strongly affected both society, such as the private sector, farmers' asso- fuel price policy and use of the savings to fund ciations and organizations, and the media, there compensatory programs for the poor (World would not be a change in the corruption and Bank 2005g; IEG 2009, box 4.1). patronage that, in their opinion, allowed cartels to operate. According to them, there needed to PSIA timing in regard to the country's be more faith in the private sector and in farm- decision-making process. A PSIA that is timed ers' organizations, as they could force govern- to coincide with the decision-making process in a ment and the public administration to actually country is likely to be influential. The Cambodia implement policy changes. social concessions of land reform PSIA (World Bank 2004a) came at an opportune time for three Broad public consultation and participation may reasons: Existing institutional arrangements not always be appropriate if the goal is to affect for transferring land to the poor were proving government decisions. The Indonesia fuel price inadequate; a growing consensus was emerging PSIA of 2005 (World Bank 2005g) addressed a in support of social land concessions; and the situation where fuel subsidies had become a government was interested in investigating the severe drain on the budget and had a regressive impacts of such a program. The semistructured impact. Yet earlier attempts to raise fuel prices interviews conducted for this evaluation also had provoked riots. The PSIA analyzed various showed the importance of getting the timing pricing options and their implications for the right. budget and for poor households and proposed using savings from removing the fuel subsidy to The fuel price increases in Nepal coincided with finance targeted benefits in education, health, the country's dialogue about that reform. The and village infrastructure. The analysis was PSIA's results were presented to policy makers and conducted jointly with Indonesian academics informed their decision to raise petroleum prices and think tanks but was not broadly discussed in (World Bank 2005i). In Armenia, semistructured public because of the political sensitivity of the interviews showed that stakeholders believed issue, although a public information campaign the PSIA (World Bank 2001d) was done at a time was subsequently mounted by the government. when the government was embarking on privatiz- 22 P S I A E F F E C T AT T H E C O U N T R Y L E V E L ing utilities in the country and was keen to have holds power, how this will affect the A PSIA timed to coincide information on its options. implementation of the reforms, and with country decision how best to proceed with the reform making is likely to In contrast, the Mozambique labor PSIA (Ministry given these present power relations. be influential. of Planning and Development, Mozambique, The recent joint Overseas Develop- and World Bank 2006) was carried out too late ment Institute and World Bank report on PSIAs to inform proposed labor law reforms that had regards the Republic of Yemen water PSIA as an already been widely discussed and submit- influential study (ODI and World Bank 2009). ted to Parliament. In Mongolia, interviewees in the semistructured interviews pointed out that Dissemination and follow-up beyond the although the cashmere PSIA was relevant at distribution of reports. Inadequate dissemi- the time of its initiation, delays in completion nation and follow-up reduced the effect of resulted in the loss of interest among various several of the examined PSIAs. The Ghana energy stakeholders (World Bank 2003c). By the time the PSIA (World Bank 2004e) was not disseminated PSIA was completed, relevant donor programs broadly or in a timely manner, which affected that the PSIA could have informed were almost public discussion of its findings and its effect on complete. country policies. Neither the fiscal nor the water PSIAs in Nicaragua (World Bank 2003a; World Stakeholders interviewed through semistructured Bank 2005j) were adequately disseminated. The interviews in Benin suggested that the PSIA (World Bank decided not to disseminate the Bangla- Bank 2004b) was introduced as a retrospective desh port PSIA widely (World Bank 2005e). This justification for the Bank's planned cotton sector decision prevented staff from sharing the report reform project and could have been introduced with external partners, who may have been able earlier. In Sri Lanka, where the PSIA (World Bank to use the findings in their interactions with the 2005k) was conceived after a Learning and Innova- government.1, 2 tion Loan on land titling was received with hostil- ity by civil society, stakeholders noted that the Semistructured stakeholder interviews suggested PSIA should have been done before the Bank that the lack of dissemination was a major source advocated land reform, not after. of frustration for stakeholders in Sri Lanka. The PSIA, which reported that restrictions Addressing political economy issues. The on land sales should only be partial Incorporation of more influential PSIAs effectively incorporate (World Bank 2005k), contrary to prior political economy political economy considerations and take into Bank views on the topic, had not yet considerations can also account the political motivations that inform been shared with any of the local improve effectiveness. governments' policy choices. An understand- researchers who had worked on the ing of the political economy can reveal why PSIA team or with stakeholders who had taken seemingly irrational or inefficient policies persist part in consultations for the PSIA. and can help a PSIA to suggest ways to reform them. For example, the Republic of Yemen water In contrast, in semistructured interviews in Nepal, sector reform PSIA (World Bank 2007j; box 3.3) interviewees expressed the view that the PSIA on identifies the stakeholders who have signifi- fuel prices (World Bank 2005i) was disseminated cant effect on the water resources and irrigated to the appropriate groups. The Bank discussed it agriculture reforms and assesses their possible with cabinet-level officials and various stakehold- support or opposition. It analyzes separately ers. One of the key dissemination activities--an the roles of Parliament, sheikhs and other large event--was spearheaded by an academic landowners, irrigating farmers, the Ministry of working on this topic and funded by the Bank. Water and Environment, the National Water This event brought together students' groups Resources Authority, the Ministry of Agriculture and distributors on the campus to discuss why and Irrigation, and donors. The report details who Nepal needed an automated pricing mechanism. 23 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R A government stakeholder appreciated the distri- In some circumstances, certain factors and bution of summary reports at meetings. The local approaches may be in tension with others. For newspaper also reported on the PSIA. instance, a thorough PSIA exploration of politi- cal economy issues may sometimes conflict The country case reviews undertaken for this with country ownership of findings. Where evaluation strongly suggest that dissemination high-level officials have financial interests in should not be an afterthought. Effective dissemi- particular policies that the PSIA suggests should nation generally requires more than one event, be reformed, the government may not welcome and not just after the report is printed. frank discussion of this issue. As emphasized in the IEG evaluation of ESW and Somewhat similarly, the need for buy-in by technical assistance (IEG 2008), follow-up is also appropriate operational agencies may conflict important for policy impact and can take the form to some degree with a candid analysis of institu- of lending, technical assistance, further analytical tional capacity to implement a policy or program. work, or policy dialogue. For policy impact, the Variation in the scope of desirable participation follow-up would need to be with policy makers in initiation, discussion, and dissemination has and groups that influence policy makers, includ- already been noted and is supported by IEG's ing parliamentarians, leaders of opposition ESW evaluation (IEG 2008). The 2008 revisions parties, trade unions, business associations, heads in the Good Practice Note that offers guidance of cooperatives, and civil society organizations. for PSIAs (World Bank 2008b) emphasize the need for flexibility in determining approaches In semistructured stakeholder interviews, suitable for country circumstances. However, interviewees noted that the PSIA in Egypt was initial lack of consideration of these tensions the first step and that the government would probably contributed to uncertainty within the have preferred a prior agreement in which Bank regarding the PSIA concept and to skepti- further stages of assistance were spelled out cism regarding its value. beforehand. In Mongolia, stakeholders felt that it was important for the Bank to have a project Contribution to Country Capacity for or technical assistance to follow a PSIA. In Nepal, Policy Analysis Bank staff felt that no matter how many PSIAs the The Bank has expected PSIAs to have a major Bank did, they needed to bring in money or the role in building local capacity. As early as 2003, Bank would always be a marginal player. the PSIA User's Guide noted-- In general, the Bank's dissemination strategy Building national capacity is key to improv- must reflect the policy issues and country-specific ing analytical rigor over time, in tandem sensitivities at hand, but the presumption should with strengthened country ownership. be in favor of dissemination and disclosure. Many low-income countries have limited Sometimes, of course, a government can ask for capacity and experience in areas of the Bank's advice on a highly sensitive reform critical importance to PSIAs. These areas decision on a confidential basis. In such cases, the include data collection systems, monitor- extent of dissemination must be at the discretion ing and evaluation systems, the capacity of that government. However, the 2008 PSIA Good to conduct analysis and to translate data Practice Note points out that even in such cases and analysis into policy, and the institu- the task team should summarize the key messages tional structures and mechanisms for from the distributional analysis and make them debate on such policy issues in the public available to the public (World Bank 2008b). domain. Building national capacity in these areas must be a fundamental cross- To conclude, it must be noted that not all factors cutting aspect of PSIA. Development apply equally and unambiguously in all cases. partners, including the Bank, have an 24 P S I A E F F E C T AT T H E C O U N T R Y L E V E L important role in strengthening national E).3 The recent joint ODI and World Building capacity has capacity and in filling analytical gaps. Bank report on PSIAs found that the not been a strong suit PSIA approaches that foster "learning by "record on capacity strengthening is of PSIAs. doing" should undergird development weak" (ODI and World Bank 2009). partners' assistance to countries (World Bank 2003h). The one instance among the country case reviews where the PSIA made a substantial contribution This attention to local capacity building was to country capacity was the Ghana energy PSIA reinforced in the 2004 PSIA Good Practice Note, (World Bank 2004e). In Ghana, considerable which stated, "PSIA is primarily the responsibil- training inputs were provided by external consul- ity of borrower governments. But the Bank and tants and the Bank team, and new methods were other development partners have a major role in introduced in the Kumasi Institute of Technology building local capacity for PSIA," and went on to and Environment, which was contracted as the identify three levels of capacity building for PSIAs local consultant for the PSIA. Its staff benefited (World Bank 2004f, paras. 4, 11). from the training in qualitative research, which was further enhanced through continuous Extent discussions and reviews. The PSIAs reviewed in this evaluation suggest a negligible contribution to country analytic The Electricity Company of Ghana also benefited capacity, on average, although there are a in terms of capacity building from the energy few examples of substantial contribution. PSIA. The company attached a member of its The criteria used to assess the effect of PSIAs staff to the consultant teams that undertook the on country analytic capacity are presented in survey on which much of the PSIA report was box 3.4. Appendix D provides details of these based. As a result, the company learned a great effects for the 12 PSIAs included in the country deal about their customers, for example, related case reviews. The 11 additional PSIAs assessed to billing arrangements in compound houses. by IEG through 48 semistructured stakeholder Four years later, and despite nonimplementation interviews, which did not include indepen- of the PSIA's recommendations, the company dent field verification, yielded somewhat more continues to regard the energy PSIA as having positive results--moderate, on average-- helped build capacity. although this result is based only on 25 observa- tions (23 of the 48 stakeholder interviews yielded In semistructured stakeholder interviews in no assessment of this dimension; see appendix Albania, Bank staff noted that the PSIA (World Box 3.4: Criteria to Assess PSIA Contribution to Country Capacity Substantial: PSIA contributed to the transfer of skills or method- but these skills dealt with only a limited part of the analysis (for ologies for gathering and analyzing information on distributional example, local consultants participated in the PSIA, but only impacts to unit(s) within the government or private institutions in the gathering of qualitative data and not in measurement or (for example, ministry staff gain skills to carry out PSIA). monitoring of impacts). Moderate: PSIA contributed to the transfer of some skills or Negligible: Little or no transfer of skills and/or methodologies was methodologies, such as gathering data or stakeholder inter- acquired at the institutional level (for example, PSIA is conducted views, to unit(s) within the government or private institutions, by a foreign consulting firm with little to no input from locals). Source: World Bank. 25 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Bank 2004k) improved capacity of the local practice capacity building is recognizing capacity NGO that had never conducted a PSIA before. building as a goal in its own right where such a The Bank conducted several training workshops goal is justified by the Country Assistance Strat- with the local NGO director and six or seven egy. Although a PSIA may or may not include a local sociologists, shared PSIA resource books, capacity building objective, in cases where such and continued the dialogue to share knowledge an objective is justified, the PSIA needs to explic- going forward, but Bank staff noted that more itly state it and specify what kinds of capacity it was needed. intends to promote and the organizations to be targeted. Capacity building cannot be treated as PSIAs have used two main mechanisms to build a by-product. country capacity. Most PSIAs have involved local stakeholders (such as government There will also need to be a strategy to achieve Capacity building needs ministries, local consultants, or local the intended capacity building over the intended to be recognized as an NGOs) on the PSIA team, relying on time period. Ensuring that there is demand for objective in its own learning by doing to build capacity capacity from governments will have to be an right and not be treated (appendix table F.12).4 The level of important part of the strategy, especially because as a by-product. involvement of local stakeholders the impetus for PSIAs and evidence-based policy and the degree of resulting capacity making may not come from client governments. building has, however, differed across PSIAs. Where capacity building is a PSIA goal, at least The other mechanism has been capacity some team members will need to be selected for building through formal training. Training their ability to work with and transfer skills to workshops on the use of analytical software host country personnel. Two important aspects were held in the context of the Mali cotton of the capacity building effort include a needs sector PSIA work, and training in participa- assessment to precede the specific capacity tory monitoring was provided in the case of building intervention and outcome monitoring the Zambia land, fertilizer, and roads PSIA to be concurrent with it. (World Bank 2005l). For the most part, these mechanisms have made a limited contribution Allowing sufficient time for capacity to to capacity building because the number of be built. Capacity building of any lasting sort skills covered and the length of time spent in cannot be hurried and usually needs to be on a training have not generally been significant. slower track than a typical PSIA. It also needs to be reinforced over time. Depending on country Furthermore, PSIAs have sometimes been circumstances, a programmatic approach that carried out primarily by Bank staff or have simply enables successive capacity building investments been contracted out to foreign consultants, as over a longer period may be more effective in in Mongolia (World Bank 2003c) or Nicaragua building capacity than a one-off effort. (World Bank 2003a), thereby contributing very little to local capacity building. There is often a tension between the objective of making a real-time contribution to a country's Explanatory factors decision-making process (for which the window There are two key factors affecting a of opportunity can close quickly) and building There are tensions PSIA's contribution to country analytic country analytic capacity (which is typically a between the objectives capacity: slower and longer process). This tension means of informing country that a single approach cannot be used to pursue policies in real time and Having an explicit capacity build- both these objectives. If capacity building is the building lasting country ing objective and a strategy to objective, then the PSIA approach needs to be analytic capacity. achieve it. The first feature of good specifically tailored to it. 26 Chapter 4 Evaluation Highlights · The PSIAs reviewed in this evalu- ation suggest on average a mod- erate effect on Bank operations, with some outstanding examples. Interviews conducted for this study indicate that there is not a common understanding among Bank staff about what a PSIA is or should be. · Operational staff have not been suf- ficiently engaged in PSIAs, which may have reduced the PSIAs' po- tential relevance to country assis- tance programs. · Staff directly involved with PSIAs see a number of corporate benefits, notably the creation of an important body of knowledge through PSIA guidance. · Quality assurance, monitoring, and evaluation of the overall effective- ness of PSIAs have been weak. 27 Villagers playing traditional Cambodian music for tourists in Angkor Wat. Photo by Masaru Goto, courtesy of the World Bank Photo Library. PSIA Effect within the Bank T his chapter assesses the effect of PSIAs within the Bank at two levels: Bank operations (including strategy and analytical work) and thinking and practice across the Bank. The latter can clarify the reasons for PSIA effect (or lack thereof) on Bank operations, one of the operational objectives of PSIAs. This chapter draws on the 12 PSIAs covered in the country case re- views, semistructured stakeholder interviews for an additional 11 PSIAs, and interviews with 30 senior Bank staff and managers. Effect on Bank Operations ments for tobacco, which hinder the Implementation of the The PSIAs reviewed in this evaluation suggest a efficiency of the sector and limit the PSIA approach has had moderate effect on Bank operations, on average, pass-through of international prices considerable limitations. with some outstanding examples of substantial to smallholders" (World Bank 2007c). effect. The criteria used to assess the effect of Likewise, in Nicaragua the PSIA on fiscal reform PSIAs on Bank operations are presented in box 4.1. (World Bank 2003a) influenced the Bank's Poverty Appendix D provides details of these effects for Reduction Support Credit and is also referred to the 12 PSIAs included in the country case reviews. as an analytical underpinning. The 11 additional PSIAs assessed by IEG through 48 semistructured stakeholder interviews, which In addition, the Cambodia social concessions did not include independent field verification, of land reform PSIA (World Bank 2004a) had yielded somewhat less positive but still moderate substantial influence on the design of a major results on average (appendix E). Bank investment project. The PSIA presented information on a range of elements central to the land reform issue that the government was Two Malawi PSIAs (World Bank 2003f, 2004h) had addressing. Before 2003, many in the government a substantial effect on Bank operations by inform- felt that efficiency in agricultural investments ing the first Poverty Reduction Support Grant to would best be stimulated by supporting large- Malawi. Both the tobacco and agriculture market scale agricultural development. The economic closure PSIAs (World Bank 2003f, 2004h) are cited concession program was one manifestation of in the grant as key analytical underpinnings that this belief. By focusing on potential impacts of contributed to identifying "several weaknesses smallholder development and identifying how in current marketing and institutional arrange- smallholder agriculture could succeed, the 29 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Box 4.1: Criteria to Assess PSIA Effect on Bank Operations Substantial: Major elements of the PSIA are reflected in the a result of a PSIA, the Bank gives greater attention to politi- content of Bank lending program, country strategy, or analytical cal economy analysis in the subsequent Country Assistance work (for example, a Bank lending document specifically refers Strategy). to the PSIA as informing the design of the project). Negligible: There is little or no impact on Bank lending, country Moderate: Some findings, but not the main points or rec- strategy, or analytical work (for example, PSIA findings appear ommendations of a PSIA, are reflected in the Bank lending in a Country Economic Memorandum, but as add-ons and with program, country strategy, or analytical work (for example, as the PSIA's main findings buried). Source: World Bank. PSIA helped change this perception and built lack of consensus between the PSIA and the subsequent support for a smallholder-based Country Economic Memorandum with respect project, Land Allocation for Social and Economic to the emphases they placed on the growth Development, which received Bank financ- of the private sector in agriculture, the impact ing. The PSIA also helped improve relations of land titling on the poor and vulnerable, the between the Bank and the government, and the benefits and drawbacks of fertilizer subsidies, trust built during the process contributed to a and the weight they placed on market solutions smoother and more consensual project design. for farmers in remote areas. Another reason for the negligible impact may have been the lack of The Bangladesh port PSIA (World Bank 2005e) sufficient participation by in-country Bank staff had a moderate effect on Bank operations. There in the PSIA and their resulting lack of ownership was little collaboration internally within the Bank, of its findings. notably between the social development unit in the Bank's South Asia Region, where the report Effect on Thinking and was housed, and the transport sector responsible Practice across the Bank for port strategy. Although some Bank staff say that the PSIA findings led them not to intervene Extent in the port sector, others point to the uncertain- This evaluation has also sought to gauge the effect ties regarding political buy-in as the reason for of the PSIA experience on thinking and practice the lack of follow-up to the PSIA. There was, regarding poverty and social impact analysis however, a positive unintended effect from this across the Bank, based on interviews with 30 PSIA: it likely encouraged greater acceptance of senior Bank staff and managers (such as country political economy analysis in the Bank in general directors, advisors, and others). They reveal only and more political economy analysis of reforms modest uptake of the PSIA as a robust practice in the preparation of a subsequent Country across the Bank, although staff directly involved Assistance Strategy in particular. with PSIAs identified some benefits that the experience has brought to the Bank (box 4.2).1 In Zambia, apart from the section on rural infrastructure, the substance of the 2005 land, Most Bank management and staff interviewed fertilizer, and rural roads PSIA (World Bank 2005l) felt that the utility of PSIAs is yet to be proven had a negligible impact on Bank operations. and that the uptake of PSIAs by country One of the likely reasons was internal policy directors and operational teams remains differences within the Bank, specifically, the dependent on individual inclinations rather 30 PSIA EFFECT WITHIN THE BANK Box 4.2: Corporate Benefits Identified by PSIA Proponents Although there was not a consensus among Bank staff inter- only for PSIAs but also for other analytical work. This has viewed about the extent of PSIA effects on thinking and practice led to increased knowledge about the differential impacts across the Bank, Bank staff and donors directly involved with of reforms and how they can be addressed. "More people PSIAs cited a variety of corporate benefits since fiscal 2002: are aware that all reforms have differential impacts and that they should be acted on. The understanding has always been · Innovative analytic framework. PSIAs have sought to provide there, but giving it form and providing the tools does affect how an innovative analytical framework that aims to bring economic people think and what guidance they have available," said and noneconomic analysis and quantitative and qualitative one interviewee. Another said, "The initiative, the branding, evidence together. They have emphasized a participatory pro- the creation of a business line propelled more thinking about cess in addition to the analytic approach. These innovations distributional issues." are captured in two quotes from Bank and other donor agency · Improved Bank risk management. "The Bank is now more staff: "PSIAs have been transformational in bringing a focus forthcoming, explicit, and systematic in recognizing and man- on a more holistic view of development beyond a narrow eco- aging risks from distributional issues." nomic view" and "Like poverty reduction strategy papers, the · Enhanced Bank credibility. Some staff see improved Bank main contribution of PSIAs has been on the process side." credibility as a result of the introduction of PSIAs, especially · Knowledge and tools. The analytic work published as part anchoring poverty and social impact analysis in a Bank Op- of PSIAs provides an important body of knowledge. In addi- erational Policy (OP 8.60). "This makes an important statement tion, guidance and tools in the form of the PSIA User's Guide, about the Bank and makes for a different view of the Bank sourcebook, and Good Practice Note are seen as useful not [from] the outside." Sources: IEG interviews with Bank staff and donors directly involved with PSIAs. than on the need to reflect established practice. social and institutional analysis and a participa- As expressed by one country office staff member, tory process. although poverty and social impact analysis is now reflected in DPLs, "it still needs to be made Emphasis on flexibility. Most Bank staff wanted a normal part of doing business." This limitation flexibility in assessing the poverty and social is expressed in the following ways. consequences of policy reforms. They felt a PSIA with all the "bells and whistles" was not needed in Differing views on the nature and every--or even most--cases. Is there consensus? importance of PSIAs. There is a vast differ- "Yes, for poverty and social analysis, ence between the institutional view of a PSIA but no, for one way of doing it bundled There is lack of a common as defined on the Bank's PSIA Web site (World together [conducting economic as understanding among Bank 2008e), the PSIA User's Guide (World well as social, institutional, and politi- Bank staff about what Bank 2003h), and the PSIA Good Practice Note cal analysis, using both quantitative a PSIA is or should be. (World Bank 2008b) and Bank staff understand- and qualitative methods, focusing ing of it. Among interviewees, there was little on analytical aspects as well as process, and so homogeneity of views about what a PSIA is, how forth], in all cases," said one respondent. Bank important it is, or how to do it. Most operational staff interviewed disagreed about the importance staff interviewed lacked a common understand- of specific aspects of the PSIA approach. ing about the objectives and processes of the PSIA approach. PREM staff generally tended to Explanatory factors favor a focus on economic analysis, and SDN staff Seven factors appear to have had significant tended to emphasize mixed methods, including bearing on the extent of the effect of PSIA 31 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R on Bank operations and, more broadly, on directors are not yet convinced of the utility of thinking and practice in Bank analytical work the approach relative to other things that have and the design of country strategies. These to be done to develop an effective country factors are-- program. Overall, the success of PSIAs in inform- ing country assistance programs has depended · Understanding of PSIA as only a product on individual inclinations of Bank staff rather · Buy-in of country directors than on established practice. · PSIA team composition and financing · Complexity and ambiguity of PSIA guidance PSIA team composition and financ- · Regional quality assurance processes ing. A number of interviewees noted that · Sufficient engagement between the PSIA team PSIAs conducted by short-term staff and with and operational staff in headquarters or coun- outside funding have often failed to attract try offices the attention of Bank staff who are respon- · Link to Bank operations or policy decision. sible for country lending or analytical work. For example, in Zambia, some members of Understanding of PSIA as only a product. the PSIA team were short-term consultants If PSIAs are seen as a product and an end and therefore not employed long enough in themselves, the focus on informing Bank to ensure that the findings remained on the operations or country policies might be lower Bank and donor agendas. Furthermore, heavy than otherwise expected. Some interviewees external funding with a minor Bank budget referred to large research studies that received contribution in this case led the PSIA to be funding from the Incremental Fund but that perceived as being external and not sufficiently were not embedded in the policy-making owned by the Bank. In contrast, the Morocco process of the client country. "PSIAs were water supply and sanitation PSIA and the 2007 always meant to be part of a policy process, but water sector DPL teams overlapped, so the the Bank went off track with massive studies," PSIA (World Bank 2006a) directly informed said one interviewee. For example, a study on that DPL. the history of land reform in Vietnam, which provides a technically sophisticated and interest- Complexity and ambiguity of PSIA ing examination of Vietnam's past land reforms guidance. The Bank has produced a substantial (Do and Iyer 2003), was categorized as a PSIA. body of guidance on the PSIA approach, and these Large research papers like this have led to the materials have been refined over time to incorpo- perception of the PSIA as a "Cadillac product" rate lessons learned. However, most Bank staff rather than an approach that involves analysis interviewed have regarded the materials (such as of poverty and social impacts of policy reforms, the 2003 PSIA User's Guide, or 2004 PSIA Good as originally intended. Practice Note) as too onerous and demanding: "Bells and whistles about PSIAs are a turn-off." Buy-in from country Buy-in of country directors. The 2004 PSIA Good Practice Note (World Bank directors has been weak. Country director buy-in for PSIAs has 2004f) was seen as creating unrealistic expecta- been weak. Several country directors tions and a high standard against which PSIAs interviewed complained that they barely have would be judged. enough time for the basic analytical blocks such as Country Policy and Institutional Assessments Tensions and ambiguities inherent in the PSIA and therefore view PSIAs as a luxury. The result guidance may also have contributed to skepticism has been scant incorporation of PSIA work into among Bank staff. These tensions include the their country programs. Sector directors appear inconsistency, in many circumstances, between to be more positive about PSIAs, but interviews strong country ownership and frank examina- with network and country staff and country tion of political and institutional obstacles and directors suggest that, for the most part, country capacities affecting proposed policies and the 32 PSIA EFFECT WITHIN THE BANK inconsistencies between exerting timely effect others. Operational staff found that The 2004 Good Practice on country (and Bank) decisions and building the Mozambique labor PSIA (Ministry Note was seen as creating country analytic capacity. In other words, many of Planning and Development, unrealistic expectations Bank staff may view PSIA guidance as unrealis- Mozambique, and World Bank 2006) and a high standard tic. Revisions in the 2008 PSIA Good Practice did not assess the shortcomings of the against which PSIAs Note (World Bank 2008b) address some of these new law or examine alternatives, that would be judged. concerns, but are not yet fully known among it lacked data and made claims it could Bank staff. The tensions between the capacity not support, and that it did not clearly assess who building objective and other PSIA operational the beneficiaries or those adversely affected by objectives remain to be addressed. the proposed reform might be. Moreover, some operational staff disagreed with the findings, Regional quality assurance processes. affecting their follow-through. In Ghana, there According to interviews with Bank managers and was insufficient ownership of the PSIA among the staff, the focus of decision meetings on PSIAs Bank's Energy Unit staff, most of whom did not has typically been on the technical quality of consider distributional issues--the focus of the assessment. Few questions are asked about the PSIA--among their main priorities.4 intended effect of PSIAs or the strategy for achiev- ing that effect. For PSIAs to attain their operational Link to Bank operations or policy decision. objectives, Regional management needs to send A PSIA that is not conceived in the context of a clear signals--through questions asked--from proposed Bank-supported policy reform, analyti- the concept stage onward, especially related to cal work, or lending program will typically have process and intended effect. In addition, staff only a moderate effect on Bank lending, performance assessments need to factor in PSIA strategy, or advice. The Bank's research The flexibility granted effect, not just technical quality. A strengthened and analytical interests may sometimes by the 2008 PSIA Regional quality assurance process that asks the generate supply-driven or general Good Practice Note in right questions about a PSIA's fit with the country analyses not directly linked to the Bank's determining approaches assistance program and a PSIA's effectiveness is operational work. The Indonesia micro- suitable for country crucial. macro reform PSIA (World Bank 2002) circumstances is likely to was a retrospective look at the poverty increase PSIA receptivity More systematic use of peer reviewing might effects of the financial crisis and was not among Bank staff. focus PSIAs better. Appendix table F.132 shows connected to a lending operation. The that the PSIA Concept Notes (World Bank 2001b) Bangladesh port PSIA (World Bank 2005e) also were peer reviewed in 40 percent of the cases in was not linked to a specific lending operation but IEG's portfolio review. The draft PSIA was peer focused on an analysis of the political economy. reviewed in 53 percent of cases, although there were several cases for which the existence or lack Recent Developments of peer review was difficult to discern from the Two developments have occurred during documentation. this evaluation--revision of the Bank's Good Practice Note and discussions on the new Sufficient engagement between the PSIA Multi-Donor Trust Fund for contin- team and operational staff both in headquar- ued funding of PSIAs. In view of Bank operational ters and in country offices. One of the most the findings of this evaluation, both staff have not been striking findings of this evaluation is the limited initiatives will need to go further sufficiently engaged in success in engaging Bank operational staff-- than they currently do. the content and process both at headquarters and in country offices--on of PSIAs at headquarters PSIA content and process and the associated 2008 PSIA Good Practice Note or in country offices lack of buy-in from operational staff within the The 2008 PSIA Good Practice Note and therefore have Bank.3 IEG's country case reviews found this (World Bank 2008b) is an improve- not sufficiently an issue in Mozambique and Ghana, among ment over the 2004 Good Practice bought into PSIAs. 33 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R The initial lack of Note (World Bank 2004f) in that on what constitutes a PSIA and what should be attention to flexibility it grants considerable flexibility to called something else. There is no consensus yet may have contributed Regions in determining the balance among Bank staff about what a PSIA is or should to uncertainty about among economic, social, institu- be. This lack of consensus has a strong influence the PSIA concept tional, and political analyses (and against conducting the relevant analysis and its and to skepticism between quantitative and qualita- quality assurance.5 regarding its value. tive techniques), as well as between analytics and such process issues as New Multi-Donor Trust Fund for PSIAs stakeholder participation and disclosure. This The Bank and donors are discussing the Multi- flexibility will allow the scope and content of the Donor Trust Fund for PSIAs. The use of such PSIA to be better tailored to the specific context earmarked funds for PSIAs requires that the and may also help avoid the perception of the Bank give special attention both to the balance PSIA as a "Cadillac product." between its own budget and trust fund use and to a strategy for continued engagement beyond However, although flexibility can help improve the end of earmarked funding. The risks associ- the relevance of the PSIA to a country context, ated with earmarked funds are similar to those Bank staff need to provide the rationale related to other infant industry approaches-- for specific choices made. This issue is not overinvestment and misallocation during the recognized in the 2008 PSIA Good Practice Note. protection period and a return to underinvest- Providing such a rationale can help ensure that ment when protection ends. Bank staff from the different networks will not continue to emphasize their respective disciplin- The Bank needs to clarify its objectives for ary approaches in undertaking PSIAs. PSIA and decide how much and which of the PSIA work it will fund from its own budget and The 2008 Good Practice Another shortcoming of the 2008 for what it will rely on trust funds. Requiring Note grants flexibility PSIA Good Practice Note is that all trust fund allocations to be matched by a to the Regions for PSIA it has not yet fostered a common substantial Bank contribution (especially from content and process. understanding among Bank staff the country unit budget) would help ensure about what a PSIA is. In fact, the main Bank ownership of the work that is financed debates among various groups in the Bank in by the trust funds and its grounding in country revising the Good Practice Note have centered assistance programs. 34 Chapter 5 35 Three generations at a family-owned store in Kabul, Afghanistan. Photo by Michael Foley, courtesy of the World Bank Photo Library. Lessons and Recommendations T he PSIA approach has correctly emphasized the importance of under- standing the institutional and political constraints to development and the need to build domestic ownership of policy reforms in addition to assessing the distributional impact of policy actions. However, implemen- tation of the approach has had considerable limitations. Nevertheless, the PSIA experience so far suggests several lessons, and some notable successes have modeled what PSIAs can accomplish when done right. The lessons and recommendations in this chapter are intended to help improve future imple- mentation of the PSIA approach and realize its potential. Lessons Both design and timing of PSIAs need to take account of the political context. Informing country policies and Adoption of evidence-based policy making can building country capacity mean major shifts in power relationships. Partic- ularly where policy making has not typically been Participation is important, but how much, when, and transparent or there are powerful stakeholders with whom depends on the specific context. in the sector being analyzed, the PSIA will need PSIAs can adopt different degrees of participa- to take these political economy considerations tion and involve varying stakeholder groups into account. An understanding of the politi- that reflect the particular contexts in which cal economy of policy making in a country can they are undertaken. The exact degree and help ensure that the PSIA fits local decision- nature of stakeholder participation that is making processes and is appropriately timed and appropriate in a particular case will depend designed. on, among other things, the type of policy under consideration, counterpart capacity, There may be trade-offs between methodological political sensitivity of the policy, and the sophistication and capacity building. operational objectives of the PSIA. Some Depending on the existing level of capacity in a stakeholder consultation will be necessary in country, building capacity through learning by all cases where stakeholder buy-in is critical to doing or short training programs may not be the success of reform. an immediate option, especially in PSIAs that 37 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R adopt overly sophisticated methodologies. If It is important to ensure ownership for PSIA work the PSIA uses complex methodologies, the Bank within the relevant Bank operational units. will need to consider significant pre-invest- Early buy-in for PSIA work from the relevant ments in training before local stakeholders can constituencies within the Bank is as essential be effectively involved in a learning-by-doing as buy-in from country stakeholders. Otherwise capacity building effort. For example, in Mali, there is little chance of the analysis having an the Social Accounting Matrix was technically effect on country strategy, lending, or analytical too sophisticated for most participants; more work, almost regardless of the technical quality intensive training would have been required for of the analysis. participants to actually use the tool and to fully understand the results. PSIA experience shows that staff ownership depends at a minimum on communication Where capacity building is a priority, the Bank about the scope and content of a PSIA between should consider whether there are more effective the analytic team and relevant operational staff; alternatives to a one-time PSIA that also has the ownership is better when there is close consul- objective of informing country policies. tation with or actual participation by the Bank's Given the tensions between informing the country staff in the analysis. Where a PSIA's policy process in a timely manner and building analysis runs counter to the prevailing views of capacity, the same approach is unlikely to operational staff, a process of dialogue will be effectively serve both objectives. Therefore, the important. Strong leadership from Bank manage- PSIA will need to be tailored to pursuing the ment will be necessary to ensure that findings particular objective in question. In many cases, that run counter to the common assumptions technical assistance or other programmatic of one sector unit in regard to another are not instruments that are designed as a series or as buried or ignored, and to resolve differences. an incremental package of interventions may be a more effective approach to capacity building Earmarking of resources has pros and cons. than a one-time PSIA. The Bank is currently The availability of special earmarked funds can exploring incremental approaches in three have both positive and negative effects. On the pilot countries whereby a number of different positive side, interviews with Bank staff and products and approaches can be combined in managers indicated that earmarked funding for a sequence over time. The joint ODI and World PSIAs was essential to getting the PSIA work Bank report on PSIAs outlines some design going, and early PSIA work contributed to a body features of an incremental approach (ODI and of knowledge on poverty and social impact that World Bank 2009). otherwise would not have materialized. Informing Bank operations and thinking and On the negative side, earmarked funding has practice across the Bank sometimes resulted in supply-driven analysis as researchers accessed the earmarked funds A clear understanding of the PSIA approach and its for projects that are not necessarily focused value is needed to obtain support for this analytic on assessing the poverty and social effects of work and to ensure the necessary allocation of specific policy reforms but that are more general Bank resources for it. background analyses. Furthermore, the availabil- The lack of clarity among Bank staff about what a ity of earmarked funds may have postponed PSIA is can create problems for quality assurance consideration by senior management of whether and monitoring of the quantity, quality, and and how to sustain such analysis with resources effectiveness of PSIAs. Ambiguity can also hinder from the Bank's budget. By using earmarked trust the appropriate allocation of funds for relevant funds, PSIAs are kept outside of mainstream Bank analysis. work and could, therefore, be excluded from the 38 L E S S O N S A N D R E C O M M E N D AT I O N S work program agreements at the beginning of and tailor the approach to those objec- each year, with possible adverse effects on PSIA tives, ensuring that the concept note-- relevance to the country program. · Contains a clear statement of the opera- tional objectives of the PSIA with respect There could also be issues of ownership, as noted to the intended effect (not just the topics/ by one interviewee: "Because there was relatively issues to be analyzed) minor input from [the Bank's] country staff and · Indicates how its approach--in particular, the funding came from a trust fund instead of stakeholder engagement, team composi- the Regional budget, there was a lack of Bank tion, partner institutions, budget, and time ownership of this PSIA." As the Bank prepares to frame--has been tailored to meet the op- obtain and execute the Multi-Donor Trust Fund erational objectives and provides the ratio- currently under negotiation, it will be important nale for the choices made to identify how the possible negative effects of · Shows how any tensions and trade-offs earmarking can be avoided. among the operational objectives will be reconciled Recommendations · Discusses if the intended dissemination au- This evaluation makes four recommendations dience and strategy are consistent with the to strengthen the Bank's work using the PSIA stated operational objectives. approach, whether done as a freestanding · Improve integration of the PSIA into the analysis or embedded in other analytical work. Bank's country assistance program by-- · Shifting significant decision-making and · Ensure that staff understand what the funding authority to the Regional Vice PSIA approach is and when to use it. Bank Presidencies to ensure that the PSIA top- management can do this by providing ics, scope, and approach are consistent clear guidance (perhaps through updat- with the country assistance program and ing of the 2008 PSIA Good Practice Note) that PSIAs ask questions that are relevant and actively disseminating this guidance, to policy particularly on-- · Requiring that all earmarked funding for · Whether and how the PSIA approach differs PSIAs be matched by a substantial contribu- from other distributional analyses, includ- tion from the country unit budget. ing whether the inclusion of the word "so- · Strengthen PSIA effectiveness through en- cial" in Poverty and Social Impact Analysis hanced quality assurance, including-- suggests the need to include a different · Subjecting PSIAs to systematic review by type of analysis Regional management at concept and com- · Whether or not PSIAs should be linked pletion stages to ensure relevance and fit of to specific reforms and identify beneficia- the PSIA to the country assistance program ries and those adversely affected by the as well as consistency of the proposed ap- reform proach with operational objectives, in ad- · What criteria should be used to determine dition to ensuring technical quality when the PSIA approach is appropriate · Ensuring that the Bank establishes a moni- for a particular operation in a country toring and self-evaluation system designed program. to assess whether PSIAs are being under- · Clarify the operational objectives of each taken where appropriate and are achieving PSIA with regard to its intended effect, their stated operational objectives. 39 Three Indian women. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. 40 Appendixes 41 Colombian children. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. APPENDIX A: PSIA TIMELINE 1987: The World Bank's Operational Guidelines practical changes in policy making that could required president's reports supporting lead to significant improvements in people's structural adjustment loans to "pay particular lives.1 attention to ... an analysis of the short-term impact of the adjustment program on the urban 2000: A letter from Oxfam International and rural poor, and measures proposed to allevi- addressed to International Monetary Fund (IMF) ate negative effects" (World Bank 1987). Managing Director Kohler and World Bank President Wolfensohn expressed concern that 1987­92: Jointly executed by the World Bank, the Fund and Bank had not met the commit- the African Development Bank, and the United ments (made by their respective boards some Nations Development Programme, the Social 15 months earlier in the context of developing Dimensions of Adjustment in Africa Program Poverty Reduction Strategy Papers [PRSPs]) to aimed at helping participating African countries assess the impact of proposed reforms in Fund/ integrate poverty and social concerns in the Bank programs before undertaking reforms (ex design and implementation of their adjust- ante impact assessments). ment programs to mitigate the burden on the poor in the process of structural adjustment. 2001: The Joint (IMF and World Bank) Implemen- The program's mandate was to strengthen tation Committee for PRSPs acknowledged the the capacity of African governments to design gaps in the analysis of policy impacts within client appropriate programs and projects in this countries and asked the Bank to take the techni- regard. It also aimed at strengthening the cal lead in helping developing countries fill this analytical capacity of governments to carry analytical gap. out empirical studies to assess the evolution of socioeconomic conditions of population 2002: Poverty and Social Impact Analyses (PSIAs) groups over time. were formally launched within the Bank in fiscal 2002 when a PSIA internal concept note was 1996: The World Bank, in conjunction with written. The concept note stated PSIA objectives national governments and a worldwide network as to continually inform policy dialogue, choice, of almost 1,000 civil society organizations, and implementation, within the overarching launched the Structural Adjustment Participa- objective of promoting sustainable poverty tory Review Initiative to assess the economic and reduction and social inclusion. social impact of structural adjustment policies on various social groups in the borrowing 2002: The World Bank and the Department countries. The goal was to improve understand- for International Development (DFID) initiated ing about the impact of adjustment policies PSIA pilots (and completed six each over the next and how the participation of civil society can two-year period). enhance economic policy making. The initiative was to be both a review of past experience and 2003: The donor network for PSIA was established a forward-looking exercise designed to identify by a range of multilateral and bilateral donor 43 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R agencies, based on the North Sea Manifesto. The resources and focuses more on specific projects network's objective was to support country-led, and programs than on major policy reforms. participatory, and evidence-based policy making for poverty reduction. 2007: Oxfam met with President Zoellick and requested that PSIA be strengthened and used as 2004: The IMF formally set up a PSIA group of consensus-building tools to carry out analysis and five experts within an existing division in its Fiscal promote pro-poor policies. According to Oxfam, Affairs Department. PSIA could be mechanisms to not only carry out useful diagnostic studies on the social impacts of 2004: The World Bank put International Bank poverty, but to help develop policy agreements for Reconstruction and Development surpluses between governments and the Bank, as well as to into a PSIA Incremental Fund of $5.8 million. ensure civil society participation in the formula- tion of poverty reduction efforts. This could lead 2004: The World Bank adopted Operational to greater policy choices being considered and Policy 8.60 on Development Policy Loans (DPLs) greater country ownership. Oxfam expressed and the accompanying Good Practice Note, concerns that (i) the Bank does not do enough "Using Poverty and Social Impact Analysis to PSIAs; (ii) it does not do enough to build country Support Development Policy Operations." capacity and to foster country ownership; (iii) there is insufficient disclosure of PSIA results by the 2006: Oxfam wrote to President Wolfow- Bank; and (iv) PSIAs focus too much on mitigating itz requesting that multidisciplinary PSIA be the impacts of a predetermined reform and not conducted in good time and that analyses enough on analyzing alternative policy options. consider a range of options, use local research- ers, be made public, and be monitored. It 2007: The World Bank, DFID, and the German also requested that while this was happening Agency for Technical Cooperation produced in some cases, it happen in most cases going "Tools for Institutional, Political, and Social forward. Analysis of Policy Reform: A Sourcebook for Development Practitioners" (World Bank 2007: Efforts of the 2003 donor network for 2007g). PSIAs were integrated with the Task Team on Ex Ante Poverty Impact Assessments of the Organi- 2008: The Bank updated the Good Practice sation for Economic Co-operation and Develop- Note "Using Poverty and Social Impact Analysis ment's Development Assistance Committee's to Support Development Policy Operations." Network on Poverty Reduction. Task teams have a two-year span, so this team will come to a close 2008: A fresh round of dedicated PSIA Trust soon. Funds came under consideration--a new Multi- Donor Trust Fund for PSIAs is being discussed 2007: The Development Assistance Committee among the Bank, DFID, GTZ, and some other approved the practical guide to ex ante Poverty donors to support country-led PSIAs with an Impact Assessment, a "light" version of PSIA that emphasis on developing country capacity for is less demanding in terms of time and financial policy analysis. 44 APPENDIX B: UNIVERSE OF BANK-FUNDED PSIAs1 Fiscal year Country Reform Sector classification of funding 1 Albania Pensions Social Protection 2002 2 Albania Energy tariffs Energy and Mining 2002 3 Albania Decentralization and priv. of water Water Supply and Sanitation 2003­04 4 Albania Education Education 2004 5 Albania Irrigation and drainage Rural Sector 2002 6 Angola Fuel and utility Energy and Mining 2004 7 Argentina Heads of Households Transition Project Social Protection 2006 8 Armenia Utility pricing and the poor Energy and Mining 2002 9 Azerbaijan Education Education 2005 10 Azerbaijan State-owned enterprises Public Sector Governance 2004­05 11 Azerbaijan Utility pricing Multisectoral 2004 12 Bangladesh Social protection Social Protection 2004 13 Bangladesh Chittagong Port Transport 2004 14 Benin Cotton Rural Sector 2003 15 Bolivia Teacher salaries Public Sector Governance 2004 16 Bolivia Public expenditure Multisectoral 2004 17 Bolivia Pension Public Sector Governance 2004 18 Bolivia Fuel and liquefied petroleum gas prices Energy and Mining 2004 19 Bosnia-Herzegovina State-owned enterprises Public Sector Governance 2005 20 Botswana Livestock Rural Sector 2005 21 Brazil Urban reforms Urban Development 2005 22 Burkina Faso Growth inequality linkages Economic Policy 2004 23 Burkina Faso Cotton sector Economic Policy 2002 24 Burkina Faso, Benin, Poverty Analysis Macroeconomic Simulator Economic Policy 2005 Guinea, Mauritania modeling 25 Burundi Coffee liberalization Rural Sector 2005 26 Cambodia Rice tariffs Economic Policy/Trade 2002 27 Cambodia School subsidies Education 2005 28 Cambodia Social concessions of land Social Development 2004 (continued on next page) 45 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Fiscal year Country Reform Sector classification of funding 29 Cameroon Overall public expenditure program Economic Policy 2003 30 Cameroon Health, human capital, and prospects for Health, Nutrition and Population 2006 inclusive economic growth for Cameroon 31 Cape Verde VAT Economic Policy 2005 32 Cape Verde Utility tariffs Multisectoral 2005 33 Cape Verde Targeting of social programs Social Protection 2006 34 Chad Cotton Rural Sector 2003 35 China Urban Dibao in China: Building on Success Social Protection 2005 36 Congo, DR Forestry Rural Sector 2005 37 Congo, DR Mining sector Energy and Mining 2004 38 Congo, DR Public sector reform for basic service Public Sector Governance 2004 provision 39 Côte d'Ivoire Public expenditure prioritization Economic Policy 2004 40 Côte d'Ivoire Trade of cocoa and coffee Economic Policy/Trade 2005 41 Croatia Enterprise restructuring Private Sector Development 2006 42 Croatia Design of social impact assessment methodology Multisectoral 2006 (country-systems approach to PSIA) 43 Djibouti Energy sector Energy and Mining 2004 44 Dominica Public sector Public Sector Governance 2003 45 Ecuador Safety net restructuring Social Protection 2005 46 Egypt Social policy Social Protection 2006 47 El Salvador CAFTA Economic Policy/Trade 2005 48 Eritrea Electricity Energy and Mining 2005 49 Ethiopia Road construction impact Transport 2004 50 Ethiopia Employment Creation Effects of the Addis Ababa Urban Development 2006 Integrated Housing Programme 51 Ethiopia Public expenditure prioritization Transport 2004 52 Ethiopia Distributional aspects of service delivery Health, Nutrition, and Population 2007 53 Ethiopia Health care financing study (distributional Health, Nutrition, and 2006 and health impacts of health sector Population development plan interventions) 54 Ethiopia Enhancing human development outcomes Public Sector Governance 2006 through decentralized service delivery 55 Ethiopia Electricity tariffs Energy and Mining 2006 56 Georgia Utilities (water, gas, and electricity tariff Multisectoral 2002 increases) 57 Ghana Energy pricing and subsidies Energy and Mining 2004 58 Guatemala Fiscal reform Economic Policy 2005 59 Guatemala CAFTA Economic Policy/Trade 2006 46 APPENDIX B: UNIVERSE OF BANK-FUNDED PSIAs Fiscal year Country Reform Sector classification of funding 60 Guyana Water sector Water Supply and Sanitation 2002 61 Guyana Bauxite Energy and Mining 2003­04 62 Guyana Sugar sector Living Standards Measurement Rural Sector 2004­05 Study 63 Honduras Minimum wage policy (proposal was on civil Public Sector Governance 2004­05 service reform) 64 Honduras CAFTA Economic Policy 2003­04 65 Honduras Fiscal analysis/taxes Multisectoral 2003/04/05 66 India Agricultural marketing reform impact on India's Rural Sector 2006 farmers 67 Indonesia Macro reforms Economic Policy 2003 68 Indonesia Local government Economic Policy 2005 69 Indonesia Social protection Social Protection 2004 70 Indonesia Fuel subsidy Energy and Mining 2004 71 Indonesia Rice tariff Economic Policy/Trade 2002 72 Indonesia Severance pay Economic Policy 2007 73 Kenya Commodities markets Public Sector Governance 2006 74 Kenya Improving targeting of public expenditure to the Economic Policy 2006 poor 75 Kenya Pension Social Protection 2005 76 Kenya Health user fees and waivers Health, Nutrition, and Population 2005 77 Kenya Maize prices Rural Sector 2005 78 Kosovo State-owned enterprises Public Sector Governance 2005 79 Kyrgyz Republic Electricity Energy and Mining 2006 80 Lao PDR Analysis of the distributional impact of public Economic Policy 2006 expenditures, with a focus on ethnic minorities and women 81 Lao PDR Public expenditure and revenue management and Economic Policy 2004 increase in utility tariffs 82 Lesotho Rural electrification Energy and Mining 2005 83 Lesotho Electricity sector Energy and Mining 2004 84 Madagascar Service delivery--Institutional analysis Public Sector Governance 2003 85 Madagascar Health care and the poor Health, Nutrition, and Population 2006 86 Madagascar Agriculture (rice, fertilizer, infra, Rural Sector 2003 transaction, extension) 87 Madagascar Enhancing land tenure security Rural Sector 2006 88 Madagascar Welfare impact of higher energy prices in Energy and Mining 2006 Madagascar 89 Malawi Electricity Energy and Mining 2005 (continued on next page) 47 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Fiscal year Country Reform Sector classification of funding 90 Malawi Tobacco marketing Rural Sector 2004 91 Malawi Agriculture market closures Rural Sector 2006 92 Mali Cotton sector Rural Sector 2004 93 Mauritania Oil and gas impact on the economy Energy and Mining 2006 94 Mauritania Mining services Energy and Mining 2004 95 Mauritania Water Water Supply and Sanitation 2002 96 Mauritania Health Health, Nutrition, and Population 2002 97 Mexico Implementation of water and forestry policies in Environment 2006 Mexico 98 Moldova Energy sector Energy and Mining 2004 99 Moldova Energy price changes Energy and Mining 2006 100 Mongolia Energy sector Energy and Mining 2004 101 Mongolia Cashmere (trade) Economic Policy/Trade 2004 102 Mongolia School year Education 2005 103 Montenegro Water and sanitation Water Supply and Sanitation 2005 104 Montenegro Poverty and environmental impacts of Energy and Mining 2007 electricity price 105 Morocco Housing (slum upgrading) Urban Development 2005 106 Morocco Water sector Water Supply and Sanitation 2006 107 Morocco Water supply and sanitation (Phase 2); Water Supply and Sanitation 2007 tariff reforms supported by water Development Policy Lendinga 108 Mozambique Primary school fees reduction Education 2004 109 Mozambique Labor market Public Sector Governance 2006 110 Namibia Extension and application of the Poverty Analysis Economic Policy 2006 Macroeconomic Simulator framework to Namibia 111 Nepal Fuel prices Economic Policy 2005 112 Nepal Infrastructure Transport 2005 113 Nicaragua Public investment program (proposal was written Public Sector Governance 2004 for public sector management reform) 114 Nicaragua CAFTA Economic Policy/Trade 2003 115 Nicaragua Education for All Education 2003 116 Nicaragua Fiscal reform Economic Policy 2003­04 117 Nicaragua Water Energy and Mining 2005 118 Niger Energy sector privatization Energy and Mining 2004 119 Pakistan Safety net review Social Protection 2005 120 Pakistan Education Education 2005 121 Romania Mining Energy and Mining 2005 48 APPENDIX B: UNIVERSE OF BANK-FUNDED PSIAs Fiscal year Country Reform Sector classification of funding 122 Rwanda Tea sector privatization Rural Sector 2003­04 123 Rwanda Poverty Analysis Macroeconomic Simulator Economic Policy 2005 modeling 124 Senegal Groundnut sector liberalization Rural Sector 2004 125 Serbia Welfare and labor impacts of mining sector Energy and Mining 2005 126 Serbia Gender analysis of pension reform Social Protection 2006 scenarios 127 Sierra Leone Mining sector Energy and Mining 2005­06 128 Sierra Leone Minimum wage policy Economic Policy 2006 129 Slovakia Distributional impact of reform of tax and benefit Economic Policy 2007 system 130 Sri Lanka Health expenditure business impact analysis Health, Nutrition, and Population 2003 131 Sri Lanka Agricultural trade policy and poverty reduction Rural Sector 2006 132 Sri Lanka Land Rural Sector 2005 133 Sri Lanka Welfare--cash transfer scheme Social Protection 2002 134 Sudan Increased public spending on health and Health, Nutrition, and Population 2006 education services and utilization by the poor in Northern Sudan 135 Sudan Oil price increase and principles and options for Energy and Mining 2007 compensation 136 Tajikistan Energy Energy and Mining 2006 137 Tajikistan Cotton farmland privatization Rural Sector 2004 138 Tanzania Local revenue Public Sector Governance 2005­06 139 Tanzania Crop boards Rural Sector 2004 140 Tanzania Low electricity tariffs for the rich Energy and Mining 2007 141 Timor-Leste Policy note on safety nets Social Protection 2007 142 Turkey Labor market Public Sector Governance 2003 143 Turkey Social security Social Protection 2004 144 Uganda Land policy in northern Uganda and Rural Sector 2006 implications for resettlement and recovery 145 Uganda, Tanzania, Local government tax reform in Uganda, Economic Policy 2006 Kenya Tanzania, and Kenya 146 Ukraine Fiscal decentralization Public Sector Governance 2005 147 Ukraine Energy sector impact Energy and Mining 2004 148 Ukraine Health and education Health, Nutrition, and Population 2005 149 Uruguay Participatory monitoring (multisectoral Public Sector Governance 2006 Development Policy Lending) 150 Venezuela, R. B. de Poverty and social impact of the Misiones Social Development 2006 Initiative in Venezuela (continued on next page) 49 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Fiscal year Country Reform Sector classification of funding 151 Vietnam Land reform Public Sector Governance 2005 152 Vietnam World Trade Organization accession Economic Policy/Trade 2004 153 Vietnam Labor market impact of state-owned Public Sector Governance 2002 enterprise 154 Yemen, Rep. of Energy sector analysis ­ social Energy and Mining 2003 assessment/training 155 Yemen, Rep. of Water sector Water Supply and Sanitation 2006 156 Zambia Land reform/titling; removal/reduction Rural Sector 2004 of fertilizer subsidy; rural roads Sources: World Bank SDN and PREM Anchors. Note: Bold entries are the 58 randomly sampled PSIAs. This list does not include the initial group of pilot PSIAs. CAFTA = Central America Free Trade Agreement; PSIA = poverty and social impact analysis. a. As no documents were available for the then ongoing phase II of the Water Supply and Sanitation PSIA, the Phase I PSIA was assessed in the portfolio review instead. 50 APPENDIX C: METHODOLOGY This appendix contains a description of the fulfilling the Ten Elements of Good Practice methodology for each of the six evaluative instru- PSIAs, as identified on the Bank's PSIA Web site ments used in this study: the portfolio review, and in the User's Guide. More specifically, the country case reviews, semistructured interviews information gathered was the following. with country stakeholders and Bank staff, interviews with senior Bank staff and managers, Type a thematic review of donor involvement in PSIAs, · PSIA document date and start date and a literature review. · Description of focus of PSIA · Was a specific policy reform analyzed? If not, I. Portfolio Review then what type of study was it? (for example, To conduct a detailed desk review of PSIAs, a background work) stratified random sample of the total universe · Was the PSIA a stand-alone piece or chapter/ was taken. Starting with the 156 PSIAs provided section of another report? by the Bank's Poverty Reduction and Economic · Was the PSIA multisectoral? If so, what other Management and Sustainable Development sectors were involved? Network Anchors, and removing 16 PSIAs whose status was incomplete, a stratified, Inputs statistically representative, random sample · What activities were undertaken in the prepa- of 58 PSIAs was taken of the remaining 140. ration stages of the PSIA? Stratification was first by funding year then · Was there a peer review of the PSIA concept by network. The sample is significant at a 95 note? percent confidence interval at ±10 percent · Was there a peer review of previous PSIA margin of error. drafts? · Were there comments from other stakehold- Information was then collected on these ers on previous drafts of the PSIA? 58 PSIAs based on several factors: (i) the · What was the skills mix of the study team? PSIA report or document itself (in some · What was the study team's budget? cases this was a chapter in a Bank economic · How much time was allocated for the PSIA? and sector work [ESW] such as a Poverty · Composition of study team authors Assessment); (ii) any funding proposals · Composition of overall study team or progress reports available for the PSIA; (iii) a questionnaire sent to all task managers of Element 1: Asking the Right Questions the 58 PSIAs, for which there were responses for · Why did the study team pick the topic for 37 PSIAs; (iv) project documents for PSIA-linked the PSIA? operations; (v) information recorded in the · What were the PSIA's stated objectives? PSIA's file on Project Portal, when available; and · What was the timing of the PSIA regarding (vi) any further relevant information provided by individual Bank operations? anchor staff. The information collected related · What was the timing of the PSIA in regard to to basic data on the PSIAs and their record in policy implementation in the country? 51 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Element 2: Identifying Stakeholders · If there were data limitations, how did the PSIA · Did the PSIA identify the appropriate range of team overcome them to still analyze the reform's stakeholders who might be either beneficiaries impact (collect more data, adapt the analytical or adversely impacted by the policy reform(s) approach, use "off-the-shelf " survey instruments, supported by the PSIA? recommend a pilot study, and so forth)? · Were the distributional impacts on women · If key data limitations existed, did the PSIA and other traditionally vulnerable populations recommend a strategy to overcome them for explicitly discussed? If not, was a justification future analyses of the poverty and social impact offered for their exclusion? of the reform policy? · Were these stakeholders actually consulted during the course of the study? Element 6: Analyzing Inputs · Given the nature of the reform, was the ap- · What quantitative tools were used in the propriate range of stakeholders who would study? affect the success of the policy implementa- · What qualitative tools were used in the tion (that is, act as proponents or opponents study? of the reform) identified? · If high indirect impacts were expected, were · Were these stakeholders actually consulted the tools appropriate for capturing those during the course of the study? impacts? Element 3: Understanding Transmission Channels Element 7: Enhancement and Compensatory Measures · What transmission channels of the proposed · Did the PSIA reveal that the proposed reform reform (that is, employment, prices, access to would have or did have adverse consequences goods and services, assets, and transfers and for the poor or other groups? taxes) did the PSIA discuss? · What were the policy recommendations? · Did the PSIA highlight whether the impacts · Were alternative policy options presented? through these transmission channels would · Did the PSIA suggest altering the design of the occur directly or indirectly? policy to include complementary measures · Did the PSIA specify whether the impact on that would enhance the positive effects of the stakeholders will differ between the short run reform for the poor? and the long run? · Did the PSIA suggest altering the design of the policy to include complementary measures Element 4: Assessing Institutions that would mitigate the negative effects of the · Did the PSIA consider how the structure of ex- reform for the poor? isting market and/or social institutions would · Did the PSIA suggest a compensatory mecha- mediate the impact of the proposed policy nism for losers of the reform? reform? · If so, did the study discuss ways to design these · Did the PSIA identify all the main agencies/ mechanisms to ensure appropriate targeting of institutions that are responsible for imple- beneficiaries and to avoid distorting incentives menting the policy reform? If so, what spe- that would compromise the reform's imple- cific aspects of these agencies/institutions mentation? were assessed (for example, responsibilities, · Did the PSIA recommend suspending the re- incentives, capacity, accountability flows, or form or delaying its implementation until a resource flows)? later period? · Was there any prioritization of the policy rec- Element 5: Gathering Data and Information ommendations? · Was the PSIA's data-collection approach most appropriate, given the nature of the reform Element 8: Assessing Risks being analyzed and the analytical tools being · Did the PSIA take into account risks to the re- employed? form program and/or risks emerging from the 52 APPENDIX C: METHODOLOGY impact of the reform (for example, institutional · Was the PSIA intended to inform a Country As- risks, political economy risks, exogenous risks, sistance Strategy (CAS)? If so, which one? or other country risks)? · Did the CAS refer to the findings/recommenda- · If so, did the PSIA incorporate these risks into its tions of the PSIA? discussion of policy choices and policy design? · Was the PSIA intended to inform an investment loan? If so, which one? Element 9: Monitoring and Evaluation · Did the investment loan document refer to the · Did the PSIA suggest key indicators that should findings/recommendations of the PSIA? be collected ex ante to monitor and evaluate · Was the PSIA intended to inform an ESW? If the reform's progress? so, which one? · Was a time frame suggested for how frequently · Did the ESW refer to the findings/recommen- these indicators should be updated? dations of the PSIA? · If key information was deemed necessary for · Was the PSIA intended to inform an additional monitoring and evaluating impacts but was un- ESW? If so, which one? available at the time of the study, did the PSIA · Did the ESW refer to the findings/recommen- offer suggestions for how to fill this knowledge dations of the PSIA? gap? · Was the PSIA intended to inform any other op- · Did the PSIA offer any recommendations for erations/loans/strategies? If so, which ones? conducting a process evaluation to understand · Did these additional operations/loans/strate- how and why the policy reform had a particular gies/documents refer to the findings/recom- outcome? mendations of the PSIA? · Did the PSIA suggest ways to implement a · Did Country Assistance Evaluations from the monitoring and evaluation system? Independent Evaluation Group (IEG) mention the impact of the PSIAs? Element 10: Fostering Policy Debate and Feeding · Did the IEG CAS Completion Report mention Back into Policy Choice impact of the PSIAs? · When, how, and to whom was the PSIA dis- seminated? Impact: Country policies · If another language besides English was widely · Was the PSIA intended to inform a PRSP? spoken in the country, was the PSIA translated · If so, did the PRSP reflect the findings/recom- into that language? mendations of the PSIA? · How well organized and written was the · Even if the PSIA and supplementary docu- study? ments did not indicate that the PSIA was to · How operationally oriented were the findings , inform a PRSP was there a PRSP available (as opposed to being too academic/research that it could inform? If so, which one? (This oriented)? would only refer to PRSPs that were concur- · Did the PSIA discuss how lessons learned from rent or subsequent to the start date of the implementing and evaluating the reform could PSIAs.) be integrated into the policy process? · If there was a PRSP that the PSIA could inform, did that PRSP reflect the findings/recommenda- Impact: Bank operations tions of the PSIA? · Was the PSIA intended to inform a PRSC? If so, · Was the PSIA intended to inform any other which one? country policies? · Did the PRSC refer to the findings/recommen- · If so, did the policy document reflect the find- dations of the PSIA? ings/recommendations of the PSIA? · Was the PSIA intended to inform another DPL? · Was the PSIA focus country included in any IEG If so, which one? sector or thematic studies? · Did the DPL document refer to the findings/ · If so, what did the study say about the impact recommendations of the PSIA? of the PSIA? 53 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R II. Country Case Reviews Were other donors active on the same Eight countries were chosen purposively for issue? Regional, sectoral, and fiscal year coverage: Ban- · The roles of the Bank and the government gladesh, Cambodia, Ghana, Malawi, Mali, Mozam- in initiating and defining the focus of the bique, Nicaragua, and Zambia. Within these PSIA. What were the main motives of each? countries, 12 PSIAs were examined in depth (see (For instance, was the PSIA supposed to feed table C.1). The country case reviews involved vis- into a PRSP or into other analytic or policy its to the countries concerned, except to Cam- choice exercises in the Bank and/or the gov- bodia and Mali. In Cambodia and Mali, extensive ernment?) interviewing of stakeholders was undertaken, sig- · When and how was the PSIA carried out? How nificantly more than for the semistructured inter- was responsibility divided/shared between views with country stakeholders and Bank staff. government and the World Bank? Was there much/some/little consultation within the gov- The country case reviews included an analysis of ernment, with nongovernment stakeholders, the effect of the PSIA on country policies and Bank or with other donors (if appropriate)? operations and the contribution of the PSIA to · What were the main findings and recommen- country capacity for policy analysis. In countries dations of the PSIA? Did it analyze policy al- where more than one PSIA was examined, the ternatives? Did it identify risks of proposed effect of each PSIA was reported on separately. To policies? Did it suggest ways to buffer negative assess these three topics, the country case review impacts and enhance positive ones? examined the following elements. Effect on country policies Process and content of the PSIA · How much and what kind of effect did the PSIA · What was the policy issue or problem that have on government policies and actions and the PSIA addressed? Why was it important? on the in-country policy debate? What is the Table C.1: PSIAs Covered by the Country Case Reviews Fiscal year of Country Reform Sector classification funding Bangladesh Chittagong Port Transport 2004 Cambodia Rice tariffs Economic Policy/Trade 2002 Cambodia Social concessions of land reform Social Development 2004 Ghana Energy pricing and subsidies Energy and Mining 2004 Malawi Tobacco marketing Rural Sector 2004 Malawi Agriculture market closures Rural Sector 2006 Mali Cotton sector Rural Sector 2004 Mozambique Primary school fees reduction Education 2004 Mozambique Labor market Public Sector Governance 2006 Nicaragua Fiscal reform Economic Policy 2003­04 Nicaragua Water Energy and Mining 2005 Zambia Land reform/titling; removal/reduction of fertilizer subsidy; rural roads Rural Sector 2004 Source: IEG. 54 APPENDIX C: METHODOLOGY evidence for this? If there had been no PSIA, academics and researchers, 2 officials from donor what difference would that have made? agencies, and 19 Bank staff) chosen purposively · Explanations for PSIA effect or lack thereof on to include stakeholders familiar with the PSIA the government. process. One stakeholder was interviewed on · Good/bad practice examples and lessons two PSIAs, resulting in a total of 48 stakeholder learned. interviews. These interviews covered 11 PSIAs in 10 countries (and were additional to the ones Effect on the World Bank covered in the country case reviews). The 11 · How much and what kind of effect did the PSIA PSIAs are listed in table C.2. have on World Bank decisions and operations? What is the evidence of this? The topics covered in the interviews were as · If there had been no PSIA, what difference follows: would that have made? · Explanations for PSIA effect or lack thereof on · Respondent's profile the World Bank. · PSIA content and process · Good/bad practice examples and lessons · Initiation and choice of the PSIA topic learned. · Timing · Soundness of conclusions Contribution to country capacity · Participation · How much and what kinds of impact did the · Dissemination PSIA have on the capacity of government agen- · Donor involvement cies with regard to collecting and analyzing · Effect on country policies/programs and on data on poverty, distributive impact, or other in-country analytical capacity social effects of policies? Was there any impact · Accountability on nongovernmental agencies, such as think · Lesson learning tanks or universities? · Effect on World Bank · Did the PSIA have any impact on consultation · Accountability and coordination among government agencies · Lesson learning or with nongovernmental organizations? · Explanations for PSIA impact or lack thereof The results of these interviews are illustrative on capacity. only and are not considered statistically · Good/bad practice examples and lessons representative. learned. IV. Interviews of Senior Bank Staff Those interviewed were government officials and Managers in ministries and agencies responsible for Semistructured interviews were conducted with the PSIA or within whose sector the PSIA fell, 30 senior Bank staff and managers (for example, members of civil society organizations involved country directors or advisors) chosen purposively in the sector or subject matter covered by the for their knowledge of PSIAs. PSIA, and staff of the World Bank and other donor agencies. The topics discussed included the following: III. Semistructured Stakeholder · Respondent involvement with PSIA Interviews with Country · Prominence of PSIAs in the Bank's Country Stakeholders and Bank Staff program/dialogue Semistructured telephone interviews were · Choice of PSIA topics conducted with 47 stakeholders (9 govern- · Overall PSIA effectiveness ment officials, 4 private sector representatives, · PSIA work since termination of the Incremental 2 nongovernmental organization [NGO] staff, 11 Fund 55 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Table C.2: PSIAs Covered by the Semistructured Interviews Country Short-hand name Full PSIA title Date of document Albania Irrigation and drainage reform PSIA of the Irrigation and Drainage Rehabilitation projects and 2004 the Water Resource Management Project in Albania Armenia Utility pricing and the poor Utility Pricing and the Poor: Lessons from Armenia 2001 Benin Cotton reform Cotton Sector Reforms: A Poverty and Social Impact Analysis 2004 Egypt Social policy reform Egypt--Toward a More Effective Social Policy: Subsidies and 2005 Social Safety Nets Indonesia Social protection reform Chapter 6 of Indonesia Poverty Assessment: "Making Social 2008 Protection Work for the Poor" Indonesia Macro reforms Examining the Social Impact of the Indonesian Financial Crisis 2002 using a Macro-Macro Model Moldova Energy sector reform Sharing Power: Lessons learned from the Reform and 2004 Privatization of Moldova's Electricity Sector--Poverty and Social Impact Analysis Mongolia Cashmere (trade) From Goats to Coats: Institutional Reforming Mongolia's 2003 cashmere sector Nepal Fuel prices Nepal Poverty Assessment Background Paper: Socio Economic 2005 Impact of Fuel Prices in Nepal Sierra Leone Minimum wage policy Review of Civil Service Minimum Wage and Senior Executive 2006 (draft) Service Options Sri Lanka Land reform Poverty and Social Impact Analysis (PSIA) of Sri Lanka's Land 2007 Policy Reforms, Socio-Economic Impact Assessment Final Report Source: World Bank. · World Bank incentives for PSIA agency staff, World Bank staff, and other PSIA · Long-term benefits from PSIA chapter in the stakeholders about their experience with donor Bank involvement in carrying out PSIAs. · The future of PSIA work. VI. Literature Review V. Thematic Review of Donor Involvement The literature review summarized findings of The thematic review of donor involvement was World Bank and other donor, NGO, and academic conducted by means of a desk review of litera- publications, policy notes, working papers, confer- ture on and by donors actively supporting and ence proceedings, and other relevant documents conducting PSIAs and by interviewing 15 donor analyzing PSIA origins and record to date. 56 APPENDIX D: DESCRIPTION OF PSIA EFFECTS-- COUNTRY CASE REVIEWS PSIA Effect on Country Policies The Bank has recently attempted to gauge the impact of the PSIA on government policy and Bangladesh--Chittagong Port has found some policy changes consistent with The Chittagong Port Authority (CPA) has the PSIA's findings. However, there is insufficient addressed some of the specific points from evidence to attribute these policy changes to the the PSIA that were incorporated in the January PSIA. Overall, this PSIA had a moderate effect on 2007 Bank Policy Note (World Bank 2007a). For country policies. instance, the CPA has taken actions to minimize the stuffing and unstuffing of containers and Cambodia--Rice Tariffs has increased container storage charges. There is no concrete evidence that the PSIA However, the measures taken to date do not (ACI 2002) had an effect on the government address the major recommendation of the PSIA of Cambodia's debates or policies. Although (World Bank 2005e), which was to undertake a the Poverty Reduction Strategy (World Bank more comprehensive refiguring of the manage- 2006c) mentions the need to increase agricul- ment structure of the port, which has not been tural productivity and improve access to inputs acted on. and credit and to improve infrastructure and the functioning of the financial system across There was lack of engagement with key govern- the board, none of these points is brought ment bodies. Although the PSIA team got up in the specific context of rice production permission to undertake the study from the or marketing, the topics of the PSIA; they are Ministry of Shipping and CPA, neither of those generic problems that are widely recognized in agencies participated in the design, analysis, or Cambodia. The agriculture section of the 2006 drafting of the PSIA. Moreover, because the Bank Poverty Reduction Strategy makes no mention decided not to actively disseminate the report, it of rice-specific reforms. was not sent to the Ministry of Shipping or the Chittagong Port Authority for comment until This lack of uptake is most likely due to (i) limited one year and nine months after it was ready. local ownership of the PSIA, (ii) limited involve- Senior and long-standing officials interviewed ment of local parties in analysis, (iii) the fact that in both organizations said they were not aware this PSIA was not linked to a particular program of such a study or its findings. Furthermore, the proposal or reform, and (iv) lack of in-country final report was also not shared at any stage with Bank staff, who championed the report after other stakeholders, including external partners, completion. Overall, this PSIA had a negligible who may have been able to use the findings in effect on country policies. some of their own interactions with the govern- ment.1 Transparency International, the Interna- Cambodia--Social Concessions of Land Reform tional Finance Corporation, and the Bangladesh Land reform was a key PRSP objective, and results Garment Manufacturers and Exporters Associa- from a PSIA on social land concession (World tion were some of the groups that were eager to Bank 2004a) provided critical analytical inputs see the PSIA, even three years later. that helped implement this national priority. In 57 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R particular, the PSIA sustained broad interest in of clear guidance in the legal framework for and momentum around the land issue. The state local decision makers. These institutions were land management subdecree of 2007 addressed strengthened in subsequent years, partly as many of the deficiencies of the prior land laws a result of the high profile the PSIA afforded and was implemented after completion of the these institutional weaknesses. The focus on PSIA, although several years later. According to institutions necessary for reform success as- Bank and other interviewees, the process toward sisted in stimulating further policy changes; creation of the subdecree was fostered by the prior to 2003, the government knew that addi- knowledge and consensus gained during the tional legislation was needed (such as the 2007 PSIA. This subdecree provided the necessary legal subdecree) and the PSIA provided concrete framework for identifying and managing land guidance about elements of this legislation. concessions at the local level. The PSIA identified The analysis also produced important infor- clear problems with the existing legal framework mation about complementary services and and sustained movement toward resolution of support needed to ensure that small-holder these problems. households would benefit following receipt of concession land. The government used this in- Furthermore, the PSIA helped build support formation in subsequent dialogue with donors within the government for a smallholder-based about projects needed to support the social agricultural development scheme. Prior to 2003, land concession program. The availability of many in government felt that efficiency in agricul- complementary services, such as technical as- ture and investments in the sector would best sistance, input supplies, marketing services, be stimulated by supporting large-scale agricul- and so forth represents an essential compo- tural development. The economic concession nent of reform sustainability. program was one manifestation of this belief. · The inclusiveness of the process (with govern- In contrast, by focusing on potential impacts ment, NGOs, and civil society participating) of smallholder development and identifying and the transparent, qualitative nature of the how smallholder agriculture could succeed, the analysis won the confidence of the govern- PSIA helped change this perception and built ment. Very little (if any) of the analysis relied subsequent support for a smallholder-based on abstract models or methods, and the sim- project, Land Allocation for Social and Economic plicity of the analysis stimulated acceptance. Development (LASED), which received Bank Stakeholder participation was broad and ac- financing. tive. The primary government partner, the Ministry of Land Management, Urban Planning, This effect is due to several factors: and Construction, was directly responsible for planning and implementing land policies; its · The central importance of land reform to the participation built a strong institutional base of Cambodian government contributed to the support within the government. The Ministry relevance of the PSIA and enabled its direct of Land Management, Urban Planning, and insertion into the national policy dialogue. The Construction was also engaged in the analysis ex ante nature of the analysis and its timing of available land and, from this experience, was relative to key policy decisions helped increase able to understand the multiple problems as- its relevance. sociated with identifying and quantifying such · The PSIA was very open about institutional land. First-hand experience was invaluable. weaknesses, particularly at the level of techni- This ministry was engaged from the very start cal support units for local land use allocation of the PSIA and continued its involvement committees. This information helped frame through the end and beyond. a clear path for successful implementation of the program. The PSIA also identified key Overall, this PSIA had a substantial effect on weaknesses in enabling legislation and a lack country policies. 58 APPENDIX D: DESCRIPTION OF PSIA EFFECTS ­ COUNTRY CASE REVIEWS Ghana--Energy Pricing and Subsidies the incentive system have not yet been realized. A holistic study of the energy sources of the poor Although this PSIA dealt with these important proposed by the PSIA (World Bank 2004e) has issues, a longer-term process of follow-up to not been undertaken. Although there have been the findings and a careful analysis of the politi- substantial improvements in access to electricity cal economy surrounding ADMARC would be supplies, most of these improvements are attrib- necessary to bring change. Overall, this PSIA had uted to the self-help program, which predated a negligible effect on country policies.4 the PSIA. Arrangements for compound house dwellers remain unchanged. The recommended Mali--Cotton Sector Reform educational programs were not undertaken. The PSIA work has not had much of an impact Bank management reported that after the on government policy for several reasons. PSIA was completed, the Ministry of Finance Because the government had already made its opened a budget line for the lifeline subsidy-- pricing decision in January 2005, when the PSIA although the PSIA team expected this measure was still in draft form, and because the PSIA to help ensure that this subsidy would be paid, it does not engage with specific issues around the recognized that this measure was not sufficient. privatization of the Malian Textile Development The recommended system of monitoring was Company, it is unclear what specific government not implemented. policy the study could have informed. Although the draft PSIA might have been used by the Bank In large part, the lack of uptake of the recommen- in its discussions with the government, a clear dations may be caused by a lack of clear consen- linkage between its findings and the govern- sus within the government about the importance ment's ultimate pricing decision on cotton is not of the issues covered in the PSIA. There was also evident.5 no natural counterpart on the Ghanaian side who would champion the report after its comple- A working paper primarily provides descriptive tion. Overall, this PSIA had a negligible effect on information and poverty estimates based on country policies.2 simulations and models that cannot easily be transformed into policy advice. The other outputs Malawi--Tobacco Marketing Reform that have been generated from the PSIA work, The tobacco PSIA (World Bank 2004h) advocated including the poverty maps and 2006 survey data, a wholesale overhaul of the legal and regula- highlight problems in the Sikasso region but stop tory framework governing the sector, but little short of identifying remedies and point to the or no action has been taken to achieve this. The need for complementary work. Furthermore, PSIA has not led to increased influence of the there has been a significant lag time between Malawian government or smallholder farmers conducting the analysis and writing up the report, in the trading of tobacco. The PSIA helped raise which is still not complete. The working paper awareness among farmers about price setting also has not been held to a deadline; it has now processes and the government's pricing policy been in the working stage for more than three for tobacco, but the policy was unable to have an years. Stakeholders have not received any draft impact on buyer practices. Overall, this PSIA had since May 2006. Overall, this PSIA work has had a a moderate effect on country policies.3 negligible effect on country policies.6 Malawi--Agriculture Market Closure Mozambique--Primary School Fee Reduction The analytical work contained in the PSIA (World The qualitative findings reported earlier were one Bank 2007c) has not had much effect on the of the factors that influenced the government to measures taken by the Malawian government. The abolish fees. The extensive information on supply- Malawi Agricultural Development and Marketing and demand-side factors influencing access as Corporation (ADMARC) has undergone some discussed in the PSIA (World Bank 2005h) has reorganization, but the proposed changes to shaped government thinking on education. Many 59 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R of its concerns regarding access to education and had already sent a draft law to Parliament. In by the most vulnerable members of society are addition, a new president, who owed his political echoed in the second strategic plan of education. support to organized labor, had just come into Through the PSIA, government officials learned office, and it was highly unlikely that a study would about the additional expenses parents pay when have influenced the government's direction with they send their children to school and the variation regard to labor law reform. However, one of in access across districts. In addition, the govern- the very few recommendations that the report ment has a greater understanding of gender and made was that if the government decided to education, as well as the particular challenges revise the labor law, it should consider transition that orphans face regarding education. provisions to ease the impact of the reforms on existing workers. The government adopted this The impact the draft PSIA had on the government recommendation, and the draft labor law was can in part be explained by two factors: govern- amended to include generous transition arrange- ment support or buy-in, and auspicious timing. ments for labor. The PSIA was undertaken in collaboration with the Ministry of Education and, in particular, the The PSIA provided limited actionable recom- National Institute for Educational Development, mendations, most likely due to the high level of which participated extensively in the design, controversy surrounding the reform, the rela- research, and analysis of the qualitative half of the tively late timing of the PSIA, and the involvement study. This inclusiveness, which was most likely of the Ministry of Planning and Development but facilitated by the Ministry of Education's commit- not the Ministry of Labor. Although the Ministry ment to universal primary education, enabled of Planning and Development did participate in uptake of the findings by government. the PSIA, the absence of the Ministry of Labor, under whose responsibility the new policy would The timing of the PSIA also facilitated uptake, fall, left a narrow channel for government uptake as the PSIA was being undertaken while the of the results. Overall, this PSIA had a moderate government was in the midst of developing an effect on country policies. Education Sector Strategic Program for 2005­10. Overall, this PSIA had a substantial effect on Nicaragua--Fiscal Reform country policies. As an ex post assessment, the PSIA (World Bank 2003a) came too late to inform policy formula- Mozambique--Labor Market Reform tion, and it is difficult to discern what effect, if any, The effect of the PSIA on government policy is it had on the government. Judging by interviews not clearly attributable to the PSIA (World Bank with government officials, legislators, other 2006h). Aside from the recommendation on Nicaraguans involved in the broader poverty- transitional provisions, there was little buy-in from reduction strategy process, and representatives the Ministry of Labor. The PSIA was undertaken at from the international donor community, the a time when a proposed reform to the labor law fiscal reform PSIA does not appear to have had was already under consideration by the Council much effect. Overall, this PSIA had a negligible of Ministers. The mandate of the PSIA was also effect on country policies. limited to comparing the previous and proposed labor laws, clarifying the major differences Nicaragua--Water Reform between the old policy and the new reform. It is Despite the centrality of access to water in likely that, regardless of the PSIA, the government the poverty reduction strategy and the Millen- would have passed the reform anyway. nium Development Goals, the water PSIA itself (World Bank 2005j) had little effect in govern- The government had already negotiated with the ment circles, either among those then in power private sector and the trade unions to change or among those who succeeded them. The the labor law, had already done its own research, ideological tide was just about to turn, and the 60 APPENDIX D: DESCRIPTION OF PSIA EFFECTS ­ COUNTRY CASE REVIEWS type of decentralization-cum-privatization of shaping government policy. The government management/ownership of the water works that had a proposed land law that the PSIA analyzed the PSIA advocated was totally at odds with the in detail; the government was preparing a tenor of the soon-to-be administration. The last transport policy at that time that the section on year of the administration was not a good time rural infrastructure could have contributed to; to tackle this controversial issue in Nicaragua. and the fertilizer support program had been in place several years. The PSIA provided a useful Moreover, the PSIA, having been conducted by assessment of the benefits and drawbacks of the a third-country consultant and lacking participa- program on which the government could have tory methods to engender domestic support, based a revised approach. left no domestic constituency to lobby for the reform when the new administration came into A number of reasons may explain why the report's power. Overall, this PSIA had a negligible effect findings were not more influential: on country policies. · Both the fertilizer program and the land issue Zambia--Land, Fertilizer, and Rural Roads are highly politicized. Since the final report The findings of the PSIA (World Bank 2005l) came out in 2005, just one year prior to a did not directly inform subsequent government general election, it is likely that the govern- policy on land, fertilizer, and rural infrastructure. ment preferred to maintain the status quo. The Fertilizer Support Program, the Food Se- Since the publication of the PSIA in 2005, the curity Pack, and maintaining customary land government has done little with regard to revising tenure practices were key instruments of the transport and land policies, although most policy to win support from rural chiefs and government officials demonstrated awareness residents in 2006. In fact, according to one that these are important issues. The government agricultural specialist in the World Bank, the has not implemented any new land or fertilizer Zambian government raised the fertilizer sub- initiatives; furthermore, it has not addressed the sidy level to 60 percent just before the 2006 defects in the fertilizer support program. In fact, election. This strategy was politically success- the abuses in the fertilizer support program have ful, as election results indicate that the ruling grown worse according to representatives of the party gained 43 percent of the vote in 2006 U.S. Agency for International Development, Civil from support from rural areas. Thus, fear of Society for Poverty Reduction, the European the consequences of changing agricultural Union, and the National Farmers' Union. subsidies or the land policy prior to the 2006 election help explain why the government did In contrast, the government did not proceed with not adopt the recommendations on fertilizer the draft land policy that the PSIA had criticized. and land in the PSIA. Though no government official referenced the · Since the PSIA was undertaken, copper prices PSIA's effect on this issue, the PSIA task team have soared, offering an alternative source of hypothesized that the government's inaction on revenue to agriculture. The government has the land issue may have been informed by the therefore begun to turn its attention to rene- report's findings. The PSIA's position on the draft gotiating contracts and managing the impact land policy echoes the views of many of the civil of copper revenue, as has the Bank. society organizations that pressure the govern- · According to many of the donors and civil so- ment. In that sense, it may have been one of the ciety organizations interviewed, key local and reasons the government did not proceed with national elites benefit from the subsidies in the the reform. fertilizer support program and from inaction on the land issue. At first glance, the context of the PSIA appears · In addition to the political climate and elite in- to have been very favorable with regard to terest in the status quo, it is likely that some of 61 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R the content of the findings and the process of 2004a) did not accurately measure land availabil- the report explain its limited effect. Regarding ity, any learning-by-doing is likely to have been land, the report's recommendations disagreed weak. Overall, this PSIA had a negligible effect with the government's policy position. Whereas on country capacity. the draft land policy proposed incorporating more land under state control, the PSIA sharply Ghana--Energy Pricing and Subsidies disagreed, noting the possible negative effects In the Kumasi Institute of Technology and on the authority of chiefs, the weak capacity Environment, which was contracted as the local of the state to grant titles, and the possible consultant for the PSIA (World Bank 2004e), negative impact on the poor if the state were considerable training inputs were provided to move to a system of individual titling. by external consultants and the Bank team, · There is also the issue of which ministry was and methods new to the Institute's personnel involved in the PSIA. Although the PSIA exam- were introduced. The Institute's staff benefited ined critical issues relating to rural agriculture, from the training in qualitative research, and the Ministry of Agriculture and Cooperatives competence was further enhanced through did not participate in or contribute to the PSIA. continuous discussions and reviews. Instead, the Ministry of Finance and National Planning was the government sponsor of the The Electricity Company of Ghana also benefited study, perhaps because the PSIA was prepared in terms of capacity building from the energy PSIA in association with the Country Economic (World Bank 2004e). It attached a member of its Memorandum. Because the Ministry of Agri- staff to the consultant teams that undertook the culture and Cooperatives did not contribute to survey on which much of the energy PSIA report the study beyond consultations with individual was based. As a result, the company learned project leaders, there was little ownership and a great deal about its customers, for example, little knowledge of the findings of the PSIA by regarding billing arrangements in compound this ministry. houses. Four years on, and despite nonimple- mentation of its recommendations, the company Overall, this PSIA had a moderate effect on continues to regard the energy PSIA as having country policies. had positive value, saying, "We now have the capacity and need to do more of such studies-- PSIA Contribution to Country Capacity but we don't have the funding." Bangladesh--Chittagong Port Overall, this PSIA had a substantial effect on The PSIA team did not involve any other stakehold- country capacity. ers in the design or preparation of the report. Overall, this PSIA (World Bank 2005e) had a Malawi--Tobacco Marketing Reform negligible effect on country capacity, either in the Three of the four papers that make up this government or outside. PSIA (World Bank 2004h) were written by local policy analysts, including analysts in the Cambodia--Rice Tariffs Centre for Social Research of the University of The Ministry of Agriculture, Forestry, and Fisher- Malawi and a Malawian consulting firm, O&M ies guided the PSIA team, but there was little Development Consulting Ltd. However, there enhancement of their own capacity as a result is little evidence that the Bank provided much of this PSIA (ACI 2002). Overall, this PSIA had a training or enhanced the existing capacity negligible effect on country capacity. of these agencies and, especially in the case of the value chain analysis in the PSIA, these Cambodia--Social Concessions of Land Reform technically more complicated components of Ministry staff participated in the analysis of the PSIA had less local input. Overall, this PSIA available land, but because the PSIA (World Bank had a negligible effect on country capacity. 62 APPENDIX D: DESCRIPTION OF PSIA EFFECTS ­ COUNTRY CASE REVIEWS Malawi--Agriculture Market Closure Mozambique--Primary School Fee Reduction The ADMARC PSIA (World Bank 2003f) was Partly as a result of the PSIA (World Bank 2005h), partially conducted by local experts. Three the Ministry of Education and Culture now background studies were prepared by staff of appreciates the importance of including both the Centre for Social Research of University of qualitative and quantitative components, accord- Malawi and by Wadonda Consulting. However, ing to one former and one current official. Several it is unclear if the existing capacity of these members of the PSIA team gained experience agencies was enhanced as a consequence of using qualitative techniques such as interview- the PSIA. The more complicated quantita- ing and stakeholder analysis. In addition, team tive components did not have local input, and members developed some expertise with regard capacity building was far less evident in govern- to survey design and implementation, and they ment. Overall, this PSIA had a negligible effect report that they may now be able to do these on country capacity. types of studies in the future. Mali--Cotton Sector Reform This moderate effect is likely due to the extensive The 2004 survey results, which the PSIA work participation of the Ministry of Education and the analyzed, were administered with the help of a National Institute of Educational Development in national research institute, but the analysis and the design, research, and analysis of the qualita- write-up was done by Bank staff. One workshop tive half of the PSIA. The fact that no locals were in May 2005 offered a half-day session on cotton involved in the econometric analysis, however, is that included the results from a 2004 survey. A the likely reason for the lack of quantitative skill second workshop in February 2006 that lasted transfer. Overall, this PSIA had a moderate effect three days introduced the Social Accounting on country capacity. Matrix (SAM) tool and the SAM analysis of the cotton sector to Malian government officials, civil Mozambique--Labor Market Reform society, and donors. A final workshop, specifically Although the Ministry of Planning and Develop- devoted to the issues of cotton and poverty, was ment worked with Bank staff on this PSIA (Ministry held in May 2006. of Planning and Development, Mozambique, and World Bank 2006), the experience of ministry Although the SAM tool was developed by staff with this PSIA highlights that they did not Bank staff, a workshop helped expose partici- feel they had the mandate to comment on pants to how they could potentially use this another ministry's (in this case, Labor) proposed new tool and highlight the links among differ- reforms. That adversely affected the nature of ent sectors of the Malian economy. The short their involvement and, consequently, the skill length of the workshop, however, did not transfer. Overall, this PSIA had a negligible effect enable most participants to gain new analytic on country capacity. skills. Most participants were not able to use the SAM tool themselves but were appreciative Nicaragua--Fiscal Reform of Bank staff efforts to show the capabilities of The Ministry of Finance claims to have only a the tool. SAM was too technically sophisticated vague recollection of a fiscal PSIA study done by for most participants, and more intensive some foreign consultants a few years back and training would be required for participants to denies having learned any new technical skills; actually use the tool and fully understand the the ministry recognizes the possibility that a CD results. The Bank worked with the National and a booklet or manual from someone at the Statistics Department on its survey, contribut- World Bank may have been received. ing to capacity development, but it is unclear if this was explicitly part of the PSIA on cotton. Because this PSIA (World Bank 2003a) was Overall, this PSIA work had a moderate effect conducted by a consulting firm and without on country capacity. substantial government involvement, it appears 63 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R to have left no sustained increase in capacity. operation--neither the Export Infrastructure Overall, this PSIA had a negligible effect on Development Program nor any of the Develop- country capacity. ment Support Credits done after the PSIA. Nicaragua--Water Reform Within the Bank, there was little collaboration Given the limited scope and exposure of the between the social development unit in the water PSIA (World Bank 2005j) and the fact that Bank's South Asia Region, where the report was it was carried out by a contracted external consul- housed, and the transport sector in Washington, tant, it is unlikely that it would have had much DC, which was responsible for port strategy. Key impact in expanding the country's analytical persons in the transport sector who dealt with capacity. Overall, this PSIA had a negligible effect port issues were not involved in the preparation on country capacity. of the report, nor was it shared with or dissemi- nated to them directly. The lack of involvement Zambia--Land, Fertilizer, and Rural Roads or ownership by transport sector staff may The PSIA (World Bank 2005l) does not appear explain why Bank sector work continued to push to have contributed significantly to building for privatization or why the recommendation country capacity, as there was little government for more radical reform was not taken forward. participation. Some civil society organizations Sector staff, for instance, felt that the PSIA, which reportedly received training, but it was limited to was an "anthropological" study, never translated several people. However, none of the civil society into a project or technical assistance financing, organizations contacted was able to confirm that as the Bank's transport program was stretched any training had taken place. quite thin. The decision not to disclose the report publicly or even circulate it widely within the Within the government, the lack of capacity Bank resulted in it not being linked to a lending development was most likely caused by the low program or shared with other donors.7 levels of government participation. Although the Ministry of Finance and National Planning was said However, within the Bank country team with to be sponsoring the study and also supplied a whom the PSIA was shared, it was successful in cross-sectoral counterpart team to work with the encouraging more political economy analysis of PSIA team, government input was minimal. Beyond reforms. Some Bank staff report that the PSIA the initial consultations regarding the PSIA's area created demand within the Bank in Bangladesh, of focus, the government did not participate and South Asia more generally, for conducting extensively in data gathering, analysis, or revision political economy analyses prior to designing or of the study. No government officials, departments, implementing policy advice. This kind of analysis or ministries are cited among the PSIA authors or was prominent in the CAS that followed, which PSIA team participants. Overall, this PSIA had a notes that the Bank would reorient its processes negligible effect on country capacity. to ensure a governance dialogue through- out project, analytical, and advisory activities PSIA Effect on Bank Operations preparation. For example, the CAS refers to a series of sector-specific political economy and Bangladesh--Chittagong Port governance studies that have been initiated, The PSIA (World Bank 2005e) seems to have had such as an upcoming study of the power sector. some effect on World Bank analytical and advisory Overall, this PSIA had a moderate effect on Bank work, but not on Bank lending or strategy. Although operations. some Bank staff say that the PSIA led them not to intervene in the port sector, others point to the Cambodia--Rice Tariffs uncertainties regarding political buy-in that were It is possible that the PSIA (ACI 2002) influenced the reason for the lack of follow-up to the PSIA. the Bank's general policy directions--the 2005 The PSIA was not linked to a specific lending CAS, for example, mentions problems of a weak 64 APPENDIX D: DESCRIPTION OF PSIA EFFECTS ­ COUNTRY CASE REVIEWS financial sector, poor infrastructure, and low rice land-related programs were active participants productivity, all of which were identified in the in and promoted the PSIA. There was strong PSIA--but there is no reference to the PSIA in the consultation between the PSIA team and staff CAS, and interviewees were unable to confirm designing the Bank operation and consulta- that the PSIA had any effect on World Bank tion between Regional staff and central net- thinking. Overall, this PSIA had a negligible effect work anchor staff. This helped ensure that on Bank operations. the product was useful to their needs. Their interest contributed to a more lasting impact Cambodia--Social Concessions of Land Reform for the PSIA. A major impact of the Cambodian land PSIA · The focus on elements of a successful reform (World Bank 2004a) on the World Bank came program, on quantifying landlessness and on through the PSIA's effect on the design of the identifying complementary services, helped Bank-supported LASED project. LASED benefited frame subsequent World Bank programs. The from the land PSIA in several ways. First, the PSIA PSIA identified several risks and challenges yielded facts and information that reduced the related to social land concessions, such as cost and shortened the time involved in project the need for complementary services. These preparation. For example, for information on complementary services and investments form identification of environmental concerns related a key component of LASED. The focus on insti- to land transfer, partly as a result of the PSIA, tutional weaknesses also informed subsequent LASED now contains specific provisions stating Bank programs. that land needs to be evaluated from an environ- · The inclusiveness of the process helped in- mental perspective prior to its distribution. crease influence within the Bank as staff recog- nized the usefulness of consultation and were Second, the PSIA informed the debate about glad to support a project that built goodwill the holding size necessary to meet food security with the government of Cambodia, donors, needs of households; the current size of land and civil society. concessions is now larger than it would have been in the absence of the PSIA. Third, approximately However, the relatively weak use of quantitative 25 percent of the project resources are now techniques may have limited some of the uptake devoted to capacity building; this lack of capacity, in the Bank. In particular, the PSIA never measured particularly at the local level, was identified as a potential impacts of the program, mainly because critical weakness in the PSIA. The remaining 75 of inadequate data. Overall, this PSIA had a percent of the resources are devoted to invest- substantial effect on Bank operations. ment in communities; this investment is geared toward providing the infrastructure and supple- Ghana--Energy Pricing and Subsidies mentary services necessary to ensure that social The energy PSIA (World Bank 2004e) had little land concessions succeed. These deficiencies effect on the Bank's CAS, successive PRSC were highlighted in the PSIA. loans, or important ESW then under way. Though the Energy Development and Access The PSIA also contributed to improving relation- Project addressed issues of disparities in access ships between the World Bank and government, to electricity, it did not substantially reflect the and the trust built during the process contrib- approaches supported by the PSIA in terms uted to a smoother and more consensual project of special provisions for the poor in scaling-up design. access. The project did include some support for off-grid solar photovoltaic schemes, but this was This effect can be attributed to several factors: not a focus area of the PSIA.8 · The analysis was well placed within the World The PSIA's limited effect within the Bank came Bank. In-country staff who were engaged in from differences in approach between those 65 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R responsible for the PSIA and those responsible pricing structures affect poverty in producers' for Bank energy interventions. Most energy households. Some of the PSIA's findings are specialists in the Bank did not include the referenced in Structural Adjustment Credit IV issues of PSIA among their main priorities. No and Poverty Reduction Support Credit 1, but one in Washington or Accra had responsibility these operations are not anchored by the PSIA for promoting the messages of the report once work. According to Bank management, the PSIA it was produced, as was illustrated by a low informed the Bank's position on cotton prices. awareness of the report within the Accra office. Overall, this PSIA work had a moderate effect Overall, this PSIA had a negligible effect on Bank on Bank operations. operations. Mozambique--Primary School Fee Reduction Malawi--Tobacco Marketing Reform There has been no Bank lending in the primary The program document for the First Poverty (or secondary) education sectors during or Reduction Support Grant to Malawi refers after this PSIA (World Bank 2005h), nor is this directly to the tobacco PSIA (World Bank 2004h) PSIA reflected in any analytical work. Overall, as a key analytical underpinning and states that, this PSIA had a negligible effect on Bank along with a previous study, "The PSIA identi- operations. fied several weaknesses in current marketing and institutional arrangements for tobacco, Mozambique--Labor Market Reform which hinder the efficiency of the sector and The PSIA on labor (World Bank 2006h) has had a limit the pass-through of international prices to limited impact on Bank operations. With regard smallholders." to its effect on Bank staff, it appears to have been largely ineffective. Most Bank staff who were not The PSIA's successful input into a Bank operation part of the core team expressed dissatisfaction may be linked to its contribution in identifying with the quality of the report. They said that it a policy hurdle. Furthermore, this effect may did not assess the shortcomings of the new law have been facilitated by good communication or examine alternatives, that it lacked data and between the PSIA team and Regional Bank staff. made claims it could not support, and that it did Overall, this PSIA had a substantial effect on Bank not clearly identify the beneficiaries and those operations. adversely affected by the proposed reform. Overall, this PSIA had a negligible effect on Bank Malawi--Agriculture Market Closure operations. The program document for the First Poverty Reduction Support Grant to Malawi refers Nicaragua--Fiscal Reform directly to the ADMARC PSIA (World Bank 2007c) Even though the PRSC-I did not contain any as a key analytical underpinning, stating that the explicit tax conditionality, the PSIA (World Bank component of the PRSC program on improving 2003a) was referenced in the PRSC's analytic the functioning of agricultural markets benefited section, and the PRSC did stress the need to from the PSIA on ADMARC reforms. improve the administration and collection of taxes and improve the allocation and focus of The PSIA's successful input into a Bank operation social expenditure--both of which were consis- may be linked to its being carried out by Regional tent with the PSIA findings and recommenda- staff. Overall, this PSIA had a substantial effect on tions. Overall, this PSIA had a substantial effect Bank operations. on Bank operations. Mali--Cotton Sector Reform Nicaragua--Water Reform The PSIA's contribution has been to provide Involvement in the PSIA (World Bank 2005j) data on cotton-producing households and may have weighed in the Bank's decision to the SAM model that shows how various raise its level of engagement in the water sector, 66 APPENDIX D: DESCRIPTION OF PSIA EFFECTS ­ COUNTRY CASE REVIEWS as reflected in the Bank's subsequent Country ing subsidies and titling were controversial within Partnership Strategy for Nicaragua, which the Bank. The PSIA was prepared in association allocates $80 million to the sector over the with the Country Economic Memorandum, but 2008­12 timeframe. However, there is no clear the PSIA and memorandum differed regarding attribution of this change to the PSIA in those the emphases they placed on the growth of a documents. The Country Partnership Strategy private sector in agriculture, the impact of land does not even mention the water PSIA, though titling on the poor and the vulnerable, and the it does mention others. Overall, this PSIA had a benefit and drawbacks of fertilizer subsidies and moderate effect on Bank operations. the stress they placed on market solutions for farmers in remote areas. Zambia--Land, Fertilizer, and Rural Roads Apart from the section on rural infrastructure, One further reason for limited effect is ownership, the substance of the PSIA (World Bank 2005l) did but this time on the part of the Bank. Because not inform Bank operations. A Bank-financed there was relatively minor input from in-country project to support commercial smallholders, by Bank staff and the funding came from a trust providing extension services and making other fund instead of the Regional budget, there was a investments, was conceptualized before the PSIA lack of ownership of this PSIA among the Bank's was completed and was not informed by the PSIA operational staff. Overall, this PSIA had a negligi- findings. The report's recommendations regard- ble effect on Bank operations. 67 Money changer on bank of Senegal River. Photo by Scott Wallace, courtesy of the World Bank Photo Library. 68 APPENDIX E: SEMISTRUCTURED STAKEHOLDER INTERVIEWS-- SUMMARY OF RESULTS Semistructured telephone interviews were Lanka. Applying the same criteria as was applied conducted with 47 stakeholders (9 government to the country case reviews, the semistructured officials, 4 private sector representatives, 2 NGO interviews indicated the following effect of PSIAs staff, 11 academics and researchers, 2 officials on country policies, country analytic capacity, from donor agencies, and 19 Bank staff) chosen and Bank operations. There was a high propor- to include stakeholders familiar with the PSIA tion of missing information for country analytic process. These interviews covered 11 PSIAs in capacity--23 of the 48 stakeholder interviews 10 countries (and were additional to the ones yielded no assessment of this dimension. One covered in the country case reviews: Albania, stakeholder was interviewed on two PSIAs (in Armenia, Benin, Egypt, Indonesia (two PSIAs), Indonesia), resulting in a total of 48 stakeholder Moldova, Mongolia, Nepal, Sierra Leone, and Sri interviews. Table E.1: Summary of Semistructured Stakeholder Interviews Substantial Moderate Negligible Unable to assess Effect on country policies 10 24 11 3 Contribution to country 4 9 12 23 analytic capacity Effect on Bank 7 11 13 17 operations Source: IEG. 69 Indian girl. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. 70 APPENDIX F: PORTFOLIO REVIEW TABLES The data in this appendix are based on a review task managers of the 58 PSIAs, of which responses of the statistically significant random sample of 58 were received for 37 PSIAs; project documents PSIAs selected out of the universe of 156 PSIAs.1 for PSIA-linked operations; information recorded The sources for the review of these 58 PSIAs are in the PSIA's file on Project Portal, when available; the PSIA report or document itself (in some cases and any further relevant information provided by a chapter in a Bank ESW such as a Poverty Assess- anchor staff, such as additional budget informa- ment); any funding proposals or progress reports tion. To minimize judgment effects, data classifi- available for the PSIA; a questionnaire sent to all cation was subject to a second-order review. Table F.1: Did the PSIA Explicitly State Operational Objectives? Incidenceb Incidenceb from portfolio review (from task manager questionnaire (excluding task manager responses conducted Operational objectivea questionnaire responses) ex post of the PSIA) Inform country policies and/or debate 41 (71%) 28 (76%) Increase country capacity for policy analysis 7 (12%) 18 (49%) Inform Bank operations 8 (14%) 20 (54%) All of the above objectives stated 0 (0%) 12 (32%) None of the objectives not explicitly stated 13 (22%) -- Source: Portfolio review. Note: N = 58 for column 1, and N = 37 for column 2. PSIA = Poverty and Social Impact Analysis. a. Only explicitly stated operational objectives are taken into account. b. Most PSIAs list more than one objective, so the percentages add up to more than 100 percent. 71 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Table F.2: Did the PSIA Use Quantitative and/or Qualitative Methods? Methodology used Incidence Quantitative analysis only 25 (43%) Qualitative analysis only 4 (7%) Both methods used 28 (48%) Unknown 1 (2%) Source: Portfolio review. Note: The tally for PSIAs that used both methods includes any occurrence in which both a quantitative and qualitative instrument was used but does not attempt to assess the extent to which (if at all) these were done in tandem or influenced each other. N = 58. PSIA = Poverty and Social Impact Analysis. Table F.3: Did the PSIA Identify Stakeholders? Incidence a. Did the PSIA identify beneficiaries or those adversely affected by the reform? Explicitly identified beneficiaries or those adversely affected 20 (35%) No explicit identification of beneficiaries or those adversely affected, but data or results disaggregateda 29 (51%) No discussion or disaggregation of beneficiaries or those adversely affected 8 (14%) b. Did the PSIA identify proponents or opponents of the reform? Yes 24 (42%) 33 (58%) Nob Source: Portfolio review. Note: The Rwanda Poverty Analysis Macroeconomic Simulator training is not included, so N = 57, not 58. PSIA = Poverty and Social Impact Analysis. a. In these cases, the text of the PSIA did not discuss or identify beneficiaries or those adversely affected, but data used and/or presented in the analysis was disaggregated in some way (for example, poor vs. non-poor, urban vs. rural, consumers vs. producers). b. Some of these did identify beneficiaries or those adversely affected, but only those that analyzed whether these groups would be proponents or opponents of the reform/policy are tallied here. Table F.4: Did the PSIA Identify Institutions? Yes Somewhat No 21 14 22 Did the PSIA identify which institution(s) would be responsible for implementing the reform? (37%) (24%) (39%) Source: Portfolio review. Note: The Rwanda Poverty Analysis Macroeconomic Simulator training is not included, so N = 57, not 58. PSIA = Poverty and Social Impact Analysis. 72 A P P E N D I X F : P O R T F O L I O R E V I E W TA B L E S Table F.5: Did the PSIA Analyze Risks? Yes No Did the PSIA take into account institutional or other risks to implementing the reform program (or in the case of background work PSIAs, risks posed by current policies)?a 37 (65%) 20 (35%) Source: Portfolio review. Note: The Rwanda Poverty Analysis Macroeconomic Simulator training is not included, hence N = 57 not 58. PSIA = Poverty and Social Impact Analysis. a. Risks to implementation include institutional capacity risks, political economy risks (for example, interest groups opposing reform), market risks (for example, weak credit markets, exog- enous price shocks), weather risks (drought), and so forth. Table F.6: Did the PSIA Incorporate Monitoring and Evaluation? Yes No Did the PSIA suggest key indicators or a data collection method necessary for monitoring and evaluation of the reform, policy, or sector? 16 (28%) 41 (72%) Did the PSIA suggest ways of implementing an M&E system? 10 (18%) 47 (82%) Source: Portfolio review. Note: Although some PSIAs did call for a monitoring and evaluation system to be created, only those that gave recommendations for indicators, data collection, a methodology, or other plan of action were tallied. The Rwanda Poverty Analysis Macroeconomic Simulator training is not included, so N = 57, not 58. PSIA = Poverty and Social Impact Analysis. Table F.7: Did the PSIA Involve Stakeholder Participation? Type of activity PSIAs that included activity Focus groups/group interviews/stakeholder interviews/participatory rural appraisal/ 32 (56%) workshops/forums, etc. No consultations mentioned with stakeholders 21 (37%) Unclear or not applicable 4 (7%) Source: Portfolio review. Note: The Rwanda Poverty Analysis Macroeconomic Simulator training is not included, so N = 57, not 58. The PSIA was judged to have incorporated stakeholder participation if it explicitly mentioned focus groups, group interviews, stakeholder interviews, participatory rural appraisal, workshops, forums, and so forth with individuals and/or groups who would have been positively or adversely affected by the reform or regarded as proponents or opponents of the reform. PSIA = Poverty and Social Impact Analysis. Table F.8: At What Stage Was the PSIA Disseminated? Number Concept paper 15 (41%) Background work 14 (38%) Draft report 28 (76%) Final report 22 (60%) Source: Task manager questionnaires. Note: N = 37. PSIA = Poverty and Social Impact Analysis. 73 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R Table F.9: What Form Did Dissemination Take? Type of dissemination activity Number of PSIAs that included activity PSIA findings shared with government 32 (55%) PSIA findings shared with civil society/NGOs 20 (35%) Workshop held 19 (33%) Dissemination to in-country sector specialists 2 (3%) Report publicly available (Bank website) 23 (40%) Dissemination activities planned, but no information on if they 11 (19%) materialized No evidence of dissemination 12 (21%) Not applicablea 3 (5%) Source: Portfolio review. Note: N = 58. PSIA = Poverty and Social Impact Analysis. a. The PSIA is a chapter of another report, so dissemination of the PSIA work itself is impossible to track. Table F.10: Where English Is Not Widely Spoken, Was There a Translation of the PSIA into the Local Language? Yes 24 (60%) No 16 (40%) Source: Portfolio review. Note: N = 40 because no information is available for the rest of the PSIAs, or English was widely spoken in the concerned countries. PSIA = Poverty and Social Impact Analysis. Table F.11: How Many PSIAs Are Available on the Internet? Is the PSIA on the Internet? Number Yes 33 (58%) No 24 (42%) Source: Portfolio review. Note: PSIA = Poverty and Social Impact Analysis. 74 A P P E N D I X F : P O R T F O L I O R E V I E W TA B L E S Table F.12: Did the Study Team Include a Local Organization or Researcher(s)? Yes 41 in totala (71%) Local consultants or researchers 30 (73%) Local NGOs 3 (7%) Government ministries/agencies 14 (34%) No 10 (17%) Information unavailable 7 (12%) Source: Portfolio review. Note: The "study team" includes those who were involved in any part of the PSIA process, beyond supplying pre-existing data, as noted in the PSIA itself, the task manager questionnaire, or PSIA progress report. N = 58. NGO = nongovernmental organization; PSIA = Poverty and Social Impact Analysis. a. Totals more than 41 because some PSIAs included local organizations or researchers in more than one of the above categories. Table F.13: What Was the Incidence of Peer Review? Yes No/no indication Was the PSIA Concept Note peer reviewed? 23 (40%) 34 (60%) Was a draft of the PSIA peer reviewed? 30 (53%) 27 (47%) Source: Portfolio review. Note: The Rwanda Poverty Analysis Macroeconomic Simulator training is not included, so N = 57, not 58. PSIA = Poverty and Social Impact Analysis. 75 Woman from Mali. Photo courtesy of the World Bank Photo Library. 76 ENDNOTES Management Response PSIAs were the only ones identified as PSIAs by Bank 1. IEG notes that its evaluation supports flexibility management and provided to IEG. and recognizes that "[t]he 2008 PSIA Good Practice 3. A consortium of NGOs developed several Note is an improvement over the 2004 PSIA Good criticisms in a paper entitled "Blind Spot: The Contin- Practice Note in that it grants considerable flexibility to ued Failure of the World Bank and IMF to Fully Assess the Regions... This flexibility will allow the scope and the Impact of Their Advice on Poor People" (Oxfam content of the PSIA to be better tailored to the specific International and others 2007). These criticisms are context...." that the Bank does not do enough PSIA, that it does 2. IEG notes that the evaluation recognizes the not do enough to build country capacity or to foster evolution and refinement of the guidance materials. country ownership, that there is insufficient disclosure The evaluation covers PSIAs conducted over fiscal of PSIA results by the Bank, and that PSIA focuses too 2002­07 and includes evidence from interviews of much on mitigating the impacts of a predetermined Bank staff conducted in fiscal 2009, which point to reform and not enough on analyzing alternative policy the persistence of key problems such as lack of buy-in options. from operational staff and lack of understanding of 4. IDA deputies at the fourth IDA-15 replenishment what the PSIA approach is. meeting in November 2007 "encouraged management 3. http://siteresources.worldbank.org/INTPSIA/ to strengthen the preparation of country and sector Resources/GPN_August08_final.pdf. strategies and improve the conduct of poverty and social impact assessments" (World Bank 2007b). Chapter 1 5. Management notes that all three World Bank 1. Management notes that in the late 1990s and reports--the Development Policy Operations early 2000s, the Bank was in the process of reform- Retrospective, Assessing the use of Poverty and Social ing its policy governing adjustment lending (OP Impact Analysis in World Bank Development Policy 8.60) and over the course of the year 2000 prepared Loans, and PSIA--Reviewing the Link to In-Country a report, "Adjustment Lending Retrospective" (World Policy and Planning Processes--will be completed by Bank 2001a, pp. 25­45), that served as background for early fiscal 2010. The Development Policy Operation the update of the adjustment lending policy and its Retrospective has been prepared by Operations Policy conversion to an Operational Policy/Bank Procedures and Country Services once every two years--review- (OP/BP) format. The poverty and social focus of adjust- ing the Bank's experience with Development Policy ment lending was analyzed at length and in a candid Operations, including compliance with aspects of OP way in that report, including the distributional effects 8.60 that relate to poverty and social impact analysis of adjustment programs. but not intended as a comprehensive self-evaluation 2. Management notes that the 156 PSIAs reflect of PSIA. mainly those financed by trust funds or stand-alone 6. Management notes that, although supporting pieces of distributional analysis; these do not include capacity development in this area, as in almost all all the poverty, social, and distributional impact analysis areas of Bank assistance, is important, it was never embedded in other World Bank ESW (such as Poverty the primary objective of PSIA work. IEG notes that Assessments, Public Expenditure Reviews, and gender this evaluation does not refer to capacity building as and social assessments). IEG notes that these 156 the "primary" objective of PSIAs and that the Bank's 77 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R 2004 Good Practice Note states "...the Bank and other Chapter 3 development partners have a major role in building 1. Although the PSIA is on the Bank's external local capacity for PSIA" (World Bank 2004f, para. 4). Web site (World Bank 2005e), this fact is not widely Furthermore, it also states, "Countries may require known and the Bank has not made efforts to actively substantial support from the Bank and other develop- disseminate it. Transparency International, staff in the ment partners to carry out PSIA. Building an in-country International Finance Corporation, and the Bangla- constituency and capacity for PSIA will strengthen desh Garment Manufacturers and Exporters Associa- country ownership and increase the prospects for tion were among the groups that were interested in mainstreaming PSIA work" (para. 11). the PSIA but unaware of its availability. 2. Management notes that the unique political Chapter 2 circumstances in which the PSIA was undertaken led 1. Management notes that IEG did not share its to the management decisions on consultation and classification of individual pieces of PSIA work. As dissemination. a result, management is unable to comment on the 3. Management notes that the assessment of a accuracy or relevance of the IEG categorizations negligible contribution to county analytic capacity is or ratings in this chapter or in chapters 3 and 4 and made in general terms and does not take into account appendix F. IEG notes that it offered management the whether capacity building was one of the stated data on which the classification is based, including PSIA objectives of the PSIA. reports, trust fund proposals, and task manager survey 4. Management notes that IEG did not share its results, and offered to meet with Bank management to classification of individual pieces of PSIA work. As review how a subset of the PSIAs had been classified. a result, management is unable to comment on the IEG also clarifies that these classifications are relevant accuracy or relevance of the IEG categorizations to only one paragraph in each chapter. or ratings in this chapter or in chapters 3 and 4 and 2. "N" here is 57 PSIAs and not 58, because the appendix F. IEG notes that it offered management the Rwanda PAMS training PSIA (World Bank 2007h) is data on which the classification is based, including excluded. PSIA reports, trust fund proposals, and task manager 3. Management notes that in some cases, however, survey results, and offered to meet with Bank manage- a general sector analysis is essential for devising differ- ment to review how a subset of the PSIAs had been ent options for reform. Offering different options for classified. IEG also clarifies that these classifications reform, as opposed to evaluating the likely impact of are relevant to only one paragraph in each chapter. a specific reform, is an important role that PSIA can play, depending on the country, sectoral context, and Chapter 4 country demand. General sectoral analysis would be 1. Management notes that poverty and social and relevant for PSIA in such cases. distributional analysis is not meant to be a "formal" 4. The assessment is made in general terms and practice across the Bank, but rather a set of tools does not differentiate between the reports that should used in stand-alone reports and integrated into have used a quantitative approach and those that regular ESW products, such as Poverty Assessments, should have used a qualitative or a combined mixed- Public Expenditure Reviews, gender assessments, and methods approach. Country Economic Memoranda. 5. The 60 percent figure includes the "somewhat" 2. Management notes that IEG did not share its category; see appendix table F.4. classification of individual pieces of PSIA work. As 6. The PSIA was judged to have incorporated a result, management is unable to comment on the stakeholder participation if it explicitly mentioned accuracy or relevance of the IEG categorizations focus groups, group interviews, stakeholder or ratings in this chapter or in chapters 3 and 4 and interviews, participatory rural appraisal, workshops, appendix F. IEG notes that it offered management the forums, and so forth with individuals and/or groups data on which the classification is based, including PSIA that would have been positively or adversely affected reports, trust fund proposals, and task manager survey by the reform or regarded as proponents or opponents results, and offered to meet with Bank management to of the reform. review how a subset of the PSIAs had been classified. 78 ENDNOTES IEG also clarifies that these classifications are relevant ing the topic, monitoring production, and discussing to only one paragraph in each chapter. its findings. There were extensive consultations with 3. Management notes that the supporting material key decision makers, including chairing of the PSIA in the paragraph highlights several PSIAs for which steering committee by the Minister of Energy; identi- the operational staff had issues with the content or fication of the PSIA with the Ministry of Finance and coverage. This does not support a "striking finding." Economic Planning; consultation with the multistake- There is often debate within the Bank on the coverage holder steering committee on the Terms of Reference, of analytical work, and that is healthy. on the selection of the firm, on the draft report, and on 4. Management notes that, although there was a the findings through numerous meetings. In addition, time at the start of the Ghana work when a change it should be noted that after the PSIA work, the Ministry in energy unit task managers plus a heavy work load of Finance and Economic Planning included a discrete may have led to less than full engagement, over the budget line for a lifeline subsidy. medium term the energy unit did use the results of the 3. Management disagrees with this analysis. Signifi- PSIA, notably in informing the Ghana Energy Develop- cant changes have been introduced, directly influenced ment and Access Project. by the tobacco PSIA. The governing legislation and 5. Management notes that the Good Practice Note regulatory framework for tobacco institutions were (World Bank 2008b) reflects corporate guidance to staff amended, resulting in strengthened representation of on what the PSIA approach encompasses and indicates tobacco farmers on the Tobacco Control Commission that PSIA is a set of analytical tools and methods that Board. Several levies were also reduced substantially, does not lend itself to universal standards, given the resulting in increased returns to farmers. Four satellite range of potential issues and the variety of methods auction floors were also opened to bring markets needed for such analysis. closer to farmers, resulting in further reductions in marketing costs. The Malawi government has become Appendix A notably more interventionist in the trading of tobacco 1. Management notes that in the late 1990s and since the PSIA, although this was not the recommen- early 2000s, the Bank was in the process of reforming dation of the PSIA. The government's continued its policy governing adjustment lending (OP 8.60) and policy of setting minimum prices, which was also not prepared over the course of the year 2000 a report, recommended by the PSIA, has had a substantial impact Adjustment Lending Retrospective (World Bank on buyer practices but is likely seriously undermining 2001a), that served as background for the update of the viability of the sector in the medium term. the adjustment lending policy and its conversion to an 4. Management disagrees with this analysis. The Operational Policy/Bank Procedures (OP/BP) format. PSIA informed substantial changes in the scope and The poverty and social focus of adjustment lending role of ADMARC, including the transfer of ADMARC was analyzed at length and in a candid way in that warehousing assets to a separate company for report, including the distributional effects of adjust- subsequent lease to the private sector, which is ment programs (see pp. 25­45). continuing. As noted, this is a longer-term process that is currently being supported in a programmatic Appendix B fashion by a Bank Poverty Reduction Support Credit 1. The list of 156 Bank-funded PSIAs over fiscal series. 2002­07 was identified by the Bank's PREM and SDN 5. Management notes that, in fact, there were Anchors. several meetings held in 2004 on issues of cotton pricing. Additionally, the results of the PSIA were a key Appendix D input in informing dialogue with government on the 1. Management notes that the unique political need to undertake price reforms and were also used circumstances in which the PSIA was undertaken led in several Development Policy Operations from 2004 to the management decisions on consultation and onwards. dissemination. 6. Management notes that the team working on 2. Management notes that the government was very the PSIA advised and helped the Mali Statistical Office much engaged, including key decision makers, in select- put in its national survey for the first time a module 79 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R on income that will specifically allow analysts to distin- alternative energy sources or off-grid energy sources guish the distributional impact of cotton price changes for rural areas. Though these issues are beyond the on incomes of producers versus other groups. scope of this PSIA, they merit further examination in 7. Management notes that the unique political subsequent research" (World Bank 2004e, p. xvii). circumstances in which the PSIA was undertaken led to the management decisions on consultation and Annex F dissemination. 1. Management notes that IEG did not share its 8. Management notes that the statement that classification of individual pieces of PSIA work. As the PSIA had little effect on the power sector credit a result, management is unable to comment on the is inaccurate; the Ghana Energy Development and accuracy or relevance of the IEG categorizations Access Project was influenced by the PSIA, specifi- or ratings in this chapter or in chapters 3 and 4 and cally in the component that provides for low-cost appendix F. IEG notes that it offered management the energy solutions, including off-grid sources, that will data on which the classification is based, including PSIA benefit the northern populations. IEG notes that the reports, trust fund proposals, and task manager survey PSIA specifically states that analysis of off-grid energy results and offered to meet with Bank Management to sources was beyond its scope. The PSIA states, "This review how a subset of the PSIAs had been classified. includes analysis of pricing, access and efficiency of IEG also clarifies that these classifications are relevant biomass fuels and kerosene, as well as exploration of to only one paragraph in each of chapters 3 and 4. 80 BIBLIOGRAPHY ACI (Agrifood Consulting International). 2002. sohn and Mr. H. Köhler." Overseas Develop- "Rice Value Chain Study: Cambodia." World ment Institute, London, 22 December. Bank Report, ACI, Bethesda, MD. Oxfam International, Save the Children U.K., Angel-Urdinola, Diego, and Quentin Wodon. 2007. CAFOD, Christian Aid, New Rules for Global "Assessing the Targeting Performance of Social Finance, Water Aid, Eurodad, TRICAIRE, Programs: Cape Verde." In Public Finance Bretton Woods Project, and Norwegian Church for Poverty Reduction: Concepts and Case Aid. 2007. "Blind Spot: The Continued Failure Studies from Africa and Latin America, ed. of the World Bank and IMF to Fully Assess the Blanca Moreno-Dodson and Quentin Wodon, Impact of Their Advice on Poor People." Joint 417­40. Washington, DC: World Bank. NGO Briefing Note. C3 Management and Economic Consulting. 2006. Sechaba Consultants. 2004. "Lesotho: Poverty "Review of Civil Service Minimum Wage and and Social Impact Analysis of Electricity Sector SES Options." Final report to Government of Reform." Report for World Bank, Sechaba Sierra Leone, C3 Management, London. Consultants, Maseru, Lesotho. Do, Quy-Toan, and Lakshmi Iyer. 2003. "Land Wolfenson, J. D. 1997. "The Challenge of Rights and Economic Development: Evidence Inclusion." Annual Meeting Address, Hong from Vietnam." Policy Research Working Paper Kong, September 23. 3120, World Bank, Washington, DC. World Bank. Forthcoming. "The Development IEG (Independent Evaluation Group). 2009. Policy Lending Retrospective." Operations, Climate Change and the World Bank Group. Policy and Country Services Report, World Phase I: An Evaluation of World Bank Bank, Washington, DC. Win-Win Energy Policy Reforms. Washington, ------. 2009a. "Assessing the Use of Poverty and DC: World Bank. Social Impact Analysis in World Bank Develop- ------. 2008. Using Knowledge to Improve ment Policy Loans." World Bank, Washington, Development Effectiveness: An Evaluation of DC. World Bank Economic and Sector Work and ------. 2009b. "Projects & Operations." http:// Technical Assistance, 2000­2006. Washing- go.worldbank.org/0FRO32VEI0. ton, DC: World Bank. ------. 2008a. "Achieving Better Service Delivery Ministry of Planning and Development, through Decentralization in Ethiopia." World Mozambique, and World Bank. 2006. "Job Bank Working Paper No. 131, World Bank, Creation in Mozambique: Is Labor Law Reform Washington, DC. the Answer?" World Bank, Washington, DC. ------. 2008b. "Good Practice Note: Using ODI (Overseas Development Institute) and Poverty and Social Impact Analysis to Support World Bank. 2009. "Poverty and Social Impact Development Policy Operations." World Bank, Analysis: Reviewing the Link with In-Country Washington, DC. Policy and Planning Processes." Draft synthe- ------. 2008c. The Impact of Macro-Economic sis report, World Bank, Washington, DC. Policies on Poverty and Income Distribution: Oxfam International. 2000. "PRSPs and Poverty Macro-Micro Evaluation Techniques and Impact Assessments: Letter to Mr. J.D. Wolfen- Tools. Washington, DC: World Bank. 81 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R ------. 2008d. "Mauritania: Poverty and Social Analysis." Report No. 43082, World Bank, Impact Analysis--Policy Note on Reforms Washington, DC. to the Provision of Ancillary Services in ------. 2006a. "Analyse des Impacts Sociaux the Mauritania Mining Sector." Report No. et sur la Pauvreté des Reformes Envisagées 40019-MR, World Bank, Washington, DC. dans les Domaines de l'Alimentation en Eau ------. 2008e. "Poverty and Social Impact Potable et de l'Assainissement." [Morocco] Analysis." http://www.worldbank.org/psia. World Bank, Washington, DC. ------. 2007a. "Bangladesh: Revival of Inland ------. 2006b. Analyzing the Distributional Water Transport: Options and Strategies." Impact of Reforms: Volume II: A Practi- World Bank Policy Note. Report No. 38009, tioner's Guide to Pension, Health, Labor World Bank, Washington, DC. Markets, Public Sector Downsizing, Taxation, ------. 2007b. "Chairman's Summary: Fourth Decentralization, and Macroeonomic IDA-15 Meeting." Dublin, November 12­13. Modeling. Washington, DC: World Bank. http://siteresources.worldbank.org/IDA/ ------. 2006c. "Cambodia: Halving Poverty by Resources/CSummaryDublin.pdf. 2015? Poverty Assessment 2006." Report No. ------. 2007c. "First Poverty Reduction Support 35213-KH, World Bank, Washington, DC. Grant: Support to the Malawi Growth ------. 2006d. "Côte d'Ivoire: Contributions and Development Strategy." IDA Program à l'Analyse de la Pauvreté (Contributions Document, Report No. 40169-MW, World to Poverty Analysis)." Report No. 36625-CI, Bank, Washington, DC. World Bank, Washington, DC. ------. 2007d. "Northern Uganda Land Study: ------. 2006e. "A Gender Analysis of Pension Analysis of Post-Conflict Land Policy and Land Reform Scenarios in Serbia." World Bank Administration--A Survey of IDP Return and Report, Washington, DC. Resettlement Issues and Lessons: Acholi and ------. 2006f. "Guyana: A Poverty and Social Lango Regions." Northern Uganda Recovery Impact Analysis of Changes in the Sugar and Development Program Report, World Regime." World Bank Poverty and Social Bank, Washington, DC. Impact Analysis, Washington, DC. ------. 2007e. "Poverty and Environmental Impacts ------. 2006g. "IBRD Program Document for a of Electricity Price Reforms in Montenegro." Proposed Loan in the Amount of 403 Million World Bank Policy Research Working Paper No. to the Republic of Turkey for a Programmatic 4127, World Bank, Washington, DC. Public Sector Development Policy Loan." ------. 2007f. "Sierra Leone--Mining Sector Report No. 36274-TR, World Bank, Washing- Reform: A Strategic Environmental and Social ton, DC. Assessment." World Bank, Washington, DC. ------. 2006h. "Job Creation in Mozambique: Is ------. 2007g. "Tools for Institutional, Political, Labor Law Reform the Answer?" PREM Report, and Social Analysis of Policy Reform: A Source- World Bank, Washington, DC. book for Development Practitioners." World ------. 2006i. "Local Government Taxation Bank, Washington, DC. Reform in Tanzania: A Poverty and Social ------. 2007h. "Training and Advice on the Impact Analysis: Report on Economic and Rwanda Poverty Analysis Macroeconomic Sector Work." Report No. 34900-TZ, World Simulator (PAMS)." Report to World Bank and Bank, Washington, DC. Republic of Rwanda Ministry of Finance and ------. 2006j. "Making Government Work for Economic Planning, Kigali, Rwanda. the Poor." In Making the New Indonesia ------. 2007i. "Uruguay: Strengthening Partici- Work for the Poor, 221­66. Washington, DC: patory Monitoring and Evaluation of Social World Bank. Policy: Report of Phase I." Report No. 40110-UY, ------. 2006k. "Making Social Protection Work World Bank, Washington, DC. for the Poor." In Making the New Indonesia ------. 2007j. "Yemen's Water Sector Reform Work for the Poor, 177­220. Washington, DC: Program: A Poverty and Social Impact World Bank. 82 BIBLIOGRAPHY ------. 2006l. "Mauritania Poverty and Social ------. 2005b. "Democratic Republic of Congo: Impact Assessment: Consumer Assessment of Economic and Sector Work: Governance and Reforms to the Provision of Ancillary Services Service Delivery." Report No. 32205-ZR, World in the Mauritania Mining Sector." PREM Bank, Washington, DC. Report, World Bank, Washington, DC. ------. 2005c. "Egypt--Toward a More Effective ------. 2006m. "Nicaragua: The Impact of the Social Policy: Subsidies and Social Safety Net." Fiscal Equity Law Reform." In Poverty and Report No. 33550-EG. World Bank, Washing- Social Impact Analysis of Reforms: Lessons ton, DC. and Examples of Implementations, ed. ------. 2005d. "El Salvador Poverty and Social Aline Coudouel, Anis A. Dani, and Stefano Impact Analysis on CAFTA: A Partial Equilib- Paternostro, 29­66. Washington, DC: World rium Estimate of the Treaty's Welfare Impact Bank. on the Salvadoran Population." Working ------. 2006n. Poverty and Social Impact Paper No. 34069, World Bank, Washington, Analysis of Reforms: Lessons and Examples DC. from Implementation. Washington, DC: ------. 2005e. "Improving Trade and Transport World Bank. Efficiency--Understanding the Political ------. 2006o. "Poverty and Social Impact Economy of Chittagong Port." Bangladesh Analysis for the Water Sector Reform in Development Series Paper No. 6, World Bank, Montenegro and Social Assessment for the Washington, DC. Montenegro Tourism Development Project." ------. 2005f. "Indonesia--Poverty and Social Poverty and Social Impact Analysis, World Impact Analysis: Social Protection Reform." Bank, Washington, DC. Summary document, World Bank, Washing- ------. 2006p. "Social Protection Options to ton, DC. Mitigate the Impact of Energy Sector Reforms ------. 2005g. "Indonesia: PSIA I--Fuel Subsidy." in Tajikistan: A Poverty and Social Impact Summary document, World Bank, Washing- Analysis." Environmentally and Socially ton, DC. Sustainable Development Department ------. 2005h. "Mozambique: Poverty and Social Report, World Bank, Washington, DC. Impact Analysis--Primary School Enrollment ------. 2006q. "Sri Lanka: Welfare Reform." and Retention: The Impact of School Fees." In Poverty and Social Impact Analysis of Report No. 29423, World Bank, Washington, Reforms: Lessons and Examples of Implemen- DC. tations, ed. Aline Coudouel, Anis A. Dani, and ------. 2005i. "Nepal Poverty Assessment: Socio Stefano Paternostro, 149­212. Washington, Economic Impact of Fuel Prices in Nepal." DC: World Bank. Background Paper, World Bank, Washington, ------. 2006r. "Tanzania: Crop Boards Reform." DC. In Poverty and Social Impact Analysis of ------. 2005j. "Nicaragua--Desconcentración Reforms: Lessons and Examples of Implemen- y Descentralización de los Servicios de Agua tations, ed. Aline Coudouel, Anis A. Dani, and Potable y Alcantarillado--Análisis de Pobreza Stefano Paternostro, 491­520. Washington, e Impacto Social." Final report, World Bank, DC: World Bank. Washington, DC. ------. 2006s. "Ukraine Energy and Poverty ------. 2005k. "Poverty and Social Impact Note." Report to World Bank, Washington, Analysis of Sri Lanka's Land Policy Reforms: DC. Socio-Economic Impact Assessment." Final ------. 2005a. Analyzing the Distributional report, World Bank, Washington, DC. Impact of Reforms: Volume I: A Practitioner's ------. 2005l. "A Poverty and Social Impact Guide to Monetary and Exchange Rate Policy, Analysis of Three Reforms in Zambia: Land, Utility Provision, Agricultural Markets, Land Fertilizer, and Infrastructure." Social Develop- Policy, and Education. Washington, DC: ment Papers, Paper No. 49, World Bank, World Bank. Washington, DC. 83 A N A LY Z I N G T H E E F F E C T S O F P O L I C Y R E F O R M S O N T H E P O O R ------. 2005m. "République de Djibouti: Analyse ity: A Multi-Market Model Poverty and Social d'Impact Social et Pauvreté Secteur Energé- Impact Analysis for Madagascar." Poverty and tique." Report No. 32260-DJ, World Bank, Social Impact Analysis, World Bank, Washing- Washington, DC. ton, DC. ------. 2005n. "Turkey Labor Market Study." ------. 2004m. "Santé et Pauvreté en Maurit- Report No. 33254-TR, World Bank, Washing- anie: Analyse et Cadre Stratégique de Lutte ton, DC. Contre la Pauvreté." Poverty and Social Impact ------. 2004a. "Assessment of Potential Impacts Analysis, World Bank, Washington, DC. of `Social Land Concessions.' Final Report." ------. 2004n. "Sharing Power: Lessons Learned Report No. 34475. World Bank, Washington, from the Reform and Privatization of Moldova's DC. Electricity Sector Poverty and Social Impact ------. 2004b. "Benin--Cotton Sector Reforms: Analysis." Poverty and Social Impact Analysis, A Poverty and Social Impact Analysis." Report World Bank, Washington, DC. No. 29951-BJ, World Bank, Washington, DC. ------. 2004o. "World Bank Poverty and Social ------. 2004c. "Does Public Investment Help Impact Analysis: Impact of Hydrocarbon Price Spur Growth and Reduce Poverty in Nicaga- Increases--A Vision from the Perspective of gua?" World Bank Poverty and Social Impact MECOVI Surveys." Bolivia PSIA, World Bank, Analysis, Washington, DC. Washington, DC. ------. 2004d. "Evaluation of the Distributional ------. 2003a. "The Distributional Impact of the Impact of the Honduran Tax Reform." World Nicaraguan Fiscal Equity Law." World Bank, Bank Report, Washington, DC. Washington, DC. ------. 2004e. "Ghana Poverty and Social Impact ------. 2003b. "District-Level Service Delivery Analysis: Electricity Tariffs: Phase I." World in Rural Madagascar: Accountability in Health Bank, Washington, DC. and Education." Working Paper, World Bank, ------. 2004f. "Good Practice Note: Poverty and Washington, DC. Social Impact Analysis." World Bank, Washing- ------. 2003c. "From Goats to Coats: Institu- ton, DC. tional Reform in Mongolia's Cashmere ------. 2004g. "Identifying Opportunities to Sector." Report No. 26240-MOG, World Bank, Increase Efficiency in the Trade and Transport Washington, DC. Sectors." Draft Concept Note, World Bank, ------. 2003d. "The Incidence of Public Washington, DC. Education Spending in Nicaragua: The Impact ------. 2004h. "Malawi--Tobacco Sector Perfor- of the Education for All­Fast Track Initiative." mance Audit: The Perceptions and Views of World Bank Report, Washington, DC. Smallholder Tobacco Farmers on the State ------. 2003e. "Poverty and Social Impact of Play in the Tobacco Sector." Report to the Analysis: Indonesia Rice Tariff." World Bank World Bank, Washington, DC. Poverty and Social Impact Analysis, Washing- ------. 2004i. "Nicaraguan Agriculture and ton, DC. CAFTA." World Bank Report, Washington, ------. 2003f. "Reforming the Malawi Agricul- DC. tural Development and Marketing Corpora- ------. 2004j. "Operational Policy 8.60: Develop- tion: Synthesis Report of the Poverty and ment Policy Lending." http://go.worldbank. Social Impact Analysis." Report No. 27512, org/YMN1C0RWR0. World Bank, Washington, DC. ------. 2004k. "Poverty and Social Impact Analysis ------. 2003g. "Toolkit for Evaluating the of the Irrigation and Drainage Rehabilitation Poverty and Distributional Impact of Projects and the Water Resource Manage- Economic Policies." http://go.worldbank.org/ ment Project in Albania." World Bank Report, Q199F135V0. Washington, DC. ------. 2003h. "A User's Guide to Poverty and ------. 2004l. "Rice Prices, Agricultural Input Social Impact Analysis." http://go.worldbank. Subsidies, Transaction Costs, and Seasonal- org/IR9SLBWTQ0. 84 BIBLIOGRAPHY ------. 2002. "Examining the Social Impact of Level Data from Vietnam." World Bank Policy the Indonesian Financial Crisis Using a Micro- Research Working Paper No. 2599, World Macro Model." Paper presented at OECD Bank, Washington, DC. seminar "How Are Globalisation and Poverty ------. 2001d. "Utility Pricing and the Poor: Interacting and What Can Governments Do Lessons from Armenia." World Bank Technical about It?" Paris, December 10. Paper No. 497, World Bank, Washington, DC. ------. 2001a. "Adjustment Lending Retrospec- ------. 1987. "World Bank Manual Circular OP tive." Working Paper No. 44666, World Bank, 87/88: Guidelines for Preparing and Process- Washington, DC. ing Adjustment Loans and Credits." World ------. 2001b. "Concept Note: Social Impact Bank, Washington, DC. Analysis (SIA) of Macroeconomic and World Bank, Government of Ethiopia, Irish Structural Policies." Draft document, World Aid, UNICEF, USAID, WHO. 2007. Reaching Bank, Washington, DC. or Escaping the Challenge: Financing the ------. 2001c. "State Ownership and Labor Health MDGs in Ethiopia. Washington, DC: Redundancy: Estimates Based on Enterprise- World Bank. 85 IEG PUBLICATIONS Annual Review of Development Effectiveness 2009: Achieving Sustainable Development Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank's Approach to Global Programs Assessing World Bank Support for Trade, 1987­2004: An IEG Evaluation Books, Building, and Learning Outcomes: An Impact Evaluation of World Bank Support to Basic Education in Ghana Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy Climate Change and the World Bank Group--Phase I: An Evaluation of World Bank Win-Win energy Policy Reforms Debt Relief for the Poorest: An Evaluation Update of the HIPC Initiative A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995­2005 The Development Potential of Regional Programs: An Evaluation of World Bank Support of Multicountry Operations Development Results in Middle-Income Countries: An Evaluation of World Bank Support Doing Business: An Independent Evaluation--Taking the Measure of the World Bank­IFC Doing Business Indicators Egypt: Positive Results from Knowledge Sharing and Modest Lending--An IEG Country Assistance Evaluation 1999­2007 Engaging with Fragile States: An IEG Review of World Bank Support to Low-Income Countries Under Stress Environmental Sustainability: An Evaluation of World Bank Group Support Evaluation of World Bank Assistance to Pacific Member Countries, 1992­2002 Extractive Industries and Sustainable Development: An Evaluation of World Bank Group Experience Financial Sector Assessment Program: IEG Review of the Joint World Bank and IMF Initiative From Schooling Access to Learning Outcomes: An Unfinished Agenda--An Evaluation of World Bank Support to Primary Education Hazards of Nature, Risks to Development: An IEG Evaluation of World Bank Assistance for Natural Disasters How to Build M&E Systems to Support Better Government IEG Review of World Bank Assistance for Financial Sector Reform An Impact Evaluation of India's Second and Third Andhra Pradesh Irrigation Projects: A Case of Poverty Reduction with Low Economic Returns Improving Effectiveness and Outcomes for the Poor in Health, Nutrition, and Population Improving the Lives of the Poor through Investment in Cities Improving Municipal Management for Cities to Succeed: An IEG Special Study Improving the World Bank's Development Assistance: What Does Evaluation Show: Maintaining Momentum to 2015: An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh New Renewable Energy: A Review of the World Bank's Assistance Pakistan: An Evaluation of the World Bank's Assistance Pension Reform and the Development of Pension Systems: An Evaluation of World Bank Assistance The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank's Support Through 2003 The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support Power for Development: A Review of the World Bank Group's Experience with Private Participation in the Electricity Sector Public Sector Reform: What Works and Why? An IEG Evaluation of World Bank Support Small States: Making the Most of Development Assistance--A Synthesis of World Bank Findings Social Funds: Assessing Effectiveness Sourcebook for Evaluating Global and Regional Partnership Programs Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000­2006 Using Training to Build Capacity for Development: An Evaluation of the World Bank's Project-Based and WBI Training The Welfare Impact of Rural Electrification: A Reassessment of the Costs and Benefits--An IEG Impact Evaluation World Bank Assistance to Agriculture in Sub-Saharan Africa: An IEG Review World Bank Assistance to the Financial Sector: A Synthesis of IEG Evaluations World Bank Group Guarantee Instruments 1990­2007: An Independent Evaluation The World Bank in Turkey: 1993­2004--An IEG Country Assistance Evaluation World Bank Engagement at the State Level: The Cases of Brazil, India, Nigeria, and Russia All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual section, please visit http://www.worldbank.org/ieg. ISBN 978-0-8213-8293-6 SKU 18293