FAST TRACKBRIEF June 3, 2009 56817 The IEG report "How Effective Have Poverty and Social Impact Analyses Been?" was discussed by CODE on June 3, 2009 How Effective Have Poverty and Social Impact Analyses Been? The Poverty and Social Impact Analysis (PSIA) approach, introduced by the World Bank in FY02, aimed to help the Bank and its client countries anticipate and address the possible consequences of proposed policy reforms, especially on the poor and vulnerable, and to contribute to country capacity for policy analysis. According to the Bank, by FY07 it had undertaken 156 PSIAs in 75 countries and 14 sectors. Total Bank and other donor support to PSIAs over FY04-06 was $15 million. Development literature has emphasized the importance of understanding the institutional and political constraints to development, building domestic ownership of policy reforms, and assessing the distributional impacts of policy actions. The PSIA approach has correctly emphasized these aspects. The Bank has produced a substantial body of guidance on how to address these aspects, and that guidance has been refined over time to incorporate lessons learned. Implementation of the PSIA approach has, however, had considerable limitations. There have been tensions between the various operational objectives assigned to PSIAs. The tensions concern inconsistencies between informing country and Bank policy decisions in a timely way and building country analytic capacity. PSIAs have had limited ownership by Bank staff and managers and have often not been effectively integrated into country assistance programs. Quality assurance, monitoring, and evaluation of the overall effectiveness of PSIAs have been weak. While there have been some outstanding examples of success, PSIAs reviewed in this evaluation have, on average, had a moderate effect on country policies and Bank operations, and a negligible effect on country analytic capacity. To improve PSIA effectiveness, the evaluation recommends that the Bank take measures to ensure that staff fully understand what the PSIA approach is and when to use it, clarify the operational objectives of each PSIA, and ensure that the approach and timeline adopted are aligned with those objectives. Quality assurance mechanisms should be strengthened to ensure that PSIAs are designed to achieve the intended effects. The evaluation also recommends that the Bank shift significant decision-making and funding authority for PSIAs to the Regional Vice Presidencies and ensure that PSIAs are grounded in country assistance programs. Introduction The development community first became concerned about the impacts of economic policy reforms on the Evaluation Background Although the Bank has submitted progress reports to poor during the structural adjustment period of the donors regarding the implementation of PSIAs, it has not 1980s. Then and in the 1990s, external groups yet completed a comprehensive self-evaluation of the PSIA continuously criticized the IMF and World Bank for a experience. This IEG evaluation, requested by the Bank's failure to properly assess the impacts of the policy Board of Executive Directors, represents the first reforms they supported. In response, the Bank in FY02 independent evaluation of the PSIA experience. introduced Poverty and Social Impact Analysis (PSIA). The Bank had been conducting distributional analysis in The PSIA experience is highly relevant today. The recent some of its economic and sector work even before this financial crisis and global slowdown are likely to put time. As early as 1987 the Bank's Operational Guidelines pressure on governments and donors to undertake far- required analysis of the short-term impact of adjustment reaching reforms in the medium term. In order to protect programs on the urban and rural poor. According to the the poor and enhance benefits to them, key reforms will revised Operational Policy (OP 8.60) on Development need to be underpinned by systematic analysis of their Policy Lending (DPL) issued in 2004, DPLs with likely expected poverty and social impacts. and significant poverty and social consequences were The Bank has envisaged several roles for PSIAs, mainly in required to summarize in their program documents the elaboration and implementation of poverty reduction relevant analytic knowledge of these consequences. But strategies in developing countries, supporting in-country the Operational Policy does not mandate that the PSIA capacity building, and informing Bank operations. These approach be used to undertake an assessment of these roles can be distilled into three operational objectives that consequences. form the basis for the questions addressed in this Over FY02-07, the Bank supported 156 pieces of evaluation: analytical work using one or more elements of the PSIA approach (hereafter called PSIAs), partly driven by the 1. What effect have PSIAs had on country policies availability of earmarked funds both from within the (including policy debate)? Bank and from DFID, GTZ, Norway, Italy, Belgium, 2. What contribution have PSIAs made to the and Finland. The PSIAs were done mainly in the context development of country capacity for policy analysis? of DPLs, but some also aimed to inform Country 3. What effect have PSIAs had on Bank operations Assistance Strategies, investment loans, and analytical (including strategy and analytical work)? work. This evaluation covers analytical work identified as PSIAs by the Bank's Poverty Reduction and Economic What Is a PSIA? Management (PREM) and Sustainable Development The Bank defines PSIA as "analysis of the distributional (SDN) Network Anchors. Analyses not identified as PSIAs impact of policy reforms on the well-being of different by these Anchors are not covered. stakeholder groups, with particular focus on the poor and vulnerable." The PREM/SDN Anchors identified 156 analyses as PSIAs over the FY02-07 period. This evaluation draws its The Bank sees the innovative aspects of the PSIA as "the findings from: a portfolio review of a statistically application of the tools and techniques of social and representative sample of 58 PSIAs out of the universe of economic analysis to analyze impacts of economy-wide 156 PSIAs; in-depth country case reviews of 12 PSIAs in policy reforms before those reforms are carried out (ex- eight countries with Regional, sectoral, and fiscal year ante analysis), and more systematic use of that analysis to coverage; and interviews with key informants. The latter inform policy advice and policy design." include interviews with country stakeholders and Bank staff The Bank has identified two key elements of PSIA: "First, associated with an additional 11 PSIAs in 10 countries and an analysis to determine the distributional impacts, and interviews with senior Bank staff and managers. second, a process that engages appropriate stakeholders in policy-making." Main Findings Objectives. The portfolio review found that about one- Source: "A User's Guide to Poverty and Social Impact Analysis," fifth of the PSIAs have not explicitly identified which of World Bank, 2003; "Poverty and Social Impact Analysis of Reforms: Lessons and Examples from Implementation," World Bank, 2006; the three operational objectives they intend to pursue. "Good Practice Note: Using Poverty and Social Impact Analysis to Among those that have identified operational objectives, Support Development Policy Operations," World Bank, 2008. informing country policies has been the most frequently stated, followed by informing Bank operations, and 2 increasing country capacity for policy analysis. Most PSIAs the government that have policy jurisdiction over the areas have had more than one operational objective. About a covered by the PSIA, engagement with appropriate third of the respondents to the task manager survey nongovernmental stakeholders, timeliness in relation to indicated that their PSIA pursued all three of the country decision-making processes, sensitivity to the objectives. politics of reform, and active dissemination beyond the distribution of reports. One or a few of these factors have Pursuit of the multiple operational objectives of PSIAs can been present in most PSIAs, but the presence of a majority create tension and raise unrealistic expectations of what a of these factors--which has occurred only in some PSIA can achieve. For example, PSIAs seeking to inform PSIAs--is important for substantial effect. government policy decisions must adjust to the timing of the decision process, but often that is inconsistent with the Contribution to Country Capacity. The PSIAs reviewed approach required to build country analytic capacity. suggest a negligible contribution to country analytic capacity, on average, with a few positive examples. The Content and Approach . PSIA practice has frequently main reason for this finding is that most PSIAs have departed from the initial guidance on how PSIAs should be treated this objective as a byproduct rather than a core conducted. The portfolio review shows the following: concern, and there has been a tension between the need to · Even though PSIAs originated out of concern provide timely inputs to policy decisions and the longer- about the impact of reform programs, about one- term sustained engagement needed to build capacity. third of the PSIAs in the portfolio review did not Where capacity building has been an explicit objective it examine well-specified reforms but were more has not always been backed by an appropriate strategy to general sector or macroeconomic analyses. achieve it. Many PSIAs have involved local ministries, · About one-third of the PSIAs explicitly identified consulting firms, NGOs, academics, or local consultants, beneficiaries or those adversely affected, and while others have held training workshops to transfer skills. However, the time allocated to either of these approaches about half did not although they included disaggregated data or results. has generally been insufficient to build lasting capacity for policy analysis. The few PSIAs reviewed that have · About 60 percent of the sampled PSIAs identified successfully helped build capacity have taken a more the institutions responsible for implementing the deliberate approach. Given that in many--perhaps most-- reform. situations, the time frame and approach required for · Over half of the sampled PSIAs included some capacity building are inconsistent with the approach needed sort of stakeholder participation, but no to inform timely government decision making, greater consultations were mentioned for about two-fifths selectivity in operational objectives is desirable and more of the PSIAs. effective approaches to building country analytic capacity PSIAs have had widely varying characteristics due in part to will also be needed. an unclear understanding of PSIAs among Bank staff. At Effect on Bank Operations. The PSIAs reviewed suggest the same time, analytical work possessing PSIA-like a moderate effect on Bank operations, on average, with characteristics has not always been classified by the Bank as some outstanding examples of success. Key factors that "PSIA." These errors of omission and commission are have inhibited PSIA effect on Bank operations have likely to have inhibited quality assurance and lesson included the ambiguity of the PSIA concept, insufficient learning. country director buy-in resulting in lack of grounding of Effect on Country Policies. Tracing links between PSIA the PSIA in the country assistance program, and weak analyses and country decisions is often difficult, especially engagement between PSIA teams and other operational where the PSIA is only one of many possible influences. staff. PSIA effects may also be diffuse and lagged--a report may Interviews with senior Bank staff and managers indicate have no discernable impact on immediate action, yet may that the uptake of PSIAs by country directors and affect ideas and debate that shape future policy choices. operational teams remains dependent on individual This evaluation focuses only on near-term effects of PSIAs. inclinations rather than reflecting established practice. The PSIAs reviewed in this evaluation had a moderate There has been only modest acceptance so far of the PSIA effect on country policies, on average, although there have as a robust practice across the Bank, although staff directly been some outstanding examples of success Informing a involved with PSIAs see a number of corporate benefits policy process is not easy and requires the convergence of a from the experience, notably the creation of an important number of factors. These include the operational focus of body of knowledge through PSIA guidance and an the PSIA, a match between PSIA topic and country appreciation of the importance of process issues in addition priorities, government ownership, engagement of parts of to analytics. However, operational staff interviewed lacked 3 a common understanding about the objectives and · What criteria should be used to determine when the processes of the PSIA approach. PREM staff generally PSIA approach is appropriate for a particular tended to focus on economic analysis and SDN staff operation in a country program. tended to emphasize mixed methods, including social and institutional analysis and a participatory process. Clarify the operational objectives of each PSIA with regard to its intended effect and tailor the The 2008 PSIA Good Practice Note is an improvement approach to those objectives, ensuring that the over the 2004 PSIA Good Practice Note in that it helps to concept note: lighten PSIA guidance, which was previously seen by staff as overly demanding. The 2008 PSIA Good Practice Note · Contains a clear statement of the operational provides Bank staff with the flexibility to determine, based objectives of the PSIA with respect to the intended on country context, the balance between economic, social, effect (not just the topics/issues to be analyzed); institutional, and political analysis (and between · Indicates how its approach, in particular stakeholder quantitative and qualitative techniques) and between engagement, team composition, partner institutions, analytics and such process issues as stakeholder budget, and time frame, has been tailored to meet participation and disclosure. But the 2008 PSIA Good the operational objectives, and provides the Practice Note does not require staff to provide a rationale rationale for the choices made; for the particular choices made. Providing a rationale is · Shows how any tensions and trade-offs among the especially important to ensure that Bank staff from the operational objectives will be reconciled; and different Networks do not continue to place undue · Discusses if the intended dissemination audience emphasis on their respective disciplinary approaches in and strategy are consistent with the stated undertaking PSIAs. operational objectives. The PSIA approach has correctly emphasized the importance of understanding the institutional and political Improve integration of the PSIA into the Bank's country assistance program by: constraints to development and the need to build domestic ownership of policy reforms in addition to assessing the · Shifting significant decision-making and funding distributional impact of policy actions. But implementation authority to the Regional Vice Presidencies to of the approach has had considerable limitations. Some ensure that the PSIA topics, scope, and approach notable successes have modeled what PSIAs can are consistent with the country assistance program accomplish when done right. The recommendations below and that PSIAs ask policy-relevant questions; and are intended to help improve PSIA implementation and · Requiring that all earmarked funding for PSIAs be realize its potential. matched by a substantial contribution from the country unit budget. Recommendations This evaluation makes four recommendations to strengthen Strengthen PSIA effectiveness through enhanced the Bank's work using the PSIA approach, whether done as quality assurance, including: free-standing analysis or embedded in other analytical · Subjecting PSIAs to systematic review by regional work. management at concept and completion stages to Ensure that staff understand what the PSIA ensure relevance and fit of the PSIA to the country approach is and when to use it by providing clear assistance program, and consistency of the proposed guidance (perhaps through updating of the 2008 approach with operational objectives, in addition to PSIA Good Practice Note) and actively ensuring technical quality; and disseminating this guidance particularly on: · Ensuring that the Bank establishes a monitoring and self-evaluation system designed to assess if PSIAs · Whether and how the PSIA approach differs from are being undertaken where appropriate and are other distributional analyses, including whether the achieving their stated operational objectives. inclusion of the word "social" in Poverty and Social Impact Analysis suggests the need to include a different type of analysis; · Whether or not PSIAs should be linked to specific reforms and identify beneficiaries and those adversely affected by the reform; and 4 About Fast Track Briefs Fast Track Briefs help inform the World Bank Group (WBG) managers and staff about new evaluation findings and recommendations. The views expressed here are those of IEG and should not be attributed to the WBG or its affiliated organizations. Management's Response to IEG is included in the published IEG report. The findings here do not support any general inferences beyond the scope of the evaluation, including any inferences about the WBG's past, current or prospective overall performance. The Fast Track Brief, which summarizes major IEG evaluations, will be distributed to selected World Bank Group staff. If you would like to be added to the subscription list, please email us at ieg@worldbank.org, with "FTB subscription" in the subject line and your mail-stop number. If you would like to stop receiving FTBs, please email us at ieg@worldbank.org, with "FTB unsubscribe" in the subject line. Contact IEG: Director-General, Evaluation: Vinod Thomas Directors: Cheryl Gray (IEG-WB) Manager: Monika Huppi (IEGSE) Task Manager: Soniya Carvalho (IEGSE) Copies of the report are available at: http://www.worldbank.org/ieg/psia IEG Help Desk: (202) 458-4497 E-mail: ieg@worldbank.org 5