LE A R N ING PR O DUC T Lessons from Environmental Policy Lending Lessons from Environmental Policy Lending IEG Learning Product June 29, 2016 © 2016 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522- 2625; e-mail: pubrights@worldbank.org. Director-General, Independent Evaluation : Ms. Caroline Heider Director, IEG Financial, Private Sector and Sustainable Development : Mr. Marvin Taylor-Dormond Manager, IEG Sustainable Development : Ms. Midori Makino Task Manager : Mr. Stephen Hutton ii Contents Preface................................................................................................................................. v Acknowledgments............................................................................................................. vii Summary ............................................................................................................................ ix 1. Introduction ..................................................................................................................... 1 2. What does the portfolio look like? .................................................................................. 2 3. In what context is policy lending most effective? .......................................................... 7 4. Political economy in DPF design .................................................................................. 10 5. Lessons on results frameworks ..................................................................................... 14 6. Lessons on DPF preparation ......................................................................................... 27 7. Conclusions ................................................................................................................... 36 References ......................................................................................................................... 37 Appendix A. The Environmental DPF Portfolio .............................................................. 38 Appendix B. Case studies ................................................................................................. 44 Appendix C. Methodology................................................................................................ 58 Boxes Box 1: Political economy in energy and environmental policy lending in Turkey .......... 11 Box 2: Main findings from IEG Learning Product on DPF results frameworks .............. 18 Box 3: Examples of relatively clear objectives from Environmental DPOs .................... 23 Box 4: Selected outcome-oriented results indicators from environmental DPOs ............ 24 Tables Table 1: Outcome ratings of environmental DPF ............................................................... 6 Table 2: IEG Criteria for assessing the prior actions (PAs) of development policy operations .......................................................................................................................... 19 Table 3: Prior action examples ......................................................................................... 20 iii Figures Figure 1: Number and volume of environmental DPOs, FY2000-16 ................................. 2 Figure 2: Types of prior actions supported by environmental DPOs, FY 2000-2016 ....... 4 Figure 3: Lessons from self-evaluations and independent validations/evaluations of closed environmental DPOs, FY 2000-2016 ...................................................................... 3 v Abbreviations AFD French Agency for Development BNDES Brazilian National Bank for Economic and Social Development CEA Country Environmental Analysis DPF Development Policy Financing DPO Development Policy Operation ENR Environment and Natural Resources Env PRL Programmatic Loan for Environmental Sustainability GP Global Practice ICR Implementation Completion Report ICRR Implementation Completion Report Review IEG Independent Evaluation Group IMF International Monetary Fund JICA Japan International Cooperation Agency M&E monitoring and evaluation NREG National Resources and Environmental Governance PA prior action PEA Political Economy Analysis PPAR Project Performance Assessment Report PRL Programmatic Reform Loan SCD Systematic Country Diagnosis SDN Sustainable Development Network SEA Strategic Environmental Assessment TA Technical Assistance TTL Task Team Leader UNFCCC United Nations Framework Convention on Climate Change vi Preface This report is one of a series of learning products on Development Policy Financing (DPF) being undertaken by the Independent Evaluation Group (IEG), covering a range of issues such as results frameworks, macro-fiscal frameworks, public expenditure reviews, environmental and social risk management, political economy, and in this case environmental policy lending. A forthcoming product will cover the use of political economy analysis in DPF design and implementation. Consequently these topics are not covered separately in this report. This learning product aims to provide operationally relevant lessons from World Bank Development Policy Operations (DPOs) to inform the design of environmental and other sectoral development policy lending. The focus is not to evaluate the performance of the DPF instrument, but rather to draw lessons on the “how” and “why”, and to share what works and what doesn’t. The specific questions to be answered were: 1. How has policy lending been used for environmental goals? 2. In what context can policy lending for the environment be most effective? 3. What constitutes high-quality design? 5. What contributes to high-quality monitoring and evaluation? The main audiences are operational task teams preparing DPF operations, especially those with environmental objectives. Many lessons are also relevant for country management units, the Bank’s operational policy and country services vice presidency, and client governments. The report draws on a portfolio review of 64 World Bank environmental DPOs, four field-based case studies in Brazil, Ghana, Indonesia and Turkey, a desk-based case study for Mexico, desk reviews for 38 closed operations, and more than 20 interviews with operational staff and managers who have been involved in environmental DPF (see Appendix C). vii Acknowledgments The report was prepared by Stephen Hutton (Evaluation Officer, IEG) under the supervision of Marvin Taylor-Dormond (Director, IEG), Stoyan Tenev (Acting Director, IEG) and Midori Makino (Manager, IEG). Maria Shkaratan carried out the portfolio review, John Eriksson produced a detailed desk review of environmental DPO experience in Mexico and provided overall advice and guidance, and John Redwood contributed to sections on climate change operations in Vietnam and Indonesia and acted as senior advisor. Vibhuti Narang Khanna provided program support. We are grateful for comments and advice from Carter Brandon, Andrew Stone, Zeljko Bogetic, Chris Nelson, Daniel Musiitwa, Aghassi Mkrtchyan, and Moritz Piatti. Kirk Hamilton, Giovanni Ruta and Fernando Loayza were peer reviewers for this product. The team gratefully acknowledges the contributions of present and former World Bank Group managers and staff who assisted in the report, especially interview subjects. ix Summary Sustainable development, including environmental sustainability, is at the core of the World Bank Group’s strategy. Environmental policy is a crucial driver of environmental outcomes and of development and poverty outcomes, because policy frameworks affect incentives and alter the behavior of public and private sector agents. Policy lending has been a major part of the World Bank’s lending operations for decades, supporting economic policy and institutional reforms. In the past most policy lending operations were multi-sector, but over time the number of operations in specific sectors has increased, particularly for policy lending with environmental goals. This report reviews the World Bank’s experience with Development Policy Financing (DPF) in the Environment sector, broadly defined.1 This product seeks to offer lessons from evaluation of this experience and inform stakeholders on how to design and implement this instrument, outlining some of the tensions and tradeoffs that must be grappled with in design. The main audience is Bank teams helping governments to prepare and implement DPF with environmental goals, particularly those staff who are relatively new to the instrument. The World Bank’s environmental policy lending has grown rapidly since 2005. These operations have supported policy actions across a broad range of subsectors, including climate change mitigation, climate change adaptation and disaster risk management, environmental protection, pollution management, institutional strengthening, and specific sectoral reforms in energy, transport, water, agriculture, forestry, and other sectors. Many operations are multi-sectoral but use an environmental lens. Despite the variation in the types of policies supported, environmental policy lending operations have tended to generate lessons focusing on a common set of issues. These center around issues of political economy, of operation design and preparation, and of institutional strength and capacity. Many of these lessons are not unique to environmental policy lending but rather to the instrument, and may apply to other sectoral DPF operations or even to multi- sectoral operations. Key insights Environmental development policy lending is most effective when used in a way that plays to the strengths of the instrument. Environmental policy lending can be most effective when policy issues are the main barrier to improving environmental outcomes, rather than capacity or other issues. It offers advantages for achieving sector-wide or multi-sectoral goals across many projects. It can be most effective when the Bank has prior knowledge of the country and sector and strong institutional relationships, which may be developed through use of other instruments. It is useful for those policy issues that need attention from high-level decision makers, especially in financing and planning ministries. Its flexibility allows the Bank to take advantage of opportunities as they arise, 1 Environmental Development Policy Operations were defined as those either mapped to the Environment sector or Environment and Natural Resources (ENR) Global Practice, or which had an environmental or natural resource management theme as a primary or secondary theme (see Appendix C). x when the timing for reform in a country is particularly promising, given the presence of a reformist government or champion; but this can be a risky approach. It can be most effective when used in combination with other instruments. A few key design and implementation considerations tend to determine the extent to which environmental policy lending can be effective. Policy lending is most effective when there is a clear political theory of change for how the operation will influence policy outcomes. Examples include supporting policy reforms that would not happen without the World Bank operation, or in other cases, influencing prioritization, timing, or technical quality. Policy lending is more effective with a strategy for achieving institutional buy-in and complementary use of other instruments. The design of results frameworks lies at the core of DPF design. This process requires intensive dialog and debate between Bank teams and governments. It involves a number of tradeoffs and tensions: between ambition and realism, between additionality and country ownership, between depth and breadth. The strongest policy actions are those that are relevant, critical, additional, and measureable, as described in a separate IEG Learning Product on Results Frameworks in DPF. Programmatic series offer a number of advantages, including the ability to induce or support longer-term government commitment to reforms. Yet they can be more effective if they endeavor to include substantial policy actions from the first operation, if they ensure that spacing between operations matches the time needed to complete reform actions, and if careful considerations are made about decisions to drop indicative triggers from future operations. Monitoring and evaluation systems for environmental DPF have often been weak. Policy lending faces inherent difficulties in designing monitoring and evaluation frameworks because of the disconnect between the substantial length of time needed to observe results and the brief time after which operations are evaluated. Yet even given this challenge, there are ways in which the quality of monitoring and evaluation in environmental DPF could be improved. Objectives have often been imprecise or unclear, and indicators have not provided a direct or adequate reflection of the objectives or sub- objectives with which they were associated. Results frameworks sometimes end up measuring processes rather than results or impact. This review offers advice on selection of objectives and indicators, and notes some pitfalls to avoid. Analytical work and technical assistance are important to the success of environmental policy lending operations. Analytical work plays a key role as a diagnostic and in providing the evidence base on which to persuade decision makers. Technical assistance is often critical for development of reforms and completion of policy actions. Yet despite unanimous agreement on its importance, sufficient analytic and advisory work is not always present. One cause of this has been tightening budgets and declining availability of trust funds. Another has been the issue of timing and the reliance on previous analytical work rather than new work commissioned specifically for the operation. And a third has been the unwillingness of many governments to borrow for technical assistance. 1 1. Introduction 1.1 Sustainable development, including environmental sustainability, is at the core of the strategy of the World Bank Group, which states that action on this goal is required to secure the future of the planet, ensure social inclusion, and set a solid foundation for the well-being of future generations (World Bank 2013a). Environmental sustainability is also a critical part of achieving inclusive growth and poverty reduction, because environmental degradation has a range of negative effects that harm the poor. Environmental policy is a crucial driver of environmental outcomes and, in turn, of development and poverty outcomes: policy frameworks affect incentives and alter the behavior of public and private sector agents. As noted in the World Bank Group’s Environment Strategy (World Bank 2012), policies can be critical in enabling the private sector to use natural resources sustainably, to support inclusive and resilient decision making, to improve governance risks, to remove perverse incentives, and to encourage sustainable growth policies. Supporting policy reforms remains a high priority under the strategy. 1.2 The World Bank offers three main categories of financing: Investment Project Financing directly finances specific investments; Program For Results Financing uses country systems and disburses based on achievement of specific results; and Development Policy Financing (DPF) supports a government program of policy and institutional actions. The DPF instrument is intended to achieve development results primarily through the supported policy reforms and associated policy dialog and support. 1.3 Policy lending has been a major part of the World Bank’s lending operations for decades, supporting economic policy and institutional reforms. The DPF instrument established in FY 05 has been used in different ways. Many Development Policy Operations (DPOs) aim to support economywide, multi-sector reforms that include, for example, governance, financial sector, and trade and competitiveness issues. In the past, most DPOs were multi-sector operations. However, the number of sectoral DPOs has grown substantially in number and commitment amounts over time. Lending through sectoral DPOs has grown, particularly for policy lending with environmental goals. 1.4 This learning product focusses on the World Bank’s experience with DPOs in the Environment sector, broadly defined. For the purposes of this review, environmental DPOs were defined to be any policy lending operation mapped to the Environment and Natural Resources (ENR) Global Practice or, prior to that, the Environment Sector Board, or any other policy lending operation with an environmental or disaster risk management theme as the primary or secondary theme (see Appendix C). This experience covers a wide range of sectors, including climate change mitigation and adaptation, green growth, natural resource management, disaster risk management, forestry, environmental policy, and others. Much of the experience is very new, with 25 of the 64 operations yet to be evaluated. Many of the active programs are among the first environmental DPOs in their country or region. Many operations were designed and implemented by teams that included staff with relatively little policy lending experience, and so the opportunity for learning is substantial. 2 1.5 This product is not an evaluation of the environmental DPF portfolio; it does not seek to assess the overall impact of these operations, but rather to use this experience to offer lessons on how to design and implement this instrument to maximize its effectiveness. The conception of an effective DPO used through this paper is one that achieves relevant development objectives through policy reform relative to a counterfactual where the DPO did not exist. Yet there are seldom definitive answers on best practice in DPF design. The key challenges often involve tensions where either extreme can cause problems: between ambition and feasibility, between country ownership and Bank additionality, between breadth and depth, and others. Thus the lessons that follow are always context-sensitive; the art of policy lending involves determining where impact is likely to be highest under particular circumstances and constraints. 2. What does the portfolio look like? Environmental policy lending represents an important part of the Bank’s work. Environmental policy lending began with a small number of operations during the 1990s, but commitments grew rapidly in the 2000s to a peak, at the time of the global financial crisis, of nearly US$3 billion in both 2009 and 2010 (Figure 1). Though commitments declined after the crisis, the numbers of environmental DPOs have remained high by historical standards. Bank lending through environmental DPF has totaled over US$14 billion since 2000. Although the environment sector was the second smallest in terms of total lending commitments in the former Sustainable Development Network (SDN), in recent years it was the second largest of the SDN sectors in terms of DPF commitments. Of SDN sectors, environment has the highest proportion of commitments coming from DPF (39 percent of ENR global practice commitments since 2000) as compared to other instruments. Since 2000, environmental DPF has accounted for roughly 9 percent of the Bank’s DPF commitments. Figure 1: Number and volume of environmental DPOs, FY2000-16 Number and volume of environmental DPOs, FY2000-16 3,500 9 3,000 8 7 2,500 6 2,000 5 1,500 4 3 1,000 2 500 1 - - 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 Number of operations (right axis) Commitments $million (left axis) Source: IEG portfolio analysis. Note that operations prior to FY2005 are adjustment loans, not formally DPOs. 3 2.1 Environmental DPOs have supported a wide range of policy actions. The 64 operations in the portfolio collectively supported 396 environmentally relevant prior actions covering climate change mitigation, climate change adaptation and disaster risk management, environmental protection, pollution management, institutional strengthening, and specific sectoral reforms, along with another 131 policy actions supporting other types of actions (see Figure 2, and Appendix A Table 5). A database published as a companion to this review can help project designers locate specific examples where operations have supported particular types of policies. 2.2 Environmental DPOs are frequently multi-sectoral in nature, especially those designed as climate change or green growth operations. Many environmental issues are outside the control of traditional environment agencies, and are rather in the energy, water, agriculture, transport, and industry sectors. The ability to address environmental aspects of these sectors jointly through a common approach is a strength of the instrument. 2.3 Despite the wide variation in the types of policies supported, environmental DPOs have tended to generate lessons focusing on a common set of issues. These include issues of political economy, operation design and preparation, and institutional strength and capacity (see Figure 3, and Appendix A Table 6). Many of these lessons are generic to the instrument, and could also apply to other sectoral DPOs or even to multi-sectoral operations. They offer the potential for learning within the Bank across institutional boundaries. A database published as a companion to this review can help project designers locate specific examples where particular issues have played a critical role. Lessons from individual project evaluations were heavily focused on use of the DPF instrument and on process issues; they rarely touched on technical policy issues of specific environmental subsectors. 2.4 Performance ratings for environmental DPOs have been similar to those of the DPF portfolio as a whole. The World Bank’s 2015 DPF Retrospective noted that 81 percent of DPF operations during 2012–14 were rated Moderately Satisfactory or above (World Bank 2015); the 2012 Development Policy Lending Retrospective noted a similar figure of 84 percent for operations during 2010–12 (World Bank 2013b). Of the environmental DPF identified in this IEG report, roughly 60 percent have been evaluated with an ICR and ICRR. Of those operations, 81 percent were rated Moderately Satisfactory or above (see Table 1). It is difficult to draw any meaningful conclusions about comparative performance for subdivisions within the portfolio; given the small size a change in ratings by one increment (e.g. Moderately Satisfactory to Moderately Unsatisfactory) for a small number of operations could reverse any apparent patterns. 4 Figure 2: Types of prior actions supported by environmental DPOs, FY 2000–16 Source: IEG portfolio review. Note: Number in parentheses is number of instances in the portfolio of a type of prior action. 5 Figure 3: Lessons from self-evaluations and independent validations/evaluations of closed environmental DPOs, FY 2000–16. Source: IEG portfolio review. A “case” is a single standalone operation or programmatic series. Note: M&E = monitoring and evaluation; TA = technical assistance. 6 Table 1: Outcome ratings of environmental DPOs Number of operations Rating All Mapped to Mapped to Programmatic Standalone operations ENR GP Other GPs series Satisfactory 13 6 7 8 5 Moderately Satisfactory 19 7 12 8 11 Moderately Unsatisfactory 3 2 1 1 2 Unsatisfactory 4 1 3 1 3 Total 39 16 23 18 21 Percent rated MS+ 82% 81% 83% 89% 76% Source: IEG portfolio review of Environmental DPF. Notes: Figures are numbers of operations unless otherwise specified. No operations were rated Highly Satisfactory or Highly Unsatisfactory. ENR GP is the Environment and Natural Resources Global Practice. 7 3. In what context is policy lending most effective? 3.1 DPF can be a powerful instrument for influencing reform efforts:  …when policy issues are the main barrier to improving environmental outcomes. Environmental goals are heavily affected by policy frameworks, which set incentives and regulations that affect decision by the private sector. Many environmental challenges include governance, public good, and externality issues where policy interventions are needed to achieve efficient outcomes. Many countries have major gaps in their environmental policy frameworks and, thus, scope for improvement. If the binding constraints are not policy issues but rather issues of capacity, enforcement, or others then DPF may not be an effective instrument. For example, in an energy and environment DPO series in Turkey the Bank chose to open an engagement on environment through policy lending. An IEG evaluation concluded that policy barriers were not the most significant ones; rather, the main challenges were weak capacity and enforcement of existing regulations, so raising environmental standards did not have much impact.  …for achieving multi-sectoral or sector-wide goals. One of the great challenges of environmental management is that most issues are inherently multi-sectoral, crossing energy, water, agriculture, urban, industry, and transport sectors in addition to environmental management Here the policy lending instrument offers major advantages, in that it can help to address environmental issues in these sectors in a way that is more difficult for other instruments. Environment agencies are usually weak and have little influence on high level government policy making. An environmental DPO can help the Bank to engage on environment issues with other, stronger ministries, particularly finance and planning. With investment lending it can be difficult to influence sector-wide outcomes, and although there can be positive spillovers from demonstration or other effects, most impacts are within the bounds of the project. Available lending envelopes may be enough to fund a single large infrastructure project, while equivalent financing from policy lending could change the incentives for all projects in that sector. For example, in Himachal Pradesh in India, Bank staff argue that it was more effective to use policy lending to change how cumulative impact assessments were done for the hydropower sector at a policy level than to engage in hydropower investment lending, where available financing would likely be consumed by a single dam. Climate change and green growth DPOs can help to create an overarching narrative vision for a reform process, tying together efforts in disparate sectors with a common lens of sustainable development. Policy dialog attached to operations has the potential to help decision makers realize how environmental issues affect the whole economy and development agenda.  … when the Bank has prior knowledge of the country and sector and strong institutional relationships. Policy lending requires a substantial degree of technical 8 knowledge in the country and sector in order to support effective policies. It needs an understanding of the political and institutional dynamics, especially within government agencies: knowledge of which agencies and what people within them have formal and informal authority over particular decisions. And it requires a degree of institutional and personal trust, to facilitate access to senior officials and willingness within government to collaborate with the Bank. DPF can work better as part of a sequence where sectoral knowledge and engagement with counterparts has been built up by prior activities. In Turkey, a strong foundation of energy sector investment lending laid the groundwork for successful energy policy lending, but a relative lack of engagement on the environment made progress more difficult. In Himachal Pradesh, India, previous investment lending and a previous multi-sector fiscal DPO in the state supported the willingness of the government to engage in an environmental DPO. In Vietnam, prior engagement in investment lending in the water sector helped with development of a climate change DPO that added an environmental dimension to water resource management. It is more difficult to start an engagement with policy lending. If these prior forms of interaction don’t exist, then engagement is likely to need a longer and more intensive preparatory stage, which may also require additional resources.  …for issues that need attention from high-level decision makers, especially in finance and planning ministries. Line ministries may be weak (especially for environment) or have limited ability to implement large-scale reform programs. Environmental issues have often not been a priority for finance ministries, which do not always recognize the substantial economic, poverty, health and other impacts of environmental issues. The potential for the substantial financing amount associated with budget support helps to get the attention of senior decision makers and the opportunity to convince them of the advantages of identifying environmental issues as priorities. DPOs can help to influence budget directions, by, for example, helping to protect environmental funding during periods of budget cuts. 3.2 DPF can allow the Bank to take advantage of opportunities as they arise, but this poses risks.  Sometimes an opportunity arises when a new government or champion emerges which has an appetite for environmental reform, and who can generate buy-in from line ministries and agencies. It may make sense for the Bank to consider supporting policy lending in such cases, even when the existing base is weaker. This possibility argues for an ongoing minimal level of analytic work and knowledge generation in countries where the Bank believes such an opportunity might occur. However, an opportunistic approach can be highly risky and vulnerable to the loss of particular champions, and an understanding of these risks by the Bank can help to set expectations appropriately. 9  Yet it is difficult to support significant environmental policy reforms under emergency circumstances. DPF is designed as an instrument for achieving development impact through policy reforms, but its ability to offer fast-disbursing budget support also makes it appealing for client countries and Bank country management units to use the instrument to provide emergency fiscal support in the face of natural disasters, or macroeconomic or financial shocks. However, these circumstances tend not to be a context in which it is possible to prepare new substantial reforms. There are incentives to select prior actions that are either relatively minor or are already existing or well-advanced be likely to proceed even in the absence of the DPO. In such cases the additional impact of the DPO on policy reforms is modest, and the benefit is largely limited to that of the fiscal transfer. For example, in Samoa a 2013 DPO was rapidly prepared in the wake of a severe cyclone. The operation provided much-needed rapid fiscal support to the government on grant terms. But many of the specific prior actions of the operation either were weak (including minor changes to planning guidelines and very preliminary preparation for transport reforms—which were not implemented), would have occurred in much the same manner even in the absence of the DPO, or had been completed already before the conception of the operation (IEG 2016b). It did also, however, include support for a significant policy action implementing a housing reconstruction loan scheme. Although DPOs could be used to support substantial reforms on disaster risk management, this usually requires time that is not available under emergency circumstances when pressures for rapid disbursement are high. 3.3 The effectiveness of environmental DPF has been dependent on country context.  It has proven difficult to get country support for environmental policy lending in high-poverty contexts. Sometimes these are cases with the most serious environmental issues – but they also face the most serious other development challenges, so it can be difficult to convince governments to give priority to the environment. Doing so requires making a strong economic case, which may be easier for environmental policy lending with a focus on natural resource sectors, where improved management can be a source of growth. Environmental DPF has been easier to support in large middle-income countries, where environmental quality and sustainability are a higher priority, and where the size of the Bank’s portfolio is larger and more able to support standalone operations in specialist areas.  The timing of a DPO can be a significant determinant of its impact. Operations may be more effective if they are aligned with timing of national development plans and decision-making contexts. The Bank needs to have sufficient knowledge of political processes to understand the point in the cycle of a government when it is most open to being influenced on policy decisions, and to take advantage of opportunities as they arise. For example, the substantial portfolio of environmental policy lending in Mexico was motivated in part by periodic financial crises that made the government interested in budget support, and in part by Mexico’s intensive efforts to meet the 10 requirements for joining the OECD, in addition to severe domestic environmental problems and growing awareness of threats posed by climate change. 3.4 DPF is often most effective when used in parallel or coordination with other instruments. Bank staff and managers interviewed by IEG often stressed the importance of “twinning” DPOs with investment loans wherever possible to ensure greater likelihood of their effectiveness, because each instrument offers its own strengths. Prior investment lending can help to build the relationships, trust, political and technical knowledge of the country and sector, and institutional capacity that are needed to support a policy dialog and reforms. Parallel or subsequent investment lending can help to support implementation of policies. For example, in a climate change DPO in Indonesia, more progress was made in renewable energy and disaster risk management in part because of stronger prior Bank institutional engagement in those areas, supported by ongoing or expected investment lending projects. Whereas in the same series, forestry actions made less progress in part because of tensions and lack of engagement between the Bank and the forest ministry, though powerful competing economic and political interests in the forest sector (absent in other sectors) also explain much of the difference. 3.5 The DPF instrument has a number of limitations for pursuit of environmental goals. Although it can help achieve policy changes and bring policy frameworks to the point where they are ready to be implemented (such as by identifying the necessary enabling regulations and authorizations), it has seldom been used in the environment sector to improve enforcement. It can rarely achieve substantial results in a short time because policy reform can be a slow process with many potential obstacles and/or layers of approvals. It is limited by the constraints of country capacity and political economy. It is not the best instrument for supporting long-term capacity building. It should not be used with the expectation of saving Bank budget: DPOs can still be very costly to prepare. 4. Political economy in DPF There is a need for a clear political theory of change 4.1 The policy lending instrument can be most effective when there is a clear understanding of the underlying theory of change for the operation by the Bank. There are two related but distinct results chains within an operation design. One is technical, relating to how the specific policy actions supported in an operation will lead to the desired development outcomes (this is discussed in chapter 5 below). But the other is political, relating to how the existence of a Bank DPO would lead to changes in what would occur in the absence of the operation. A political theory of change will identify how the DPO is intended to have a particular desired impact, and how it will influence stakeholders to support this impact. This political theory of change may not be explicitly documented, given political sensitivities, but should be clearly understood and identified by the Bank task team to inform a strategy for maximum potential influence. This theory is key to ensuring that policy lending leads to development or environmental impact. Political economy analysis (PEA) may help to provide knowledge on the political feasibility of reforms and to identify 11 risks and mitigation actions; an IEG review of the use of formal PEA notes that this tool may be under-used by the Bank (IEG 2016c). 4.2 A DPO political theory of change can include impacts through a number of channels:  Change occurs because of a policy reform that would not otherwise happen without DPO.  Technical quality of reform is improved owing to Bank support through DPO.  Timing of reform is accelerated by Bank financing of DPO.  Sequencing of reforms is improved because of Bank DPO engagement.  Stakeholder buy-in of reform is strengthened by Bank DPO engagement. 4.3 Policy reforms that would not otherwise happen can sometimes occur in cases when government needs substantial technical advice from development partners in the design of an intervention strategy. In other cases it can occur because the large quantity of budget support funding means that the Bank can elevate existing reform agendas in line ministries (especially weak environment ministries) up to finance and planning ministries or other senior decision makers. Sometimes there are cases where a government intends to support particular reforms, but has struggled to get them done, outside support and pressure from a DPO can provide needed impetus. For example, in Morocco a green growth DPO helped win the approval of a regulation on coastal zone management. The regulation had been drafted in 1996, but had not previously been of high enough priority to be formally adopted; the presence of the Bank DPO helped to put the policy on the agenda of senior decision makers. But it need not be the case that planned reforms would not happen without the Bank; there are many other ways in which the Bank can have a positive impact by influencing when and how reforms occur. 4.4 Policies might be of higher technical quality because of Bank technical assistance (TA) and policy dialog. The Bank can help governments to set priorities and thus help some reforms happen earlier than they would otherwise. It can influence the timing or sequencing of reforms, to ensure that necessary prerequisites are in place. It can act as a relatively neutral third party to bring confidence to governmental and nongovernmental stakeholders in joint policy consultation and formulation. An energy and environmental DPO series in Turkey demonstrates several of these pathways (Box 1). Box 1: Political economy in energy and environmental policy lending in Turkey The Bank supported a three-operation programmatic series in Turkey on energy policy (especially in the power sector), climate change strategy, and environmental policy. On energy policy, the operation had a clear theory of change. The country had strong ownership of its reform agenda; it was not the case that reforms would not occur without the Bank’s DPO series. But the Bank added value by helping the Treasury and senior officials to prioritize the most critical electricity sector reforms and to add impetus to a reform agenda that was country-owned but was stalling. The high level of financing 12 (roughly US$2.1 billion) also helped to get the attention needed on electricity policy reforms from top government officials, including the cabinet and the high planning council; and the need to complete particular actions by a deadline in order for loans to proceed helped to apply pressure to implement reforms in a timely fashion. The Bank was well respected in government and the private sector, and Bank support for the reforms encouraged investor confidence by helping to reassure investors that market rules would be fair. The Bank also influenced reform sequencing, helping to ensure that privatization of distribution companies and energy market development would be done before privatization of generation companies, so that potential bidders for generation assets would know that there would be a commercial market with economic pricing for their product. However, an IEG evaluation concluded that the political theory of change was weaker for most of the environmental policy prior actions. The evaluation noted that on many environmental regulations the government had been motivated primarily by the requirements of harmonizing Turkey’s environmental regulations with the EU acquis and by its own priorities rather than by the Bank’s policy lending or dialog, and that, except in water resource management, the Bank was not very involved in the design of policies, concluding that the Bank DPO had only modest impacts on improving environmental management. Source: IEG PPAR case study (see Appendix B). 4.5 A range of other political change theories have been successful. The Bank may help to convince politicians that policies supported by their civil service are of high quality and that international best practice is being followed. It can give political traction to reform agendas supported by line ministries. The Bank can help to ensure the robustness of a policy framework, making sure that it contains all elements necessary for implementation. In improving air quality management in Brazil, the Bank helped to ensure that the policy framework included an appropriate legal authorizing environment, and that it established clear accountability elements identifying responsibilities. A strategy for achieving institutional buy-in can resolve political economy challenges 4.6 Bank task teams argue that the main political economy issues of environmental DPF have been those within and between government agencies, rather than risks from private sector lobbying. (Important exceptions are reforms in the forest sector and for energy subsidies, where private sector political economy issues are central.) Some teams note cases where private industry, wanting to reduce ambiguity in rules, has been supportive of environmental policy reforms with Bank involvement. Clear rules and regulations help to reduce investment uncertainty and help prevent governance challenges common in discretion-based systems. 4.7 Finance and planning ministries are the primary counterparts for the Bank for DPF, but significant line ministry engagement is also required. Policy lending provides general budget support which is attractive to finance ministries. But line ministries can be reluctant to embrace the DPF instrument because they do not receive direct financial support yet 13 remain tasked with implementation responsibilities, leading sometimes to an attitude of “What’s in it for me?” This perception can be particularly strong when the Bank switches from investment lending to policy lending in a given sector. Environment ministries can sometimes feel threatened by environmental reform efforts that shift the focus away from their traditional areas of influence and expertise toward reforms in water, energy, and transport that are outside of their control. 4.8 Bank teams have found some useful strategies for mitigating this challenge by highlighting the advantages to line ministries of the instrument. Strong personal and institutional relationships between the Bank and line agency leaders can help them realize that they retain substantial influence and control over the agenda. Agency leaders can be helped to recognize that the DPF modality allows them to use the Bank to obtain greater attention from senior government decision makers for their own priorities. Bank DPF and policy engagement sometimes convinces finance or planning ministries that additional budget for line ministries is required. Provision of TA directly to line agencies can help to get agencies on board by making them feel that they are getting tangible support directly from the Bank. In multisector environmental DPOs, where there are policy spillovers across sectors, agencies may be more disposed to buy in when they realize that the DPO offers them the opportunity to pursue policy reforms that promote their agendas but require the cooperation of other ministries. For example, in a green growth DPO series in Morocco, the choice of specific policies helped to motivate institutional buy-in for the larger program. Every action in the policy matrix was supported by at least one government agency, but most required support and input from other agencies as well. The environment ministry was willing to buy into a water law reform and pollution investment fund driven mainly by the water ministry, because the environment ministry needed support from the water ministry on coastal zone management reform. Moving forward on these reforms in parallel at a similar pace helped to boost cooperation because of mutual advantage. 4.9 Earmarking budget support funds directly to line ministries or agencies is not an effective strategy. The temptation may arise to try to gain the support of line ministries by explicitly or tacitly supporting the direct channeling of DPF lending to those ministries or their agencies. A background paper for the Bank’s most recent Environment Strategy conducted a stakeholder assessment on environmental DPOs and found that most officials in client environment agencies criticized the absence of earmarking of sectoral funding in Bank policy lending (World Bank 2010). But DPF is intended to provide general support to the overall government budget; Bank guidance advises staff that DPF funds cannot be earmarked to a specific sector. Finance ministries are welcome to use DPF proceeds however they wish, including to increase financial support for environmental or other agencies, which may be desirable when done through regular budgeting processes. But if the Bank is seen as involved in a specific channeling of resources to line agencies this can have negative consequences. It can insert the Bank into the relationship between the finance ministry and the line ministries and their agencies, hampering normal interactions around budget allocation processes. If agencies become heavily reliant on donor budget support, then there are risks to the sustainability of their operations when sector budget support operations are closed. Finally, it can serve to undermine the Bank’s approach to managing risks of adverse environmental and social effects in DPF, which focuses exclusively on the effects of policies rather than on the 14 impacts of financing. This notwithstanding, IEG has identified cases where a degree of earmarking appeared to have occurred in Brazil, Cameroon, Ghana, and Vietnam (IEG 2015a) and have triggered some of these consequences. 4.10 One of the most commonly cited lessons is the need for reform champions in government and political buy-in from key decision makers. Bank staff and evaluations emphasize the need for support from not just a senior figure with authority, but that of other secondary officials across ministries. An effective champion is one who can use influence strategies to achieve change by building coalitions within government. One of the most substantial operational risks for DPF is that of the loss of key champions; this is a common reason for unsuccessful or discontinued programs. Mitigating this risk requires building a broad support base, and recognizing that cases where reforms depend on a single champion face an inherent operational risk. In Ghana, the finance minister during the design phase was a major supporter, but his death early in implementation was a factor in the lack of progress. In Indonesia likewise, a key champion was the finance minister who then left that position. A decline in government commitment that followed (and had a number of causes) meant that the planned programmatic series did not proceed beyond the first operation, though some policy dialog was continued. In Himachal Pradesh, the Bank team argued that the stable presence, throughout the series, of key senior officials was a major factor behind successful implementation. Stable institutional political economy in the civil service has proved helpful in helping to sustain government commitment through a change in government. 5. Lessons on results frameworks Designing policy areas, actions, and related objectives involves tensions and tradeoffs 5.1 Policy areas and prior actions are the core of the DPF instrument. The decisions around selection of these actions are arguably the most important in determining the impact of the operation. Policy areas and their associated sub-objectives define the scope of the DPO and establish the structure of the results framework. 5.2 In choosing policy areas there can be tradeoffs between depth and breadth.  As noted in chapter 3, the ability to support environmental policy reforms across multiple sectors can be a strength of environmental DPF, especially in pursuit of climate change or green growth goals. A broad design allows many areas to be addressed through a common environmental lens.  In several cases, one of the most substantial contributions of a Bank environmental DPO has been to establish a platform for cross-sectoral cooperation. Often line agencies are deeply siloed, and a DPO has set up working groups and workshops at a policy or strategy level that enabled previously unseen levels of interaction across ministries. Bank teams interviewed by IEG had mixed views on the effectiveness of intersectoral working groups. They have been very powerful when it helps agencies to improve 15 how they manage cross-sectoral issues, and can have spillover effects well beyond those of the specific policies supported by the policy lending operation. In Ghana, a DPO covering forestry, mining, and environmental management greatly improved coordination across the relevant agencies, for example on the environmental impacts of mining, and on resolving disputes around mining in forest areas. In Colombia, improved cooperation between ministries was one of the most significant impacts of an environmental DPO series. But working groups were less effective if they were either at so high a level that the group was not a priority for members, or if they were at so low a level that members had little authority or influence. In Mexico, inter-sectoral working groups played a useful coordination role in early environmental DPOs, but in a third operation the process deteriorated as implementation progressed, in part because of a complex institutional context driven by the decision to cover six sectors (agriculture, energy, forestry, housing, tourism, water).  There are thus risks from program designs that are too broad. A larger number of policy areas typically increases the number of agencies involved, which can dramatically increase coordination challenges. It also increases the risks of non- completion of the series: the more agencies and policy areas there are, the greater the likelihood that failure to complete a particular prior action will delay or even derail the process. Implementation of policy reforms often requires engagement with sub- national agencies, which further increases coordination challenges.  In Morocco, reducing the number of agencies to work with was a deliberate strategy in the process of simplifying the policy matrix. In the first environmental DPO series in Brazil, the complexity of the operation, which covered 15 different policy areas across many different government agencies, was overly ambitious, and contributed to the inability of the series to advance beyond the first operation. A climate change resilience DPO in Mexico supported a strategy for sustainable tourism in three states, but had implementation difficulties because of a lack of collaborative relationships between states and difficulty in engaging with a plethora of entities at the municipal level. Too many policies can make a program too complicated, and with a fixed level of staff and budget resources can reduce the ability of the Bank to provide detailed support in each area. There is a risk that multi-sectoral DPO operations can become “Christmas tree” operations that seek to undertake too many things, and then dilute the effectiveness of the Bank’s engagement—for example there may not be strong synergies between combining environment and education in same program. Coordination is a particular challenge when mixing authorities from different levels of government (for example, national versus state, province or municipal government). Challenges in managing the different responsibilities of federal versus state agencies for aquifer water complicated implementation of a DPO supporting climate change adaptation in the water sector in Mexico. 16  The breadth of design can be dependent on factors such as the degree of centralization, hierarchy, capacity, and others. It is easier to operate a multi-sectoral DPO in a country with a high degree of top-down centralization. In more decentralized cases the larger number of agencies that must be engaged to implement a reform makes this more difficult. The Bank supported multisector climate change DPOs in both Indonesia and Vietnam. Progress was slower in Indonesia, where various levels of public administration needed to be mobilized to properly implement specific policy actions, as compared to more centralized Vietnam. 5.3 Within each area, the Bank and government jointly select the specific policy actions to be supported. In selecting policy actions, a number of factors should be considered.  The primary concern is to choose the actions that fit the technical theory of change and results chain of the operation: the supported actions should collectively lead to the desired objectives. Specific prior actions in environmental DPF have often been too weak to make a major contribution to achievement of their associated policy area objective. The actions should be policy changes, not merely production of other outputs. A common process is to start with a large number of potential actions, and then work with government through a process of winnowing. This process requires intensive engagement and debate with relevant line ministries and agencies. It can be useful to consider a number of criteria and tradeoffs that may inform this process. Selected actions may be those reforms that are highest priority or have the most impact. Or they may be those where the Bank presence will make the most difference – not necessarily the most important reforms, but those where the existence of time- sensitive disbursement conditions are more likely to have an additional effect relative to what would occur without the operation, or those where the Bank has the greatest ability to improve the technical quality of the policies.  The choice of actions will often involve tradeoffs between ambition and realism. More ambitious actions will increase the potential impact of the reforms. The Bank wants to choose policies that are difficult to achieve—otherwise the country would be able to do so on its own. There is a risk that reducing ambition too much can dilute the impact while still requiring substantial financing. But overly ambitious actions may not be feasible to implement, especially in the foreseen timelines. The Bank can increase impact by gently pushing government agencies slightly outside their comfort zone, but not so far as to overwhelm agency capacities or seriously damage client relationships. This inevitably involves making informed judgment calls based on the information available at the time.  Discussions on the selection of actions involve inherent and, preferably, productive tensions. Government agencies may seek to commit to less as a way to minimize the burden on them and the potential for failure. Bank quality review processes tend to push teams to seek stronger actions in order to increase the potential impact and justify financing commitments. A degree of customization and compromise based on the client relationship is needed. Sometimes the Bank will have to accept that some 17 needed reforms cannot be achieved in a planned operation. In Mozambique, the Bank initially wanted to include a policy reform on forest carbon in a climate change DPO series, given that forests played a major role in domestic energy supply, timber exports, livelihoods, and environmental services. Yet despite significant efforts the Bank did not manage to bridge the gap between the agriculture and environment agencies (which bore responsibility for forests and climate change and reducing emissions from deforestation and degradation (REDD), respectively) and so concluded that the policy area had to be dropped.  There can sometimes be tensions between additionality and ownership: The Bank wants to ensure that reforms have strong ownership from government agencies, but if strong ownership already exists in all government parties then potentially a Bank DPF is not needed, and the reforms would occur on their own or could be supported by standalone TA. If the Bank or other development partners push too hard for reforms that government had not planned on making or does not believe in, then this can lead to low government ownership and makes the reforms unlikely to succeed. In some cases Bank teams have deliberately not chosen policies that might generate substantial public opposition when there were concerns that this might affect the sustainability of those measures. Policies that are highly partisan can face substantial risk of being reversed. A green growth DPO in Colombia illustrates one approach where the Bank has made a meaningful additional impact on government-owned reforms. The specific policy actions were already present in the government’s national development plan. The Bank added value not by coming up with new policies, but by helping the government to identify and prioritize the most critical issues, and by helping it to put specific regulations and plans in place to make legal changes implementable.  There are sometimes incentives for Bank task teams to select policy actions that are convenient, rather than high-impact. This can lead to an operation that proceeds and disburses, but does little to advance the program objectives. Selection of policies that have been completed before the preparation of the DPO (or are so well advanced that the Bank cannot make a meaningful contribution) should be avoided. An IEG evaluation of an environmental management DPO in Brazil concluded that there was little evidence that the Bank’s DPO made a contribution to significant progress on environmental reforms undertaken by the Brazilian government, because many of the operation’s prior actions had been completed long before preparation of the loan even began—sometimes years before. In this case IEG found that it was implausible that the Bank DPO had a causal impact on adoption of the reforms.  Prior actions should also be framed in a way that is clear. For example, in some cases, operations have included prior actions to approve a particular regulation or order referred to by number, without including a description of what that measure actually was. Though it is clear and specific, this formulation lacks transparency: observers not familiar with the details of the operation are unlikely to understand what is being supported. Overall, DPF documentation could be improved; sometimes program 18 documents and ICRs make high-level general statements without clearly describing the actual contents of supported policies or how they would achieve their objectives. In 2015, IEG produced a Learning Product on results frameworks for DPF operations (IEG 2015b). The main findings of this review are relevant to designers of environmental DPF (Box 2). Box 2: Main findings from IEG Learning Product on DPF results frameworks Some DPOs suffer from lack of clear statements of objectives and outcomes. Their results frameworks lack explicitly stated outcomes, while results indicators in many cases fall short of meaningfully measuring a DPO’s impact. Some straightforward solutions are available to improve the results framework presentation, and DPO results orientation and evaluability. The quality of prior actions is critical for the robustness of the results frameworks of DPOs. The review highlights recent improvements in prior actions and suggests further steps to improve their quality: • Avoid actions that do not support significant policy changes, such as draft regulations at early stages of preparation (before approval by governments), agency-level actions with little or no tangible implication for overall policy, statements of intentions, repeated prior actions on recurrent government functions that lack additionality, and “pilot” actions without a clearly defined scaling-up strategy. • Avoid policy actions unrelated to Bank engagement with the client country, which are contrary to the Bank’s approach to budget support and undermines the additionality of DPOs. The report suggests formulating guidance and standards on prior actions to improve the results orientation of DPOs. • Excessive use of flexibility in a programmatic series can compromise a DPO’s focus on results. Dropping essential triggers or accepting partially met triggers that do not capture the true character of intended reforms may substantially undermine the quality of results frameworks. Although maintaining flexibility in DPOs is important, in many cases a better balance between flexibility and rigor would improve a DPO’s focus on results. The review also notes that recent changes in the presentation of medium-term reform programs in a programmatic series have improved the clarity of results frameworks. Source: IEG 2015b 5.4 IEG informally uses four criteria for considering the quality of prior actions (Table 2). These provide a useful framework for thinking about how to produce a strong results framework. Table 3 describes examples of policy actions taken from environmental DPOs, assessed against these criteria. 19 Table 2: IEG Criteria for assessing the prior actions (PAs) of development policy operations Criterion What Does IEG Look for? Relevance The extent to which PAs are relevant to objectives and associated outcomes. Institutional depth The extent to which PAs are sufficient to achieve policy changes or and criticality reforms, as compared to those that will mean many subsequent steps to generate and outcome. Policy actions that are excessively process- oriented, easily reversible, or that only indicate intentions should be avoided. Additionality The extent to which the World Bank adds value to the specific PA relative to a counterfactual without the DPO. Measurability The extent to which the expected impact of PAs is measurable. This largely depends on the quality of the M&E framework and the links between results indicators and PAs. Source: IEG 2015b Note: M&E = monitoring and evaluation; PAs = prior actions. 20 Table 3: Examples of prior action Policy action Assessment on criteria Made satisfactory progress in the negotiations with Making “satisfactory progress” in the European Union on a voluntary partnership negotiations does not have high agreement concerning the “Forest Law measurability or criticality. It is Enforcement, Governance and Trade” initiative, quite subjective and so difficult to and has defined the elements of such agreement by assess completion or noncompletion, December 2007. (Ghana) and follow-up actions are required to reach an agreement. Prepared, through the Recipient’s Minerals This prior action did not have high Commission, a proposal for new guidelines on criticality; preparation of a social responsibility in mining activities, which proposal for guidelines takes into account, inter alia experiences with accomplishes little by itself. Two alternative livelihood programs and community subsequent prior actions in the development schemes in the mining sector, for series were required to get to consultation with relevant stakeholders. (Ghana) issuance of the (nonbinding) guidelines. Forest legal framework strengthened through the This prior action had little enactment of three key legal acts: Public Forest additionality, because the laws in Management Law which promotes forest question had been passed years management in public land; Atlantic Forest Law before the design of the operation, which promotes conservation of this highly and had been used as prior actions endangered biome; and, Resolution 3545 of the for a previous Bank DPO in Brazil. National Monetary Council that regulates bank lending to agribusiness (Brazil) The bidding process for the first two lots of The action has high relevance and, distribution companies launched by the criticality– privatizations had to be Privatization Administration, with winning bidders completed, and privatized for the first two distribution companies determined companies had to abide by their in July 2008 and for the next two distribution contracts, which included companies in September 2008. (Turkey) investment and loss reduction targets. Moderate additionality was provided by the specific deadlines – the Bank DPO helped to ensure timely progress. Monitoring and evaluation systems can be improved 5.5 Policy lending faces inherent difficulties in designing monitoring and evaluation frameworks and in selecting indicators because of the disconnect between the length of time 21 needed to observe results and the point at which operations are evaluated. Many effects of policies take years to occur. For example, some policy reforms are designed to influence downstream behavior; they may alter incentives that will then affect subsequent public investment and private decisions and so will affect outcomes gradually over a long period. Operations may support substantial legal or policy shifts at the highest levels that then require substantial follow-on work in regulation and supportive policies in order to be implemented. Yet self-evaluations must be finalized a year after closure of an operation or series, and many DPF operations close shortly after approval and disbursement, so they rely on data that are generated sometimes only a matter of months after adoption of the policy. Independent validation systems from IEG rely on the same data sources. Designers thus face the challenge of designing systems that are deep enough to generate evidence on outcomes or impact, but also modest in design and focused on the short term. 5.6 Monitoring and evaluation systems for most environmental DPOs have been weak. For the 31 operations for which IEG has assigned a monitoring and evaluation quality rating, 21 had ratings of negligible or modest, with only 10 having a rating of substantial; none had a rating of high. IEG desk reviews often note objectives that are imprecise or unclear (and so are often not meaningfully evaluable), and indicators that did not provide a direct or adequate reflection of the objectives or sub-objectives with which they were associated. Results frameworks sometimes end up measuring processes rather than results or impact. Task teams interviewed by IEG often report that they struggle with the design of monitoring and evaluation systems. Indicator selection is sometimes seen more as a compliance burden than as something that adds value to the operation. Some DPO teams argue that Bank budget for DPF is concentrated in the preparation phase, which is when most of the work occurs, but that limited budgets during implementation make it difficult to support an intensive monitoring and evaluation program. Utilization of monitoring and evaluation is rare; systems are used largely for reporting purposes and for tracking progress on triggers for future operations in a programmatic series rather than for assessing the need for course correction. This lack of interest in monitoring and evaluation by the Bank may have been a contributing factor to the lack of prioritization, funding, data collection, reporting, or field verification by governments observed in some operations, and consequently to a missed opportunity to benefit from monitoring and evaluation systems. Objectives should be clear and evaluable, and should balance ambition and realism 5.7 The choice of program objectives is important both for helping to create a storyline and vision for a program and in establishing goals against which progress will be assessed. The Bank’s self-evaluation and independent evaluation systems are objectives-based; performance is judged against achievement or likely achievement of objectives. These objectives are the overall program development objectives, not the individual objectives and targets for specific policy areas. 5.8 As much as possible, objectives should be outcome-oriented; they should capture the goals that determine environmental and economic wellbeing. Yet in choosing objectives, there is a tension between the level of ambition and realism as to what can be demonstratively achieved at the time of evaluation. 22  If objectives are pitched too high, they may be unachievable and, thus, subject to poor ratings from accountability systems. In Gabon, a forestry and natural resource management operation had overly ambitious objectives, with a stated goal to "increase the contribution of natural resources to national income, and to help reduce the country’s heavy dependence on declining oil resources while protecting the resource base and improving management efficiency.” These objectives reflected the broad goals of the government’s long-term strategy rather than what was realistically achievable within an operation that covered the review and cancellation of logging permits, forestry company restructuring, and a range of other governance and resource management policy measures. More achievable objectives could have reflected the real design of the operation—perhaps to improve the sustainability, efficiency, and transparency of management of forests and other natural resources. Sometimes, moreover, DPO objectives include goals that are not achievable by the actual actions supported.  If objectives are pitched too low, then there comes a problem of insufficient relevance or evaluability. Sometimes objectives have been framed in terms that are unnecessarily weak, such as “to adopt a policy on X” rather than targeting a specific goal directly. The best objectives are those that are an achievable stretch. It is useful to ask: what would success or failure look like? If the objective is framed in terms such that it cannot fail (for example, by being automatically achieved by the completion of prior actions), then it is not meaningfully evaluable, and evaluation systems will add little value. Yet it is very hard to evaluate the level of ambition fairly ex post; it becomes too easy to assume that anything that wasn’t achieved was unrealistic. 5.9 Together with ambition is the level of clarity and evaluability. An IEG portfolio review for this report found that many objectives were unclear or unevaluable. Even among clear objectives, many use a passive framing of “to support the government achieve X” rather than using a more active framing that implies that the operation will lead to concrete achievements. Many objectives included relatively clear primary objectives buried within larger statements containing overarching objectives or “by” or “through” statements about means and methods. Objective statements can be more evaluable if they separate the specific program development objectives intended to be achieved by an individual operation or programmatic series from higher-level motivating goals (such as to “balance socioeconomic development with environmental protection and improvement”), or from statements about processes or methods. The Bank’s formal objectives need not specifically include higher- level objectives of a major government program of which the operation supports only a part. Objectives could be clearer if they can be more specific instead of including language like “mainstreaming of environment in policy,” “promoting inclusive green growth,” or “greening physical capital,” which is much harder to usefully assess. Macroeconomic goals that are driven by the budget support financing aspect of DPF (rather than by the policy reforms) such as “to support the government’s stimulus program” are usually better avoided as objectives; they are inputs rather than outcomes and are not meaningfully measureable or evaluable. Goals such as promotion of “long-term sustainable growth” are not likely to be 23 achievable within the evaluation time frame. Box 3 gives examples of objectives that are relatively clear and evaluable, demonstrating the feasibility of this approach. Box 3: Examples of relatively clear objectives from Environmental DPOs “…to develop a lower carbon, more climate-resilient growth path.” (Indonesia) The objective makes clear that the goal is to support climate change mitigation and adaptation, but without adding the likely unreasonable burden of achieving a reduction in greenhouse gas emissions by the end of evaluation. “(1) …increasing renewable energy supply, promoting energy efficiency through cogeneration, reducing gas flaring and venting, and piloting the reform of the rural agricultural electricity subsidy; (2) improving the efficiency of the vehicle fleet and road transport operations in Mexico; (3) strengthening the market for energy–efficient housing; and (4) mainstreaming climate change considerations into land–use and forestry activities.” (Mexico) Long lists of sub-objectives are generally undesirable, but some inclusion of some multiple objectives may be unavoidable in complex operations that cover many sectors. “…to support Samoa in recovering from the immediate impacts of cyclone Evan…and to help Samoa build resilience against such shocks in the future.” (Samoa) This clearly states and separates the desired goals of immediate support and future risk reduction; the challenge is in selecting indicators that can show at the time of evaluation that resilience reductions will be likely to be achieved. To “…a) enhance energy security; b) integrate principles of environmental sustainability, including climate change considerations, in key sectoral policies and programs; c) improve the effectiveness and efficiency of environmental management processes.” (Turkey) The objective would be clearer still if accompanied by definitions of energy security and of principles of environmental sustainability. “…to improve the economic, environmental and social performance of the municipal solid waste sector.” (Morocco) This lays out the three criteria on which success will be judged. “…to strengthen public and private efforts to achieve socially-, economically-, and ecologically-sustainable use of national forest and wildlife resources.” (Cameroon) This clear and outcome-oriented objective was expected to be achievable because the two- tranche operation would have substantial time between initial policy actions and closure and evaluation. Indicators can be selected to assess achievements 5.10 Results indicators are useful because they provide evidence on achievement of desired outcomes, which allows for performance assessment and course correction, and 24 because the systems for reporting tend to create incentives among program implementers to focus on those metrics that are measured.  Indicators are most useful when they accurately reflect and are framed in terms of outcomes. For many key outcomes it may not be possible to observe changes by the time of evaluation, or to reasonably attribute changes in them to a specific operation. However, it is often possible to select indicators that measure progress on intermediate outcome indicators, and sometimes even for more downstream outcomes. Intermediate outcome indicators are most effective if the link between the intermediate outcome and the downstream final outcome is strong and uncontroversial: for example reducing sulfur content in fuel is highly likely to have positive downstream effects on air quality.  Output indicators can be useful in establishing that a results chain is being implemented. They can help to build confidence that policy changes are being implemented at an operational level, or that changes in targeted outcomes are the result of DPO policy actions. For example, an operation might have a prior action related to passing a particular piece of legislation, and then use an indicator to track the creation of associated regulations needed to implement the policy as a proxy for success in implementation. It can be useful to try to include a measure of quality in indicators: for example, to look at the number of hectares of forest under management plans complying with Forest Stewardship Council or other similar standards, rather than just looking at the number of hectares of forest cover under management plans. Box 4 shows some examples of indicators that have been used in environmental DPOs to indicate progress on intermediate or actual outcomes. These demonstrate that it is possible to have outcome-oriented indicators even within limited timeframes. Box 4: Selected outcome-oriented results indicators from environmental DPOs Percentage of municipal solid waste collected and disposed in sanitary landfills (Morocco) This is a good outcome indicator, though other evidence would be needed to demonstrate that is attributable to the policy reforms. Number of smart [power] meters installed (Poland) The link between installation of smart meters and reduction in nontechnical losses is well understood, so installation—and utilization—of smart meters would be a reasonable proxy. However, nontechnical losses could also be a useful indicator if enough time elapses between installation and the point of evaluation. Tons of industrial pollution (BOD5) abated per year (Morocco) This is a good outcome measure, provided that there is a robust methodology for calculating abatement. Subsidy expenditures for diesel, gasoline and industrial fuel as a percentage of GDP (Morocco) Tracking expenditures on subsidies is a useful way of assessing whether changes in subsidy policy were implemented. 25 Implementation of an integrated public transport network through which at least 90% of the households in each of the 10 service areas will be located at less than 800 m from a stop served at least by 5 buses per hour and/or a metro line (from 6AM to 10 PM) This is a clear and specific indicator that demonstrates completion of a planned network system—though a better measure of access would capture whether households were actually using the service. Percentage of eligible surface that is covered by climate risk insurance (Mexico) This tracks expansion of insurance coverage—though a better indicator might track not just whether insurance was available but whether it was used. Number of water quality monitoring sites established in 13 river basin organizations (Mexico) Measuring the establishment of sites gives evidence that a policy change has been operationalized—but a more useful indicator could be one that captures whether sites are reporting data as expected. Percentage of district roads that are constructed or upgraded from 2014 onwards that are in compliance with revised design standards. (Mozambique) This helps assess whether revised standards are being applied in practice, if it is accompanied by a verification process that confirms compliance. Number of vehicles converted to natural gas and number of service stations in Lima with natural gas supply installed and operating (Peru) Conversion of vehicles to natural gas is a reasonable short-term proxy for likely reductions in emissions. Energy savings by heavy industries compared to forecast under business as usual scenario. (Vietnam) Comparison to a business-as-usual forecast is more meaningful than comparing to an initial baseline in sectors where changes are likely occurring even absent policy reform. 5.11 A number of other factors have weakened results framework designs in some environmental DPF.  In many cases, results frameworks rely almost purely on outputs, which give little confidence that downstream results will be achieved. In some cases outcome indicators are so far downstream that it is difficult to know whether or not these could be the results of the targeted policy reforms. Some operations have had indicators that track the production of prior actions from the same operation; these indicators are not useful for assessing progress because, by definition, prior actions are completed in order for the operation to be approved and for the indicators to come into force. Sometimes indicators are added that track completion of outputs that are not related to the policy reforms in question (such as an output from a project 26 funded by another donor), as a form of soft conditionality. Sometimes indicators are chosen with weak, missing, or purely qualitative baselines. Sometimes robust results indicators were dropped during the preparation process in sometimes tense negotiations with government. Inevitably, a judgment call is required by task teams on how far outside of their comfort zone government can be pushed to collect a particular indicator.  Indicators should be clearly identifiable, ideally in quantitative terms. Qualitative indicators can be highly subjective and so subject to pressures to provide positive spin. For example, an indicator in a natural resource management DPO in Gabon was that “Previous progress is consolidated, and further steps enjoy continued political backing,” with no definition of what consolidation of progress would mean, or how political backing would be judged. Indicators can be more effective if they can be measured continuously rather than being binary, so the indicator can be used as a measure of incremental progress rather than a blunt “yes/no.” Qualitative indicators may be appropriate in some cases, but require clear attention to measurability such as by establishing progress benchmarks or developing a scaling system.  Some operations included too many indicators. In Ghana, the results framework contained roughly 60 indicators which were to be assessed and monitored on an annual basis; in Cameroon the matrix included roughly 100 indicators, each with its own verification measure. The risk in such cases that the resulting monitoring process both overwhelms the capacity of client agencies and crowds out meaningful policy dialog from Bank-client interactions. The tendency towards too many indicators seems to be more common in operations cofinanced by multiple donors, where there can be political incentives to add large numbers of indicators corresponding to wide-ranging donor priorities, or in operations where potential prior actions or triggers considered earlier in the design process are stripped out as formal conditionalities but are added as indicators.  Sometimes it has been a practice to establish DPO indicators based on the indicators already being used in country programs. This is understandable, given its relative ease when the data are already being generated and tracked; yet if the country monitoring and evaluation framework is inadequate then the Bank DPO will also be weak. Country program indicators may also have different timeframes from a Bank DPO and so may need different indicators. 5.12 It is often easier to select more effective indicators for programmatic series than for standalone operations. Programmatic series are typically two to three operations over a multiyear period, and evaluation does not occur until after closure of the final operation. More time will pass between the adoption of a policy change and the point at which evaluation occurs, so it is easier to observe changes in more downstream indicators. Task teams interviewed by IEG strongly argue that it is difficult to have indicators that can demonstrate changes in environmental outcomes from a standalone DPO. Yet even for programmatic series the time horizon is relatively short, especially for actions completed later in the series. Although DPF operations can close shortly after disbursing, in some cases 27 the Bank has deliberately kept operations active for up to a year after approval, to allow more time for policy engagement and monitoring. 5.13 Monitoring and evaluation systems are not being set up to assess longer-term impacts of Bank engagement. A fundamental challenge is that the Bank’s monitoring and evaluation systems are heavily concentrated at the level of the individual operation or series; this leads us to judge success or failure based on short-term evidence. Formal evaluation systems rarely support the ability to make judgements of medium- to longer-term impacts. This is a particular problem for DPF because of the relatively shorter time between production of outputs and assessment of results. Sometimes the idea is suggested that subsequent operations can monitor progress, but in the formal system it does not make sense for a project to monitor results that are not attributable to that project. Information about longer-term impacts exists as tacit knowledge held by particular staff members who have been working in a particular country and sector. But this approach poses greater risks of subjectivity and potential conflict of interest, means that knowledge can be lost as a result of staff turnover, and makes information more difficult for “outsiders” to access then it would be from a formal evaluation. This can result in a situation where large sums are being disbursed with a lack of evidence on overall impact. 5.14 The clearest example of this challenge is in Mexico, where the Bank approved eight environmental policy lending operations over 11 years, and disbursed more than US$4 billion. Though strategically linked, each operation was formally a standalone operation and a self-evaluation was produced after each one. So indicator selection was built around the timeframes of the individual operations. Sometimes there was overlap between indicators in two adjacent operations (for example, on CO2 emissions), but then the next operation moved on to a different topic with different indicators; the combined pool of monitoring and evaluation data is consequently quite heterogeneous. This means that even in aggregate, Bank documents provide little evidence on which to assess overall improvement in terms of environment-related physical and human outcomes. 5.15 Thus, there is a risk that the Bank is perpetually living in the short term. The Bank could work to find opportunities for joint assessments with the country of longer-term policy- related performance, but this would require data collection on a longer-term basis for a meaningful assessment. Current Bank systems do not support this approach. A forthcoming evaluation of the Bank’s self-evaluation systems (IEG 2016a) notes how the Bank Group’s self-evaluation systems have favored accountability over learning, and recommends that the Bank expand the use of voluntary evaluations that respond to the learning needs of management and teams. This approach could include, when warranted, a retrospective learning-oriented assessment of results from large clusters of operations in a given country and sector, such as for environmental interventions in Mexico. 6. Lessons on DPF preparation Analytical work and technical assistance are critical underpinnings 6.1 Analytical work plays a central role in DPF design and preparation. 28  Up-front analysis can help to identify the most critical environmental issues faced by a particular country. It can play a key role in supporting policy dialog with senior officials and in advocacy by the Bank of appropriate strategies. It can help to provide the evidentiary base needed to convince decision makers. In particular, the Country Environment Analysis (CEA) can play a major role in supporting policy dialog: CEAs have helped to establish environmental priorities, carry out institutional analysis, and support cost-benefit analysis (World Bank 2008). DPOs are required to draw on analytical work under Bank operational policy, and all DPOs cite lists of prior studies; but, in practice, the degree to which existing analytical work actually meaningfully informs program design is uneven.  The technical quality of analytical work is sometimes a major part of the Bank’s comparative advantage. Sometimes government agencies lack the expertise to do high-quality analytical work on their own. In other cases, higher-capacity governments have a wealth of analytical work and expertise that can serve as the basis for reform; the Bank needs to make sure not to consider only Bank-produced analytic work in informing design. If analytical work is done in a collaborative way with clients and drawing on their own work, then this can increase ownership and uptake within government, as occurred in Colombia.  In some cases analytical work has directly informed DPO design. In Ghana, analytical work and a CEA on the economic costs of natural resource degradation played a key role in galvanizing the finance minister and senior finance ministry officials to support policy reforms on natural resource and environmental governance. This also helped to shift the scale of Bank action from a proposed single action in a multisectoral budget support operation to a freestanding environmental programmatic series covering forestry, mining, and environmental management. Analytical work also produced specific recommendations, most of which mapped directly into policy actions in the DPO series. Similarly, in Morocco, analytical work showing the client the benefits of green growth (with work covering water, fisheries, energy, pollution, and climate change) was central to convincing the government to support the operation.  In other cases the connection between analytical work and specific policy actions has been indirect but still substantial. In Colombia a high-quality CEA identified environmental health issues (such as those from indoor and outdoor air pollution) as the most costly forms of environmental degradation, falling most heavily on the poor. Dissemination of this analytical work at a critical time helped to get environmental issues onto the country’s policy agenda by influencing politicians and public opinion. The Bank was thus able to have a significant influence on environmental policy reform, despite a relatively modest prior engagement in environmental investment lending. A CEA likewise helped to inform design of a climate change series in Indonesia; and a study on climate-resilient development emphasizing the Bank’s role was also important in Vietnam. In Mozambique, the Banks’s country study on the economics of climate change adaptation raised the profile of the issue and established the need for action, convinced decision makers, and influenced the government’s 29 thinking and agenda. Government then chose sector priorities and proposed policy actions based on this goal. 6.2 Yet sometimes preexisting analytical work is not relevant. It can be difficult to rely on the existing base of analytical work that was not designed with the goal of informing policy lending. Bank staff argued that quality of CEAs has been often excellent but inconsistent in quality: for example, only a few of the early CEAs supported detailed analysis linking environmental priorities to poverty reduction, assessed political economy constraints, or analyzed resource flows or transparency (World Bank 2008). Resource constraints mean that fewer CEAs are now produced than in the past (as documented in IEG 2015a). New tools such as the comprehensive Systematic Country Diagnosis (SCD) cannot be expected to provide relevant analytical support; of necessity the SCD is a much broader document, with many sectors and issues competing for space. 6.3 Technical assistance is often critical in supporting design and implementation of policy reforms.  TA can contribute technical knowledge on international best practice that may be lacking. TA can support the country’s own processes and cycles for decision making, and can finance parts of the policy development process which may not be well covered by existing agency budgets. TA accompanying a DPO can have strategic influence beyond that of standalone TA support. For example, it can improve buy-in and commitment from ministries implementing reforms. Conversely, a lack of accompanying TA can have a serious effect on the ability of governments to implement policy reforms on expected timetables. Interviews with Bank staff suggest that insufficient or low-quality TA can be major causes for poor performance of DPF.  Care needs to be taken that TA is provided to the right institution. In Mozambique, TA was originally provided to a weak institution that lacked decision-making power and influence, which meant the support had little impact. Recipient-executed TA is not always what is needed: Bank staff note that ministries are sometimes too slow to make hiring decisions, and reluctant to pay higher prices for international expertise. TA may be insufficient to support institutions that are extremely weak.  TA should also not be expected to substitute for a lack of institutional commitment or client ownership. In Indonesia, parallel sector TA by other donors in the implementing agencies was not sufficient to support implementation of some policy actions in a timely manner, because of the lack of agency buy-in. 6.4 Bank staff working on DPF are nearly universal in their call for substantial analytical work and TA. Yet often they note that what occurs is insufficient. Why? Several answers have been suggested.  One constraint on analytical and advisory work has been that of budget resources. Bank budget availability is tight and has continued to tighten over recent years. Bank staff report that budget decisions have tended to prioritize preparation and 30 supervision, meaning that analytical work is often what is cut. TA and analytic work have often been reliant on trust funds, which have been valuable for both their financial support and flexibility. But many of these resources have declined over time; for example, trust funds for Country Environment Analysis and Strategic Environmental Assessments were exhausted in 2012, and so the number produced has declined. Some trust funds have shifted toward climate change goals, which do not always meet client needs. The ad hoc nature and timing of trust fund resourcing also creates challenges; often Bank teams will apply for funding, but most proceed with appraisal without knowing whether the resources will be forthcoming. Bank teams emphasize that greater predictability and reliability would assist them in effective operational planning.  A second constraint has been that of timing. Often there are strong pressures for rapid preparation and disbursement of a DPO, but this means that it is very difficult to undertake new analytical work to support operational design; hence, most often the operation must rely on whatever work existed at the time of conception, which may not have been undertaken with policy lending in mind. Analytical work prepared without a DPF in mind may focus on filling knowledge gaps but may not be relevant for the areas where political will for policy reform exists that might be supported by a DPO. Sometimes timing constraints have meant that analytical work was not completed until after a DPO operation was approved, and so largely too late to influence policy design. These challenges argue for an approach to DPF that emphasizes greater advance planning rather than opportunism: for example if a CPF identifies the opportunity for environmental policy lending then it could also ensure that the necessary analytical work was programmed in advance.  A third has been unwillingness of governments to borrow for TA, sometimes seeing these as unproductive investments. Governments are rarely willing to spend funds from the budget support on TA; high-quality TA sometimes requires expensive international consultants. The Bank has limited leverage with which to induce governments to adopt TA, because it cannot credibly threaten not to proceed with the operation in the absence of TA. However, the Bank could do more to demonstrate how appropriate TA leads to better results.  Some staff argue that the Bank’s internal incentives are a barrier to prioritizing knowledge work. They note that there are few perceived rewards for strong analytical or other knowledge work and that internal incentive systems (performance reviews, management attention and visibility, and promotion possibilities) are driven by loan approvals and disbursements. Programmatic series over advantages over standalone operations 6.5 Programmatic series offer a number of advantages over standalone operations.  Bank staff argue that the structure of a programmatic DPO series, which explicitly ties future budget support to achievement of particular policy triggers, increases the 31 leverage of the Bank to obtain sustained commitment for more ambitious reforms, and helps to provide needed incentives for continued reform impetus. This can be particularly important for environmental policy reforms, which often take a long time to develop and to achieve outcomes. The structure can help to support a more forward-looking policy dialog. And the longer timeframe can allow for more effective monitoring and evaluation.  However, there are risks from a programmatic series design that postpones the most substantial reforms to subsequent operations. If all planned operations in the series do not proceed, then the Bank may disburse substantial funds with relatively little impact. In two separate environmental DPOs designed as programmatic series in Brazil, one reason for their limited impact was that many of the most significant environmental reforms were left to the second and third operations. When each series lapsed after the first operation (and after disbursing US$500 million for the first series and US$1.3 billion for the second), this left the series with modest impacts. In a programmatic series on climate change in Indonesia, major actions on energy subsidy reform were left to the second operation, which never occurred.  Programmatic series have typically included operations spaced one year apart, but this is not a formal requirement. Sometimes environmental DPF series have struggled with designs that spaced operations one year apart, because of the difficulty of completing complex reforms within a year. In Ghana, the single year between disbursement of one operation and assessment of triggers for the next meant that only moderate progress could be made under each operation. Combined with the large number of outputs and targets being tracked, much of the supervision process focused on verification rather than client support and engagement. A short program time horizon also led to selection of very intermediate outcomes for program development objectives, which meant that the program framework and monitoring and evaluation progress were not focused on addressing the main outcomes of concern in the sector (resource degradation, environmental damage, etc.) because of the limits to which reforms could be accomplished in a single year.  Programmatic series can offer flexibility between operations by adding, modifying, or removing triggers; however, this requires careful consideration. If the Bank is too willing to drop any trigger that might not be completed, then the credibility of the instrument is undermined and conditionalities can become meaningless. Yet sometimes particular triggers really are unachievable, and it may not be worth cancelling an entire DPO series because of one failed trigger. The Bank has to make a judgment call on whether any dropped triggers are really infeasible, and on whether they are of sufficiently marginal importance to the program that it can proceed without them. Two cases illustrate this tension. In Turkey, the Bank dropped actions on Strategic Environmental Assessment and on amendments to a gas market law, judging that these were not achievable. There was pressure to drop a third action on approval of an energy efficiency strategy, but the Bank resisted because the action was seen as critical, and the strategy approval was then completed in time. In Indonesia, a climate change programmatic series was left incomplete after failure to 32 achieve some triggers. The Bank held firm in insisting on compliance with that the triggers established for the proposed second loan, judging correctly that it would be difficult to achieve the program objectives without them. Other design and preparation factors influence implementation 6.6 Consultations. Consultation with stakeholders is formally a responsibility of the client, but the Bank can play a role in supporting and increasing the credibility of consultation processes. Extensive consultation and engagement contribute to better design and generate goodwill on the part of the client. Two series of environmental DPOs in Brazil helped to contrast different extremes in this case: in the first series, the Bank team carried out a long-term high-quality technical engagement with government ministries and sectoral agencies, and made a concerted effort to consult with an array of civil society organizations. In a second series, a rushed preparation process meant that little consultation was carried out, which contributed to design weaknesses, limited effectiveness and ownership in government, and raised opposition from civil society. Many civil society groups do not understand the Bank’s policy lending instrument and how it differs from investment lending; engaging them can be a means of getting them to contribute to the process rather than to act as opponents. Civil society groups are familiar with consultation processes around investment lending, but frequently find DPF design to be a black box, where little information is made public until the operation is approved and the reforms have already been implemented. 6.7 Environment and social risk management. An assessment of environmental and social risk management in environmental DPF is beyond the scope of this report. An IEG learning product published in 2015 (IEG 2015a) examined implementation of the environmental and social requirements for all DPF during 2005–14 and identified opportunities for improving the systems for managing these risks. But it is important to note that these risks apply across many types of DPF operations, and issues in implementing environment and social risk requirements identified in the report are not specific to environmental DPF. 6.8 DPF offers the opportunity for the Bank to help countries improve their country systems for managing environmental and social risks, or to make sure that mitigation measures are built into supported policies. For example, in Himachal Pradesh, one consequence of Bank involvement in plans for water resource management reform supporting hydropower development was to improve how environmental and social effects were managed. In particular, the Bank helped the government to establish a local area development fund (financed out of dam construction costs) to support communities affected by dams. This reportedly made a significant difference in changing the social acceptability of hydropower projects. In Colombia, the Bank used a green growth DPO as an opportunity to support a strategic environmental assessment to consider a range of downstream impacts and identify options for improvement, which influenced the risk assessment of the operation. 6.9 Bank staff note that concerns about environmental and social risks have hindered the ability of the Bank to conduct policy reforms in areas such as forestry. These areas do have significant potential for adverse environmental or social effects, especially on vulnerable 33 groups such as indigenous peoples and other traditional forest users. Outcomes depend in part on the robustness of enforcement programs and site-level social conflicts, and on activities in the informal sector, which may be difficult to influence with policy lending. Similarly, concerns about potential resettlement have hampered the Bank’s ability to engage on land use planning policies, which are critical for disaster risk management and effective climate change adaptation. The Bank could work to find ways of using DPF to support improvements in country systems and capacity for managing these risks. 6.10 Team composition. Bank task team leaders (TTLs) with DPO experience argue that team composition is a major determinant of program success.  They note the need for the team to include people with policy experience, institutional knowledge, and understanding of the country and language. Some staff called specifically for greater use of environmental economists. As with other instruments, soft skills and the ability to manage client relationships and influence institutions are critical. Some Bank staff argue that the recent and rapid expansion of environmental DPF has meant that there are at present relatively few environmental staff with substantial expertise in policy lending even though the number of environment- related DPOs have increased in recent years. Most environment specialists have experience primarily with investment lending and many are specialized in specific environmental areas where the Bank has been traditionally active, such as biodiversity conservation, but feature less in the DPF portfolio. This means that training and learning on DPF could be a priority. Some staff suggested that the Bank should establish a global solutions group for environmental DPF or policy.  Multisector environmental DPOs will benefit from drawing on staff from the relevant global practices. These staff can bring specialized technical knowledge, greater political buy-in within the Bank, and access to sector-specific trust fund resources. Staff from other practices may be attracted to an environmental DPF task team by the opportunity to engage in policy dialog if previous Bank engagement in that country was largely limited to investment lending. Some staff perceive that the Bank’s new structure can make it more difficult to work across silos because there is competition for resources and leadership. This was somewhat easier in the previous structure where there was a single Sustainable Development Network Director in each region.  Turnover in staff, especially team leaders, can cause problems. Meaningful policy dialog requires trust, which requires personal relationships between key development agency and government officials. Implementing agency officials have sometimes found it difficult to work with new people from development agencies whom they perceive as not understanding the background of the program or respecting agreements made already. Improvements in handover processes and replacement of experts by staff with similar expertise could ameliorate the problem of staff rotation and turnover. 6.11 Working with other donors. The Bank has often tried to harmonize policy lending with other development partners, either through cofinancing, parallel financing, or other 34 coordination mechanisms, but has also often chosen to work bilaterally with the government. Harmonization has advantages in increasing the leverage and visibility of the agenda, improving coordination, reducing wasteful duplication, and reducing transaction costs and compliance costs for clients.  The Bank can play a convening role in multi-donor budget support, increasing the willingness of other donors to be involved by mitigating their risk perception of a policy lending operation, or helping to coordinate other donors and develop agreement on a set of shared priorities.  If donors are not well coordinated then their actions can have unintended consequences on each other. In Indonesia, one of several reasons that the Bank’s programmatic series did not go past the first of four planned operations was the prospect of a US$1 billion grant for Reducing Emissions from Deforestation and Forest Degradation (REDD+) from the Norwegian government. This shifted incentives and meant that the Bank’s joint policy lending with France and Japan was no longer a priority for the government.  In some cases, other development partners have provided the majority of funding. In such cases the Bank can still exercise positive influence on program design while recognizing the need to compromise and collaborate with other donors’ priorities. In the multi-donor climate change DPO series in both Indonesia and Vietnam, the Bank joined already existing budget support programs from other donors, but was able to improve the operations by making the policy reform triggers more ambitious. For example in Indonesia, the Bank only agreed to join the program after the government accepted the inclusion of politically more difficult policy measures with respect to the reduction of energy subsidies and improved forest governance. In some other cases the Bank appears to have joined existing budget support of other donors with little additional impact. Attribution can be unclear, where multiple donors are trying to take credit for an entire set of reforms. Sometimes the same actions are used to satisfy conditionalities for multiple parallel policy lending operations by different donors. 6.12 Working with partners can also involve tradeoffs, complications, and delays.  Harmonization can be challenging when other donors do not share the Bank’s view of policy lending instruments. Sometimes other donors are willing to accept as disbursement triggers weaker actions that are not really reforms, such as the publication of a study. The Bank has more intensive processing requirements and preparation than most other donors, especially because of the need to seek Board approval for each subsequent stage of a programmatic series. Other partners can come to regard the Bank as difficult to work with; however, they also sometimes value the Bank’s quality assurance and review process as offering significant design improvements and quality assurance.  In Ghana there were tradeoffs associated with donor harmonization. Collaboration in the design phase was strong, and parallel financing of a common program and policy 35 matrix strengthened and elevated environmental agendas. Yet coordination partially broke down during supervision, in part because of high staff turnover among the Bank and other donors. Parallel financing efforts were introduced in some subsectors, and some development partners tried to add items to the program that were outside the main objectives. The need for consensus among all development partners and government agencies made it extremely difficult to implement any changes. In Indonesia, though Bank policy dialog with government continued after the end of the jointly financed programmatic series, donor coordination broke down in the absence of an alternative mechanism.  Other development partners have advantages and disadvantages relative to the Bank. Other donors can be a valuable source of grant funding for TA. Sometimes bilateral donors are not in a position to provide financing support at the scale of the Bank, but can provide TA on more favorable terms than the Bank can muster. Yet expectations that other development partners will provide needed TA have not always been met, as other donor priorities can change based on their internal institutional issues. In Ghana, the lack of accompanying TA was a major drawback, and may explain why agencies struggled to implement reforms on the planned timetable. Sometimes other donors, especially bilaterals, may not have the level of specialist technical expertise and knowledge that the Bank can command. Bank staff sometimes argue that bilateral agencies are relatively more concerned with disbursements and processes, and lack the Bank’s focus on results. 6.13 Interactions with the International Monetary Fund (IMF) should be seen as a standard part of any budget support, including environmental DPF. A separate IEG learning product on macro-fiscal frameworks in DPF noted that weaknesses of certain elements of macro fiscal frameworks were more likely in the absence of substantial collaboration with the IMF (IEG 2015c). 36 7. Conclusions 7.1 Policy lending offers many advantages for pursuing environmental goals. Environmental challenges are often driven by externalities where policy interventions are needed to reach efficient outcomes. Environmental issues often span multiple ministries, so the ability of policy lending to act as a platform for change across multiple sectors is very valuable. And environment ministries are often relatively weak, so the ability to elevate critical environmental development issues to senior decision makers in finance and planning ministries is critical. Yet these characteristics also make issues of institutional political economy of high importance, so operations benefit from a clear political theory of change and a strategy for achieving institutional buy-in. And they increase coordination challenges and the risk of stretching an operation too thinly by covering too many policy areas and agencies. 7.2 Environmental policy reforms are often highly complicated and require significant time to design; – therefore it can be difficult to support significant environmental policy reforms under emergency circumstances. There is a risk that operations prepared in those circumstances may select policy actions that have already been completed or otherwise offer little ability for the Bank to influence, in which case the DPO may have little impact beyond that of the fiscal transfer. 7.3 As in other sectors, environmental policy lending is heavily reliant on effective analytical work and TA. Although Bank teams are in near-universal agreement on the importance of these factors their provision is viewed as inadequate. Bank staff argue that this has been to the result of insufficient budget allocation for analytic work, tight Bank budgets across the board, a decline in the availability of trust funds such as for CEAs, timing difficulties related to the pressures for rapid disbursement, clients’ unwillingness to borrow for TA, and incentives in the Bank that do not favor knowledge work. 7.4 The lessons generated from the Bank’s ICR evaluations of environmental DPF have focused on design and process issues around use of the DPF instrument rather than on sector- specific technical issues, rarely contributing on the wide range of subsectors in which the Bank has supported environmental reforms. This is a potential missed opportunity for improving the knowledge base on sector-specific experiences. 7.5 Finally, the Bank could consider strategies for improving the quality of results frameworks, including monitoring and evaluation systems. There are inherent difficulties in demonstrating impacts from DPF on outcomes in a limited time frame, but there are opportunities for Bank teams to select program objectives that are clear and evaluable and to use performance indicators that give evidence of early implementation effectiveness. The Bank could also consider mechanisms for assessing longer-term results, separate from the “short-termism” of the project-based evaluation system. 37 References IEG (Independent Evaluation Group) .2015a. Managing Environmental and Social Risks in Development Policy Financing. Washington, DC: World Bank. ________. 2015b. The Quality of Results Frameworks in Development Policy Operations. Washington, DC: World Bank. ________. 2015c. Quality of Macro-Fiscal Frameworks in Development Policy Operations. Washington, DC: World Bank. ________.2016a. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group (ROSES). Washington, DC: World Bank. ________. 2016b. Cluster Country Program Evaluation on Small States: Pacific Island Countries Program Evaluation (FY05–15) Volume II. Washington, DC: World Bank. ________. 2016c (Forthcoming). The Role of Political Economy Analysis in Development Policy Operations. Washington, DC: World Bank. World Bank. 2008. Strengthening Policy Dialog on Environment: Learning from Five years of Country Environment Analysis.” Environment Department Papers, Institutions and Governance Series, No. 114. Washington, DC: World Bank. ________. 2010. The Role of Environmental DPOs in Supporting Environmental Sustainability Assessing Experience, Learning Lessons. Unpublished. Washington. ________. 2012, “Toward a Clean, Green, Resilient World for All”, World Bank Washington DC ________. 2013. World Bank Group Strategy. Washington, October 2013. ________.2013b, “2012 Development Policy Lending Retrospective: Results, Risks and Reforms”, World Bank Washington DC ________. 2015. 2015 Development Policy Financing Retrospective: Results and Sustainability . Washington DC: World Bank. 38 Appendix A. The Environmental DPF Portfolio The following are the 64 operations identified under the working definition of an environmental DPO; those policy lending operations either mapped to the ENR GP, or had an environment and resource management or natural disaster management theme as their primary or secondary theme. Annex Table 1: Operations mapped to the ENR GP, with a published ICR: Approval Project ID Name Country FY P057927 ENV/PRIV SUPPORT SAL Bulgaria 2000 P070196 GA-Natural Res Mgmt DPO (FY06) Gabon 2006 P074539 MX Programmatic EnvSAL Mexico 2003 P079748 MX ENVDPO II Mexico 2006 P080829 BR 1st. PRL for Environmental Sustainab. Brazil 2005 P081397 CO Prog Dev Policy Ln for Sust. Dev Colombia 2005 P095205 BR 1st Prog. DPO for Sust. Env Mgmt Brazil 2009 P095510 MX Environmental Sustainability DPO Mexico 2009 P095877 CO 2nd Sustainable Dev DPO Colombia 2007 P101301 CO 3rd Sust. Dev DPO Colombia 2009 P102971 GH-Environmental Governance (FY07) Ghana 2008 P110849 MX Climate Change DPO/DDO Mexico 2008 P113172 GH-NREG DPO Ghana 2009 P118188 GH:Natural Resources Env Governance DPO3 Ghana 2010 P120313 Indonesia Climate Change DPO Indonesia 2010 P126449 BR MST Piaui Green Growth and Inclus DPO Brazil 2012 Annex Table 2: Operations mapped to the ENR GP, with no ICR Approval Project ID Name Country FY P101471 PE DDO First Prog. Environ DPO Peru 2009 P115101 MXSupplement to Env Sustain. DPO Mexico 2009 P116152 PE 2nd Prog. Env DPO Peru 2010 P118713 PE 3rd Prog. Environmental DPO Peru 2011 P122667 VN-Vietnam Climate Change DPO Vietnam 2012 P124041 IN: HP DPO Green Growth India 2013 P127201 VN-Climate Change DPO II Vietnam 2013 P127956 MA-Inclusive Green Growth DPO Morocco 2014 P128434 MZ:Climate Change DPO Mozambique 2013 P131775 VN-Climate Change DPO III Vietnam 2014 P143032 DPO 2 - Inclusive Green Growth in HP India 2014 P146398 Second Climate Change DPO Mozambique 2015 39 Approval Project ID Name Country FY P146981 BR Piaui Productive and Social Inclusion Brazil 2016 P149747 MA- Inclusive Green Growth DPO2 Morocco 2016 P150475 Colombia DPO for Sustainable Development Colombia 2016 Annex Table 3: Operations mapped to other GPs, with published ICR Approval ProjectID Name Country FY P071103 KH-Poverty Reduction and Growth-1 (DPO) Cambodia 2008 P074073 TZ-PRSC2 (intermediate) Tanzania 2005 P082412 CL -Santiago Urban Transport Adj Chile 2006 Congo, P091990 DRC - Transitional Support Credit (DPO) Democrat 2006 P095575 SL-GRGG (DPG 1)-Programmatic Series Sierra Leone 2007 P104937 MA-SOLID WASTE SECTOR DPO Morocco 2009 P104990 RW- PRSG 4 DPO FY08 Rwanda 2008 P105287 VN - PRSC 7 Vietnam 2008 P115608 MX Framework for Green Growth DPO Mexico 2010 P121651 ESES DPO 3 Turkey 2012 P065351 COAL ADJ 2 Poland 2002 P070656 CM-Forestry & Env DPO (FY06) Cameroon 2006 P073020 CM GEF Forest & Env DPO (FY06) Cameroon 2006 P095840 MA-Water Sector DPO Morocco 2007 P101724 Vietnam PRSC 6 Vietnam 2007 P112625 ECONOMIC RECOVERY DPO Moldova 2010 P115426 ENERGY EFFICIENCY DPO Poland 2011 P117651 ESES DPO2 Turkey 2010 P120134 MX DPO Adapt. Climate Change in WtrSct Mexico 2010 P120170 MX Strengthening Social Resilience to CC Mexico 2012 P121800 MX MEDEC Low-Carbon DPO Mexico 2011 P076830 Jamaica Emergency Recovery Loan Jamaica 2002 P118636 Economic Crisis Recovery Support Credit Samoa 2010 Annex Table 4: Operations mapped to other GPs, with no ICR Approval ProjectID Name Country FY P117279 LR- RRSP 3 - Budget Support Liberia 2011 P127955 MA-Solid Waste Sector DPO3 Morocco 2013 P147166 HT Strengthening Governance Haiti 2014 P150941 BI Eight Economic Reform Support Grant Burundi 2015 P148083 RS:Strengthening Fiscal & Water Mgmt DPO Brazil 2014 P148642 MA-Solid Waste Sector DPO4 Morocco 2015 40 P144377 Samoa Development Policy Operation Samoa 2014 P150751 BO DRM DPC and DPO Bolivia 2015 P147152 1st Programmatic Resilience Building DPC Grenada 2014 P151821 Grenada Resilience Building DPC 2 Grenada 2016 A portfolio review of prior actions carried out for this product developed a typology of policy actions (Annex Table 5). The review also developed a typology of the types of lessons offered by self-evaluations, desk validations, and field-based evaluations (Annex Table 6). 41 Annex Table 5: Prior actions supported by environmental DPOs, FY 2000–16 Number of Number of prior actions prior actions Type of policy in operations in operations mapped to mapped to ENR GP other GPs Climate change strategy and action plan 12 4 Climate change Climate change mitigation: LULUCF 14 7 Climate change mitigation: energy, demand side 6 8 Climate change mitigation: energy, supply side 10 9 Climate change mitigation: transport and urban 15 0 Climate change adaptation: general 2 9 adaptation Climate change Climate change adaptation: urban and infrastructure 1 1 Climate change adaptation: agriculture 4 3 disasters Natural Disaster risk management 9 5 Pollution: general 9 0 Pollution Pollution: industry and mining 6 5 Pollution: sanitation 12 12 Environmental protection: general 40 2 Environmental protection Environmental protection: coastal and marine resources 12 1 Environmental protection: water resources 24 8 Environmental protection: land resources 4 4 Institutional strengthening: environmental and climate strengthening change cross-sectoral coordination and financing Institutional 13 1 Institutional strengthening: monitoring systems and tools 6 3 Institutional strengthening: social impact, social inclusion, and public consultations 15 9 Sector reform and governance Sector reform and governance: agriculture, aquaculture 5 14 Sector reform and governance: oil and mining 8 4 Sector reform and governance: water and sanitation 2 12 Sector reform and governance: energy 4 4 Sector reform and governance: transport and logistics, telecom 5 5 Sector reform and governance: forestry 10 15 Sector reform and governance: tourism 3 0 Non-environmental policies 16 115 Source: IEG portfolio review. 42 7.6 Annex Table 6: Lessons from self-evaluations and independent validations/evaluations of closed environmental DPOs, FY 2000–16 Number Type of lesson of cases Government ownership and leadership: champions, support by top-level 16 officials, political support Government ownership, institutional support, and interagency 29 coordination Political economy Embedding policy reforms in the government agenda 7 Government involvement in monitoring progress 5 Reform continuity, consistency, and dialog (Bank-Government) 15 Stakeholder involvement, transparency, communication 12 Working with subnational governments 3 Need for a strong implementation agency 7 Earmarking of DPF funds to line agencies 3 strengthening Institutional Need for institutional capacity and importance of capacity building 9 Building a culture of legal compliance 1 Synergy of bringing sectors and institutions together 6 Need to focus on key reforms, limit complexity in terms of policies and 12 agencies involved Importance of flexible approach 7 Pilots and phasing approach useful for large operations 3 DPO design and preparation Cooperation between the Bank and the IFC 1 Multi-donor involvement and coordination 13 Analytical work and knowledge-based policy design 9 Technical assistance needed to support DPF 18 Investment operation vs. DPF or to complement DPF 9 Results framework and M&E issues 22 Ex-post evaluation and supervision 6 Programmatic series vs. single operation 10 Mainstreaming climate change across sectors 4 Design issues in programmatic series 6 Resource requirements of DPF 2 Project-specific lessons and "other" lessons 23 Source: IEG portfolio review. A “case” is for a single standalone operation or programmatic series. Note: M&E = monitoring and evaluation. 43 44 Appendix B. Case studies B.1: Natural Resource and Environmental Governance in Ghana Ghana’s growth is heavily reliant on natural resources, many of which have suffered from significant resource degradation, particularly in the forest sector. Resource degradation had occurred because of a history of poor governance and management of natural resources, weak environmental protection, and limited community involvement. The World Bank had been involved in previous efforts to improve natural resource management in Ghana in the late 1990s and early 2000s, but these were not successful and the World Bank withdrew from the sector. One reason for the poor performance of investment projects was the number of policy barriers, so the Bank with other development partners decided to re-engage in the sector with a Development Policy Operation: the Natural Resources and Environmental Governance Program (NREG). The operation represented an innovative attempt to bring a new approach to a set of challenging subsectors and to do so in a harmonized manner with other major development partners. The Program Development Objectives of the US$40 million programmatic series were to a) Ensure predictable and sustainable financing for the forest and wildlife sectors and effective forest law enforcement; b) improve mining sector revenue collection, management, and transparency; c) address social issues in forest and mining communities; and d) mainstream environment into economic growth. Though relevant to the broad goals of the Country Assistant Strategy and the Government’s goals for the program, the objectives focused on inputs, outputs, and intermediate outcomes, which reduced their relevance. Setting achievable objectives that were sufficiently relevant was a difficult challenge when the expected duration of the program was three years. The program supported efforts to improve environmental and natural resource management under three policy areas: forest and wildlife, mining, and environmental protection. Major elements included increased revenue collection from forestry royalties, creation of a system for tracking the legal origin of timber consignments, benefit-sharing schemes with land users and investors, a mining revenue task force, mine audits, mining regulatory reform, more collaborative forest management, a social conflict tracking tool, increased use of environmental impact assessment and strategic environmental assessment tools, and development of a climate change strategy. Though program documents did not always identify a clear theory of change, the elements of the program contributed to the program objectives. However, the design did little to address informal forestry and mining, it did not address land tenure issues that were a barrier to private sector investment in forestry, and it was not clear how program activities would reduce social conflict. The operation was largely implemented as designed, and prior actions and triggers were completed (except for one that was dropped). Changes in government and elections led to some delays but did not disrupt the overall program. Implementing agencies were highly committed to the program, and improved their cooperation and coordination. A critical issue in management of funds was the decision to keep them in a ring-fenced account, and to earmark them specifically to the implementing agencies. This decision was made by the government, but arguably with the tacit support of the Bank and other development partners. The decision had three significant negative consequences: It acted to insert the Bank and other development partners into the relationship between government and agencies, and led agencies to hold development partners, rather than the government, responsible for providing sustained financing. It undermined the ability of the operation to contribute to a mature budget dialog between the finance ministry and line ministries. And it encouraged the government to effectively delegate responsibility for carrying out the program to the implementing agencies. 45 Progress was made in a number of areas. Policy changes and internal investments increased revenue collection by the Forest Commission. Forestry policies and plans were revised, but competitive bidding for timber permits has stalled, and private investment in plantations has been less than was hoped. Planned increases in forestry royalty rates were blocked by the forestry industry. Significant progress has been made on a wood tracking system, but as of 2014 it had not yet been completed and no export licenses had been issued. Informal forestry remains a significant problem and strategies for managing it remain at the pilot stage. Though overall forest cover is increasing, forest degradation continues. Changes to mineral royalties increased the revenue generated from the mining sector, though increasing production and rising gold prices were also major causes. A commitment to transparency for large mining company revenues through the Extractive Industries Transparency Initiative existed prior to the program, but incremental progress was made in expanding transparency. Little support was provided to small-scale miners, because planned mining cooperatives on new sites were not established. There were significant improvements in relationships between government and civil society on natural resource management. Consultation and collaboration with forest communities was increased, and royalty payments to traditional authorities and district assemblies were published. A social conflict tracking tool and guidelines on social responsibility for large mining companies were established. But monitoring and evaluation systems did not track social conflict, and a planned social assessment was not carried out, so it is difficult to assess whether social conflict has declined. Incremental improvements were made to the environmental impact assessment process. Strategic environmental assessments were carried out in a number of sectors and these have fed into policy development in some cases. A national climate change strategy was established and climate change was required to be considered in key national and local-level policies. But capacity limitations mean the ability of these policies to have an impact is unclear. The choice of a Development Policy Operation instrument was justified, but there were a number of problems with the way it was implemented, caused in particular by the earmarking of funds, and the decision to use single-year operations within the series combined with a complex design with many actions to be achieved and a lack of supporting technical assistance. The operation had no clear exit strategy for what would occur to line agency funding once the program ended. The decision to harmonize budget support with other development partners was positive, but harmonization was incomplete and partially broke down during implementation. The Bank’s participation after the original approval consisted primarily of assessment of the large number of targets and triggers at the expense of ongoing policy and technical advice. Monitoring and evaluation systems were weak, focusing largely on recording production of outputs rather than on assessing progress on outcomes. Many indicators were vague, and baselines were weak. The system was improved during implementation but the core problems remained. The experience of NREG offers a number of useful lessons about using development policy operations for sectoral interventions, about donor harmonization, and about natural resource management. These include: • Earmarking funds to specific agencies can undermine the rationale of a development policy operation by inserting the Bank into the relationship between the finance ministry and the line 46 ministries and their agencies. When agencies are heavily reliant on donor budget support, then there are risks to sustainability if there is no long-term plan for addressing what will happen at the end of the program. • There is a risk that in sectoral development policy operations with one-year operational cycles, policy and technical advice to the client can be crowded out by processing requirements. • Sectoral development policy operations may need complementary technical assistance and support. • Donor harmonization has tradeoffs: it can help to provide a unified platform for sector reform, but can reduce the flexibility of programs, and differences in rules and expectations across agencies pose significant challenges. Source: http://ieg.worldbankgroup.org/Data/reports/Ghana_NRM_PPAR_889590PPAR0P1000Box38528 5B00PUBLIC0_0.pdf 47 B.2 Energy sector and environmental sustainability series in Turkey The World Bank has had a long engagement in the energy sector in Turkey, with support through investment lending and policy dialog. The government of Turkey has pursued an agenda of market-oriented electricity sector liberalization since 2001. In the mid-2000s, analytic work showed that Turkey faced the prospect of electricity shortages as early as 2009–11. The Bank aimed to help the government to alleviate these shortages by supporting an increase in the pace of the reforms, which were intended to increase private sector investment in electricity generation and distribution and energy efficiency. The Programmatic Electricity Sector Development Policy Loan, approved in June 2009 as the first in what was intended to be a two-operation programmatic series, was intended to provide this support and to serve as a platform for continued policy engagement. The objectives of this operation were to “address the projected electricity supply-demand imbalance”. The design of the programmatic series was highly relevant to this objective, with a coherent design of mutually reinforcing prior actions covering electricity sector market development, pricing reform, renewable energy development, distribution company privatization, energy efficiency, and other measures. After 2008, the government gave higher priority to environmental issues, particularly after the opening of the environmental chapter of the European Union acquis and accession to the Kyoto Protocol. The government requested World Bank assistance on environmental policy through the electricity programmatic DPO series. The design of the series was expanded from two operations to three, a number of new prior actions and triggers on environmental issues were added, and the objectives of the series for the Second Environmental Sustainability and Energy Sector Development Policy Loan and the Third Environmental Sustainability and Energy Sector Development Policy Loan were to a) enhance energy security; b) integrate principles of environmental sustainability, including climate change considerations, in key sectoral policies and programs; c) improve the effectiveness and efficiency of environmental management processes. The energy sector objective increased its relevance by expanding to cover energy security beyond the electricity sector, but the environmental objectives were less relevant, because the main weaknesses in environmental management were in implementation rather than policy. The electricity sector design remained strong, but relatively little was added to address energy issues outside of the electricity sector. The design of the climate change pillar was relatively strong, because a national climate change strategy and action plan were important steps in addressing climate change. But there were weaknesses in the design of the environmental pillar. Some individual prior actions were relatively minor policy changes, and many prior actions may have had little additional impact because these changes were already being pursued by Turkey motivated by EU harmonization. The main environmental challenges in Turkey were due to weaknesses in implementation and enforcement rather than policy, and so a DPO might not have been the right instrument for environmental engagement, especially absent parallel technical assistance. The series was implemented largely as designed, though triggers on strategic environmental assessment and on gas market law amendments were dropped. An important trigger on energy efficiency, which was at risk of being dropped, was preserved and the policy was adopted. The reforms played a major role in addressing electricity supply-demand imbalances. The government’s strategy of using policy reforms to encourage private sector investment in electricity generation has been generally successful, and the Bank DPO played an important 48 supporting role. However, there have been some wintertime power shortages, driven in part by gas supply shortages, and there has been less progress on energy efficiency. The electricity reforms led to increased electricity supply security, but there was little impact on other aspects of energy security, including natural gas supply. The climate change strategy and action plan supported under the operation are important steps, though the climate change strategy involves substantial increases in greenhouse gas emissions, particularly from coal power development, in the medium term. However, the effects on improving the effectiveness and efficiency of environmental management were smaller, because many of the prior actions supported by the DPO were relatively modest, or were motivated by EU harmonization goals with little additional impact from the Bank’s involvement. In particular, little progress has been made on increasing the degree of public participation in environmental decision making. There were a number of weaknesses in the design of the monitoring and evaluation system. Many of the indicators were designed in a qualitative manner that made judging their fulfillment ambiguous or subjective, and many indicators tracked outputs rather than outcomes. The actual measured values did not track the targeted indicators in some cases, and there were no indicators for some key outcomes. The program offers a number of lessons:  The Bank can maximize its development impact by concentrating its strategic engagement including its lending and advisory support in sectors with track records of success.  A well-designed programmatic DPO can be a key instrument in the Bank's long-term engagement that leverages the Bank’s strengths on technical quality, convening power, and credibility to help support sectoral reforms that can yield substantial impacts.  Prior actions should focus selectively on those reforms that are critical to achieving project objectives but are difficult to undertake because of political or institutional resistance. Prior actions should also ensure that they are additional to what would occur in the absence of the DPO operation.  DPOs can achieve good outcomes when they serve as the culmination of a process of engagement rather than the initial engagement in a new sector.  A comprehensive yet well-integrated set of market reforms can provide credible signals and incentives to private investors.  Changing laws and regulations may not have much impact on environmental outcomes when environmental management agencies are weak and lack implementation and enforcement capacity. Source: http://ieg.worldbankgroup.org/Data/reports/PPAR_- _Turkey_1st_Programatic_Electricity_DPO.pdf 49 B.3 Climate change policy loan in Indonesia Indonesia is the third largest emitter of greenhouse gases in the developing world after China and India. These emissions stem largely from deforestation, peatland conversion, and associated fires, together with electricity generated by coal-fired power plants and the consumption of fossil fuels in the energy and transport sectors, also associated with high fuel subsidies and rapid urbanization. Comprising more than 13,000 islands, Indonesia is also one of the most vulnerable countries to the rising adverse impacts of global climate change, including extreme weather events—tropical storms and droughts—and sea level rise, particularly on account of the concentration of much of its population in lowland areas. In recognition of this, in 2007 the Indonesia government developed a National Action Plan for Addressing Climate Change. Even prior to that, it had signed an agreement with the Government of Japan that resulted in a multi-year policy-based loan to support Indonesian efforts to deal with climate change, based on the National Action Plan. The French Government joined this initiative shortly thereafter, but even though the World Bank participated in some of the initial meetings and provided technical and analytical support for this program, it decided not to cofinance it during the first two years of its implementation. The main reasons for this were that the agreed policy matrix was considered to be too fragmented and insufficiently ambitious with respect to forest governance and energy subsidy reform. The Bank agreed to participate in the funding of the second phase of the ongoing climate change policy loan program in 2010 after the Government agreed to incorporate more significant policy reforms and after a presidential pledge that the country would reduce its greenhouse gas emissions by 26 percent by 2020. The Bank approved an initial DPO of US$200 million alongside contributions from the Japan International Cooperation Agency (JICA) and the French Development Agency (AFD) of US$300 million in both 2009 and 2010. This was the first of what was expected to be a four-loan programmatic series to support what the Bank denominated the Indonesia Climate Change (CC) DPO Program, whose objectives were to support the government’s efforts to develop a low- carbon, climate-resilient growth path. A number of prior actions were recognized in three main policy areas—mitigation, adaptation and disaster preparedness, and cross-sectoral and institutional issues— and 11 subareas to justify approval and disbursement of the first loan. Four triggers and other “indicative” policy actions were agreed by the government and the development partners for the second loan, as well as tentative indicative actions for the third and fourth ones, expected to occur in 2011, 2012, and 2013, respectively. For a variety of reasons, the Bank DPO series did not extend past the initial loan. According to government officials interviewed by IEG, the program failed to go forward because of a presidential decision not to borrow for climate change, even though the resources transferred by the Bank and the Japanese and French governments were for general budget support and were not used to support investments to address climate change per se. This decision was reportedly taken in response to a “consensus” at the United Nations Framework Convention on Climate Change (UNFCCC) that developing countries should only receive grant money, rather than loans, to address climate change, although in practice it was also a reflection of several other factors. These factors included: (i) the loss of critical program “champions” within the Indonesian government when the Minister of Finance departed and other high-level personnel changes occurred within the National Development Planning Agency, which was responsible for program coordination; (ii) the availability of budget support finance from alternative sources, including the Bank, through 50 other DPOs; and (iii) the near-simultaneous offer by the Norwegian government to provide up to US$1 billion in grant funding for implementation of Indonesia’s incipient Reduction of Deforestation and Degradation (REDD) program, which was also being supported by the Climate Change DPO. In addition, the government had failed to meet two of the four triggers previously agreed for the second loan, while achievement of a third had been substantially delayed. Consequently, only one of the four anticipated operations was presented to the World Bank Board. The Japanese and French Governments also ceased their funding for the policy-based program after 2010, though the other development partners, especially JICA, continued to provide TA grants to some of the line ministries responsible for implementing parts of the original program. Although one of the triggers for the second loan was met in a timely way, two others were not met in the form originally envisaged in the Program Document, the fourth was delayed, and the experience with other indicative actions for planned operations two, three, and four was mixed. Several of the expected results could not be assessed because the needed baseline surveys were not carried out or the agency responsible for their monitoring discontinued collection of the required data, while others were only achieved in part. Some results indicators, moreover, only partly reflected the policy actions that were to be taken. Although performance in some policy subareas, such as those related to renewable energy, water resource management, and natural disaster risk management, was generally positive, this was less true in others, especially those concerned with peatland conservation, REDD (now REDD+, which includes conservation, sustainable forest management, and enhancement of forest carbon stocks in addition to the reduction of deforestation and forest degradation), and forest governance. Nor was it possible to establish an intergovernmental fiscal transfer mechanism to provide incentives for local governments to take priority climate change actions, including the strengthening of forest management activities, which had been one of the triggers for the second loan. More generally, available data suggest that Indonesia’s greenhouse gas emissions have continued to rise in recent years, at least through 2012, because of persistently high rates of deforestation, peatland conversion, and fires, as well as growing fossil fuel-based energy consumption. Electricity subsidies were finally reduced somewhat as of late 2013 and geothermal energy investments increased, in part with financial support from the World Bank and the Clean Technology Fund. However, the share of renewables in Indonesia’s energy mix remains very low (about 3 percent) and is expanding very slowly; coal and oil continue to strongly predominate. Forest and land use management also persist as major challenges, while REDD+ implementation has advanced very slowly and had very limited results on the ground to date. Several of the triggers for the proposed second loan proved overly ambitious, given the strong institutional and political economy constraints encountered by the program. The latter included the frequent tendency of subnational governments to fail to implement central government decisions when they went against local vested economic and political interests, such as those related to peatland use, palm oil concessions, and curbing fires and deforestation, together with strong public resistance to cutting energy subsidies. Government commitment in the Ministry of Finance and Planning agency was initially strong, but it declined significantly after the changes in top-level personnel; commitment in the various line ministries and agencies involved in DPO implementation was uneven from the start and remained so throughout the brief life of the second phase of the program. A joint evaluation by AFD and JICA, issued in June 2014, observed that even the additional TA grants provided by these donors to some of the participating ministries proved to be an insufficient incentive for them to proactively implement some of their sector-specific policy obligations, while others were effectively impeded by uncooperative local governments, empowered by the country’s recent decentralization. 51 Despite these frustrations, the Indonesia CC DPO experience provides a number of important lessons. Among them are: • Both a strong “champion” and broad institutional commitment are needed for DPO policy actions to be effectively implemented; it is, thus, important to fully understand the incentives involved for the various government entities that are to be engaged in DPO implementation. In this regard also it is essential to fully understand the potential political economy, as well as the institutional, constraints that can impede or delay policy implementation; this has implications for the up-front risk analysis and the DPO appraisal process more generally. • This is especially important in DPOs with environmental, including climate change objectives, which are inherently cross-sectoral or multisectoral in nature, and, therefore, tend to depend on a broader range of participating institutions, both at the national and subnational levels, than single-sector or macroeconomic/fiscal DPOs. • Programmatic DPOs can encounter many of the same obstacles to development effectiveness, including varying and changing levels of government and implementing agency commitment and implementation delays, as can be met with in investment loans. • DPOs for climate change and other complex development challenges are more effective as part of a broader targeted multi-instrument Bank assistance strategy, including the use of investment loans and TA, as a way of ensuring greater Borrower interest and ownership and establishing a longer-term relationship and policy dialogue. • Even when a DPO is unsuccessful in terms of its own expected results, it may play a positive and strategically important role as part of an evolving longer-term Bank-Borrower partnership to help address an emerging complex development challenge such as climate change Source: http://ieg.worldbankgroup.org/Data/reports/PPAR_Indonesia.Clmt_.ChgDPO.pdf 52 B.4 Environmental management in Brazil Brazil Programmatic Reform Loan for Environmental Sustainability (2004) The First Programmatic Reform Loan for Environmental Sustainability (Env PRL) was the first World Bank budget support program in Brazil with a purely environmental policy objective, and was part of a set of environmental policy loans to countries in the Latin America and Caribbean Region that were some of the first in the World Bank. The Env PRL was designed as a programmatic series of up to three loans to the Federal Government for a total of approximately US$1.2 billion. The first loan, for US$502 million, was approved in August 2004 and disbursed in a single tranche and closed in December 2004. Only the first PRL was completed; the second and third loans did not materialize. An Environmental Technical Assistance Loan (Env TAL) was prepared to accompany the Env PRL and provide support through studies, workshops, and other capacity-building activities. However, this loan did not become effective until 2006, nearly two years after the approval of Env PRL 1. Brazil faced a variety of environmental challenges associated with deforestation and the need to combine agricultural growth, environmental protection, and sustainable development. Increases in urbanization and industrialization had brought additional environmental challenges. Finding a balance between economic development and sustainable management of the environment was therefore a key national priority for Brazil, and for the World Bank’s program. The principal objective of the Env PRL was “to support Brazil’s goal of balancing economic growth with social development and the maintenance and improvement of environmental quality.” The program also had two specific objectives: 1) increasing the effectiveness and efficiency of Brazil’s environmental management system (EMS), including a green, brown and blue agenda, and 2) mainstreaming environmental sustainability in selected sector policies and programs. These objectives were necessary for achieving the principal objective but were unlikely to be sufficient, particularly on the social side. The Env PRL was designed with 15 different policy areas and multiple government agencies in addition to the official implementing agencies—the Ministry of Finance and the central Ministry of Environment. The general approach under the Env PRL series, drafting laws and strategies under the first operation, approving legislation under the second, and implementing the reforms under the third is causally linked and logically correct in most—though not all—cases. However, a risk of noncompletion was created by predominantly targeting the federal level with “backloaded” reform agendas that would not be fully implemented until the second or third operation. The Env PRL is credited with contributing to the raising of the profile and agenda of the Ministry of Environment and associated agencies, resulting in improved engagement with other important parts of the government, such as the Ministry of Finance. This was of high importance for advancing the sustainability agenda in Brazil. The reduction of deforestation in the Atlantic Forest and the introduction of a system for water charges were also important achievements. In other areas, such as the protection of the Cerrado Forest, improvement of the environmental licensing process, and improved management of hazardous chemicals, evidence suggests that significant challenges remained. Some of the reforms supported by the Env PRL are credited with making substantial contributions to environmental mainstreaming. This is particularly true with regard to the impressive progress that Brazil has made in reducing deforestation in the Amazon, which is a signal achievement of great importance for Brazil and for the global environment. In the area of environmental sanitation, there has been some progress through the innovative “payment-for-results” sanitation program, and zoning in the Amazon, though with little apparent impact on the ground at the time of evaluation. In other policy areas such as energy and tourism, the planned mainstreaming reforms were never carried out. The failure to introduce a Strategic Environmental Assessment approach into river basin and 53 hydroelectric investment planning, and the lack of results in mainstreaming environmental sustainability in financial institutions, were missed opportunities. Although the second and third operations in the Env PRL series did not materialize as planned, Env PRL 1 and the accompanying Env TAL gave impetus to the reform agenda, resulting in the continuation of many of the reforms and important contributions to improved environmental management and mainstreaming of environmental sustainability in Brazil. Those included the essential strengthening of the Federal Ministry of Environment, and the significant reduction in deforestation of the Amazon and Atlantic Forests. Reforms in some other areas did not achieve results as planned. The Brazilian government continued to make progress on many—but not all—of the policy areas under the Env PRL program, indicating good commitment to the overall environmental management and mainstreaming agenda. The World Bank team that prepared the Env PRL was praised by both government counterparts and civil society representatives for the quality of the preparation process, including the extensive and long-term engagement of the World Bank team with the Brazilian environmental policy reform agenda and with stakeholders both inside and outside of government during preparation. On the other hand, the design of the Env PRL was highly ambitious, involving too many policy areas and associated agencies. There is also a question as to whether a three-loan series was the best design option. Although the Borrower continued with a number of the reforms after the Env PRL 1, this was more serendipity than good planning on the part of the Bank. The preparation of the Env TAL that was meant to accompany the Env PRL lagged behind that of the Env PRL. Supervision of the Env PRL series was not adequate. After Env PRL 1 was disbursed and closed, there was inadequate attention to continuing the dialogue and engagement from the World Bank side to ensure that the programmatic series would continue as planned for a second and third operation. Monitoring and evaluation under the Env PRL were particularly weak. The Bank did not evaluate the Env PRL with a full ICR for the series, so despite the large size of the loan, its innovation, and its potential implications for a subsequent environmental DPO, there was little accounting of what worked and what did not under the Env PRL, what the outcomes were, and what lessons could be gleaned for future operations. Brazil has made significant progress in improving environmental sustainability in certain areas— particularly the signal achievement of reducing deforestation in the Amazon. There is evidence that this had much to do with actions taken by the Brazilian government, particularly the strengthening of “command and control” enforcement measures under the Amazon deforestation prevention program supported by the Env PRL. The government also took steps to strengthen the environmental management system and central institutions in the country by restructuring, raising the profile of, and increasing staffing for the Ministry of Environment. Despite the cancelation of the second and third operations under the Env PRL series, the Brazilian government also continued to make progress in mainstreaming environment in some of the other sectors. At the same time, the government made little progress in other key policy areas described under the Env PRL, such as improving the performance of the environmental licensing system overseen by the Brazilian Institute of Environment and Natural Resources, integrating a Strategic Environmental Assessment approach into hydroelectric investment planning, and mainstreaming environmental sustainability in financial and fiscal policies. There was also no progress reported in the planned mainstreaming of environment in the tourism sector. The cancelation of the Env PRL series adversely impacted the functioning of the high-level Management Committee composed of representatives of the seven ministries involved. The Brazilian government also reportedly resisted the associated Env TAL loan, which was an important complement to the Env PRL. 54 This assessment includes a number of lessons, which are summarized below: • Risks associated with not completing the required ex-post evaluation of an operation could influence the quality and effectiveness of subsequent operations. The World Bank did not prepare the required full ICR for the Env PRL series. As a result, there was no proper accounting for the Env PRL, and there were adverse impacts on the subsequent First Programmatic Development Policy Loan for Sustainable Environmental Management (SEM DPL) environmental DPO series as well. The World Bank should have ensured that the evaluation of the previous series was completed before embarking on preparation of a new series in the same sector. • Extensive engagement and consultations by the World Bank in preparing a DPO contribute to better design and generate goodwill on the part of the client. The team that prepared the Env PRL was praised for its long-term, high-quality technical engagement with government ministries and sectoral agencies, and for its efforts to consult with an array of civil society organizations. • It is important to be selective and realistic about what can be achieved in the context of a sectoral DPO. Considering that this was the first DPO series in Brazil focused on the environment, the complexity of the issues, and the many government agencies that they involved, the World Bank and the Borrower needed to be more cautious about program design and policy reform priorities, in order to avoid being overly ambitious, as was evident with the 15 different policy areas of the Env PRL. Source: http://ieg.worldbankgroup.org/Data/reports/brazil-env-prl-ppar.pdf 55 Brazil Sustainable Environmental Management Development Policy Loan (2009) NOTE: The following is based on an IEG field-based project evaluation. The findings were disputed by World Bank management and the government of Brazil. See the full document available at http://ieg.worldbankgroup.org/Data/reports/brazil-sem-DPO-ppar.pdf for the Bank management response and Borrower government comments. The First Programmatic Development Policy Loan for Sustainable Environmental Management (SEM DPL 1) was designed as a programmatic series of two loans to the Federal Government of Brazil for a total of approximately US$2 billion. The first loan was for a total of US$1.3 billion divided into two tranches of US$800 million and US$500 million on International Bank for Reconstruction and Development (IBRD) terms. The loan was approved in March 2009, became effective in June 2010, and was disbursed in June and December 2010 and then closed. The planned second loan in the series, SEM DPL 2, did not materialize and was eventually canceled. It is important to understand the history and evolution of the operation and the context in which it was prepared. The operation was initially proposed as a “BNDES PAC-Env DPO”—a development policy loan to the Brazilian National Bank for Economic and Social Development (BNDES) to support the government’s scaling-up of infrastructure investments while also improving BNDES’s environmental and social policies, which was considered a significant challenge. The Brazilian government and the World Bank next concluded that BNDES was not eligible for a DPO. Instead, preparation began on a US$1 billion Financial Intermediary Loan—a type of investment loan—to BNDES called the “BNDES Environmental and Social Sustainability Project,” with 99 percent of the funds going to finance BNDES investment operations and 1 percent for technical assistance to strengthen BNDES’s environmental and social safeguards. Internal World Bank reviewers expressed serious concerns about BNDES’s ability to comply with requirements on environmental and social safeguards, interest subsidies, financial management, and procurement. At the time of the global financial crisis in 2008, an agreement was reached to change the design of the operation again by transforming it into an environmental DPO to the federal government. The reformulation incorporated some of the reform agenda contained in an earlier World Bank–financed environmental DPO—the 2004 First Programmatic Reform Loan for Environmental Sustainability (Env PRL). The Ministry of Environment was added as an implementing agency, the objective was framed at a national scale, and the size of the program doubled to US$2 billion to create the SEM DPL. BNDES remained an implementing agency and a major focus of the policy actions. BNDES was also on-lent the entire amount of the DPO funds from the federal government. The evolution of this operation—including significant changes to the financing instrument, the objectives, and the implementing agencies—created issues with the project logic, the delay in effectiveness, and the Bank’s reputation. The objectives for the SEM DPL series were stated as follows: “The SEM DPL series supports the GOB’s [Government of Brazil’s] concerted efforts to strengthen environmental management, with particular attention to: improvements in the overall environmental management system, sustainable management of agricultural lands, forests, and water resources; reduction of deforestation in the Amazon; reduction of the environmental degradation of land and water resources that are key determinants of the well-being of the poor; and, promotion of renewable energy.” The objectives of the SEM DPL could have been clearer but they were undoubtedly relevant, given Brazil’s tremendous environmental wealth of global importance, its rapidly developing economy that is highly dependent on commodities, and the challenges it has faced in balancing the tradeoffs between the two. The design of the operation suffered from disjointed project logic. This is likely a consequence of the rapid change in direction from preparing a loan to BNDES to preparing the broader SEM DPL, and 56 the limited opportunities for the Bank to engage on the broader reform agenda with government agencies and stakeholders outside of BNDES, given the speed with which the operation had to be prepared. The series of three policy actions and the associated outcome indicator for each policy area inconsistently combined national-level policies and actions and outcomes specific to BNDES. For example, for the sub-objective “Improve sustainability of natural resources management,” the First Tranche Prior Action was to strengthen the federal legal framework by enacting specific forest management laws (for example, for managing the Atlantic Forest); the Second Tranche Release Condition was focused exclusively on BNDES forest programs and guidelines; and finally, the series outcome indicator was a general measure of the surface area of public and private forests sustainably managed, with no direct link to BNDES or the Atlantic Forest. The SEM DPL had major shortcomings in the achievement of its objectives. For a loan—including a DPO—to be effective in achieving its objectives, it should successfully achieve not only the outputs represented by the various policy actions, but also the associated outcomes for the series to which those outputs are designed to contribute. Moreover, these outputs and outcomes should be attributable to the World Bank’s related engagement under the loan with the client on the reforms. Although Brazil has made substantial progress in strengthening environmental management in some areas, there is little evidence that the SEM DPL contributed to this progress, because results instead were rather part of an environmental reform process that has been active for decades. Many of the SEM DPL’s prior actions were implemented before preparation of the loan even began—in a number of cases, years before—and so are clearly not attributable to the operation. Some are the same as policy actions supported by the Env PRL DPO approved in 2004. Details on assessing the efficacy of the loan are provided in the full evaluation. The operation also had a number of other weaknesses. The World Bank carried out no analytical work on the critical area of BNDES’s environmental and social management system. There was also little collaboration or coordination with partners—particularly the International Finance Corporation (IFC)—which had been working with BNDES to improve its approach to environmental and social management by adopting the Equator Principles before the SEM DPL was prepared. World Bank supervision did not ensure the implementation of the promised monitoring and evaluation system, or provide the promised technical assistance to BNDES. Monitoring and evaluation was weak. The Bank’s ICR provided no information on feedback from stakeholders, and no beneficiary assessment. The 13-month delay in effectiveness of the operation undermined part of the rationale for a DPO as fast-disbursing budget support, and was not dealt with in a timely manner. According to the World Bank ICR, the delay was mainly owing to the government having forgotten to include the loan in its budget, and to an “extraordinarily lengthy” senate approval process. The cancelation, after the first loan, of an environmental DPO series that the government had committed to—for the second time in a matter of years—combined with the lack of progress on a number of environmental policy areas, indicates uncertainty and a degree of lack of commitment by the government to the SEM DPL reform agenda. The earmarking of DPF funds to BNDES potentially undermined the approach of the DPO instrument to management of environmental and social risks. This approach considers only environmental and social effects of specific policies supported by the operation, on the grounds that financing is being provided for general budget support and not for specific investments. However, in this case evidence gathered by IEG from multiple sources confirmed that the US$1.3 billion were then on-lent by Brazil’s Treasury to BNDES with the express intent of financing BNDES’s investment projects. Because this was done in the context of DPF, World Bank investment lending safeguards were not applied. High-profile investments financed by BNDES—including hydroelectric dams in the Amazon rainforest—continue to experience an array of environmental and social problems, and a lack of transparency on safeguards by international standards. 57 The assessment includes a number of lessons, which are summarized below: • Particularly for DPOs focused on reforms in a sector—rather than on macroeconomic stability—the impacts of the actions supported can often not be adequately perceived within the short timeframe of the loan, and the tight deadline for submitting an ICR after closure often does not allow for additional outcome evidence to be available and collected. It would be preferable to require that DPO operations wait a reasonable period either before closing or before producing an ICR—at least one year—so as to allow for adequate monitoring and evaluation. • Back-loading of reforms in a DPO programmatic series can increase the risk of later loans in the series being canceled without full realization of the objectives of the series. • In the future, when attempting to support reforms in state-owned banks like BNDES that finance both public and private sector investments, it will be important to take a “One World Bank Group” approach. The program would have benefitted from closer collaboration between IFC and World Bank. IFC had been working with BNDES to improve their environmental and social standards before the SEM DPL, but their efforts were later sidelined. • The level of government targeted by DPO reforms should be consistent with the outcomes intended and the client country’s institutional structure. In Brazil, although many national-level laws and policies are approved by the central government, implementation often depends in large part on states and municipalities. But the SEM DPL limited itself to the federal-level implementing agencies. Source: http://ieg.worldbankgroup.org/Data/reports/brazil-sem-DPO-ppar.pdf 58 Appendix C. Methodology World Bank operations are mapped to a Global Practice (and prior to the Bank’s 2013 restructuring were mapped to a Sector), and are assigned a percentage weighting across sector and theme codes based on their content. For the purposes of this review, environmental DPOs were defined to be any policy lending operation mapped to the Environment and Natural Resources Global Practice or the Environment Sector Board, or any other policy lending operation with an environmental or disaster risk management theme as the primary or secondary theme. That is to say, operations where the theme with the highest or second highest percentage weighting was either Biodiversity (80), Climate Change (81), Environmental Policies and Institutions (82), Land Administration and Management (83), Pollution Management and Environmental Health (84), Water Resources Management (85), Other Environment and Natural Resources Management (86), of Natural Disaster Management (52) were included. This definition identified 64 operations since FY2000, including four operations prior to FY2005 which formally were sector adjustment loans or structural adjustment loans rather than development policy operations. This definition excludes cases where a small number of environment-relevant policy actions were covered in larger policy lending operations. The rationale for this was to remain focused on a core portfolio of limited scope, where environment issues played a major role in the operation and where lessons were thus more likely to be relevant. However, this approach does create a bias in coverage toward large middle-income countries that have had standalone environmental operations. It also means that the learning product is unable to address the question of the relative effectiveness of putting environment actions into large multisector policy lending operations versus standalone environment-focused operations. The definition also deliberately excludes Catastrophe Deferred Drawdown Operations, which are somewhat different from other policy lending operations and will be covered by other forthcoming work from IEG on drawdown options. The Learning Product is designed to look at environmental DPOs as a case study of sectoral DPO operations. The product draws largely on existing evaluative material but with some new analysis. Specifically, the review drew on: 1. A portfolio review of all 64 Environmental DPOs over 2000–16, with the focus on the 39 operations that are closed and evaluated. 2. A synthesis of lessons from existing self-evaluations and IEG validations (ICRs and ICR Reviews). 3. Detailed case studies based on already completed evaluations with field visits (PPARs) for operations in Brazil, Ghana, Turkey, and Indonesia. A new desk-based case study with additional analysis and interviews was carried out for DPOs in Mexico 4. Document analysis and interviews with Bank staff for active and un-evaluated operations. 5. 25 Bank staff and managers were interviewed, in addition to those interviewed for the field- based project assessments. This portfolio review developed a typology of prior actions and lessons supported by operations, and identified the analytic underpinnings, use of technical assistance, 59 coordination with other related operations, presence of other donors, and quality of objectives. 60