Delphi Technique Migara Jayawardena Enno Heijndermans How Expert Panels Predict Maurya West Meiers Emerging Opportunities Ryan Watkins and Challenges in Joy Butscher Renewable Energy Shenghui Feng Noureddine Berrah IEG Methods and Evaluation Capacity Development Working Paper Series © 2022 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: Jayawardena, Migara, Enno Heijndermans, Maurya West Meiers, Ryan Watkins, Joy Butscher, Shenghui Feng, and Noureddine Berrah. 2022. Delphi Technique: How Expert Panels Predict Emerging Opportunities and Challenges in Renewable Energy. IEG Methods and Evaluation Capacity Development Working Paper Series. Independent Evaluation Group. Washington, DC: World Bank. EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Luísa Ulhoa Fernanda Macedo This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Delphi Technique How Expert Panels Predict Emerging Opportunities and Challenges in Renewable Energy Migara Jayawardena, Enno Heijndermans, Maurya West Meiers, Ryan Watkins, Joy Butscher, Shenghui Feng, and Noureddine Berrah Independent Evaluation Group January 2022 CONTENTS Abbreviations����������������������������������������������������������������������������������������������������������������iv Acknowledgments�������������������������������������������������������������������������������������������������������vi Overview��������������������������������������������������������������������������������������������������������������������� viii 1. Background and Context ������������������������������������������������������������������������������������������2 2. Rationale for Undertaking Delphi ��������������������������������������������������������������������������10 3. Designing and Planning for Delphi ������������������������������������������������������������������������16 4. Administering the Delphi Feedback Process ��������������������������������������������������������28 5. Analyzing Results ���������������������������������������������������������������������������������������������������40 Analyzing Opportunities and Challenges 44 Analyzing Actions or Solutions  51 Analyzing the Bank Group: Readiness to Support Clients 59 6. Formulating Conclusions����������������������������������������������������������������������������������������64 7. Final Remarks on the Delphi Technique�����������������������������������������������������������������74 Bibliography����������������������������������������������������������������������������������������������������������������76 Appendix A | Sample Response Templates for Round 1 (Blank)—Ideas Generation for Opportunities and Challenges����������������������������������������������������������������������������� 80 Appendix B | Analysis of Delphi Results������������������������������������������������������������������94 ii  ABBREVIATIONS CO2 carbon dioxide DG distributed generation IEG Independent Evaluation Group PV solar photovoltaics QCA qualitative comparative analysis RE renewable energy SDG Sustainable Development Goal ToC theory of change VRE variable renewable energy Independent Evaluation Group | World Bank Group v ACKNOWLEDGMENTS The paper was prepared by the Independent Evaluation Group (IEG) for interna- tional development specialists, with both evaluation professionals and energy sector practitioners in mind. It delves into issues of renewable energy invest- ments and the strategic context for those investments. It remains, however, a methods-focused paper that also examines in detail the sector-specific evidence, describing the approach and process for designing and carrying out the Delphi ex- periment and the analysis of results and key conclusions drawn from the findings. This paper was produced as part of the Methods Paper Series sponsored by the Independent Evaluation Group’s Methods Advisory Function. The authors are grateful for the feedback provided by the peer reviewers: Lauren Kelly and Ray Rist. In addition, the authors would like to thank the editors and staff of the Paper Series for their comments and contributions: Ariya Hagh, Sylvia Otieno, Maurya West Meiers, Jos Vaessen. A special thanks to Amanda O’Brien, Kristen Milhollin, Fernanda Macedo, and Luísa Ulhoa for their support in editing, pro- duction, and graphic design. Although many people contributed to the Delphi exercise and the preparation of this paper, the findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and should not be attributed in any manner to the Bank Group or to members of its Board of Executive Directors or the coun- tries they represent. Independent Evaluation Group | World Bank Group vii OVERVIEW This paper describes how a panel of global experts on renewable energy (RE) was convened to anticipate the big opportunities and major challenges facing the scale-up of RE to meet sustainable development and climate goals. The panelists participated in a Delphi process through which their views and perceptions were shared, aggregated, and analyzed. The Delphi analysis was a part of a compre- hensive, multimethod evaluation (RE evaluation) that was carried out by the Independent Evaluation Group (IEG) of the World Bank Group. The primary pur- pose of the RE evaluation was to assess the institution’s performance in support- ing the development of renewable energy from 2000 to 2017, examining how well the Bank Group is positioned to help clients overcome emerging challenges and seize future opportunities for scaling up various renewable energy technologies. It is an important global development consideration, since the United Nations’ Sustainable Development Goals and the Paris Agreement on climate change ex- pect that renewable energy will be a key solution in an anticipated clean energy transition. The Delphi panel specifically helped contextualize the future of rapid- ly evolving RE markets, against which the Bank Group’s readiness to help clients could be assessed. By doing so, the Delphi panel also identified critical barriers that, when triangulated with results sourced from other methods, charted a path- way for scaling up RE for achieving global energy and climate goals. The paper is organized as follows: Chapter 1: Background and Context. This chapter sets out the energy sector context under which the overall multimethod evaluation was carried out. Chapter 2: Rationale for Undertaking Delphi. This chapter explains the strategic rationale for opting to apply a Delphi method with a global panel of experts on renewable energy, and how it strategically fit with and contributed to the overall multimethod evaluation. Chapter 3: Designing and Planning for Delphi. This chapter describes the stra- tegic planning and design undertaken before administering the Delphi process. Independent Evaluation Group | World Bank Group ix Chapter 4: Administering the Delphi Feedback Process. This chapter details how the Delphi process was administered in the case of the global expert panel on renew- able energy, highlighting how the feedback process was managed to ensure integrity of the approach and credibility of the results. This chapter is the most relevant to evaluation professionals who may want to conduct a Delphi exercise. Chapter 5: Analyzing Results. This chapter presents the analyses of the raw data, delving into issues confronted by energy practitioners to identify key sector findings. It provides a sector narrative that arises from the Delphi panel results. Chapter 6: Formulating Conclusions. This final chapter highlights important findings from the Delphi process and illustrates how they were triangulated with results from other applied methods to draw conclusions in the broader multimeth- od RE evaluation. x Delphi Technique | Chapter 4 1 BACKGROUND AND CONTEXT World Bank Group commitment to renewable energy Pathway to achieving the clean energy transition Shifting fuel sources for global production of electricity Renewable energy (RE) is central to achieving the Sustainable Development Goals (SDGs) and responding to the urgency of climate change. According to the United Nations, the availability of affordable and clean energy (SDG 7) is an explicit and interdependent goal “crucial for achieving almost all [16 other] SDGs,” such as eradicating poverty, improving health and education, supplying clean water, indus- trialization, and combating climate change. These are further emphasized through SDG 13 on climate action (UN 2018). The Paris Agreement on climate change, which became effective in 2016, places even greater prominence on scaling up RE as a key solution to limiting global temperature rise to no more than 2°C by 2100 and to “make best efforts to limit warming to 1.5°C” (World Bank, IFC, and MIGA 2016). RE has experienced a dynamic expansion over the years through the development of multiple technologies and has the potential to scale up further in the future. The major RE technologies—hydropower, solar power, wind power, geothermal power, and biopower—produce over a quarter of the world’s electricity (figure 1.1). Hydro- power, which has been the dominant technology by scale, makes up the highest pro- portion in the global RE mix, with a 62 percent share in 2017 (or 16.4 percent of total global electricity production). Around 2000, wind power began to expand at a global- ly significant scale, followed by a major scale-up of solar photovoltaics (PV) starting around 2008. Together, wind and solar PV made up 7.5 percent of global electricity produced in 2017. For the same year, biopower and geothermal—two long-standing technologies—produced less than 3 percent of global electricity. RE markets experi- enced a major transformation during the period assessed, with multiple technologies now showing a significant global impact (box 1.1). Future expansions of RE will need to navigate this anticipated continued market volatility. 4 Delphi Technique | Chapter 1 Figure 1.1. Renewable Energy Share of Global Power Generation, 2017 73.5% Nonrenewable electricity 5.6% 26.5% 16.4% Wind power Renewable Hydropower electricity 2.2% Biopower 1.9% Solar PV 0.4% Ocean, CSP, and geothermal power Source: REN21 2018. Note: CSP = concentrated solar power; PV = photovoltaic. A very large scale-up in RE is required to achieve global energy and environment goals, which needs to be set in motion immediately and sustained for several de- cades. Despite the expansion of RE, electricity produced from fossil fuels makes up nearly three-quarters of the global generation mix, since these technologies have also continued to grow. Therefore, the expansion of RE will need to accelerate and be sustained so that the growth of fossil-based technologies can be reduced without locking in a less sustainable pathway to meeting energy demand. Several studies pro- pose different pathways to achieving such goals, but all options involve an unprec- edented scale-up in RE.1 One such analysis by the International Energy Agency used in a 2019 report by the Intergovernmental Panel on Climate Change estimates that the RE share in the global energy mix will need to more than double by 2030 (consis- tent with the SDGs) and further accelerate to almost quadruple by 2040—a program of action commonly referred to as the clean energy transition (table 1.1). The 2019 Intergovernmental Panel on Climate Change report calls for “rapid and far-reaching” transitions in the energy sector (among others) to limit global warming to 1.5°C. It is estimated that such an expansion will double the present global level of investments in RE, from $300 billion to $600 billion (IEA 2017, 2019; IRENA 2017). Independent Evaluation Group | World Bank Group 5 Table 1.1. A Pathway to Achieving the Clean Energy Transition Installed Capacity Average Electricity Produced (GW) Average Capacity (TWh) Renewable Annual Addition Energy Increase (GW/ Technology 2016 2030 2040 (percent) year) 2016 2030 2040 Hydropower 1,241 1,723 2,060 2.1 34.0 4,070 5,688 6,928 Wind 466 1,706 2,629 7.5 90.0 981 4,193 6,950 Solar PV 299 1,846 3,246 10.4 123.0 303 2,732 5,265 Biopower 127 243 347 4.3 9.2 570 1,209 1,807 Geothermal 13 44 82 8.0 2.9 86 292 563 CSP 5 92 328 19.0 13.5 11 287 1,066 Source: Based on International Energy Agency forecasts for its Sustainable Development Scenario. Note: 2030 represents a target consistent with meeting the Sustainable Development Goals. CSP = con- centrated solar power; GW = gigawatt; PV = photovoltaic; TWh = terawatt-hour. The World Bank Group has mainstreamed its support to RE and is committed to the SDGs and the climate goals in the Paris Agreement. The institution has a long his- tory of financing hydropower, but its support for other RE technologies was initially modest and mostly for increasing electricity access, particularly in rural areas not served by the power grid. However, in 2004, the president of the Bank Group made a commitment at the Renewable Energy Conference in Bonn, Germany (the Bonn Com- mitment) to increase its lending for RE by 20 percent over five years (figure 1.2). This commitment coincided with the global expansion in wind and predated the boom in solar power that followed. After the Bonn Commitment, there was a notable uptick in the Bank Group’s financial support to RE covering all major technologies, averaging about $2 billion per year. The scale-up in funding extended to the greater involve- ment of the private sector, including through the International Finance Corporation and the Multilateral Investment Guarantee Agency—two institutions within the Bank Group. Although the Bank Group’s financing for RE is dwarfed by the $300 billion in annual global flows into the sector, the institution’s significant international role in supporting reforms, convening partners and mobilizing financing, and disseminating global experiences is a unique position of influence. It also explains the Bank Group’s interest in evaluating its performance in support of RE so that it is optimally geared to help client countries successfully navigate the clean energy transition. 6 Delphi Technique | Chapter 1  orld Bank Group Commitment to Renewable Energy, Figure 1.2. W 2000–17 $1.6 billiona $8.5 billiona $12 billiona (2000–05) (2006–11) (2012–17) 3.0 2.67 2.57 2.46 Commitment amount (US$, billions) 2.43 2.5 2.20 2.13 2.0 1.73 1.60 1.57 1.5 1.39 1.10 1.0 0.49 0.5 0.34 0.34 0.40 0.29 0.16 0.19 0.0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 58 projects 189 projects 281 projects Institution World Bank IFC MIGA Source: World Bank Group 2018. Note: The renewable energy portfolio evaluated includes 18 additional projects that had been approved previously and were evaluated in 2000 or later. IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency. a. Total commitment amounts do not include issuances of MIGA guarantees. Box 1.1.  The Evolution of Renewable Energy and Other Generation Technologies The global power supply has evolved over time, with renewable energy (RE) playing an increasing role. Global electricity production grew more than fourfold over the 45 years between 1970 and 2015, with shifts in the mix of generation technologies (IEA; figure B1.1.1). Thanks to its reliability and low production cost, coal grew steadily to the main nonrenewable source of global power generation, representing nearly 40% of total production by 2015. Hydropower is a long-standing RE technology that has also continued to increase, although its share of global electricity declined from 23 to 17 percent over 1970–2015, as fossil fuel use grew more rapidly over the same period. (continued) Independent Evaluation Group | World Bank Group 7  he Evolution of Renewable Energy and Other Generation Box 1.1. T Technologies (cont.)  hifting Fuel Sources for Global Production of Figure B1.1.1. S Electricity, 1971–2015 10,000,000 Global electricity production (GWh) 8,000,000 6,000,000 4,000,000 2,000,000 0 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015 Year Coal Gas Hydropower Nuclear RE w/o hydropower Oil Source: Independent Evaluation Group, based on International Energy Agency data. Note: GWh = gigawatt-hours; RE = renewable energy. Decreases in prices led to a rise in the use of oil for power generation in the ear- ly 1970s, followed by a decline later in the decade resulting from global oil shocks. Nuclear power, viewed as an alternative to fossil fuels, expanded until the 1990s, but several accidents progressively led to a slow-down in its growth out of safety con- cerns. Natural gas was used sparingly for power generation, with a global share of only 13 percent in 1971, mostly confined to countries that produced the fuel. However, technological advances in extraction and more efficient means of transportation al- lowed natural gas to overtake hydropower as the second-largest fuel source for power production by the late 1990s. (continued) 8 Delphi Technique | Chapter 1  he Evolution of Renewable Energy and Other Generation Box 1.1. T Technologies (cont.) RE technologies such as wind power and solar photovoltaics were initiallly devel- oped at small scale or for demonstration purposes before their rapid global expansion began in 2000. Electricity produced from these sources grew from negligible shares in the early 2000s to reach a combined share of 7.5 percent of global electricity pro- duced by 2017. Other more niche RE technologies, such as geothermal power, con- centrated solar power, and biopower, are not globally significant by scale, but some make important contributions in specific countries. Endnotes 1 The Intergovernmental Panel on Climate Change 2019 Report considers the following stud- ies: the Energy Technology Perspectives (ETP) (series IEA 2014, 2015, 2016, 2017), the IEA and IRENA reports (OECD/IEA and IRENA 2017), and the Shell Sky Scenario (Shell International Finance B. V. 2018). Shell Corporation, ‘Sky’ scenario (2019): www.shell.com/skyscenario, accessed April 2021. Independent Evaluation Group | World Bank Group 9 2 RATIONALE FOR UNDERTAKING DELPHI Comparative case studies Methods used in the renewable energy evaluation Semistructured interviews The Delphi technique is a widely used systematic forecasting technique for achiev- ing convergence of opinion by soliciting information from experts on a certain subject through an interactive, iterative, and anonymous approach. It was originally developed by the Rand Corporation in the 1950s, primarily for military purposes. The technique, which is rooted in the premise that “[more] heads are better than one” (Dalkey and Rourke 1972), is a group communication exercise that aims to examine a specific issue for goal setting, policy examination, and forecasting of future events (Ulschak 1983; Turoff and Hiltz 1996; Ludwig 1997). Regular surveys explore “what is”; the Delphi technique attempts to address “what could/should be” (Miller 2006)— its name is derived from the ancient Greek oracle who could predict the future. The Delphi technique typically seeks to (i) shed light on alternatives; (ii) correlate expert insight on a specific subject; (iii) provide background information for decision-mak- ing; and (iv) reveal consensus in expert opinion (Watkins, West Meiers, and Visser 2012). “The Delphi method uses informed, intuitive judgment to analyze the future.” —Rand Corporation (original developer of technique) The Delphi approach is different from regular surveys. It employs several rounds of data collection, usually through a questionnaire and controlled feedback to en- courage reflection of one’s own contributions and those of others (Hsu and Sandford 2007). During the first round, the group of experts independently share their views with a facilitator, who reviews the data and provides a synthesis or summary. The experts review this summary and are requested to provide updated inputs through one or more additional rounds. Throughout the process, the experts have a com- plete record of what insights and forecasts others have shared during each round, without attribution to any specific individual, to maintain anonymity. This allows each expert to (i) review the aggregated inputs of the group; (ii) reassess their initial judgments; (iii) generate additional insights; and (iv) clarify the information devel- oped by previous iterations (Hsu and Sandford 2007). Anonymity is an important aspect of the Delphi technique, since it ensures free expression of opinion and helps prevent potential pitfalls of conventional means of pooling group opinion, for exam- ple, reluctance to revise opinions, the influence of dominant individuals, and group pressure for conformity—either real or perceived (Dalkey and Rourke 1972). The emphasis on anonymity and confidentiality makes the Delphi technique particularly suitable for online facilitation. 12 Delphi Technique | Chapter 2 The Delphi technique seeks to find convergence of opinion by asking experts to prioritize emerging issues, which enables quantitative analysis of the opinions (Thangaratinam and Redman 2005), making the process more like problem-solving. A Delphi panel continues until a significant level of consensus is reached—typically two to four rounds. Even if consensus is not reached to an adequate level, this itself can indicate that there is not significant consensus of expert opinion related to the question being asked. The Bank Group’s Independent Evaluation Group (IEG) undertook a Delphi exercise as a part of a broader evaluation of RE. A key aim of the evaluation was to assess the institution’s readiness to help clients navigate the clean energy transition described in chapter 1. Although the objectives of the transition are clear, there is less certain- ty as to how the RE markets, which have experienced a considerable evolution since around 2000, will unfold. Therefore, as one element of the evaluation, IEG wanted to identify the key emerging opportunities related to RE that the institution should help clients seize and challenges on the horizon that developing countries would require help to overcome. To establish this future scenario, IEG convened a global panel of experts on RE. The goal was to apply the Delphi technique with the panel in an attempt to generate consensus of expert perspectives on the future of RE, against which the Bank Group’s readiness to help clients navigate the transition could be evaluated. The Delphi panel was one methodology in the multimethod RE evalua- tion to assess the Bank Group’s performance in supporting RE (see box 2.1). It is ex- pected to inform the future direction of the institution’s support to client countries in scaling up RE to meet globally established SDGs and climate goals. Box 2.1.  Multimethod Evaluation of World Bank Group Support to Renewable Energy Development The renewable energy (RE) evaluation was designed to assess the World Bank Group’s performance during 2000–17 in supporting clients in developing and scaling up their RE resources (World Bank 2020). Specifically, the RE evaluation attempted to answer the following questions: (i) In what ways—and how well—has the Bank Group contrib- uted to addressing the evolving RE needs of its clients?; and (ii) What lessons from ex- perience can be identified to strengthen the role of the Bank Group in helping clients achieve their emerging RE goals (that is, the Sustainable Development Goals and the clean energy transition)? (continued) Independent Evaluation Group | World Bank Group 13 Box 2.1.  Multimethod Evaluation of World Bank Group Support to Renewable Energy Development (cont.) Figure B2.1.1. Methods Used in the Renewable Energy Evaluation Structured RE market SLR barriers SLR benefits/ literature review review to RE impacts of RE Portfolio review 546 RE projects/ 245 CAS/CPF CBA/in-depth EVALUATION QUESTIONS and analysis investments ASA/AS strategies hydro review Comparative 9 in-depth QCA causal 19 PPARs case studies country studies analysis Semistructured Public & private World Bank Group staff Other interviews Bank Group clients survey/interviews partners Global expert Delphi panel of global panel experts on RE Source: Independent Evaluation Group. Note: AS = advisory services; ASA = advisory services and analytics; CAS = Country Assistance Strategy; CBA = cost-benefit analysis; CPF = Country Partnership Framework; QCA = qualitative comparative analysis; PPAR = Project Performance Assessment Report; RE = renewable energy; SLR = structured literature review. Multiple methods were applied to triangulate findings into robust conclusions. These methods included the following: » Structured literature review. An assessment of the evolution of RE markets; a literature review of barriers to developing RE, the energy and environment out� comes of electricity produced from RE, and development impacts. (continued) 14 Delphi Technique | Chapter 2 Box 2.1.  Multimethod Evaluation of World Bank Group Support to Renewable Energy Development (cont.) » Portfolio review and analysis. A review of 546 investment projects in the Bank Group’s RE portfolio, select World Bank advisory services and analytics and International Finance Corporation advisory services from a portfolio of 245 activities and 19 Project Performance Assessment Reports, and an in-depth review of hydropower. » Comparative case studies. Nine purposefully selected, in-depth country case studies, and a qualitative comparative analysis of the case results to validate the theory of change and identify critical pathways to expanding RE. » Semistructured interviews. Interviews with public and private stakeholders, de� velopment partners, and key Bank Group managers, and a survey of a purposive sample of Bank Group staff working on RE. In addition, a global panel of experts on RE were convened to help predict emerging trends in RE: » IEG global expert panel on RE. A structured, iterative Delphi process with a set of global experts who helped identify and prioritize emerging RE opportunities and challenges, to establish a future scenario against which the Bank Group’s capacity and position to influence can be evaluated. Independent Evaluation Group | World Bank Group 15 3 DESIGNING AND PLANNING FOR DELPHI Formulating key themes and questions Conceptualizing the Delphi process Structure of feedback template The successful application of a Delphi technique requires careful design and plan- ning, a tightly administered feedback process, and robust analysis of results to formulate conclusions. The illustration in figure 3.1 diagrams the key steps to com- pleting a Delphi process. Figure 3.1. The Delphi Process—Design Stage Design of Delphi approach Controlled feedback process Results analysis 1 2 3 4 10 11 Conceptu- Formulate Initiate Delphi Results Incorporate alize and themes and panel and Round 1 analysis into evaluation assemble data-gather- process Ideas or report panelists ing modalities generation 8 9 5 Round N Round N Round 1 Review and Synthesize Synthesize prioritize results results synthesis 7 6 Round 2 Round 2 Synthesize Review and results prioritize synthesis Source: Independent Evaluation Group. This section focuses on the first stage of applying a Delphi process (highlighted in blue in figure 3.1), which is to design and plan the approach. The aim is to develop the strategy and the preparatory steps needed to effectively capture feedback from experts and analyze the results. The specific approach that was applied in the RE evaluation is detailed for each step. 18 Delphi Technique | Chapter 3 Conceptualize the Delphi Process Conceptualizing the Delphi process requires rationalizing the purpose of using a Delphi process with a panel of experts and how the results would be used, whether in a stand-alone manner or as a part of a multimethod evalu- ation, as was the case with the RE evaluation (figure 3.1, 1). It is also useful to consider the time frame and budget needed to carry out the Delphi activities. In the RE evaluation, the Delphi technique was selected as a key method specifi- cally to help frame the responses to a key evaluation question: What lessons from experience can be identified to strengthen the role of the Bank Group in helping clients achieve their emerging RE goals (that is, scale up for meeting SDGs and the clean energy transition)? To answer this question, more than an assessment of the Bank Group’s capacity and position of influence was needed; it required a prediction of the investment climate that would prevail in developing country RE markets in the coming years. The decision by IEG to assemble a global panel of experts on RE and lead them through a Delphi process was an essential element of making such a forecast. It would help determine the Bank Group’s readiness to help clients navi- gate future RE markets based on its capacity assessed against the predictions of the Delphi panel. The findings from the Delphi process could then help draw conclusions when triangulated with results from other methods, such as a qualitative compara- tive analysis (QCA), a project portfolio review, and interviews with Bank Group staff and developing country stakeholders. Administering a Delphi panel also fit within the schedule of the overall RE evalua- tion. Although the Delphi approach was conceptualized early in the process of pre- paring the overall RE evaluation, most of the other methods were already underway by the time IEG assembled the global panel of experts in RE. It was anticipated that the Delphi approach could be completed in two to three iterative rounds before the results could be analyzed for incorporating into the RE evaluation. The anticipated costs were also budgeted, including payments for panel members and expenses for administering the Delphi process. Independent Evaluation Group | World Bank Group 19 Assemble Panel of Experts The identification of suitable panelists (figure 3.1, 1) is a critical element to the successful application of a Delphi process, since the results crucially rest on the panel. It is vital to include specialists who have strong reputations with regard to their expertise on the subject matter. This may be straightforward when exploring a narrow subject with well-understood industry norms. Ap- propriate representation can be more complex when the subject is broader (possibly at the congruence of multiple subjects, such as climate change) or less well defined. The size of panels ranges widely based on Delphi-related literature; some are more akin to smaller focus groups, and others can be large and almost surveylike. A consideration for smaller groups is ensuring that there are not significant dropouts over the multiple rounds, as this may compromise the integrity of the results. With very large groups, it may be asked whether the highest level of expertise on the subject is so widely dispersed. The RE evaluation was exploring the sector at the energy and environment nexus as a key solution to sustainably meeting energy demand and helping mitigate global climate change. This made the subject matter wider than delving into the narrower area of RE, requiring broader expertise within the Delphi panel to provide a glob- al environmental perspective. Therefore, several overarching criteria were applied when inviting experts to participate in IEG’s global expert panel on RE: ▪ Extensive knowledge, applied experience, and globally recognized expertise in issues of energy, climate change, or both. ▪ A mix of representation within the panel from developed countries, where much RE development has occurred, and developing countries, which repre- sent the lion’s share of future expansion (and make up the Bank Group’s cli- ents). ▪ A combination within the panel of experts in private sector development of RE, which is expected to make a significant contribution to the clean energy transition, and those with deep knowledge of public sector initiatives in policy and investments, which are likely to be important in facilitating the expansion of RE in developing countries. ▪ Experience addressing climate change at the global level and an understand- ing of the implications for it of scaling up RE. The aim was not to be dogmatic about the criteria but to ensure that the overall global panel of experts on RE would adequately represent these important consider- 20 Delphi Technique | Chapter 3 ations, so that panelists would take into account a diversity of perspectives related to RE in arriving at conclusions. Ultimately, IEG’s global panel of experts on RE in- cluded eight members. The panelists were from developing and developed countries, representing both the public and private sectors. The private sector was represent- ed by a RE investor in emerging markets and the leader of a global RE equipment manufacturer. Large developing countries with a significant global environmental footprint and ambitious RE targets, such as China and India, were represented, as were smaller, less polluting countries at different stages on the development spec- trum. Several experts from globally significant carbon dioxide (CO2) emitters and from those countries most urgently affected by climate change participated in the panel. Each of them had played a prominent role in the Paris Agreement negotia- tions. Some experts also represented global think tanks on energy and academia. One shortcoming in the panel was gender balance. Despite efforts by IEG, some invited female participants declined because of scheduling difficulties, and in one case, a possible conflict of interest. Perhaps inviting more female panelists earlier in the process would have resulted in better gender balance. Formulate Key Themes and Questions Based on the conceptualization, determine what themes the panel of experts is expected to explore through the Delphi process (figure 3.1, 2). Once the themes are selected, it will be important to carefully craft the specific ques- tions posed to the panel, ensuring that they are clearly formulated to prevent ambiguous interpretations. Jargon that might be variously interpreted should be clarified or entirely avoided, but commonly understood industry terminolo- gy is appropriate. In the RE evaluation, the objective was to explore in greater depth certain aspects of the theory of change (ToC) that were developed by IEG based on global (includ- ing Bank Group) experience. The ToC, elements of which are described in more detail in box 3.1, was predicated on investments in RE resulting in (i) benefits to consumers from the increase in electricity supplied, and (ii) contributions to mitigating global climate change stemming from the displacement of alternative fossil-based power generation sources that would have otherwise emitted CO2 (a causal link that was subsequently confirmed through a separate QCA). However, mobilizing investments in RE can be hindered by several key barriers, according to the ToC. Therefore, adequately addressing these barriers was vital to achieving the energy and climate goals. The dynamic evolution of RE markets over the evaluation Independent Evaluation Group | World Bank Group 21 period also changed the nature of these barriers to RE, while other opportunities for scaling up various technologies emerged. Similar market shifts are expected to continue and change the future investment landscape in the sector. Through IEG’s global expert panel on RE, the RE evaluation sought to identify these future chang- es in RE markets that may help or hinder the envisaged scale-up in the sector by proposing the following questions: ▪ What are the main emerging opportunities to further develop RE around the developing world for power generation to meet climate and SDGs (and why)? An opportunity was defined as a specific condition that is favorable or conducive to scaling up RE per the clean energy transition, including but not limited to technological advances, improving market conditions, changes in demand patterns, shifts in policy, and influences outside the RE space (such as shifts in fossil fuel prices or the impact of climate change). ▪ What are the main emerging challenges that could hold back develop� ing countries from further development of RE for power generation, and hamper their ability to meet climate and SDGs (and why)? A challenge was defined as a specific constraint or barrier that countries are likely to face in attempting to scale up RE per the clean energy transition, which, if not ad- dressed, will hamper their ability to achieve RE development goals. ▪ For the emerging opportunities and challenges that are predicted by the respective panelist, what specific key action(s) should be taken by devel� oping countries to seize each opportunity and address each challenge? 22 Delphi Technique | Chapter 3 The aim was to solicit potential solutions that could be implemented by de- veloping countries. Box 3.1.  Theory of Change for RE Evaluation The following theory of change (ToC) was developed by the Independent Evaluation Group to define the causal relationship for expanding renewable energy (RE), incor- porating the World Bank Group’s role in supporting clients achieve their energy and environmental goals. Figure B3.1.1. Theory of Change for Renewable Energy Development A B C D E Inputs Key types of barriers Development Energy and Contribution to broader addressed of RE environment benefits development impacts 1 Inadequate policies Increased Increase electricity Contributes to and regulations RE supply to grid economic growth capacity and 2 Inability to integrate generation Improve access to Supports improve- RE into power system electricity ments to quality of life, Financing Reduce energy including for people Global 3 Insufficient design living in poverty insecurity knowledge and technical standards Helps protect local and Displace Avoid global global environment Partnerships 4 Weak institutional fossil-based pollution and human capacity capacity and generation Avoid local 5 Existence of pollution Assumed significant investment contributions from risks interrelated sectors 6 Constraints to mobilizing financing Source: Independent Evaluation Group. Note: RE = renewable energy. The ToC is most easily understood beginning with the column labeled C, which de- picts an output of increasing investments in RE technologies, which essentially means the construction and operation of RE power plants and associated infrastructure such as transmission lines to evacuate the electricity for supplying consumers (that is, to (continued) load centers). Column C also illustrates that investments in power generation from Independent Evaluation Group | World Bank Group 23 renewable sources lead to the displacement of an equivalent amount of alternative generation capacity from fossil fuels. Box 3.1.  Theory of Change for RE Evaluation (cont.) Column D identifies the outcomes that result from this increase in investments in RE and the displacement of fossil-based power generation. The primary energy bene- fits are accrued to grid-based consumers from the increase in supply as a result of RE-based power generation. Additionally, RE-based electricity can also help increase access to those who currently are not connected; and, as indigenous resources, RE- based electricity can reduce energy insecurity that arises from greater reliance on fossil fuel imports. The environmental benefits are derived from the displacement of fossil fuel–based generation by RE, which helps curtail carbon dioxide greenhouse gases that would have been otherwise emitted, contributing to global climate change (that is, avoided global pollution). Additionally, the displacement of fossil fuels also helps avoid local pollution from the emission of sulfur oxide, nitrogen oxide, and par- ticulate matter, which can lead to respiratory illnesses and other health impacts. Col- umn E illustrates how the energy and environmental outcomes, interdependently with outcomes from other development efforts, contribute to larger development impacts. Although mobilizing investments in RE, as depicted in column C, is a major contributor to meeting energy needs while protecting the environment, the ToC suggests that this is dependent on adequately addressing several major barriers that affect the invest- ment climate for RE, as illustrated in column B. The ToC also suggests that a major contribution of the Bank Group to expanding RE is to help clients address these key barriers through disseminating global knowledge, extending financing, and mobilizing partnership support using its global convening capacity, as illustrated in column A. Members of the Independent Evaluation Group’s global panel of experts on RE were requested to share their expertise, primarily on addressing barriers to RE development (column B) and the World Bank’s readiness to support clients in this effort (column A), although they were free to comment on any aspect of the ToC. The previously mentioned themes aimed to elicit perspectives on the energy sectors in developing countries, irrespective of the involvement of the Bank Group. They 24 Delphi Technique | Chapter 3 reflect challenges that must be overcome by countries seeking to scale up RE, op- portunities they could seize to do so, and the actionable solutions they could under- take toward these ends. However, IEG also wanted to explore the extent to which the Bank Group can support clients in successfully implementing these actions or solutions identified by the global panel of experts on RE. It was recognized that, al- though the panelists were sector experts who were aware of the Bank Group’s global role in supporting RE, their knowledge regarding the institution could vary across the panel. However, when triangulated with evidence from other methodological sources, expert perspectives could provide meaningful insights into the readiness of the Bank Group to help clients navigate future RE markets that continue to evolve. Therefore, IEG’s global panel of experts on RE was asked to assess, for each of the actions or solutions identified (for addressing challenges and seizing opportunities), the Bank Group’s position to influence and capacity to help clients: ▪ Identify how well the World Bank Group is positioned to help clients to successfully carry out [each] action/solution. The aim was to identify the institution’s sphere of influence and the corresponding comparative advan- tage to support clients in implementing reforms, recognizing that there are other actors that can shape actions, including the clients themselves. ▪ Identify specific interventions/engagements the World Bank Group can take to support clients more successfully implement each respective ac� tion/solution. The Bank Group could support clients in a range of ways that include sharing of global experiences and knowledge transfer, financial sup- port, and mobilizing additional support from other partners based on the institution’s global convening capacity. ▪ Identify the current capacity of the World Bank Group to successfully undertake each intervention/engagement for supporting client actions/ solutions. The aim was to identify whether the institution has the expertise and experience to adequately help clients undertake their specific actions or solutions for scaling up RE. Design Data Gathering Modalities Once the questions are clearly defined, the next key step is to determine how the feedback from panelists will be gathered (figure 3.1, 2). Typically, a questionnaire or template is prepared, which can be administered electron- ically or on paper. It is at this stage that the designers of the Delphi process will determine the structure through which panelists will prioritize or score Independent Evaluation Group | World Bank Group 25 various inputs and how these results will be (statistically) evaluated to form conclusions. Panelists’ familiarity with different technological modalities is a consideration when designing feedback questionnaires or templates, as is the simplicity and ease with which they can respond to others. In the RE evaluation, the questionnaire was formulated as a Microsoft Word tem- plate, given a diverse international panel’s likely familiarity with that software. An option to develop a web-based interface was discussed and discarded, as such a prototype was not needed and the effort to manage such feedback from a relatively small panel could not be justified. The template that was created played an import- ant role in maintaining control and structure in obtaining feedback, so that results were comparable and could be aggregated with consistency. Two templates were used, one to capture perspectives on emerging opportunities and the other to solicit views on challenges facing the scale-up of RE (figure 3.2, i and ii). The blank tem- plates used for the information initially gathered from IEG’s global expert panel on RE are presented in appendix B. Figure 3.2 also highlights the main structure of each of the two templates, emphasizing the areas where expert opinions were solicited from the panel, consistent with the themes identified in the previous section. Figure 3.2. Structure of Feedback Template A B World Bank Group role Sector opportunities Client solutions and challenges Ranking Bank Group C Emerging position i opportunities Ranking Ranking Client activities or actions Emerging ii challenges Bank Group intervention Ranking Bank Group D capacity Source: Independent Evaluation Group. 26 Delphi Technique | Chapter 3 ▪ Identify specific opportunities or challenges facing the expansion of RE, including rationale for the significance of the selection (figure 3.2, A). The panelists were also requested to provide a score ranking for each selection to ascertain their initial prioritization. The responses were open ended, al- though each panelist was restricted to a maximum of 10 opportunities and 10 challenges, to maintain a sense of priority. ▪ For each of the opportunities and challenges identified, indicate up to three actions or solutions developing countries could undertake to expand RE (fig- ure 3.2, B). Ideas for actions or solutions were only solicited at the initial stage, and later stages requested the panelists to prioritize actions or solutions. ▪ The remainder of the templates focused on the Bank Group’s readiness to support clients in undertaking their respective actions or solutions identi- fied by panelists for each opportunity or challenge. This included obtaining expert opinions on how well the institution is positioned or placed, especial- ly when compared with other development partners, to influence the chang- es identified as actions or solutions (figure 3.2, C). The panelists were asked to score the strength of the institution’s place or position to undertake each intervention. ▪ The templates also solicited up to three specific potential Bank Group in- terventions or support that could help clients successfully undertake their respective actions or solutions. ▪ For the same actions or solutions, panelists were requested to provide their views on the Bank Group’s existing capacity to help client countries suc- cessfully design and implement such actions or solutions through specific interventions (figure  3.2, D). The capacity could include a combination of the institution’s skills and experience, and ability to convene and influence stakeholders and extend financial support. The panelists were asked to score the institution’s level of capacity to undertake each intervention. Independent Evaluation Group | World Bank Group 27 4 ADMINISTERING THE DELPHI FEEDBACK PROCESS Idea generation Kicking off the Delphi process Delphi panel–generated ideas for RE The objective of the second stage is to tightly administer the controlled feedback process to ensure that there is structure and order to the expert opinions solicited through the Delphi technique. Adequately administering this stage is vital for main- taining the integrity of findings and working toward convergence in opinions. Areas covered in this chapter are shaded blue in figure 4.1. Figure 4.1. The Delphi Process—Feedback Stage Design of Delphi approach Controlled feedback process Results analysis 1 2 3 4 10 11 Conceptu- Formulate Initiate Delphi Results Incorporate alize and themes and panel and Round 1 analysis into evaluation assemble data-gather- process Ideas or report panelists ing modalities generation 8 9 5 Round N Round N Round 1 Review and Synthesize Synthesize prioritize results results synthesis 7 6 Round 2 Round 2 Synthesize Review and results prioritize synthesis Source: Independent Evaluation Group. In the RE evaluation, a virtual meeting via videoconferencing proved to be more conducive, given the different international locations of the panelists. IEG initiated the meeting with a presentation that introduced the Delphi process to the panel- ists, explaining the context in which the results will be used in the RE evaluation. The presentation defined their roles as global experts in RE and clarified what was expected from the panel. The panelists were informed of the key questions they were expected to answer about scaling up RE in developing countries and the Bank 30 Delphi Technique | Chapter 4 Group’s readiness to support its clients with related endeavors. Some publicly available data on global RE markets were also distributed to ensure panelists’ shared understanding of the facts. It is important to note that panelists were informed that they were not expected to carry out any additional research unless they wanted to of their own accord. Instead, it was stressed that IEG’s interest in the Delphi exer- cise was to take advantage of their existing RE knowledge and expertise related to achieving the clean energy transition. The virtual kick-off meeting also provided an opportunity to clarify the Delphi pan- el’s feedback process. A step-by-step demonstration of the Microsoft Word question- naire template familiarized panelists with the format for providing their feedback in the first round. They were requested to complete the template individually and con- fidentially; it was confirmed that any data IEG shared with the panel in later rounds would also be confidential and aggregated without attribution to specific individuals. IEG also informed the panel of an email account that was established specifically for corresponding with the global expert panel on RE, which was the primary means of communications and where completed templates were to be submitted. Panelists’ other questions were also answered during the meeting. Additionally, the panel was given information for contacting the IEG team leader and other team members administering the Delphi process should they have questions or concerns during the feedback process. Kicking Off the Delphi Process It is important to ensure that all panelists clearly understand the purpose of their participation in the Delphi, the process and the time commitment required, and the modalities through which they will interact with those administering the process (figure 4.1, 3). Typically, this can be accomplished through a face-to-face or virtual meeting where they can ask questions and seek clarification. It is important to share with them the overall schedule and indicate the estimated required effort and time, including contingencies, since the exact number of required rounds or iterations can be uncertain. It is also important to stress the anonymity and confidentiality of the panel’s feedback or data. After the kick-off meeting, the round 1 questionnaires or templates can be shared, initiating the feedback process. Independent Evaluation Group | World Bank Group 31 An important understanding reached during the virtual kick-off meeting was the time frame for implementing the Delphi process. The schedule included a planned two iterative rounds and a contingent plan for an optional third round and a wrap- up meeting, as necessary. The IEG team recognized that there should be sufficient time to complete the tasks at each stage but that the process should not drag on unduly, especially considering the demanding schedules of a prominent group of global experts. For the same reason, both IEG and the panelists needed to adhere to the agreed schedule. The following schedule was agreed for the first two rounds, to be completed within a month: ▪ Distribution of round 1 response templates to panelists by IEG—immediate- ly after virtual kick-off meeting. ▪ Completed round 1 response templates submitted by panelists to IEG—12 days after kick-off meeting. ▪ Synthesis of round 1 responses and issuance of round 2 templates with syn- thesized information to panelists by IEG—7 days after receipt of round 1 re- sponse templates. ▪ Completed round 2 response templates, including rescoring priorities, sub- mitted by panelists to IEG—7 days after issuance of round 2 template with synthesized information. The panelists were also informed of contingent steps (a possible third round of feed- back and wrap-up meeting) in the feedback process should the need arise, although a specific time frame for these activities was not established at kick-off. Round 1—Idea Generation The next step is for the panelist to proceed individually to complete the round 1 questionnaire or template (figure 4.1, 4). This first round asks open-ended questions intended to generate ideas from the experts. The panelists include their responses in the template and submit them to the administrators for further action. In the RE evaluation, immediately after the virtual kick-off meeting the panelist re- ceived by email two questionnaire templates—one for opportunities and one for chal- lenges facing the scale-up of RE. Examples of the blank templates that were shared with the panel in round 1, which followed the structure illustrated in figure 3.2, are provided in appendix A. Although the template provided structure for data input, it 32 Delphi Technique | Chapter 4 was not populated with any data, providing panelists with an open-ended opportu- nity to freely express their individual predictions (capped at 10 priority challenges and opportunities). The panelists were also asked to prioritize their responses by allocating 100 points among them, with more points indicating an item of higher importance. As the panelists separately prepared their feedback, the IEG team com- municated with them to maintain the agreed schedule. All eight panelists submitted a completed set of questionnaires. Overall, the round 1 submission of responses proceeded smoothly and concluded successfully. Synthesis of Round 1 Results The administrators of the Delphi process now play the crucial role of synthe- sizing the various expert responses, grouping them if feasible (since multiple experts can have similar opinions; figure 4.1, 5). Any initial prioritization can also be assessed, and aggregated scores could be shared for subsequent rounds. Subject matter expertise within the team administering the Delphi process is vital at this stage to decipher responses and synthesize the information into a cohesive set reflecting the views of the overall panel. Once the synthesis is completed, a revised template is prepared. The revised template follows a similar structure, but unlike the initial one that was blank for open-ended re- sponses, it is now populated with the synthesized responses of the panelists without individual attribution (they will see the ideas of others but not know whose ideas they are). The aim is to share with the panel the synthesized results to inform the panel of the broader suite of responses by other panel- ists, which may influence any subsequent feedback and (re)prioritization. The anonymity of individuals is important, so that panelists are influenced by the ideas rather than the expert who may have originated them. In the RE evaluation, the team prepared the basic structure of the round 2 template before receiving the completed round 1 questionnaires, aware that the schedule al- lowed only seven days to complete the synthesis. Two team members were designat- ed to synthesize the feedback—one subject matter specialist in RE with experience leading similar focus groups and one from IEG’s methods team who was familiar with the Delphi technique. Once the templates were received, the first challenge was to group multiple responses in which similar opportunities or challenges were highlighted, and then categorize them into broad areas of reforms. In this case, most responses (27) were grouped according to barriers that were identified in the ToC described in box 3.1, and a few outliers (3) were grouped separately. Round 1 Independent Evaluation Group | World Bank Group 33 resulted in 14 opportunities and 16 challenges identified by the panel, as indicated in figure 4.2. Figure 4.2. Delphi Panel–Generated Ideas for RE—Round 1 The emerging opportunities put forward by the panel for Round 1, grouped by reform areas or barriers Policy and Regulation Desire to meet national and international emission reduction commitment Address local environmental concerns Achieve energy access goals Improve security of power supply with use of indigenous resources Integration of RE into Power Systems Integrated planning of power systems Utilization of technology and smart grids Firming up (variable) RE capacity by combining compli- Other Opportunities mentary RE systems or addition of electricity storage Income generating Increase distributed generation in which RE can play a potential of RE key role Integration of RE and EE in power systems Investment Risk Mitigation planning Mitigation of upfront investments costs/risks of RE through public finance and pooling of resources, for more affordable and greater private sector participation Mobilizing Finance Strong interest in developing market for RE by developer, development partners, and climate financing sources Liberalization of power markets that increase private RE interests Declining technology costs of RE and emergence of power storage 34 Delphi Technique | Chapter 4 The emerging challenges put forward by the panel for Round 1, grouped by reform areas or barriers Policy and Regulation Existing interests that may hinder development of RE Inadequate and unstable policy and regulatory infrastructure Lack of “real” commitment to decarbonize power system Need bankable projects and PPAs Integration of RE into Power Systems Difficult to integrate large amounts of RE, especially those of variable/intermittent nature, without developing flexibility of power systems, which can be costly Improvement to Design and Technical Standards Need local industry to create strong interest in RE promotion Institutional Capacity Other Challenge Need champion to drive RE development process Large land requirements for Strenghten capacity for transparency and accountability to developing RE address corruption and governance Improve systems and capacity to reduce project delays Capacity within government agencies to support and develop RE Investment Risk Mitigation Vulnerability of RE systems to extreme weather and climactic events Mobilizing Finance Affordability of offgrid equipment and aftersales care for poorer populations Difficulty mobilizing financing for RE in less lucrative, smaller markets for developers Regulatory and counterparty risks keep cost of capital high Rapid decline in technology costs could lead to slower uptake as utilities and developers take wait-and-see approach Source: Independent Evaluation Group. Note: EE= energy efficiency; RE = renewable energy. Independent Evaluation Group | World Bank Group 35 Soliciting Round 2 Responses Once the panelists receive the synthesized results, they are requested to reprioritize the set of aggregated responses from round 1, by selecting either their own response or ones identified by others (figure 4.1, 6). A scoring system is typically applied. In the RE evaluation, the panelists were presented with two round 2 templates—one for opportunities and another for challenges, for scaling up RE. However, unlike in round 1, these templates were populated with the synthesized results reflecting the combined views of the entire panel without attribution to any specific panelist. Although the round 2 templates shared round 1 input from all panelists, panelists were requested to rescore the 14 opportunities and 16 challenges. This is an important aspect of the iterative feedback process in a Delphi technique, where each panelist can now reprioritize with the benefit of input from other experts, potentially being influenced by them. The panelists rescored by allocating 100 points across each of the 14 opportunities and another 100 points across the 16 challenges; more points indicated a higher priority. The panelists were also asked to score the same oppor- tunities and challenges using a five-point Likert scale, and to likewise score the cor- responding actions or solutions, which were also prepopulated based on the round 1 synthesis. Similar Likert scale ratings were requested for the Bank Group’s readiness to support clients with specific reforms (actions or solutions), reflected as its posi- tion of influence and capacity. The prepopulated templates and scoring scales can be accessed in a supplemental annex on the IEG website. Since there was already significant convergence in round 1 among panelists, the IEG team saw an opportunity to explore some key emerging issues with the experts in greater depth. Therefore, round 2 included a separate supplemental questionnaire seeking the panel’s responses to a couple of specific RE-related questions. The ques- tions were as follows; panelists were free to respond in essay format: ▪ Given the trade-off between (i) decreasing costs of RE technologies, partic- ularly wind and solar; and (ii) increasing costs on power systems to ensure adequate flexibility for a smooth integration of variable or intermittent RE generation sources, 36 Delphi Technique | Chapter 4 ▪  How can developing countries manage this trade-off? ▪  What are the prospects for availability of economical electricity stor- age solutions (for example, thermal, batteries), and how will this af- fect the trade-off indicated above? ▪ If the Paris Climate Agreement and its emission reduction commitments are a significant opportunity that can support the development of RE, how important is the mobilization of the funds committed in the Agreement by developed countries (that is, $100 billion per year by 2020) to deploy RE gen- eration in developing countries to meet the climate change goals? Synthesis of Round 2 Results The synthesis for round 2 includes evaluating the rescored (reprioritized) ideas by the panel (figure 4.1, step 7). At this point, the administrators will need to determine whether there is sufficient convergence in prioritization of responses or whether further rounds are required. It is possible at this stage to discard ideas that no longer garner support. It is also possible to seek ad- ditional information related to any specific priorities that may be emerging. In the RE evaluation, the feedback was broadly completed promptly. Two panelists required follow-up conversations for clarifications, which delayed obtaining all of the submissions by a couple of days. More important, there were no dropouts, as all pan- elists submitted completed questionnaires or templates (including the supplemental questionnaire). In a single case, a panelist inadvertently copied other members when submitting his responses. IEG had no reason to believe that there was undue influ- ence as a result, since most panelists had already submitted their submissions by then. A quick review of the submissions confirmed that there was sufficient informa- tion and consensus in most of the submissions, and that the panel results could now be analyzed to formulate conclusions. The panelists were duly notified and thanked for sharing their expert opinions for the RE evaluation; without their commitment to RE and the evaluation, the Delphi process could not have been successful. Independent Evaluation Group | World Bank Group 37 Additional Iterative Rounds of Feedback Round 3 and any subsequent rounds will follow an iterative process similar to round 2, until a sufficient degree of consensus is developed among panelists’ predictions (figure 4.1, 8 and 9). Convergence may not be attainable, especial- ly in the case of some new or contentious subjects, which itself is a finding (that there is a significant divergence as to its importance or prioritization). Successive rounds may be time consuming and costly, and may not bring about greater alignment of perspectives—important factors for administrators to consider. It is also possible that the design of the Delphi process or the way questions are phrased are key reasons for divergence of opinions (for exam- ple, panelists may have different interpretations of a question); in such circum- stances, the Delphi process may need to be redesigned, although panelists may be reluctant to participate in a reconfigured effort. In the RE evaluation, further iterative rounds were not needed because of the sig- nificant number of ideas that were generated through the process and the relatively clear convergence across the panel on members’ collective priorities. Therefore, it was decided to use the data on hand at the end of round 2 and proceed with ana- lyzing results and drawing conclusions to be used in the RE evaluation, which are presented in the next chapter. The optional wrap-up meeting was also omitted. 38 Delphi Technique | Chapter 4 5 ANALYZING RESULTS Results analysis stage Analyzing opportunities and challenges Analyzing actions or solutions The third and final stage is to analyze results from the Delphi process and triangu- late them with other methods in the RE evaluation to draw conclusions. This chapter focuses on analyzing the results, and chapter 6 describes what conclusions were drawn from the results and how they were integrated into the broader RE evalua- tion. Greater subject matter focus was provided, given that the results needed to be analyzed using an energy sector perspective. At this stage, no further iterations nor participation by the panelists are required except to seek specific clarifications re- garding their responses as needed. The analysis is carried out by the team adminis- tering the Delphi process. The Delphi process step covered in this chapter is marked 10 in the blue-highlighted area in figure 5.1. Figure 5.1. The Delphi Process—Results Analysis Stage Design of Delphi approach Controlled feedback process Results analysis 1 2 3 4 10 11 Conceptu- Formulate Initiate Delphi Results Incorporate alize and themes and panel and Round 1 analysis into evaluation assemble data-gather- process Ideas or report panelists ing modalities generation 8 9 5 Round N Round N Round 1 Review and Synthesize Synthesize prioritize results results synthesis 7 6 Round 2 Round 2 Synthesize Review and results prioritize synthesis Source: Independent Evaluation Group. 42 Delphi Technique | Chapter 5 Analyze Results to Extract Key Findings The ideas generated and the data on prioritizing these opinions can now be summarized, represented visually, and analyzed statistically to draw key conclusions from the Delphi exercise (figure 5.1, 10). This can be typically car- ried out using spreadsheet and other data analysis software. Subject matter expertise and a basic understanding of statistics are important at this stage of the process to formulate robust conclusions. In the RE evaluation, a combination of quantitative and qualitative approaches was used to analyze results. Basic data analytics or statistics were used to calculate the average (mean) scores to rank priorities collectively for the panel and stan- dard deviations from the mean to ascertain the degree to which panelists reached consensus related to specific responses. The qualitative interpretation and contex- tualization beyond statistical measures relied heavily on sector expertise within the evaluation team to decipher the subject matter knowledge embedded in panelists’ responses. The Delphi results from IEG’s global panel of experts on RE were ana- lyzed in three steps: 1. Future opportunities and challenges for scaling up RE for achieving the clean energy transition. 2. Reform actions or solutions related to these opportunities and challenges. 3. The Bank Group’s readiness to support clients in implementing actions or solutions. Independent Evaluation Group | World Bank Group 43 Analyzing Opportunities and Challenges The first step was to analyze the major conclusions that can be drawn from the 14 opportunities and 16 challenges that were identified by the panel. The responses were analyzed based on a combination of scores on importance (indicated by average of the five gradations of Likert scale scores from very low to very high), and dispersion of scores across the panel (indicated by standard deviation from the average Likert score) as a proxy for the degree of consensus among the experts. The two parts of figure 5.2 summarize the results from the panelists’ feedback for the challenges that are expected to confront the future expansion of RE and for emerging opportuni- ties that developing countries can seize to scale up the sector. Often, opportunities identified by the panel highlighted the need to address corresponding challenges. In some instances, opportunities reflected emerging options for addressing some of the future challenges RE will face. In other instances, lower-priority items reinforced related higher-order challenges or opportunities. The following section accounts for these links in analyzing the feedback from the panelists. It presents conclusions that are an amalgamation of challenges and opportunities, broadly paraphrasing the views expressed by IEG’s global panel of experts on RE. The desire to meet national and international commitments to reduce greenhouse gases will open up greater opportunities for scaling up RE, although existing interests for maintaining the status quo can greatly hinder progress. IEG’s global panel of experts on RE noted the greater global awareness of climate change and the associated nationally determined commitments made by countries to reduce CO2 after the Paris Agreement as a high-ranking (third of 14) opportunity for scaling up RE. Greater consideration of carbon prices can also create a level playing field for RE in relation to other technologies. However, the top challenge that could stymie the expansion of RE, according to the panel, is resistance from vested interests. The opposition could be from coal and other fossil fuel–based interests that stand to be displaced by greater market penetration by RE. These industries could end up with stranded (existing) assets and suffer job losses as a result. Some power utilities may also resist a transition to increased generation from RE. This could occur when the scale-up occurs through independent power producers, which may be perceived as a loss of dispatch control within their operations, or if utilities are compelled to make additional costly investments in dispatchable capacity for integrating RE, especially variable renewable energy (VRE). 44 Delphi Technique | Chapter 5  rioritization of Challenges to and Opportunities for Scaling Up RE Figure 5.2. P a. Challenges for scaling up RE P 1 Existing interests that may hinder development of RE 3 (37.5%) 5 (62.5%) 4.38 /(0.48) P 2 Inadequate and unstable policy and regulatory infrastructure 2 (25.0%) 6 (75.0%) 4.25 /(0.43) I 3 Difficult to integrate large amounts of RE, especially those of variable/intermittent nature 1 (12.5%) 6 (75.0%) 1 (12.5%) 4.00 /(0.50) C 4 Capacity within government agencies to support and develop RE 1 (12.5%) 6 (75.0%) 1 (12.5%) 4.00 /(0.50) 5 1 (12.5%) 5 (62.5%) 2 (25.0%) F Regulatory and counterparty risk keep cost of capital high 3.88 /(0.60) P 6 Need bankable projects and PPAs 3 (37.5%) 4 (50.0%) 1 (12.5%) 3.63 /(1.1) P 7 Lack of “real” commitment to decarbonize power system 2 (25.0%) 2 (25.0%) 1 (12.5%) 3 (37.5%) 3.38 /(1.22) Independent Evaluation Group | World Bank Group 45 T 8 Need local industry to create strong interest in RE promotion 1 (12.5%) 3 (37.5%) 2 (25.0%) 2 (25.0%) 3.38 /(0.99) C 9 Improve systems and capacity to reduce project delays 1 (12.5%) 1 (12.5%) 6 (75.0%) 3.38 /(0.70) F 10 Difficulty mobilizing financing for RE in smaller countries/markets that may be less lucrative for 5 (62.5%) 1 (12.5%) 2 (25.0%) 3.38 /(0.86) developers C 11 Strengthen capacity for transparency and accountability to address corruption and governance 3 (37.5%) 5 (62.5%) 3.38 /(0.48) C 12 Need champion to drive RE development process 1 (12.5%) 2 (25.0%) 2 (25.0%) 3 (37.5%) 3.13 /(1.05) F 13 Affordability of off-grid equipment and aftersales care for poorer populations 3 (37.5%) 3 (37.5%) 1 (12.5%)1 (12.5%) 3.00 /(1.00) O 14 Large land requirements for developing RE 3 (37.5%) 2 (25.0%) 2 (25.0%) 1 (12.5%) 2.88 /(1.05) F 15 Rapid decline in technology costs could lead to slower uptake as utilities and developers take 2.88 /(0.78) 1 (12.5%) 6 (75.0%) 1 (12.5%) wait-and-see approach R 16 Vulnerability of RE systems to extreme weather and climactic events 1 (12.5%)1 (12.5%) 5 (62.5%) 1 (12.5%) 2.25 /(0.83) 0 2 4 6 8 0 1 2 3 4 5 Challenges Votes (no.) Mean Score / St.Dev. Barriers to RE Scale-Up Importance Score P Policies and I Integration of RE T Design and technical C Institutional and R Investment F Mobilizing O Others Null Very low = 1 Low = 2 Moderate = 3 High = 4 Very high = 5 regulations in power systems standards human capacity risks financing Source: Independent Evaluation Group. Note: EE = energy efficiency; RE = renewable energy; PPA = power purchase agreement; VRE = variable renewable energy.  rioritization of Challenges to and Opportunities for Scaling Up RE (cont.) Figure 5.2. P 46 Delphi Technique | Chapter 5 b. Opportunities for scaling up RE P 1 Achieve energy access goals 5 (62.5%) 2 (25.0%) 1 (12.5%) 4.50 /(0.71) F 2 Declining technology costs of RE and emergence of power storage 2 (25.0%) 5 (62.5%) 1 (12.5%) 4.29 /(0.45) P 3 Desire to meet national and international emission reduction commitment 3 (37.5%) 4 (50.0%) 1 (12.5%) 4.25 /(0.66) P 4 Address local environmental concerns 4 (50%) 2 (25.0%) 2 (25.0%) 4.25 /(0.83) F 5 Strong interest in developing market for RE by developers, development partners, and 1 (12.5%) 5 (62.5%) 1 (12.5%) 1 (12.5%) 3.75 /(0.83) climate financing sources I 6 Firming up (variable) RE capacity by combining complementary RE systems or addition of 5 (62.5%) 2 (25.0%) 1 (12.5%) 3.71 /(0.45) electricity storage P 7 Improve security of power supply with use of indigenous resources 2 (25.0%) 2 (25.0%) 3 (37.5%) 1 (12.5%) 3.63 /(0.99) I 8 Integrated planning of power systems 1 (12.5%) 4 (50.0%) 2 (25.0%) 1 (12.5%) 3.63 /(0.86) 9 3.50 /(1.12) I Increase distributed generation in which RE can play a key role 2 (25%) 2 (25%) 2 (25%) 2 (25%) F 10 Liberalization of power markets that increase private RE interests 1 (12.5%) 1 (12.5%) 4 (50.0%) 1 (12.5%) 1 (12.5%) 3.29 /(0.88) I 11 Utilization of technology and smart grids 1 (12.5%) 1 (12.5%) 5 (62.5%) 1 (12.5%) 3.25 /(0.83) R 12 Mitigation of upfront investments costs/risks of RE through public finance and pooling of 4 (50%) 3 (37.5%) 1 (12.5%) 3.25 /(0.97) resources, for more affordable and greater private sector participation O 13 Integration of RE and EE in power systems planning 3 (37.5%) 4 (50.0%) 1 (12.5%) 3.25 /(0.66) O 14 Income generating potential of RE 1 (12.5%) 5 (62.5%) 1 (12.5%) 1 (12.5%) 2.75 /(0.83) 0 2 4 6 8 0 1 2 3 4 5 Opportunities Votes (no.) Mean Score / St.Dev. Barriers to RE Scale-Up Importance Score P Policies and I Integration of RE T Design and technical C Institutional and R Investment F Mobilizing O Others Null Very low = 1 Low = 2 Moderate = 3 High = 4 Very high = 5 regulations in power systems standards human capacity risks financing Source: Independent Evaluation Group. Note: EE = energy efficiency; RE = renewable energy; PPA = power purchase agreement; VRE = variable renewable energy. Several specific barriers, consistent with the ToC for the RE evaluation, were predicted to prominently challenge the scaling up of RE as the sector quickly expands and markets continue to evolve. Therefore, to achieve the targets in the clean energy transition, countries and RE stakeholders will need to adequately address these barriers to mobilize investments in the sector. The panel stressed the following: ▪ Inadequate and unstable policy and regulatory framework. All panelists viewed the importance of this barrier as high or very high. They perceived a stable RE policy framework, in which the direction of the sector is predict- able, as a requirement for scaling up RE, especially for gaining the confi- dence of private investors. The enforcement of policies, including through an independent regulator, was seen as vital for attracting high-caliber de- velopers. The panel also recognized that fair pricing policies are needed to ensure the financial viability of utilities and to attract investors, including the availability of bankable RE projects and adequate offtake agreements. Also, lack of transparency in procurement and other related processes can result in governance issues that can stymie competition and dissuade quali- fied developers from participating in the RE sector. Inadequate policy and regulatory framework Shortcomings in the policy and regulatory environment established by gov- ernments can hinder public and private investments in RE, especially when they do not provide adequate opportunities or incentives for investors. Barrier as defined in the RE evaluation Difficulties integrating large amounts of RE, especially those of a variable or intermittent nature, without enhancing the flexibility of power systems. Nearly 90 percent of the panelists scored this barrier as having high or very high impor- tance, ranking it as one of the top three challenges for scaling up RE. This reflects the decreasing technology costs for solar PV and wind power, leading to a global ex- pansion of VRE. This trend is expected to accelerate as reflected in the sizable share of the two technologies incorporated in the clean energy transition. Integrating increasing shares of VRE requires greater flexibility in power systems to accommo- date intermittent availability of these RE resources. Although integrated planning of power systems garnered less significance on its own as a barrier, the panel acknowl- edged that a holistic approach is an important part of enhancing system flexibility Independent Evaluation Group | World Bank Group 47 to avoid bottlenecks and speed up RE deployment. Improved planning specially to integrate VRE should include building out transmission networks to supply demand centers, and where feasible, interconnection of power grids (within country or with other countries) to create future regional networks. There was extensive input by the panel on energy storage and its prospects (including from responses to additional question on the subject), which, together with other tools for integration, will have a significant impact on achieving the clean energy transition. The panel emphasized that the cost of battery storage at utility scale is already declining but acknowledged that this is presently uneconomical for most systems as a solution for integrating VRE (and that these costs should be included as part of the investment costs of VRE technologies). However, there was considerable consensus that economical battery storage is a clear and emerging trend, with some indicating a near- to medium-term time horizon when it will be cost-effective at large scale. There was also a call for developing hydropower (including pumped storage), which is presently an econom- ical solution for sustainably enhancing system flexibility. Another emerging trend is the expansion of distributed generation (DG) from RE (including ambitious targets established in countries such as China and India), which can help integrated RE by injecting electricity directly into the distribution system and bypassing potential short-term transmission bottlenecks.1 The panel was divided and unable to reach consensus on the subject, since half viewed DG from RE as important or highly im- portant and the other half thought its significance was moderate or low. Inability to integrate RE into power systems Inadequate planning, transmission bottlenecks and insufficient capacity, lim- ited scope for power trading and pooling, inability to store electricity—can all result in inflexibility of power systems to smoothly and efficiently integrate RE, especially as the share of VRE such as wind power and solar PV increases. Barrier as defined in the RE evaluation ▪ Inadequate capacity within governments to implement their RE devel- opment plans, including facilitating private investment mobilization. Most panelists viewed this as a shortcoming of high significance, as it leads to implementation delays and higher costs of projects and programs. Lim- ited experience and capacity within energy ministries and other related in- stitutions on RE-related issues was seen to disadvantage governments when negotiating with investors. The capacity constraints that extend the time 48 Delphi Technique | Chapter 5 necessary to design, build, and commission investments lead to higher costs that can compromise their economic viability. This is especially the case when mobilizing investments in advanced RE and related technologies, ac- cording to the panel. Weak institutional and human capacity In many developing countries, various institutions involved in the develop- ment of RE do not have sufficient capabilities to undertake new investments or operate ongoing projects. Barrier as defined in the RE evaluation ▪ Regulatory and counterparty risks can increase the cost of develop- ing RE. This is especially the case given the capital intensity (for example, high up-front costs) for many RE technologies. As noted by the panel, this can limit or make expensive available financing for RE projects; driving up costs of investments makes RE-based generation options less competitive. The panel indicated that countries with similar natural resource potentials have widely different levels of RE investments and costs as a result. Some panelists saw an opportunity to mitigate some of these risks through pub- lic financing (initial investments and then divesting) and pooling resources. Such initiatives were viewed as being especially useful for mobilizing private investments in RE. Existence of investment risks Even with improved policies and enhanced institutional capabilities, there may be residual risks that projects face, either on a transitional basis while re- forms are being implemented or permanent risks that are outside the control of developers, and which may discourage investments (such as commercial or off-taker risks, political risks, RE resource risks). As defined in the RE evaluation ▪ Several other challenges to overcome were identified, which may have significance in specific environments for developing RE. The need to de- Independent Evaluation Group | World Bank Group 49 velop local industries in RE was rated by some panelists as being high in im- portance. Although such efforts to develop local capacity propelled a country such as China to become a global leader in RE development, it may be a lower priority for smaller countries with less resource potential, smaller markets, and more limited human capacity. The challenge of mobilizing financing in smaller countries or markets was also raised, since these may be less lucra- tive for developers to invest in (ranked 10th out of 16 challenges). Further down the list of priorities was the potentially large land requirement for de- veloping utility-scale RE, which can be a unique but important challenge in geographically smaller countries or densely populated areas. There was also acknowledgment that mobilizing supply-side solutions to climate change through the development of RE is only a partial pathway to achieving the greater objective of mitigating climate change, and that efforts should be combined with energy efficiency initiatives for better demand-side manage- ment of electricity use. Although the overarching premise for the clean energy transition is to supply electricity sustainably through RE and avoid global environmental pollution, it is also an opportunity to achieve several other local development priorities prevalent in select circumstances. IEG’s global panel of experts on RE highlighted several energy and environmental benefits that are also part of the RE evaluation ToC that may accrue from scaling up RE (box 3.1): ▪ Achieve energy access goals. The highest-rated opportunity from expand- ing RE among the 14 identified was the ability to achieve energy access goals, echoing one of the key objectives of SDG 7. It was scored by over 60 percent of the panel as having very high importance and seen as an opportunity to connect the nearly 1 billion people presently living without electricity. The decreasing technology costs of RE were seen as a way of more affordably ex- panding electrification to people living in poverty. Off-grid RE was also seen as a solution to overcoming various institutional and other constraints fac- ing conventional grid expansion, through household-level connections and from the rollout of self-standing minigrids as a regional electrification path- way in areas with little or no access to electricity in developing countries. ▪ Address local environmental concerns. The panel noted that fossil-based generation “presently contributes to three million premature deaths, which may increase to between six and nine million fatalities by 2060,” if current trajectories continue. As a result, addressing local pollution was ranked 50 Delphi Technique | Chapter 5 fourth among the 14 opportunities identified for expanding RE, with half the panelists placing very high importance on the matter. Recent challenges fac- ing large cities such as Beijing, China, and New Delhi, India, underscore the importance of addressing local pollution from power generation and point to RE as a key solution. Ultimately, the direct benefits of avoiding local pol- lution may be as much a factor for countries developing their RE resources as global climate considerations. ▪ Improve security of power supply with the use of indigenous resources. The use of indigenous RE resources can enhance energy security by limiting exposure to the volatility of international markets, since it reduces reliance on fossil-based alternatives that are subject to price fluctuations typical of tradeable commodities. However, this ranked in the middle of the list of opportunities identified by the panel (seventh of 14). Although 25 percent of the panelists found energy security from developing RE to be of very high importance, half of the group viewed the opportunity as moderate or low in significance. Analyzing Actions or Solutions The second step was to analyze the actions or solutions that IEG’s global panel of experts on RE mapped to different challenges and opportunities to scale up RE. Therefore, the primary focus of this section is the actions or solutions that cor- respond to the priority challenges and opportunities highlighted in the previous section. The conclusions and sector narrative are drawn from the priority actions or solutions identified by the panelists based on the corresponding numeric Likert scale scores: (i) solid rating of important or very important (a Likert scale average score of 4.0 or higher marked as green in tables 5.1 to 5.5); (ii) a rating of important at the lower end (a Likert scale average between 3.5 and 3.99 marked as yellow in tables 5.1 to 5.5); and (iii) a rating of moderate importance or lower (a Likert scale average below 3.5 marked as blue in tables 5.1 to 5.5). As in the previous section, the stan- dard deviations from average scores were used to gauge the consensus within the panel for the selected actions or solutions. Panel responses were interpreted into a narrative based on subject matter expertise and sector context. Phasing out existing fossil-based power plants and avoiding construction of new ones will need to consider and address the concerns of stakeholders affected by the transition, including power utilities and investors with stranded assets. IEG’s global panel of experts in RE affirmed the need to strategically displace fossil fuels Independent Evaluation Group | World Bank Group 51 over time. The highest-rated actions included phasing out the most polluting power plants, especially in hotspots where they are a major contributor to poor local air quality. To a lesser degree, the panel also called for the cessation of constructing new fossil-based power plants and the removal of fossil fuel subsidies that exist in some 40 countries worldwide.2 The resulting unmet demand can then be supplied from RE sources. However, the panel clearly recognized that such actions will generate resis- tance from fossil-based interests, which stand to lose business and suffer potential losses as a result of stranded assets. As noted previously, these existing interests were identified by the panel as the most important challenge or barrier that could hinder the expansion of RE. Therefore, a successful transition would need to address these concerns, including alternative opportunities for investments and employment in RE and other areas. Furthermore, the panel placed a high level of importance and reached considerable consensus on the need for a long-term view and targets agreed with power utilities, since their operations can also be affected by greater use of VRE resources, increases in distributed generation, and, in some cases, the growing sup- ply from independent power producers. Ultimately, it was proposed that clear pol- icies and legislation will be required to set targets for expanding RE and displacing fossil-based generation. There was a recognition of opportunities to learn from the experiences in other countries in resolving these issues and transitioning smoothly. 52 Delphi Technique | Chapter 5 Table 5.1 provides a list of key proposed actions or solutions as prioritized by the panel for addressing existing interests that may constrain the expansion of RE.  ctions or Solutions to Address Existing Interests That May Table 5.1. A Hinder Development of Renewable Energy Priority Ranking Actions or Solutions Proposed by Delphi Panel mean (std. dev.) Phase out fossil fuel power plants over time, starting with the 4.38 (0.70) most polluting ones Closure of strategically located coal-based power stationsa 4.25 (0.66) Legislate clear long-term RE targets and agree with utility how 4.13 (0.33) to achieve these targets Existing interests may hinder development of RE Create alternative employment for areas affected by removing 4.00 (0.50) fossil electricity production Identify hotspots where power generation is a major contributor to local air quality and provide strong incentives for switch to 4.00 (0.71) REa Legislate market liberalization to allow IPPs 4.00 (1.00) Stop building fossil fuel–based power plants 3.88 (1.17) Bring off-grid options into long-term electrification plan 3.75 (1.09) Learn from successes of other countries 3.63 (0.48) Remove subsidies on kerosene 3.63 (1.22) Increase awareness on job creation potential of RE 3.38 (0.70) Experiment with off-grid electrification concessions 3.25 (1.09) Provide incentives to convert conventional power plants into RE, 3.25 (1.20) storage, or both Remove VAT and import duty on solar 3.00 (1.12) Independent Evaluation Group | World Bank Group 53 Source: Independent Evaluation Group. Note: IPP = independent power producer; RE = renewable energy; VAT = value-added tax. a. Reflects a related action or solution identified for seizing an opportunity. The significance of rectifying an inadequate policy and regulatory framework to improve the investment climate for mobilizing investments in RE was un- derscored by the strong consensus within the panel that all corresponding actions or solutions identified by them are important. This includes modernizing the policy framework for promoting RE, with new or revised legislation and poli- cies regarding national strategies on energy that align with the goals in SDG 7. The panel also called for avoiding or removing statutory or regulatory policies that can be barriers to the deployment of RE, and for fair and predictable implementation of policy through greater independence of regulators. The likely need to streamline permitting and development processes was recognized, especially for accelerat- ing RE development through the private sector. Reforms were proposed for tender processes, including consideration of competitive bulk procurement for RE capacity, and for factoring life cycle costs and the potential for avoiding greenhouse gases in investment decision-making. Such approaches need to be supported by strong an- ticorruption laws and robust and consistent enforcement so that market confidence is enhanced to attract investments. The selected developers should also be subject to adequate environmental review and land acquisition. Depending on the extent of these reforms and the existing market structure, electricity market restructuring 54 Delphi Technique | Chapter 5 may be required. Table 5.2 provides a list of proposed key actions or solutions as prioritized by the panel for overcoming an inadequate and unstable policy and regu- latory framework that affects the investment climate for RE.  ctions or Solutions to Address Inadequate Policies Table 5.2. A and Regulations Priority Ranking Actions or Solutions Proposed by Delphi Panel mean (std. dev.) Facilitate easier finance for RE by reducing policy-based risks 4.50 (0.50) to such investmentsb Create enabling environment for investments in RE—modern- 4.38 (0.48) ized legislation, energy policy, independent regulatorb Inadequate and unstable policy and regulatory infrastructure Develop stable policy frameworks that support RE develop- 4.38 (0.70) ment Amend procurement or tendering practices to allow for con- 4.13 (0.33) sideration of life cycle costs when making decisionsb Do not erect statutory or regulatory barriers to the deploy- 4.13 (0.60) ment of hybrid technologiesb Establish independent electricity regulator 4.13 (0.78) Undertake electricity sector restructuring 4.13 (0.78) Encourage rule of law through strong anticorruption laws and 4.13 (0.78) robust, consistent enforcementa Inclusion of GHG price (as a cost) in cost-benefit analysis for 4.13 (0.78) new generation capacityb Develop national RE strategies and long-term plans 4.00 (0.71) Align national energy policy to achieve targets in SDG 7 to 4.00 (0.71) receive official development assistanceb Repeat bulk procurement of RE electricity on a competitive 4.00 (0.71) basis to accelerate price reductions with increasing volumesb Streamline permitting and development processes, including 4.00 (0.87) adequate environment review and land acquisitiona Source: Independent Evaluation Group. Independent Evaluation Group | World Bank Group 55 Note: GHG = greenhouse gas; RE = renewable energy; SDG = Sustainable Development Goal. a. Reflects an action or solution related to addressing a different challenge. b. Reflects an action or solution related to seizing an opportunity. Integrated planning and strengthening grid infrastructure for transporting RE to demand centers was seen as vital for expanding RE, followed by measures for balancing power and especially for assimilating VRE. The panel placed the highest importance on prioritizing grid infrastructure investments to improve the transmis- sion and distribution networks to accommodate RE. There was a similar call to mod- ernize grid operations and governance and strengthen the capacity of grid operators. Given the anticipated expansion of VRE, the panel placed high importance on solu- tions to support the integration of solar PV and wind power. Key proposed solutions included the promotion of pumped-storage hydropower and battery storage. Inte- gration of grids within regions or countries was also viewed as an important solution for scaling up RE, since it can present more options for balancing power. Although the adoption of grid integration protocols, especially for RE DG, can ease some of the transmission bottlenecks and support load balancing in the short term, there was a divergence of views within the panel, as previously noted, on the significance of DG as a major long-term solution for scaling up RE. As the penetration of VRE increases in power systems, it may be necessary to reconfigure electricity markets to provide for price discovery of the balancing energy market. Although a key to integrating 56 Delphi Technique | Chapter 5 VRE is creating greater generation flexibility, the panel also recognized the comple- mentary role that demand-side energy efficiency measures can play. Table 5.3 pro- vides a list of proposed key actions or solutions as prioritized by the panel for better integrating RE into power systems. Table 5.3.  Actions or Solutions to Address Integration of Renewable En- ergy into Power Systems Priority Ranking Actions or Solutions Proposed by Delphi Panel mean (std. dev.) Independent Evaluation Group | World Bank Group 57 Improve transmission and distribution network 4.38 (0.70) Difficult to integrate large amounts of RE, especially those of variable/intermittent nature, Prioritize grid infrastructure investmentsa 4.38 (0.70) Integrate grid systems 4.25 (0.66) Ensure adequate grid capacity for RE; modernize operations without developing flexibility of power systems, which can be costly 4.25 (0.66) and governance of grida Balance load to optimize local RE and grid electricity 4.25 (0.66) capacitya Develop and adopt grid integration protocolsa 4.13 (0.60) Make it easier to integrate VRE sources into grid and support 4.13 (0.60) solutions that involve storagea Strengthen capacity of grid operators 4.00 (0.50) Promote pumped storage and battery storage 3.75 (0.97) Undertake long-range transmission planning 3.63 (0.86) Unlock flexibility in generation and the demand side by 3.63 (0.70) creating appropriate market incentives Reconfigure electricity markets to provide for price discovery 3.50 (0.87) of balancing power Monetize future loss reduction to pay for RE DG installations 3.50 (1.09) now Develop smart grids 3.38 (0.70) Develop risk guarantee mechanism to compensate fos- sil-based plants for capacity cost when use is too low to 3.13 (0.60) recover these costs because of preference for RE electricity Progressively move to reflect full RE costs, especially as its 3.13 (0.60) share in energy generation rises Source: Independent Evaluation Group. Note: DG = distributed generation; RE = renewable energy; VRE = variable renewable energy. a. Reflects an action or solution related to seizing an opportunity. Strengthening the capacity within government agencies would improve the implementation of public projects and also make them better partners in facil- itating private investments in RE. IEG’s expert panel on RE placed very high im- portance on the capacity to plan, implement, and monitor and evaluate RE invest- ments—spanning the spectrum of a typical project cycle. Given the fast-evolving RE markets, similar significance was placed on training businesspeople, managers, and engineers, so they are familiar with the latest developments. Although such voids 58 Delphi Technique | Chapter 5 can be filled to an extent by mobilizing domestic and foreign private actors, the ca- pacity within governments to negotiate such agreements is often lacking. The panel proposed seeking development partner assistance as a solution. They stressed that many countries have national plans but often lack the human capacity to implement them. The actions or solutions proposed by the panel for strengthening capacity were, however, limited and general, as shown in table 5.4—a deviation from the extensive suggestions made for other high-ranking challenges.  ctions or Solutions to Develop Capacity for Supporting Table 5.4. A Renewable Energy Priority Ranking Actions or Solutions Proposed by Delphi Panel mean (std. dev.) Capacity within government Capacity within government agencies to support and agencies to support and 4.50 (0.71) develop RE develop RE Training on RE for businesspeople, managers, and 4.00 (0.87) engineers Seek support from development partners to assist with 3.75 (0.97) negotiations to be undertaken Source: Independent Evaluation Group. Note: RE = renewable energy. Approaches to reducing counterparty risks can facilitate financing for RE by making projects more bankable for investors. There was significant consensus regarding the high importance placed by the panel on actions to de-risk or minimize counterparty risks faced by RE investors, including using grants or concessional finance, and consideration of life cycle costs in decision-making. The highest score given by the panel for risk mitigation actions was for developing a clear and robust investment framework, although there was considerable divergence of opinions within the panel regarding the solution. A specific proposal was that smaller coun- tries could pool similar projects within a region to generate greater investment interest through economies of scale. A less significant solution was to have a risk guarantee mechanism to compensate fossil fuel plants for underuse of capacity resulting from preference for RE, so that they will not present a roadblock for scaling up RE as an alternative. Table 5.5 provides a list of proposed key actions or solutions as prioritized by the panel for better mitigating investment risks. Independent Evaluation Group | World Bank Group 59 Table 5.5.  Actions or Solutions to Mitigate Regulatory and Counterparty Risks Priority Ranking Actions or Solutions Proposed by Delphi Panel mean (std. dev.) Create a clear and robust investment framework 4.25 (0.83) Regulatory and counterparty risks Amend procurement or tendering practices to allow for 4.13 (0.33) keep cost of capital high consideration of life cycle costs when making decisionsb Minimize counterparty risk 4.00 (0.50) Explore opportunities for de-risking investments through 4.00 (0.50) the use of grant or concessional financinga Organize small countries to pool similar projects in same 3.75 (0.97) regiona Develop risk guarantee mechanism to compensate fos- sil-based plants for capacity cost when use is too low to 3.13 (0.60) recover costs because of preference for RE electricitya Source: Independent Evaluation Group. Note: RE = renewable energy. a. Reflects an action or solution related to addressing a different challenge. b. Reflects an action or solution related to seizing an opportunity. Analyzing the Bank Group: Readiness to Support Clients The third step was to analyze the Bank Group’s role in terms of its readiness to support the clean energy transition. IEG’s global panel of experts on RE shared their perception of the Bank Group’s position (to influence the outcome) and its capacity (expertise and experience) to support clients, for each action or solution for address- ing challenges and seizing opportunities. Therefore, the primary focus of this section is the institution’s readiness to support clients in implementing various reforms (ac- tions or solutions) identified by the panel. It is important to interpret this informa- tion with greater caution, and ensure it is ultimately corroborated with other meth- odological sources. Although the panel has extensive expertise in RE development globally and each member is familiar with the work of the Bank Group, their respec- tive understanding of the institution and its functions and their own experience with 60 Delphi Technique | Chapter 5 it may vary considerably. Therefore, the views of the panel on the institution’s role may be less generalizable. The Bank Group’s position and capacity to support clients to implement specific ac- tions or solutions is scored on a different scale. The panel was presented with a four- point Likert scale (instead of a five-point scale). The scale for the Bank Group’s posi- tion to influence outcomes was 4—extremely well, 3—very well, 2—moderately well, and 1—poorly; the scale for institution’s capacity to support clients undertake actions or solutions was 4—very high, 3—high, 2—moderate, and 1—low. In both cases, each panelist had the option to not opine (by responding “Do not know” or “No opinion”), although the response rate with a perspective was 84 percent (89 percent for position and 79 percent for capacity). As for previous sections, the results were analyzed based on priorities identified through panel scores (mean, standard deviation). Overall, the Bank Group was moderately well to very well positioned and had moderate to high capacity to help clients navigate the clean energy transition. The average score for the Bank Group’s position to influence the 16 challenges was 2.61 and the 14 opportunities was 2.56, which is at the lower end of “very well”, barely surpassing “moderately well.” The institution’s capacity to support clients to undertake actions or solutions was only slightly better, for challenges at 2.66 and for opportunities at 2.73, at the threshold between “moderate” and “high.” The panel did not view the Bank Group to be extremely well positioned or possess very high capacity to influence the clean energy transition. Mitigating regulatory and counterparty risks was the only priority challenge where the Bank Group was per- ceived to be clearly very well placed with high capacity to support clients. The panel viewed the institution as very well positioned (mean 3.13, std. dev. 0.72) and having a high level of capacity (mean 3.00, std. dev. 0.88) to help clients develop a clear and robust investment framework and help minimize regulatory and counterparty risks, which likely reflects the multiple instruments available within the Bank Group for risk mitigation (guarantees, political risk insurance, concessional financing, and ability to mobilize grants). The score was also likely based on the institution’s role helping improve the regulatory and policy frameworks in client countries (another high-ranking challenge, noted previously). Risk mitigation also garnered one of the highest levels of consensus for a challenge within the panel. The Bank Group’s readiness to support clients in addressing other priority chal- lenges was perceived with greater ambiguity within the panel. Most were scored Independent Evaluation Group | World Bank Group 61 by the panel as being moderately well and very well placed and having moderate to high capacity to support clients. There was also a diversity of views that did not rep- resent a definitive consensus, which may reflect the different experiences of panelists with the institution. The Bank Group’s readiness to help clients overcoming other priority barriers include the following: ▪ Inadequate policy and regulatory infrastructure. In the case of this barrier, the Bank Group’s capacity fell just below the threshold for a clearly high rat- ing. Within the specific barrier, the institution was viewed as clearly having high capacity to help clients develop RE strategies and long-term plans and to support power sector restructuring where there are clients committed to such reforms, although its ability to help establish and support independent regulators was rated modest. The Bank Group was seen to have between moderate and high capacity for helping develop a stable policy framework for RE. ▪ Integration of RE to power systems. Although the overall rating for capac- ity to help clients overcome this barrier was mostly modest, integration of grid systems and promoting pumped-storage hydropower and battery stor- age were specific actions where the Bank Group was viewed to have high capacity (and was very well positioned). Integrated systems and long-range transmission planning capacity was also perceived to be high, although the institution was seen to be less influential with clients in this regard. The Endnotes 1 However, scaling up of solar photovoltaics, in particular, will increase the share of variable renewable energy, which can, in turn, generate the need for more flexibility in power systems through additional dispatchable capacity of energy storage options. 2 The list includes countries that provide direct subsidies for coal, oil, natural gas, and elec- tricity. There are additional countries that may provide tax breaks and other incentives for fos- sil fuels, which can also undermine the competitiveness of renewable energy. Bank Group had more moderate capacity for many other related actions or solutions, including unlocking flexibility in generation and reconfiguring power markets for price discovery for balancing power; developing smart 62 Delphi Technique | Chapter 5 6 FORMULATING CONCLUSIONS Incorporate results into the evaluation or report Results analysis stage This chapter continues with the third and final stage, focusing on the major conclu- sions drawn from the Delphi exercise and how they were integrated into the broader RE evaluation. It is based on the results analyzed in chapter 5 and continues to apply a subject matter perspective, but it triangulates the conclusions from the Del- phi process with findings from other methods to formulate conclusions. The Delphi process step covered in this chapter is marked 11 within the blue-highlighted area in figure 6.1. Figure 6.1. The Delphi Process—Results Analysis Stage Design of Delphi approach Controlled feedback process Results analysis 1 2 3 4 10 11 Conceptu- Formulate Initiate Delphi Results Incorporate alize and themes and panel and Round 1 analysis into evaluation assemble data-gather- process Ideas or report panelists ing modalities generation 8 9 5 Round N Round N Round 1 Review and Synthesize Synthesize prioritize results results synthesis 7 6 Round 2 Round 2 Synthesize Review and results prioritize synthesis Source: Independent Evaluation Group. Incorporate Results into the Evaluation or Report Once the results from the Delphi panel are analyzed, the key findings can be triangulated with the results from other methods to formulate the overall conclusions on the subject matter being evaluated (figure 6.1, step 11). These conclusions can form the basis of recommendations made to effect change. 66 Delphi Technique | Chapter 6 In the RE evaluation, the Delphi exercise with IEG’s global expert panel on RE provided useful insights and helped established the key emerging challenges and opportunities facing the scale-up of RE as envisaged in the clean energy transition. The RE evaluation was able to triangulate findings with the results drawn from other previous methods and come to several key conclusions (box 2.1). These conclusions reflect the emerging direction of RE markets and also where the Bank Group may be best placed to support clients in navigating the sector. This section summarizes the major conclusions that were reached from the multiple methods used in the RE evaluation, to which the Delphi results contributed. ▪ Addressing existing interests, especially from fossil fuel generators and power utilities, will create headwinds for scaling up RE, and require a com� prehensive approach to addressing their concerns so that investments in RE can advance. Although some RE-specific reforms are necessary to im- prove the investment climate for developing the sector, a more comprehen- sive and sustained approach is needed to address challenges that may be posed by competing and complementary sectors. Fossil fuel–powered gen- erators, along with electricity utilities, have at times displayed resistance to expanding RE, because of potential business losses, technical challenges to integration, or concerns about higher power system costs, especially when assimilating VRE. There may also be pushback from local communities and other concerned stakeholders if RE technologies such as hydropower ad- versely affect the environment and those who may be displaced are inad- equately compensated.1 These vested interests can manifest in some of the barriers identified in the ToC in box 3.1. Case studies and portfolio reviews of the Bank Group RE engagements found them to be more effective in achiev- ing development outcomes when the institution takes a more comprehen- sive and sustained approach to addressing barriers to sector expansion. ▪ Although the overarching global impetus for scaling up RE is to mitigate adverse climate impacts, local considerations such as increasing electric� ity access and avoiding local air pollution may be a greater motivator for developing the sector. However, the panel did not perceive the Bank Group to be well positioned to help clients capitalize on these aspects. Increasing electricity access through RE to nearly a billion people around the world was the top-ranked opportunity. However, the Bank Group’s capacity to support such efforts was perceived to be modest by the Delphi panel, although a re- view of the institution’s RE portfolio found increasing access an objective in Independent Evaluation Group | World Bank Group 67 22 percent of the 465 projects, with nearly 80 percent of these investments achieving their access goals. The Delphi panel had a similar perception about another high-priority opportunity: that the Bank Group was not well posi- tioned to help clients address local pollution. This conclusion is consistent with the RE portfolio review, which found few investments that aimed to avoid local air pollution. A recent IEG evaluation on pollution found that lo- cal environmental issues were underresourced within the institution, further corroborating the Delphi panel’s view (World Bank 2017). The Bank Group may want to strengthen its support to local energy considerations as a way of scaling up RE, since a review of country investment strategies as a part of the RE evaluation found them to be the primary country-level motivator for developing the sector, over global considerations. ▪ The Bank Group has a comparative advantage in disseminating global knowledge, which can significantly contribute to clients’ efforts to suc� cessfully scale up RE. The Delphi panel saw the Bank Group as very well po- sitioned to share lessons from its extensive global experience, to inform cli- ents about and help them successfully navigate the emerging RE landscape. This conclusion was further validated by a review of the RE portfolio, which found that the Bank Group shared global knowledge and international expe- riences with 78 countries in their design and approach to RE development. In addition, the institution operates several global platforms that disseminate knowledge and development experiences related to RE, such as the Energy Sector Management Assistance Program, the Carbon Finance Group, and the Asia Sustainable and Alternative Energy Program.2 The conclusions drawn regarding the major barriers to scaling up RE, as identi- fied by the Delphi panel and corroborated with findings from other methodologi- cal sources, are detailed below: ▪ A key pathway to scaling up RE, particularly through private participation, identified three major barriers to overcome: inadequate policies and regu� lations, integration of RE into power systems, and addressing any residual risks through specific mitigation measures. As previously noted, the Del- phi panel identified these three barriers as the top challenges to address in scaling up RE in developing countries (along with capacity building). A QCA carried out for the RE evaluation also independently confirmed overcoming the three-barrier combination as a key pathway to mobilizing investments in RE for achieving energy supply and global environmental objectives. A sub- sequent separate analysis by the International Energy Agency identified the same set of barriers as “the three main challenges” that need to be addressed 68 Delphi Technique | Chapter 6 to “accelerate significantly” RE growth for meeting the long-term goals es- tablished under the SDGs and the Paris Agreement (IEA 2019). Renewable electricity growth still needs to accelerate significantly to meet long-term sustainable energy goals This growth is possible if governments address the three main challenges to faster deployment: policy and regulatory uncertainty; high investment risks in many developing economies; and system integration of wind and solar PV in some countries. IEA 2019 ▪ An adequate policy and regulatory environment for RE is an essential factor for mobilizing investments in the sector—an area where the Bank Group has considerable experience, although there is a need to continu� ally adapt to evolving market conditions. The Delphi panel ranked an un- stable policy and regulatory framework as the second-most significant chal- lenge facing the expansion of RE. The QCA identified addressing the same barrier as a requisite (near-necessary) one to overcome in scaling up RE. The Delphi panel also assessed that the Bank Group has high or near high capac- ity to help client countries with several of the key reforms needed to improve policy frameworks. This finding was underscored by a portfolio review, where a third of the Bank Group’s 217 public sector interventions included support to policy and regulatory reforms.3 ▪ The integration of RE, in particular VRE technologies such as solar PV and wind power, is emerging as a major barrier to achieving the clean ener� gy transition, yet the Bank Group has limited project-level experience in this area. This is among the top three challenges for scaling up RE iden- tified by the Delphi panel. The QCA found integration to be the other es- sential (near-necessary) barrier (along with policies and regulations) to be addressed to mobilize investments in RE. Despite its significance, the Delphi panel found the Bank Group to have only moderate capacity to help with most related actions or solutions. This assessment is reinforced by a portfolio review, which found that less than 7 percent of the institution’s RE projects included activities to address this barrier.4 The limited experience with inte- grating RE also reflects the circumstance that many developing countries are only beginning to reach high shares of VRE, which is sure to exacerbate the Independent Evaluation Group | World Bank Group 69 integration challenge in the future. Among the solutions suggested by the Delphi panel in this regard, two emerging ones stand out: » Energy storage is becoming an increasingly important approach to balancing power for integrating VRE. The need for storage un- derscores the immediate importance of hydropower, especially pumped storage, and there was consensus that battery storage will progressively become an economical solution. Although bat- tery storage technologies continue to advance, with steadily declin- ing costs, there was acknowledgment that hydropower, especially pumped-storage technology, is presently economically viable and should be considered when developed to industry and international standards. The Delphi panel also identified a clear trend and predicted that battery storage technologies will continue to innovate and scale up in the near to medium term as they become more economically viable at utility scale. An in-depth review of hydropower undertaken in the RE evaluation found the Bank Group to have extensive experi- ence helping clients deploy the technology, since it makes up nearly 40 percent of its RE portfolio. Although 90 percent of evaluated hy- dropower projects complied with environment and social policies, the institution could more consistently implement best practices in these policies and development of local communities that extend beyond minimal safeguard requirements. Nevertheless, the review found sub- stantial direct economic benefits from its hydropower portfolio as a result of avoided costs from alternative generation and sizable cli- mate benefits from avoided CO2. Although the Bank Group is more recently supporting clients in deploying utility-scale battery storage, most of its experience during the evaluation period was confined to household-level solutions in projects aiming to increase electricity access. It validates the more moderate ratings by the Delphi panel for the institution’s capabilities in this regard. » DG from RE resources is a rapidly emerging approach to scaling up RE while addressing transmission bottlenecks, but there was a divergence of views within the Delphi panel on its significance. Panelists did not reach consensus regarding this emerging trend to have more dispersed generation nodes that can help scale up RE by directly supplying distribution grids, bypassing potential transmis- sion constraints. They were divided; half viewed RE DG as important or highly important and the other half thought its significance was 70 Delphi Technique | Chapter 6 moderate or low. This could be due to rapid expansion of VRE from RE DG (mostly from solar PV) progressively exacerbating the integra- tion challenge as the share of intermittent electricity increases. The Delphi panel’s view contrasted with the perspectives of Bank Group energy sector staff surveyed for the RE evaluation, where 80 percent placed high importance on RE DG. It would be prudent to carefully consider the institution’s presently expanding activities in RE DG and its implications for power systems so that potential unintended ad- verse consequences can be avoided. ▪ Mitigating RE and country-specific risks can be vital, especially for mo- bilizing private investments, and the Bank Group has a suite of options to support clients in this regard. There was considerable consensus within the Delphi panel regarding the high importance of minimizing counterparty risks as a way of mobilizing investments in RE. The significance of this barrier also highlights the need to improve the policy and regulatory environment, which can be a source of uncertainty for investors, and the need to use other approaches to minimize any residual risks. Mitigating RE risks was rated by the Delphi panel as the barrier the Bank Group is best placed to address. The panel’s high level of confidence may reflect the suite of instruments that the institution can deploy to help clients mitigate investment risks, including guarantees, political risk insurance, concessional financing, and the ability to mobilize grant funds. It likely indicates why the Bank Group was perceived to be very well positioned to mobilize financing in small countries and mar- kets, where risks can be high. ▪ Inadequate human capacity for designing, developing, and operating RE is an important barrier to overcome, but it can often be augmented through external support. Nearly 90 percent of the Delphi panel placed high importance on developing institutional capacity for developing RE, particu- larly within government agencies. The QCA also found a causal link between adequate capacity and the realization of RE investments, although it was not found to be a near-necessary condition for expanding RE. This poten- tial contradiction between the Delphi panel and the QCA results may exist because, although developing long-term internal capacity in RE is import- ant, shortfalls can also be augmented with external support on a transitional basis. The Bank Group plays a key role in this regard, since nearly half of the investments in its RE portfolio included activities to strengthen client capabilities.5 The Delphi panel opined that the institution has high capacity for training various officials on RE, although it was perceived to have less capability in helping clients negotiate with the private sector. Independent Evaluation Group | World Bank Group 71 Endnotes 1 The World Commission on Dams (2000) recommended that hydropower be implemented with a broader development objective as a goal rather than narrowly focusing only on its en- ergy benefits. The World Bank Group has adopted most of these recommendations, and the RE evaluation found that when projects adhered to these principles, they had more successful development outcomes. 2 The Asia Sustainable and Alternative Energy Program was a stand-alone multidonor-sup- ported program that operated within the World Bank during the evaluation period, although it has since merged with and now operates as a part of the Energy Sector Management Assistance Program. 3 Nearly all investment interventions to support policy and regulatory reforms within the World Bank Group portfolio were understandably undertaken through public sector projects supported by the International Development Association and the International Bank for Re- construction and Development. 4 The World Bank Group has a long history and sizable portfolio supporting transmission and distribution of electricity, which is a key part of integrating generation into power systems. Many of these investments are not classified as renewable energy projects, although some could directly or indirectly facilitate the integration of renewable energy. 5 The public sector projects aimed to improve technical design and implementation capaci- ty (including strengthening governance and fiduciary capabilities), and private sector invest- ments primarily helped comply with environmental and social requirements. 72 Delphi Technique | Chapter 6 FINAL REMARKS ON THE DELPHI TECHNIQUE The Delphi technique outlined in this paper provides a valuable means of systematically achieving a convergence of opinion from a panel of reputable subject-area experts. The method provides a robust, iterative, and anonymous group communication approach designed to shed light on alternatives, correlate expert insights on a specific subject or challenge, generate background informa- tion for decision-making, and reveal hidden consensus relevant to the forecast- ing of future events. In the context of the multimethod evaluation of renewable energy sources explored here, the technique provided an opportunity to incorporate expert knowledge and experiences from global thought leaders and practitioners to en- hance evaluative findings. Specifically, the panel perspectives helped (i) estab- lish future scenarios for the rapidly evolving industry, which helped (ii) identify critical barriers that need to be overcome to scale up RE, and (iii) formulate a set of priority reforms countries can undertake to overcome these challenges, against which (iv) the Bank Group’s readiness to help clients was assessed. Though active steps were taken to offset potential response biases from partic- ipants (for example, confirmation, satisficing, bandwagon effects), more could have been done to ensure balanced representation in the panel of experts, par- ticularly with respect to gender. In addition, it should be noted that findings may be subject to the Von Restorff effect — in appraising challenges or opportunities, respondents may uncon- sciously bias responses towards more visible, severe, or otherwise distinctive issues. This would be of greatest concern with respect to the actions or solutions proposed in response to challenges. While the predictions generated via the Delphi technique may not always prove to be accurate, the primary challenge in implementing such a process involves the respondents themselves. The panel’s findings are ultimately a product of the collective knowledge of the assembled experts. As such, they reflect the biases, misperceptions, or knowl- edge gaps the panelists may possess. Nonetheless, thoughtful question design and the rigorous implementation of the iterated data collection procedure can help minimize the risks posed by these challenges. Independent Evaluation Group | World Bank Group 75 BIBLIOGRAPHY Dalkey, N. C., and D. L. Rourke. 1972. “Experimental Assessment of Delphi Pro- cedures with Group Value Judgments.” In Studies in the Quality of Life: Delphi and Decision-Making, edited by N. C. Dalkey, D. L. Rourke, R. Lewis, and D. Snyder, 55–83. Lexington, MA: Lexington Books. Hsu, C., and B. A. Sandford. 2007. “The Delphi Technique: Making Sense of Con- sensus.” Practical Assessment, Research & Evaluation 12 (10): 1–8. IEA (International Energy Agency). 2014. World Energy Outlook 2014. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2014. IEA (International Energy Agency). 2015. World Energy Outlook 2015. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2015. IEA (International Energy Agency). 2016. World Energy Outlook 2016. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2016. IEA (International Energy Agency). 2017. World Energy Outlook 2017. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2017. IEA (International Energy Agency). 2019. World Energy Outlook 2019. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2019. IPCC (Intergovernmental Panel on Climate Change). 2019. 2019 Refinement to the 2006 IPCC Guidelines for National Greenhouse Gas Inventories. Geneva: IPCC. IRENA (International Renewable Energy Agency). 2017. Renewable Energy Statis- tics 2017. Abu Dhabi: IRENA. Jayawardena, Migara, Ryan Watkins, and Maurya West Meiers. 2019. “Oracle for the Future of Sustainable Energy: Designing a Delphi Approach to Predict Future Trends through a Panel of Global Experts.” Paper presented at the Evaluation 2019 Conference, American Evaluation Association, Minneapolis, November 11–16. Independent Evaluation Group | World Bank Group 77 Ludwig, B. 1997. “Predicting the Future: Have You Considered Using the Delphi Methodology?” Journal of Extension 35 (5). Miller, L. E. 2006. “Determining What Could/Should Be: The Delphi Technique and Its Application.” Paper presented at the Annual Meeting of the Mid-Western Educational Research Association, Columbus, Ohio, October. REN21. 2018. Renewables 2018 Global Status Report. Paris: REN21 Secretariat. Thangaratinam, S., and C. W. E. Redman. 2005. “The Delphi Technique.” The Obste- trician and Gynaecologist 7: 120–25. Turoff, M., and S. R. Hiltz. 1996. “Computer Based Delphi Process.” In Gazing into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health, edited by M. Adler and E. Ziglio, 56–88. London, UK: Jessica Kingsley Publishers. Ulschak, F. L. 1983. Human Resource Development: The Theory and Practice of Need Assessment. Reston, VA: Reston Publishing. UN (United Nations). 2018. “Regional Preparatory Meeting on Energy Issues.” Out- come Document UN E/ESCWA/RFSD/2018/CRP, United Nations Economic and Social Commission for Western Asia, New York. Watkins, R., M. West Meiers, and Y. L. Visser. 2012. A Guide to Assessing Needs: Essen- tial Tools for Collecting Information, Making Decisions, and Achieving Development Results. Washington, DC: World Bank. World Bank, IFC (International Finance Corporation), and MIGA (Multilateral Invest- ment Guarantee Agency). 2016. World Bank Group Climate Change Action Plan 2016–2020. Washington, DC: World Bank. World Bank. 2017. Toward a Clean World for All: An Evaluation of the World Bank Group’s Support to Pollution Management. Washington, DC: World Bank. World Bank. 2020. Renewable Energy: Evaluation of the World Bank Group’s Support for Electricity from Renewable Energy Resources, 2000–2017. Thematic and Sector Evaluation. Washington, DC: World Bank. http://documents.worldbank.org/ curated/en/197701602792711871/Renewable-Energy-Evaluation-of-the-World 78 Delphi Technique | Bibliography APPENDIX A SAMPLE RESPONSE TEMPLATES FOR ROUND 1 (BLANK)— IDEAS GENERATION FOR OPPORTUNITIES AND CHALLENGES -Bank-Group-s-Support-for-Electricity-from-Renewable-Energy-Resourc- es-2000-2017. World Commission on Dams. 2000. Dams and Development: A New Framework for Decision-Making. London and Sterling, VA: Earthscan Publications. Independent Evaluation Group | World Bank Group 81 [Please note that each original template had 10 rows, since panelists could submit up to 10 opportunities and 10 challenges. The samples in this attachment show fewer rows for illustration purposes only.] The Power to RE-new: Evaluation of the World Bank Group’s Support to Renewable Energy Development Delphi High-Level Global Expert Panel on Renewable Energy: Response Templates TEMPLATE 1: OPPORTUNITIES GUIDELINES FOR COMPLETING RESPONSE TEMPLATES 1. The evaluation is limited to renewable energy (RE) for producing and supplying electricity, both for grid connected, mini-grids and household- level solutions. The scope of the evaluation does not cover other areas such as RE for transport, cooking, etc. 2. For the purposes of this evaluation, RE is defined to include biomass, geothermal, hydropower (all sizes, run-of-the-river as well as those with reservoirs for storage of water), solar (photovoltaic and concentrated solar), and wind power. However, the evaluation does not intend to assess each technology for its adequacy, but rather, how various countries can best develop a combination of available RE resources to supply electricity and reap environmental benefits in line with climate and Sustainable Development Goals (SDGs). 3. The panelists are encouraged to consider information and experiences from throughout the world in formulating their advice as global experts on RE. However, the inquiries from the Global Expert Panel on RE are related to helping developing countries (that is, World Bank Group client countries) achieve their RE development goals. Round 1 Response Templates Please note: There are two response templates—one for Opportunities and one for Challenges 4. You are separately provided with two templates in Microsoft Word format to facilitate your responses. The templates are based on specific questions 82 Delphi Technique | Appendix A that are being asked of you, and they are color coded to make it easier to navigate within them. The first template is designed to capture your responses to the emerging opportunities to further scale up RE, whereas the second template is seeking your views on the emerging challenges that could hold back further expansion of RE. 5. An opportunity can be defined as a specific condition that is favorable or conducive to developing and scaling up RE, including but not limited to technological advances, improving market conditions, changes in demand patterns, shifts in policy, and influences outside the RE space (that is, shifts in fossil fuel prices, impact of climate change). 6. A challenge can be defined as a specific constrain or barrier that countries attempting to develop and scale up RE going forward are likely to face, which, if not addressed, will hamper their ability to achieve RE development goals. These could be, but are not limited to, natural resource constraints, technological limitations, technology or power systems related costs, technological or market risks, affordability and shifts in demand, changing market conditions, financing, challenges from civil society, and influences from outside the RE space (that is, improvements in fossil-fuel technologies, availability and costs of fossil fuels). 7. Please note that, at the end of each template, you will have space for sharing observations and recommendations that do not directly respond to the questions asked, but that you believe are important for the future development of RE. 8. For guidance, we have also provided you two examples of a completed matrix entry—one for emerging opportunities and one for emerging challenges to further developing RE to produce electricity. These examples can be found in Attachment 2 of the Objective, Approach, and Guidelines Note. Template 1 of 2: Emerging Opportunities to Scaling Up RE Going Forward 9. Template 1 consists of 8 columns that reflect your responses related to each opportunity selected by you. 10. The first (1) column of the template is a numerical count of the major emerging opportunities that you identify for further developing RE in Independent Evaluation Group | World Bank Group 83 countries around the world. You have space for a maximum of 10 such opportunities, although the number you enter should be based on what you determine to be opportunities that, if seized, can potentially have significance for expanding RE in a substantial way around the world. You are not required to enter all 10 unless you wish to do so. 11. The next four (4) columns are for you to provide your responses related to each (up to 10) of the RE SECTOR opportunities that you have identified: a. Column 2: Insert by typing precise detail and concise text indicating the opportunity identified by you. Please be as brief as possible but provide sufficient detail for us to understand what exactly you mean. b. Column 3: Insert by typing the rationale as to why you identified the opportunity as being important to the further development of RE. Let us know why you think it is an emerging opportunity that, if seized, will have a significant impact on expanding RE development in countries around the world. Please provide sufficient detail so that we can understand the reasoning behind your thinking. c. Column 4: Please insert by typing a numerical score between 1 and 100, to distribute a total of 100 points for each of the identified opportunities. You should award more points to opportunities that you think are more important and fewer points to the ones you feel are not as important. Please ensure that the total score in the column adds up to exactly 100. d. Column 5: For each identified opportunity, please insert by typing text to identify up to three specific actions that can be undertaken either globally or within countries in order to seize/realize the opportunity. 12. Columns six (6) through eight (8) are designated to get your views on the World Bank Group and its ability to help client countries successfully undertake each of the actions you proposed for seizing/realizing the identified opportunities: a. Column 6: For each proposed action, please indicate how well the Bank Group is positioned/placed, especially when compared with other development partners, to influence development outcomes related to RE with various stakeholders either at the country, regional, or global level. 84 Delphi Technique | Appendix A i. Please note that to respond to the above question, you are asked to answer by selecting one of five options: » 4: The Bank Group is EXTREMELY WELL positioned to help clients seize the opportunity. » 3: The Bank Group is VERY WELL positioned to help clients seize the opportunity. » 2: The Bank Group is MODERATELY WELL positioned to help clients seize the opportunity. » 1: The Bank Group is POORLY positioned to help clients seize the opportunity. » X: Do not know/no opinion. b. Column 7: Please insert by typing text to identify up to three specific potential interventions/support to seize each opportunity to develop RE resources for producing electricity. This may include activities that harness and share global knowledge, help mobilize financial resources, mitigate risks, and convene stakeholders within countries or around the world to address mutual concerns. Please be precise in your answer. c. Column 8: For each proposed action, please indicate the current capacity of the Bank Group to help client countries successfully design and implement such action in order to seize each opportunity to develop its RE resources for producing electricity. The capacity of the Bank Group may include a combination of skills and experience, ability to convene and influence stakeholders, and ability to mobilize financial resources. i. Please note that to respond to the above question, you are asked to answer by selecting one of five options: » 4: The Bank Group has VERY HIGH capacity to help clients seize the opportunity. » 3: The Bank Group has HIGH capacity to help clients seize the op- portunity. » 2: The Bank Group has MODERATE capacity to help clients seize the opportunity. Independent Evaluation Group | World Bank Group 85 » 1: The Bank Group has LOW capacity to help clients seize the oppor- tunity. » X: Do not know/no opinion. (continued) 86 Delphi Technique | Appendix A Response Template 1 of 2: Opportunities Emerging Opportunities to Developing RE Going Forward RE Sector Opportunities and Actions Main emerging op- Priority score: portunities to further Between What action(s) should develop RE in developing Rationale for selection of 0 and 100 be taken by develop- countries opportunities (Why?) (Column total must ing countries to seize No. (Please be concise) (Please provide details) be 100) the opportunity? Click here to enter an action. Click here to enter an Click here to enter your Click here to enter Click here to enter an 1 opportunity. rationale. your score. action. Click here to enter an action Click here to enter an action Click here to enter an Click here to enter your Click here to enter Click here to enter an 2 action opportunity. rationale. your rationale. Click here to enter an action Click here to enter an action ... Click here to enter an Click here to enter your Click here to enter Click here to enter an 10 opportunity. rationale. your rationale. action Click here to enter an action TOTAL: 100 Independent Evaluation Group | World Bank Group 87 Response Template 1 of 2: Opportunities (cont.) Emerging Opportunities to Developing RE Going Forward (cont.) World Bank Group Positioning and Capacity to Help Client Countries How well do you think the World Bank Group is How do you assess the positioned to help clients current capacity of the successfully carry out Bank Group to success- the action(s) to seize the What intervention(s) should the Bank Group fully implement each opportunity? undertake to help clients implement the intervention? (Using the 4-point scale in action? (Using the 4-point scale in No. the drop-down list) (Please be specific and provide details) the drop-down list) Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. 1 Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. 2 Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. 88 Delphi Technique | Appendix A Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. ... Choose an item. Click here to enter a Bank Group intervention. Choose an item. 10 Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. TEMPLATE 2: CHALLENGES Template 2 of 2: Emerging Challenges to Scaling Up RE Going Forward 1. Template 2 also consists of 8 columns that reflect your responses related to each challenge selected by you. 2. The first (1) column of the template is a numerical count of the major emerging challenges that you identify, which could hold back further development of RE around the world. You have space for a maximum of 10 such challenges, although the number you enter should be based on what you determine to be challenges of significance that, if they are not addressed, can be a major drawback to expanding RE in a substantial way around the world. You are not required to enter all 10 unless you wish to do so. 3. The next four (4) columns are for you to provide your responses related to each (up to 10) of the RE SECTOR challenges that you have identified: a. Column 2: Insert by typing precise detail and concise text indicating the challenge identified by you. Please be as brief as possible but provide sufficient detail for us to understand what exactly you mean. b. Column 3: Insert by typing the rationale as to why you identified the challenge as being a significant emerging obstacle for further development of RE. Let us know why you think it is an emerging challenge that, if not sufficiently addressed, will have a significant impact on stymieing RE development in countries around the Independent Evaluation Group | World Bank Group 89 world. Please provide sufficient detail so that we can understand the reasoning behind your thinking. c. Column 4: Please insert by typing a numerical score between 1 and 100, to distribute a total of 100 points, for each of the identified challenges. You should award more points to challenges that you think are more important and fewer points to those you believe to have a less significant impact. Please ensure that the total score in the column adds up to exactly 100. d. Column 5: For each identified challenge, please insert by typing text to identify up to three specific actions that can be undertaken either globally or within countries in order to address the challenge. 4. Columns six (6) through eight (8) are designated to get your views on the World Bank Group and its ability to help client countries successfully undertake each of the actions you proposed for addressing the identified challenges: e. Column 6: For each proposed action, please indicate how well the Bank Group is positioned/placed, especially when compared with other development partners, to influence development outcomes related to RE with various stakeholders either at the country, regional, or global level. i. Please note that to respond to the above question, you are asked to answer by selecting one of five options: » 4: The Bank Group is EXTREMELY WELL positioned to help clients address the challenge. » 3: The Bank Group is VERY WELL positioned to help clients address the challenge. » 2: The Bank Group is MODERATELY WELL positioned to help clients address the challenge. » 1: The Bank Group is POORLY positioned to help clients address the challenge. » X: Do not know/no opinion. f. Column 7: Please insert by typing text to identify up to three specific potential interventions/support to address each challenge to 90 Delphi Technique | Appendix A developing RE resources for producing electricity. This may include activities that harness and share global knowledge, help mobilize financial resources, mitigate risks, and convene stakeholders within countries or around the world to address mutual concerns. Please be precise in your answer. g. Column 8: For each proposed action, please indicate the current capacity of the Bank Group to help client countries successfully design and implement such action in order to address each challenge to developing its RE resources for producing electricity. The capacity of the Bank Group may include a combination of skills and experience, ability to convene and influence stakeholders, and ability mobilize financial resources. h. Please note that to respond to the above question, you are asked to answer by selecting one of five options: » 4: The Bank Group has VERY HIGH capacity to help clients address the challenge. » 3: The Bank Group has HIGH capacity to help clients address the chal- lenge. » 2: The Bank Group has MODERATE capacity to help clients address the challenge. » 1: The Bank Group has LOW capacity to help clients address the chal- lenge. » X: Do not know/no opinion. (continued) 92 Delphi Technique | Appendix A Response Template 2 of 2: Challenges Emerging Challenges to Developing RE Going Forward RE Sector Opportunities and Actions Main emerging challeng- es that would hold back Priority score: further development Between What action(s) should of RE in developing Rationale for selection of 0 and 100 be taken by devel- countries challenge (Why?) (Column total must oping countries to No. (Please be concise) (Please provide details) be 100) address the challenge? Click here to enter an action. Click here to enter a Click here to enter your Click here to enter Click here to enter an 1 challenge. rationale. your score. action. Click here to enter an action Click here to enter an action Click here to enter a Click here to enter your Click here to enter Click here to enter an 2 challenge. rationale. your rationale. action Click here to enter an action Click here to enter an action ... Click here to enter a Click here to enter your Click here to enter Click here to enter an 10 challenge. rationale. your rationale. action Click here to enter an action TOTAL: 100 Response Template 2 of 2: Independent Challenges (cont.) Group | World Bank Group 93 Evaluation Emerging Challenges to Developing RE Going Forward (cont.) APPENDIX B ANALYSIS OF DELPHI RESULTS How well do you think the World Bank Group is positioned to help clients How do you assess the successfully carry out current capacity of the Bank the action(s) to seize the What intervention(s) should the Bank Group to successfully imple- opportunity? Group undertake to help clients imple- ment each intervention? (Using the 4-point scale in ment the action? (Using the 4-point scale in the No. the drop-down list) (Please be specific and provide details) drop-down list) Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Independent Evaluation Group | World Bank Group 95 Click here to enter a Bank Group intervention. Choose an item. 1 Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. 2 Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. 96  Click here to enter a Bank Group intervention. Choose an item. Delphi Technique | Appendix B Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. ... Choose an item. Click here to enter a Bank Group intervention. Choose an item. 10 Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Click here to enter a Bank Group intervention. Choose an item. Figures B.1–B.7 show the analysis of the Delphi results. Figure B.1. Prioritization of Challenges to Scaling Up RE P 1 Existing interests that may hinder development of RE 3 (37.5%) 5 (62.5%) 4.38 (0.48) P 2 Inadequate and unstable policy and regulatory infrastructure 2 (25%) 6 (75%) 4.25 (0.43) Difficult to integrate large amounts of RE, especially those of I 5 1 (12.5%) 6 (75%) 1 (12.5%) 4.00 (0.50) variable/intermittent nature C 10 Capacity within government agencies to support and develop RE 1 (12.5%) 6 (75%) 1 (12.5%) 4.00 (0.50) F 14 Regulatory and counterparty risk keep cost of capital high 1 (12.5%) 5 (62.5%) 2 (25%) 3.88 (0.60) P 4 Need bankable projects and PPAs 3 (37.5%) 4 (50%) 1 (12.5%) 3.63 (1.11) P 3 Lack of “real” commitment to decarbonize power system 2 (25%) 2 (25%) 1 (12.5%) 3 (37.5%) 3.38 (1.22) T 6 Need local industry to create strong interest in RE promotion 1 (12.5%) 3 (37.5%) 2 (25%) 2 (25%) 3.38 (0.99) Independent Evaluation Group | World Bank Group 97 C 9 Improve systems and capacity to reduce project delays 1 (12.5%) 1 (12.5%) 6 (75%) 3.38 (0.70) Difficulty mobilizing financing for RE in smaller countries/markets F 13 5 (62.5%) 1 (12.5%) 2 (25%) 3.38 (0.86) that may be less lucrative for developers Strengthen capacity for transparency and accountability to address C 8 3 (37.5%) 5 (62.5%) 3.38 (0.48) corruption and governance C 7 Need champion to drive RE development process 1 (12.5%) 2 (25%) 2 (25%) 3 (37.5%) 3.13 (1.05) Affordability of off-grid equipment and aftersales care for poorer F 12 3 (37.5%) 3 (37.5%) 1 (12.5%) 1 (12.5%) 3.00 (1.00) populations O 16 Large land requirements for developing RE 3 (37.5%) 2 (25%) 2 (25%) 1 (12.5%) 2.88 (1.05) Rapid decline in technology costs could lead to slower uptake as F 15 1 (12.5%) 6 (75%) 1 (12.5%) 2.88 (0.78) utilities and developers take wait-and-see approach R 11 Vulnerability of RE systems to extreme weather and climatic events 1 (12.5%) 1 (12.5%) 5 (62.5%) 1 (12.5%) 2.25 (0.83) 0 1 2 3 4 5 6 7 8 Challenges Votes (no.) Mean Score / St.Dev. RE barrier classification Importance score P Policy and I Integration into T Improvement to design C Strengthen institutional R Mitigate investment F Mobilizing O Others Null Very low = 1 Low = 2 Moderate = 3 High = 4 Very high = 5 regulatory power system and technical standards capacity risks financing Source: Independent Evaluation Group. Note: PPA = power purchase agreement; RE = renewable energy. Figure B.2. Prioritization of Opportunities to Scaling Up RE P 3 Achieve energy access goals 5 (62.5%) 2 (25%) 1 (12.5%) 4.50 (0.71) 98  F 12 Declining technology costs of RE and emergence of power storage 2 (28.6%) 5 (71.4%) 1 (14.3%) 4.29 (0.45) Delphi Technique | Appendix B P 1 Desire to meet national and international emission reduction commitment 3 (37.5%) 4 (50%) 1 (12.5%) 4.25 (0.66) P 2 Address local environmental concerns 4 (50%) 2 (25%) 2 (25%) 4.25 (0.83) F 10 Strong interest in developing market for RE by developers, development 3.75 (0.83) 1 (12.5%) 5 (62.5%) 1 (12.5%) 1 (12.5%) partners, and climate financing sources Firming up (variable) RE capacity by combining complementary RE I 7 systems or addition of electricity storage 5 (71.4%) 2 (28.6%) 1 (14.3%) 3.71 (0.45) P 4 Improve security of power supply with use of indigenous resources 2 (25%) 2 (25%) 3 (37.5%) 1 (12.5%) 3.63 (0.99) I 5 Integrated planning of power system 1 (12.5%) 4 (50%) 2 (25%) 1 (12.5%) 3.63 (0.86) I 8 Increase distributed generation in which RE can play a key role 2 (25%) 2 (25%) 2 (25%) 2 (25%) 3.50 (1.12) F 11 Liberalization of power markets that increase private RE interests 1 (14.3%) 1 (14.3%) 4 (57.1%) 1 (14.3%) 1 (14.3%) 3.29 (0.88) I 6 Utilization of technology and smart grids 1 (12.5%) 1 5 (62.5%) 1 (12.5%) 3.25 (0.83) R 9 Mitigation of upfront investments costs/risks of RE through public finance 4 (50%) 3 (37.5%) 1 (12.5%) 3.25 (0.97) and pooling of resources, for more affordable ... O 14 Integration of RE and EE in power systems planning 3 (37.5%) 4 (50%) 1 (12.5%) 3.25 (0.66) O 13 Income generating potential of RE 1 (12.5%) 5 (62.5%) 1 (12.5%) 1 (12.5%) 2.75 (0.83) 0 1 2 3 4 5 6 7 8 Opportunities Votes (no.) Mean Score / St.Dev. RE barrier classification Importance score P Policy and I Integration into T Improvement to design C Strengthen institutional R Mitigate investment F Mobilizing O Others Null Very low = 1 Low = 2 Moderate = 3 High = 4 Very high = 5 regulatory power system and technical standards capacity risks financing Source: Independent Evaluation Group. Note: EE = energy efficiency; RE = renewable energy. Figure B.3.  Prioritization of Actions or Solutions for Addressing Challenges to Scaling Up RE No. Challenge Actions/Solutions 4 3 1 4.38 /(0.70) • Phase-out fossil fuel power plants over time, starting with the most polluting ones 1 7 4.13 /(0.33) P 1 Existing (vested) interests • Legislate clear long-term RE targets and agree with utility how to achieve these targets 1 6 1 4.00 /(0.50) • Create alternative employment for areas impacted by removing fossil electricity production 4 4 4.00 /(1.00) • Legislate market liberalization to allow IPPs 4 3 1 3.88 /(1.17) • Stop building fossil fuel base power plants 1 6 1 3.75 /(1.09) • Bring off-grid options into long term electrification plan 5 3 3.63 /(0.48) • Learn from successes of other countries • Remove subsidies on kerosene 2 3 2 1 3.63 /(1.22) • Increase awareness on job creation potential of RE 4 3 1 3.38 /(0.70) • Experiment with off-grid electrification concessions 1 2 4 1 3.25 /(1.09) • Provide incentives to convert conventional power plants into RE and/or storage 2 1 2 3 3.25 /(1.20) • Remove VAT and import duty on solar 1 1 4 1 1 3.00 /(1.12) 4 3 1 4.38 /(0.70) • Develop stable policy frameworks that support RE development 3 3 2 4.13 /(0.78) • Establish independent electricity regulator P 2 Policy and regulatory infrastructure 3 3 2 4.13 /(0.78) • Undertake electricity sector restructuring • Develop national RE strategies and long-term plans 2 4 2 4.00 /(0.71) • Developing countries should insist donors coordinate their assistance for RE 1 1 5 1 3.13 /(1.05) P 3 Commitment to decarbonize power system • Countries should use their voice as Bank Group shareholders and demand change 1 6 1 3.00 /(1.00) P 4 Bankable projects and PPAs • Streamline permitting process to speed up approvals 1 3 4 3.63 /(0.70) 4 3 1 4.38 /(0.70) • Improve transmission and distribution network 3 4 1 4.25 /(0.66) • Integration of grid systems 1 6 1 4.00 /(0.50) • Strengthen capacity of grid operators • Promote pumped storage hydro and battery storage 2 3 2 1 3.75 /(0.97) I 5 Integration of large amounts of RE • Undertake long-range transmission planning 1 4 2 1 3.63 /(0.86) 1 3 4 Independent Evaluation Group | World Bank Group 99 • Unlock flexibility in generation and the demand side by creating appropriate market incentives 3.63 /(0.70) • Reconfigure electricity markets to provide for price discovery of balancing power 1 3 3 1 3.50 /(0.87) • Develop smart grids 4 3 1 3.38 /(0.70) • Develop risk guarantee mechanism to provide fosil-based plants with capacity cost 2 5 1 3.13 /(0.60) • Progressively move to reflect full RE costs, especially as their share in energy generation rises 2 5 1 3.13 /(0.60) • Technologies transfer. Obtain manufacturing capabilities 1 2 4 1 3.38 /(0.86) T 6 Local RE industry 1 3 2 1 1 3.25 /(1.20) • Support manufacturing of renawable energy equipment C 7 Champion to drive RE • Training on RE for business people, managers, and engineers 2 2 4 3.75 /(0.83) C Anti-corruption and governance • Encourage the rule of law through strong anti-corruption laws and robust, consistent enforcement 3 3 2 4.13 /(0.78) 8 • Streamline permitting and development processess, including adequate environment review and land acquisition 3 2 3 4.00 /(0.87) C 9 Implementation delay of RE projects 3 2 3 4.00 /(0.87) • Training on RE for business people, managers, and engineers C 10 Capacity within government agencies 2 3 2 1 3.75 /(0.97) • Seek support from development partners to assist with negotiations that need to be undertaken R 11 Vulnerability of RE systems to extreme weather • Make RE systems more climate resilient 1 4 2 1 3.63 /(0.86) 2 4 1 1 3.75 /(1.20) • Support PAYGO Systems with intermidiaries who rent and service equipment that depends on RE F 12 Affordability of off-grid equipment and • Provide some targeted subsidies to poor households to be able to rent RE-related equipment 1 4 2 1 3.63 /(0.86) aftersales care • Provide training of women from rural communities to service and maintain RE systems 1 4 2 1 3.63 /(0.86) 4 4 4.50 /(0.50) • Explore opportunities for de-risking some investments, through the use of grant or concessional financing F 13 Mobilizing financing in country small markets • Organize small countries to pool similar projects in the same region 2 3 2 1 3.75 /(0.97) • Create a clear and robust investment framework 4 2 2 4.25 /(0.83) F 14 Risks keeping cost of capital high • Minimize counterparty risk 1 6 1 4.00 /(0.50) F 15 Rapid decline in RE costs causing utilities to delay • Introduce pooled prices for RE electricity 2 1 3 2 3.38 /(1.11) 3 3 1 1 4.00 /(1.00) • Require that the design of large hydro damns must be more environmentally sensitive 2 3 2 1 3.75 /(0.97) • Require resettlement with full engagement of stakeholders and flowing the state of the art guidelines • Create a licensing regime that balances renewable development with legitimate land use concerns 1 4 2 1 3.63 /(0.86) O 16 Large land requirement for RE • Do not allow biomass expansion at the expense of good agricultural land 3 2 1 1 1 3.63 /(1.41) • Priority lending for repowered RE systems 1 1 5 1 3.25 /(0.83) • Create an investment framework that creates incentives for renewable developers 3 3 1 1 3.00 /(1.00) • Retention of older (and higher) tariffs for repowered RE plans 2 4 2 3.00 /(0.71) 0 2 4 6 8 2.00 3.00 4.00 5.00 Number of votes Mean Score / St.Dev. RE barrier classification Importance score P Policy and I Integration into T Improvement to design C Strengthen institutional R Mitigate investment F Mobilizing O Others Null Very low = 1 Low = 2 Moderate = 3 High = 4 Very high = 5 regulatory power system and technical standards capacity risks financing Source: Independent Evaluation Group. Note: IPP = independent power producer; PPA = power purchase agreement; RE = renewable energy; VAT = value-added tax. Figure B.4.  Prioritization of Actions or Solutions for Seizing Opportunities to Scaling Up RE (1–7) 100  No. Opportunity Actions/Solutions 4 4 4.50 /(0.50) Delphi Technique | Appendix B • Facilitate easier finance for RE by reducing policy-related risks to such investments 1 Meet emission reduction 3 3 2 4.13 /(0.78) P • Inclusion of a GHG price (as a cost) in cost-benefit analyses of new generation capacity commitment 2 3 3 3.88 /(0.78) • Provide strong incentives for power producers to expand RE 6 2 3.75 /(0.43) • Integrate health, air pollution, and energy security discussions 2 2 4 3.75 /(0.83) • Review NCDs to ensure congruence between mitigation ambition expressed there and medium and long-term national energy transition... • Develop long term strategies for GHG emissions phaseout 1 2 5 3.50 /(0.71) • Use price on carbon as tool to meet climate change goals 3 4 1 3.25 /(0.66) • Change policy to build only renewable plants in future 2 2 1 1 2 3.13 /(1.54) • Shift all investment in thermal power to renewables and in particular higher risk new enabling technologies 2 1 3 2 3.13 /(1.45 • Use climate change negotiating fora and other sustainable energy meetings to sensitize international stakeholders 3 3 2 3.13 /(0.78) • Establish regional networks for pricing carbon/trading allowances 2 3 3 2.88 /(0.78) 3 4 1 4.25 /(0.66) P 2 Address local environmental • Closure of strategically located coal-based power stations • Identify hotspots where power generation is a major contributor to local air quality and provide strong incentives for switch to RE 2 4 2 4.00 /(0.71) concerns • Discourage use of self-generation using diesel, which is a major source of local pollution. Self-generation using RE should be encouraged 3 2 2 1 3.88 /(1.05) • Introduction of electric buses with RE-based charging in urban areas 2 2 4 3.75 /(0.83) • Raise awareness of the health risks of local air pollutants 2 4 1 1 3.75 /(1.20) • Standards and protocols for urban buses and charging infrastructure 3 4 1 3.25 /(0.66) 2 5 1 4.13 /(0.60) Achieve energy access • Support rural households to acquire RE equipment P 3 • Align national energy policy to achieve the targets set out in SDG 7 to receive official development assistance 2 4 2 4.00 /(0.71) goals • Provide an open, competitive regulatory environment for new entry into off grid solutions 4 1 2 1 4.00 /(1.12) • Integrate scaled-up PV deployment into access discussions 1 5 1 1 3.75 /(0.83) • Create a clear regulatory framework for the interconnection of mini grids with the centralized system 2 3 2 1 3.75 /(0.97) • Establish protocols for system integration, standards, and labeling of systems 1 4 3 3.75 /(0.66) • Move from heavy emphasis on grid extension to deliver energy access, to a range of “distributed” approaches 2 3 2 1 3.75 /(0.97) • Facilitate micro and mobile payments for off grid solar + storage 2 3 2 1 3.63 /(1.22) • Aggregate demand and bundle procurement 1 3 3 1 3.50 /(0.87) • Facilitate sectoral approaches to distributed PV deployment (residential, commercial, utility) 5 2 1 3.50 /(0.71) • Integrate health, air pollution, and energy security discussions 4 3 1 3.38 /(0.70) • Ensure community supply by supporting maintenance of equipment and supply systems 3.25 /(0.83) 1 1 5 1 • Introduce regulatory regimes that favor distributed energy and set fair and consistent user tariffs, enabled by intelligent subsidies 3.25 /(0.83) 4 2 2 • Institute that customer pay covers all the system costs (full cost-recovery) and fix bankrupt distribution utilities 1 3 2 2 3.13 /(1.36) • Create the enabling environment for investments in RE—modernized legislation, energy policy, independent regulator 3 5 4.38 /(0.48) 4 Improve security of power P • Build RE implementation capacity 2 4 1 1 3.88 /(0.93) systems • Develop locally based RE through support to local producers 2 3 2 1 3.63 /(1.22) • Prepare medium and long-term national development plans that speak very clearly to the role of energy sector transformation 1 4 2 1 3.38 /(1.41) • Provide time-limited incentives for RE-based electricity systems that substitute imported fuels 1 3 2 1 1 3.25 /(1.20) • Increase awareness of the benefits of locally based RE in communities 1 2 3 1 1 3.13 /(1.17) • Countries can gain best access to RE technologies by respecting intellectual property rights and at the same time getting support for the tran... 1 4 3 2.88 /(0.93) • Provide support to the key industries to implement RE programs that will help to reduce the existing high cost of energy as a production i... 3 3 1 1 2.88 /(1.27) • Prioritize grid infrastructure investment 4 3 1 4.38 /(0.70) I 5 Integrated planning of p... • Realize the transmission need and develop a holistic long-range transmission plan rather than addressing transformation on a project 1 4 3 3.75 /(0.66) 3 4 1 4.25 /(0.66) Utilization of technology and • Ensure that adequate grid capacity exists for renewable developments, modernize the operation and governance of the grid I 6 small grids • Allow energy storage to qualify for renewable energy financing incentives and tariffs 3 2 2 1 3.88 /(1.05) • Allow for capitalization of investment in digital assets including hardware and software 2 2 3 1 3.38 /(1.49) • Digitalize network operations, make full use of big data in renewables forecastings 4 3 1 3.38 /(0.70) • Do not erect statutory or regulatory barriers to the deployment of hybrid technologies 2 5 1 4.13 /(0.60) Address variable RE w/ I 7 complementary system/.. • Make it easier to integrate variable RE sources into the grid and to support solutions that involve storage of variable energy 2 5 1 4.13 /(0.60) • Provide incentives for development of hybrids 2 2 3 1 3.63 /(0.99) 0 2 4 6 8 2 3 4 5 RE barrier classification Importance score P Policy and I Integration into Null Very low = 1 Low = 2 Moderate = 3 High = 4 Very high = 5 Number of votes Mean Score / St.Dev. regulatory power system Source: Independent Evaluation Group. Note: GHG = greenhouse gas; NCD = nonconvertible debentures; PPA = power purchase agreement; PV = photovoltaics; RE = renewable energy; SDG = Sustainable Development Goal. Figure B.5.  Prioritization of Actions or Solutions for Seizing Opportunities to Scaling Up RE (8–14) No. Opportunity Actions/Solutions 3 4 1 4.25 /(0.66) I 8 Increase distributed • Load balancing so as to optimize local RE and grid electricity supply generation 2 5 1 • Development and adoption of grid integration protocols 4.13 /(0.60) • Monetize future loss reduction so as to pay for RE installation now 2 2 2 2 3.50 /(1.12) R 9 Mitigation of upfront • Establish dedicated public-sector infrastructure fund to provide construction funding 1 3 1 3 3.25 /(1.09) investment costs/risks 5 2 1 4.50 /(0.71) F 10 Strong interests by multiple • Develop the requisite internal capacity to plan, implement, monitor and evaluate the RE projects stakeholders to finance RE • Proactively pursue quality investments in RE that match the scale and the capacity of the country 2 4 2 4.00 /(0.71) • Develop comprehensive funding proposals, tailored to the needs of the respective financial instruments 2 4 1 1 3.88 /(0.93) • Develop a clear energy road map that identifies the interventions necessary to facilitate the transition away 1 5 1 1 3.63 /(1.11) from fossil fuels to RE and share • Establish a portfolio of investment opportunities in the electricity sector 1 4 1 2 3.50 /(1.00) F 11 Liberalization of power • Open electricity markets to independent power producers 2 3 3 3.88 /(0.78) Independent Evaluation Group | World Bank Group 101 markets F 12 Declining techcosts of • Amend procurement/tendering practices to allow for consideration of life-cycle cost when making 1 7 4.13 /(0.33) RE/emergence of power procurement decisions storage • Ensure the adequate financial background of the off-taker (i.e. bankability) is available to renewable 3 3 1 1 4.00 /(1.00) producers • Repeated bulk procurement of RE electricity on a competitive basis, so as to accelarate price reduction 2 4 2 4.00 /(0.71) with increasing volumes • Provide a clear policy framework for renewable investment, primarily through capacity auctions 1 4 3 3.75 /(0.66) • Undertake detailed cost-benefit analyses to identify the RE sources 1 1 6 3.38 /(0.70) • Risk reduction of RE electricity producers, especially with regard to contracting and payment guarantee 1 3 3 1 3.25 /(1.39) O 13 Income generating potential • Conducting case study 1 2 2 3 3.13 /(1.05) of RE • Selecting best practice 3 3 2 3.13 /(0.78) O 14 Integration of RE and EE in • Develop efficiency standards 2 5 1 4.13 /(0.60) power systems planning • Integrate efficiency into energy planning processes, particularly with visible domestic planning processes or 1 5 2 3.88 /(0.60) international processes • Provide finance for efficiency 1 4 3 3.75 /(0.66) 0 2 4 6 8 2.00 3.00 4.00 5.00 Number of votes Mean Score / St.Dev. RE barrier classification Importance score I Integration into R Mitigate investment F Mobilizing O Others Null Very low = 1 Low = 2 Moderate = 3 High = 4 Very high = 5 power system risks financing Source: Independent Evaluation Group. Note: EE = energy efficiency; RE = renewable energy. Figure B.6.  Prioritization of Bank Group Position and Capacity for Addressing Challenge to Scaling Up RE The World Bank 1818 H Street NW Washington, DC 20433