Qualitative Comparative Claude Rubinson Migara Jayawardena Analysis Ryan Watkins Joy Butscher Exploring Causal Links for Noureddine Berrah Scaling Up Investments in Renewable Energy IEG Methods and Evaluation Capacity Development Working Paper Series © 2022 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: Rubinson, Claude, Migara Jayawardena, Ryan Watkins, Joy Butscher, and Noureddine Berrah. 2022. Qualitative Comparative Analysis: Exploring Causal Links for Scaling Up Investments in Renewable Energy. IEG Methods and Evaluation Capacity Development Working Paper Series. Independent Evaluation Group. Washington, DC: World Bank. EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Luísa Ulhoa Fernanda Macedo This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Qualitative Comparative Analysis Exploring Causal Links for Scaling Up Investments in Renewable Energy Claude Rubinson, Migara Jayawardena, Ryan Watkins, Joy Butscher, and Noureddine Berrah Independent Evaluation Group January 2022 CONTENTS Abbreviations����������������������������������������������������������������������������������������������������������������iv Acknowledgments�������������������������������������������������������������������������������������������������������vi Overview��������������������������������������������������������������������������������������������������������������������� viii 1. Background and Context�������������������������������������������������������������������������������������������2 2. Rationale for Qualitative Comparative Analysis in Evaluating Rnewable Energy�10 3. A Road Map for Applying Qualitative Comparative Analysis ��������������������������������16 4. Application of Qualitative Comparative Analysis: Strategic Decisions and Case Study Analysis ������������������������������������������������������������������������������������������������������������22 Developing a Theory of Change  24 Identifying Country Case Studies  26 Developing a Set of Factors or Preconditions  28 Scoring the Factors or Preconditions 32 5. Application of Qualitative Comparative Analysis: Analyzing Results �������������������36 6. Application of Qualitative Comparative Analysis: Interpretation of Findings and Conclusions ����������������������������������������������������������������������������������������������������������������46 Interpreting Findings  48 Conclusion  51 Preconditions (Barriers to Developing Renewable Energy) 59 Output (Investment Leading to Development of Renewable Energy) 65 Glossary of Key Terms  About the Authors Bibliography����������������������������������������������������������������������������������������������������������������54 Appendix A | Country-Level Ratings Criteria for Qualitative Comparative Analysis��58 Appendix B | Matrix of Consolidated Precondition Barrier Scores������������������������ 68 ii  ABBREVIATIONS IEA International Energy Agency CO2 carbon dioxide IEG Independent Evaluation Group IFC International Finance Corporation IPCC Intergovernmental Panel on Climate Change MIGA Multilateral Investment Guarantee Agency PV solar photovoltaics QCA qualitative comparative analysis RE renewable energy SDG Sustainable Development Goals ToC theory of change All dollar amounts are US dollars unless otherwise indicated. Independent Evaluation Group | World Bank Group v ACKNOWLEDGMENTS The qualitative comparative analysis (QCA) methodology relies on in-depth case studies as a key data source. The country case studies for renewable energy were carried out by an experienced group of sector specialists that included (in alpha- betical order) Varadan Atur, Noureddine Berrah, Enno Heijndermans, Ihsan Kaler Hurkan, Ramachandra Jammi, Migara Jayawardena, Fernando Lecaros, Andres Liebenthal, Alexey Morozov, and Dennis Reyes. The case study work was coordi- nated by Joy Butscher. The QCA modeling was carried out by Claude Rubinson. The case study structure and QCA scoring was developed by Noureddine Berrah, Migara Jayawardena, and Ryan Watkins, who together with Claude Rubinson analyzed the QCA results. This paper was produced as part of the Methodological Paper Series sponsored by the Independent Evaluation Group’s Methods Advisory Function. The authors are grateful for the feedback provided by the peer reviewers: Lauren Kelly and Ray Rist. In addition, the authors would like to thank the editors and staff of the Paper Series for their comments and contributions: Jos Vaessen, Ariya Hagh, Sylvia Otieno, Maurya West Meiers. A special thanks to Amanda O’Brien, Kristen Milhollin, Fernanda Macedo, and Luísa Ulhoa for their support in editing, produc- tion, and graphic design. Although many people contributed to the exercise and the preparation of this paper, it is noted that the findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and should not be attributed in any manner to the World Bank Group, to members of its Board of Executive Directors, or to the countries they represent. Independent Evaluation Group | World Bank Group vii OVERVIEW This paper illustrates how qualitative comparative analysis (QCA) was used to identify causal pathways for scaling up renewable energy to meet the sustainable development and climate goals. The analysis was a part of a multimethod evalu- ation that was carried out to assess the World Bank Group’s performance in sup- porting the development of renewable energy (RE) from 2000 to 2017. The evalu- ation explored the World Bank Group’s role in helping clients achieve the United Nations’ Sustainable Development Goals and the targets in the Paris Agreement on climate change. QCA was used to validate the theory of change developed for the RE evaluation and identify pathways for scaling up RE as a key solution. The paper was prepared for international development professionals, with both evaluation specialists and energy sector practitioners in mind. It delves into issues of RE investments and the strategic context for those investments. It remains, however, a methods-focused paper that also examines in detail the approach and process for designing and carrying out the QCA and the analysis of results and key conclusions drawn from the findings. The QCA presented an opportunity to synergistically bring together energy sector and development specialists together with methods experts, which was essential to designing, conducting, and analyzing results from the methodological application. This approach was vital to developing robust conclusions that could be triangulated with other methods in the RE evaluation. Ultimately, the QCA findings, together with results from other methods, helped validate the theory of change developed for the RE evaluation and helped identify three pathways through which six key challenges to scaling up RE (precondition barriers) can be overcome so that the clean energy transition can be successfully achieved. Independent Evaluation Group | World Bank Group ix 1 BACKGROUND AND CONTEXT Scaling up renewable Expanding clean energy energy to limit global and other generation warming technologies Shifting fuel sources for global electricity production Renewable energy (RE) is central to achieving the Sustainable Development Goals (SDGs) and responding to the urgency of climate change. According to the United Nations, the availability of affordable and clean energy (SDG 7) is an explicit and interdependent goal “crucial for achieving almost all [16 other] SDGs,” such as erad- icating poverty, improving health and education, supplying clean water, industri- alization, and combating climate change (UN 2018). (These are further emphasized through SDG 13 on climate action). The Paris Agreement on climate change, which became effective in 2016, places even greater prominence on scaling up RE as a key solution to limiting global temperature rise to no more than 2°C by 2100 and to “make best efforts to limit warming to 1.5°C” (World Bank, IFC, and MIGA 2016, 1). RE has experienced a dynamic expansion over the years through the development of multiple technologies and has the potential to scale up further in the future. The major RE technologies—hydropower, solar power, wind power, geothermal power, and biopower—produce over a quarter of the world’s electricity (figure 1.1). Hydro- power, which has been the dominant technology by scale, makes up the highest pro- portion in the global RE mix, with a 62 percent share in 2017 (or 16.4 percent of total global electricity production). Around 2000, wind power began to expand at a global- ly significant scale, followed by a major scale-up of solar photovoltaics (PV) starting around 2008. Together, wind and solar PV made up 7.5 percent of global electricity produced in 2017. For the same year, biopower and geothermal—two long-standing technologies—produced less than 3 percent of global electricity. RE markets experi- enced a major transformation during the period assessed, with multiple technologies now showing a significant global impact (box 1.1). Future expansions of RE will need to navigate this anticipated continued market volatility. A monumental scale-up in RE is required to achieve global energy and environ- ment goals, which needs to be set in motion immediately and sustained for several decades. Despite the expansion of RE, electricity produced from fossil fuels make up nearly three-quarters of the global generation mix, since these technologies have also continued to grow. Therefore, the expansion of RE will need to acceler- ate and be sustained so that the growth of fossil-based technologies can be re- duced without locking into a less sustainable pathway to meeting energy demand. Several studies propose different pathways to achieving such goals,1 but all options involve an unprecedented scale-up in RE. One such analysis by the International Energy Agency used in a 2018 report by the Intergovernmental Panel on Climate Change estimates that the RE share in the global energy mix will need to more than double by 2030 (consistent with the SDGs) and further accelerate to almost quadruple by 2040—a program of action commonly referred to as the clean energy transition (table 1.1). The 2018 Intergovernmental Panel on Climate Change report calls for “rapid and far-reaching” transitions in the energy sector (among others) 4 Qualitative Comparative Analysis: Renewable Energy | Chapter 1 to limit global warming to 1.5°C. It is estimated that such an expansion will double the present global level of investments in RE from $300 billion to $600 billion (IEA 2017, 2019; IRENA 2017). Figure 1.1. Renewable Energy Share of Global Power Generation, 2017 73.5% Nonrenewable electricity 5.6% 26.5% 16.4% Windpower Renewable Hydropower eletricity 2.2% Biopower 1.9% Solar PV 0.4% Ocean, CSP, and geothermal power Source: Renewables 2018 Global Status Report Note: CSP = concentrated solar power; PV = photovoltaic. A monumental scale-up in RE is required to achieve global energy and environ- ment goals, which needs to be set in motion immediately and sustained for several decades. Despite the expansion of RE, electricity produced from fossil fuels make up nearly three-quarters of the global generation mix, since these technologies have also continued to grow. Therefore, the expansion of RE will need to accelerate and be sustained so that the growth of fossil-based technologies can be reduced without locking into a less sustainable pathway to meeting energy demand. Sev- eral studies propose different pathways to achieving such goals,2 but all options involve an unprecedented scale-up in RE. One such analysis by the International Energy Agency used in a 2018 report by the Intergovernmental Panel on Climate Change estimates that the RE share in the global energy mix will need to more than double by 2030 (consistent with the SDGs) and further accelerate to almost quadruple by 2040—a program of action commonly referred to as the clean energy transition (table 1.1). The 2018 Intergovernmental Panel on Climate Change report calls for “rapid and far-reaching” transitions in the energy sector (among others) Independent Evaluation Group | World Bank Group 5 to limit global warming to 1.5°C. It is estimated that such an expansion will double the present global level of investments in RE from $300 billion to $600 billion (IEA 2017, 2019; IRENA 2017). Table 1.1. A Pathway to Achieving the Clean Energy Transition Installed Capacity Average Electricity Produced (GW) Average Capacity (TWh) Renewable Annual Addition Energy Increase (GW/ Technology 2016 2030 2040 (percent) year) 2016 2030 2040 Hydropower 1,241 1,723 2,060 2.1 34.0 4,070 5,688 6,928 Wind 466 1,706 2,629 7.5 90.0 981 4,193 6,950 Solar PV 299 1,846 3,246 10.4 123.0 303 2,732 5,265 Biopower 127 243 347 4.3 9.2 570 1,209 1,807 Geothermal 13 44 82 8.0 2.9 86 292 563 CSP 5 92 328 19.0 13.5 11 287 1,066 Source: Based on International Energy Agency forecasts for its Sustainable Development Scenario. Note: 2030 represents a target consistent with meeting the Sustainable Development Goals. CSP = con- centrated solar power; GW = gigawatt; PV = photovoltaic; TWh = terawatt-hour. The World Bank Group has mainstreamed its support to RE and is committed to the SDGs and the climate goals in the Paris Agreement. The institution has a long his- tory of financing hydropower, but its support for other RE technologies was initially modest and mostly for increasing electricity access, particularly in rural areas not served by the power grid. However, in 2004, the president of the Bank Group made a commitment at the Renewable Energy Conference in Bonn, Germany (the Bonn Commitment), to increase its lending for RE by 20 percent over five years (figure 1.2). This commitment coincided with the global expansion in wind and predated the boom in solar power that followed. After the Bonn Commitment, there was a notable uptick in the Bank Group’s financial support to RE covering all major tech- nologies, averaging about $2 billion per year. The scale-up in funding extended to the greater involvement of the private sector, including through the International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency—two institutions within the Bank Group. Although the Bank Group’s financing for RE is dwarfed by the $300 billion in annual global flows into the sector, the institution’s significant international role in supporting reforms, convening partners and mo- bilizing financing, and disseminating global experiences is a unique position of 6 Qualitative Comparative Analysis: Renewable Energy | Chapter 1 influence. It also explains the Bank Group’s interest in evaluating its performance in support of RE so that it is optimally geared to help client countries successfully navigate the clean energy transition. Figure 1.2. World Bank Group Commitment to Renewable Energy, 2000–17 $1.6 billiona $8.5 billiona $12 billiona (2000–05) (2006–11) (2012–17) 3.0 2.67 2.57 2.46 Commitment amount (US$, billions) 2.43 2.5 2.20 2.13 2.0 1.73 1.60 1.57 1.5 1.39 1.10 1.0 0.49 0.5 0.34 0.34 0.40 0.29 0.16 0.19 0.0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 58 projects 189 projects 281 projects Institution World Bank IFC MIGA Source: World Bank Group. Note: The renewable energy portfolio evaluated includes 18 additional projects that had been approved previously and were evaluated in 2000 or later. IFC = International Finance Corporation; MIGA = Multilat- eral Investment Guarantee Agency. a. Total commitment amounts do not include issuances of MIGA guarantees. Independent Evaluation Group | World Bank Group 7  he Evolution of Renewable Energy and other Generation Box 1.1. T Technologies The global power supply has evolved over time, with renewable energy (RE) playing an increasing role. The global electricity production grew more than fourfold over the 45 years between 1971 and 2015, with shifts in the mix of generation technologies (figure B1.1.1). Coal use, given its reliability and low production cost, grew steadily to reach a 40 percent share as the main nonrenewable source of power generation. Hydropower is a long-standing RE technology that has also continued to increase, although its share of global electricity declined from 23 to 17 percent over 1971–2015, as fossil fuel use grew more rapidly over the same period.  hifting Fuel Sources for Global Production of Electricity, Figure B1.1.1. S 1971–2015 Global electricity production (gigawatt-hours) 10,000,000 8,000,000 6,000,000 4,000,000 2,000,000 0 1970 1975 1980 1985 1990 1995 2000 2005 2010 2015 Year Coal Gas Hydropower Nuclear Re w/o hydropower Oil Source: Independent Evaluation Group, based on International Energy Agency data. Note: RE = renewable energy. (continued) 8 Qualitative Comparative Analysis: Renewable Energy | Chapter 1  he Evolution of Renewable Energy and other Generation Box 1.1. T Technologies (cont.) Decreases in prices led to a rise in the use of oil for power generation in the early 1970s, followed by a decline later in the decade due to global oil shocks. Nuclear power, viewed as an alternative to fossil fuels, expanded until the 1990s, but several accidents progressively led to a slow-down in its growth, out of safety concerns. Natu- ral gas was used sparingly for power generation, with a global share of only 13 percent in 1971, mostly relegated to countries that produced the fuel. However, technological advances in extraction and more efficient means of transportation allowed natural gas to overtake hydropower as the second-largest fuel source for power production by the late 1990s. RE technologies such as wind and solar photovoltaics were developed at small scale or for demonstration purposes, before their rapid global expansion starting around 2000. Electricity produced from wind and solar photovoltaics grew from negligible shares in the early 2000s to reach a combined share of 7.5 percent share of global electricity produced by 2017. Other more niche RE technologies, such as geothermal power, concentrated solar power, and biopower, are not globally significant by scale, but some make important contributions in specific countries. Notes 1 The Intergovernmental Panel on Climate Change 2018 report considers the following three studies: the Energy Technology Perspectives series (IEA 2014, 2015, 2016, 2017), the Inter- national Energy Agency and International Renewable Energy Agency report (OECD/IEA and IRENA 2017), and the Shell Sky report (Shell International B. V. 2018). 2 The Intergovernmental Panel on Climate Change 2018 report considers the following three studies: the Energy Technology Perspectives series (IEA 2014, 2015, 2016, 2017), the Inter- national Energy Agency and International Renewable Energy Agency report (OECD/IEA and IRENA 2017), and the Shell Sky report (Shell International B. V. 2018). Independent Evaluation Group | World Bank Group 9 2 RATIONALE FOR QUALITATIVE COMPARATIVE ANALYSIS IN EVALUATING RNEWABLE ENERGY Methods used in the renewable energy evaluation Reasons to use qualitative comparative analysis Qualitative comparative analysis (QCA) is an analytical technique that was devel- oped in the 1980s for use in the qualitative study of macrosocial phenomena. It has proven itself particularly useful in mixed methods research, although its introduc- tion to evaluation is relatively recent. QCA draws on both the variable-oriented and case-oriented methodologies as a “means of bridging quantitative and qualitative analysis” (Cragun et al. 2016). It combines the use of quantitative techniques to identify patterns within one’s data with in-depth qualitative understanding of the cases and subject matter being studied. The QCA methodology uses Boolean algebra to generate a set of inferences based on underlying data across multiple qualitative cases. Thus, the methodology integrates both quantitative and qualitative analysis, generating findings that are generalizable across a wider population. QCA can help identify causal patterns when triangulated with results from other methods. “QCA is based on two primary assumptions: change is often the result of different combinations of factors, rather than on any one individual factor; and different combinations of factors can produce similar changes.” —Charles Ragin, who is credited with developing QCA QCA can be a useful methodology for analyzing multiple cases in complex situations to identify causal links and explain conditions under which changes happen. QCA is especially useful for identifying and understanding cross-case patterns in small- and medium-N data sets (for example, 5–50 cases) with qualitative or mixed methods findings, or both. The application of the methodology requires an in-depth under- standing of circumstances within each case and must be accompanied by a robust theory of change (ToC). When these conditions are present, the results of a QCA can be interpreted and generalized across the case studies and across the population. The Bank Group’s Independent Evaluation Group (IEG) was carrying out a multi- method evaluation of the institution’s performance in supporting RE development in its client (emerging market) countries (box 2.1). QCA was one of the methods selected, to be applied in combination with country case studies, to (i) validate the ToC developed for the evaluation (or make adjustments as necessary), and (ii) iden- tify different pathways through which RE can be scaled up so countries can achieve the clean energy transition. Together with other methods, QCA helped formulate key evaluative conclusions and provided the basis for recommendations that would allow the Bank Group to improve its performance in supporting its clients as they navigate an evolving global RE landscape. 12 Qualitative Comparative Analysis: Renewable Energy | Chapter 2  ultimethod Evaluation of World Bank Group Support to Box 2.1. M Renewable Energy Development The renewable energy (RE) evaluation was designed to assess the World Bank Group’s performance during 2000–17 supporting clients in developing and scaling up their RE resources (World Bank 2020). Specifically, the RE evaluation attempted to answer the following questions: (i) In what ways—and how well—has the Bank Group contributed to addressing the evolving RE needs of its clients? and (ii) What lessons from expe- rience can be identified to strengthen the role of the Bank Group in helping clients achieve their emerging RE goals (that is, the Sustainable Development Goals and the clean energy transition)? Figure B2.1.1. Methods Used in the Renewable Energy Evaluation Structured RE market SLR barriers SLR benefits/ literature review review to RE impacts of RE Portfolio review 546 RE projects/ 245 CAS/CPF CBA/in-depth EVALUATION QUESTIONS and analysis investments ASA/AS strategies hydro review Comparative 9 in-depth QCA causal 19 PPARs case studies country studies analysis Semistructured Public & private World Bank Group staff Other interviews Bank Group clients survey/interviews partners Global expert Delphi panel of global panel experts on RE Source: Independent Evaluation Group. Note: AS = advisory services; ASA = advisory services and analytics; CAS = Country Assistance Strategy; CBA = cost-benefit analysis; CPF = Country Partnership Framework; RE = renewable energy; QCA = qualitative comparative analysis; PPAR = Project Performance Assessment Report; SLR = structured literature review. Various methods were applied to triangulate findings into robust conclusions. These methods included the following: » Structured literature review. An assessment of the evolution of RE markets; a literature review of barriers to developing RE, the energy and environment out� comes of electricity produced from RE, and development impacts. (continued) Independent Evaluation Group | World Bank Group 13  ultimethod Evaluation of World Bank Group Support to Box 2.1. M Renewable Energy Development (cont.) » Portfolio review and analysis. A review of 546 investment projects in the Bank Group’s RE portfolio, select World Bank advisory services and analytics and In� ternational Finance Corporation advisory services from a portfolio of 245 activi� ties, and 19 Project Performance Assessment Reports, and an in-depth review of hydropower. » Semistructured interviews. Interviews with public and private stakeholders, de� velopment partners and key Bank Group managers, and a survey of a purposive sample of Bank Group staff working on RE. » Independent Evaluation Group global expert panel on RE. A structured, iterative Delphi process with a set of global experts who helped identify and prioritize emerging RE opportunities and challenges. » Comparative case studies. In-depth country case studies for a set of purpose� fully selected countries and a qualitative comparative analysis of the case study results to validate the theory of change for the RE evaluation and identify path� ways for scaling up investments in RE to navigate the clean energy transition. The remaining chapters focus specifically on the QCA methodology and how it was applied within IEG’s RE evaluation: ▪ Chapter 3 provides a road map for applying QCA, ▪ Chapter 4 describes how the QCA was designed including the strategic deci- sions made by the evaluation team, the selection of in-depth case studies, and the criteria established for scoring factors and preconditions, ▪ Chapter 5 analyzes the results from the QCA modeling, and ▪ Chapter 6 interprets findings and draws conclusions from the QCA results. 14 Qualitative Comparative Analysis: Renewable Energy | Chapter 2 3 A ROAD MAP FOR APPLYING QUALITATIVE COMPARATIVE ANALYSIS Applying qualitative comparative analysis Identifying causal links among case studies Strengthening the theory of change Undertaking QCA requires several distinct and carefully designed steps that can further strengthen the ToC and identify potential causal (near-necessary, sufficient, or both sufficient) links within this framework, as illustrated in figure 3.1. 1. Developing the detailed ToC. The successful application of QCA begins with the development of a detailed and robust ToC for the subject being evaluated. The ToC should include the change that is being evaluated and key factors that would drive these changes based on the theoretical and experiential understanding of the subject. A sound ToC is essential since it hypothesizes the causal relationship and underlies the interpretation of results. 2. Identifying case studies. The qualitative data that will be used to analyze potential causal links are based on case studies. These studies will need to be sufficiently representative of the key sources of variation under scrutiny (that is, where RE expanded rapidly and where it was less prolific; where the private sector was mobilized heavily and where the public sector had a larger role). The assessments must be detailed and in-depth so that qualitative data can be accurately transposed into quantitative figures to be analyzed through QCA. It is also vital that the criteria applied across the case study assessments are clear and consistent, ensuring that the results are comparable. Figure 3.1. Process for Applying Qualitative Comparative Analysis Interpret the findings and revise the theory of change 1 Develop or use a detailed theory of change 2 6 Analyze the dataset Identify cases of interest 5 3 Score the factors 4 Develop a set of factors 18 Qualitative Comparative Analysis: Renewable Energy | Chapter 3 3. Developing a set of explanatory factors or preconditions. Factors, or preconditions, are the causal (that is, necessary, sufficient, or both) drivers that influence outputs and outcomes. They should be developed to include the factors that influence behavior change within the ToC framework. 4. Scoring the factors or preconditions. Once the ToC is developed and the factors or preconditions are identified and defined, as much as is feasible must be learned about each selected case. This may include the use of a combination of existing data, desk reviews, and site visits, including stakeholder interviews. The information gathered through the case studies is then analyzed to assign scores for the factors or preconditions for each case. This is a vital step in converting qualitative findings into quantitative data, and it should be consistently applied across all cases. The factors or preconditions may be coded as crisp sets (dichotomies scored as 0.0 or 1.0), fuzzy sets (with scores assigned within the 0.0–1.0 interval), or a combination of crisp and fuzzy sets. 5. Analyzing results. At this stage, with all of the groundwork completed, the data are analyzed using specialized software.1 The QCA software is used to analyze patterns among the factors or preconditions and outputs or outcomes.2 By using Boolean algebra, the software provides a rigorous logic-based approach to identifying patterns across multiple case studies and factors or preconditions.3 In addition to confirming causal links, the QCA software also looks for combinations of factors or pathways that can lead to various solutions. A unique aspect of QCA is that it is sensitive to equifinality and will recognize the presence of multiple distinct pathways that lead to the same solution. 6. Interpreting findings. The next step is to interpret the QCA findings. This requires referring back to the case studies and ToC to ensure that the results make sense. This stage may require seeking additional case material or revisiting the ToC, and carrying out steps 1–5 again. In this sense, the QCA can be an iterative process for seeking multiple causal pathways to address a particular problem. Independent Evaluation Group | World Bank Group 19  Cautionary Note on Applying Qualitative Comparative Box 3.1. A Analysis Although qualitative comparative analysis (QCA) presents a rigorous methodology for analyzing drivers of change through a sample of cases that may otherwise be too small for conventional statistical analysis, the approach has its limitations. They include the following: QCA relies on the strength of the underlying theory and understanding of the » subject matter that is being evaluated. » QCA depends on the quality, depth, and consistency of the evidence gathered through case studies. » The scoring of factors or preconditions can require considerable judgment and subject matter expertise, making it subjective if it is not calibrated for consisten� cy through clear application of criteria across case studies. » QCA cannot cope with missing data; therefore, all factors or preconditions must be scored. » QCA is susceptible to varying sample size, which may reduce the diversity of data sets, especially if there are fewer than five cases and potential pathways are not found in the available data. » It can be tempting to apply statistical mind-sets to QCA (such as using sample size for power analysis), but with Boolean algebra and set theory as its founda� tion, QCA must be interpreted through a deterministic lens. As with most complex methodologies, the weaknesses of QCA often result from the way in which it is applied (Simister and Scholz 2016). The following three chapters discuss the way in which QCA was systematically applied in IEG’s RE evaluation. Chapter 4 discusses the strategic decisions and steps taken to prepare for running the QCA model; chapter 5 describes the QCA results and how they were analyzed; and chapter 6 documents how QCA results were inter- preted and triangulated with results from other methodological sources to draw key conclusions in the context of scaling up RE. 20 Qualitative Comparative Analysis: Renewable Energy | Chapter 3 Notes 1 See http://grundrisse.org/qca/. 2 In very simple cases, patterns can be discerned visually. However, complex relationships, a large number of cases, or both require specialized computer software to identify causal patterns accurately. 3 Unlike conventional statistics, which are based on vector algebra, the qualitative compara- tive analysis algorithms use Boolean algebra to identify the relationships between the pre-con- ditions and outputs or outcomes. 4 APPLICATION OF QUALITATIVE COMPARATIVE ANALYSIS: STRATEGIC DECISIONS AND CASE STUDY ANALYSIS Developing a theory of change for renewable energy development Determining major barriers to renewable energy development Identifying country case studies Reviewing the energy evaluation This chapter details the steps the Bank Group’s IEG took to carry out the QCA for the RE evaluation, in line with figure 3.1 in chapter 3. Developing a Theory of Change The first step in carrying out a QCA requires the development of a ToC or the adop- tion of an existing one. The ToC should clearly define the potential causal links that will be tested and refined through the QCA process. The development of the initial ToC itself may need to be an iterative process to ensure that the theoretical concept is robust and based on sound experience. It is vital to include subject matter special- ists who have the necessary expertise to contribute to the development of the ToC. This may also be augmented with desk research (literature review). If an existing ToC is used, it may also require adjustments depending on the specific subject matter being tested through QCA. This will also require inputs from subject matter specialists and possibly coordination with the original preparer of the ToC. The ToC for the RE evaluation was developed with a team of energy sector specialists with experience developing RE, those with expertise in investment projects support- ed by the Bank Group, and evaluators who specialize in multimethod evaluations. The concept was also developed with input from a sector literature review. The re- sulting ToC is illustrated in figure 4.1. Figure 4.1. Theory of Change for Renewable Energy Development A B C D E Inputs Key types of barriers Development Energy and Contribution to broader addressed of RE environment benefits development impacts 1 Inadequate policies Increased Increase electricity Contributes to and regulations RE supply to grid economic growth capacity and 2 Inability to integrate generation Improve access to Supports improve- RE into power system electricity ments to quality of life, Financing Reduce energy including for people Global 3 Insufficient design living in poverty insecurity knowledge and technical standards Helps protect local and Displace Avoid global global environment Partnerships 4 Weak institutional fossil-based pollution and human capacity capacity and generation Avoid local 5 Existence of pollution Assumed significant investment contributions from risks interrelated sectors 6 Constraints to mobilizing financing Source: Independent Evaluation Group. Note: RE = renewable energy. 24 Qualitative Comparative Analysis: Renewable Energy | Chapter 4 The ToC is most easily understood beginning in column C in figure 4.1, which depicts the output as increasing investments in RE technologies. This essentially means the construction and operation of RE power plants and associated infra- structure such as transmission lines to transport electricity to consumers (that is, to load centers). The production of RE-based electricity is also theorized to displace an equivalent amount of alternative generation from fossil fuels, also depicted under column C. Column D identifies the outcomes that result from the increased investments in RE and the displacement of fossil-based power generation illustrated in column C. The increase in investments in and production of RE-based electricity to the power grid supplies consumers who benefit from the use of this electricity. The production of RE-based electricity can also help increase access to those who are currently not connected; and, as indigenous resources, RE-based electricity can reduce energy insecurity that arises from reliance on fossil fuel imports. As RE-based power gen- eration expands, the resulting displacement of fossil fuel–based electricity will help curtail the emission of carbon dioxide (CO2)—a greenhouse gas that contributes to global climate change. As a result, a key outcome of developing RE is the anticipated avoidance of global pollution. Additionally, the replacement of fossil fuels also helps avoid local pollution that can lead to respiratory illnesses and other health impacts. The emission of sulfur oxide, nitrogen oxide, and particulate matter from fossil fuel combustion is also avoided when electricity is produced from RE. Although columns C and D identify a causal relationship between outputs and outcomes, the mobilization of investments to develop RE resources faces major barriers in most countries, which are the preconditions or factors that will be tested through QCA. These precondition barriers, separated into six major categories based on input from subject matter specialists and from review of literature, are illustrated in column B. Theory and experience suggest that these barriers constrict the invest- ment climate for RE and need to be addressed to scale up electricity produced from RE, which in turn, will offset equivalent production from fossil-based technologies. Sufficiently overcoming barriers should mobilize investments in RE-based power generation facilities and associated infrastructure, leading to the energy and envi- ronment outcomes defined in the ToC. The QCA will assess the causal model proposed by the ToC (the three shaded columns in figure 4.1), testing the relationships between addressing precondition barriers (column B) and mobilizing investments in RE and displacing fossil-based generation (column C) and whether the results are electricity supply and global environmental outcomes being realized (column D). However, the overall ToC extends beyond the application of the QCA and conceptualizes two additional aspects. First, it notes that energy and environmental outcomes and outcomes from other development efforts Independent Evaluation Group | World Bank Group 25 are interdependent, contributing together toward an overall set of development impacts, depicted in column E. They include the contribution to economic growth, improvements in the lives of people living in poverty, and the protection of the local and global environments. Second, because the overall RE evaluation is designed to assess the Bank Group’s performance in supporting RE, column A represents the support provided by the institution to client countries to address various barriers to RE (column B). This support includes providing funding support (loans, grants, and guarantees), sharing global knowledge and experiences, and convening partners to also mobilize the same. It should be stressed that the QCA is applied to information related to columns B, C, and D, and not to columns A and E, which are evaluated through other methodological means in the overall RE evaluation (see box 2.1). Identifying Country Case Studies The next step in the QCA process was to identify case studies that would provide the data required for the analysis. For the RE evaluation, the case studies were prepared at the country level to determine the extent to which different countries addressed key precondition barriers to RE development and mobilized investments that would lead to scaling up and securing energy and environmental benefits. The case studies were prepared by sector specialists with extensive experience in RE in developing countries, although not necessarily in the specific countries that they were assigned to evaluate. The country case studies were prepared not solely for the QCA but also to serve multiple purposes in the complex RE evaluation. In fact, the case studies were conceptualized before the QCA, although its structure was based on the overall ToC developed for the RE evaluation. However, the QCA approach was formulated and fully embedded into the case study design before the country-spe- cific work was carried out. Moreover, each case study preparer became familiar with the QCA, was instructed how to tabulate the data and score against the established criteria to adhere to a QCA format, and was informed how these efforts would con- tribute to the QCA. Nine countries were purposefully selected for in-depth case work, as illustrated in map 4.1. The initial aim was to carry out up to 15 country case studies, but the evaluation faced budget constraints because preparing original case studies can be costly. However, undertaking original case work has the benefit of customizing the inquiry to meet the specific needs of the evaluation. Thus, it was decided that the nine selected countries would suffice if they were sufficiently representative and adequately in-depth, since one advantage of QCA is its ability to identify potentially generalizable inferences from a small number of cases. The selected number of cases surpassed five, which is typically considered a minimum for QCA, and with nine cas- 26 Qualitative Comparative Analysis: Renewable Energy | Chapter 4 es the evaluation applied in-depth case knowledge, particularly for understanding outlier cases, to a greater extent than cross-case analysis.1 Countries were selected for the case studies based on balancing several criteria to have an adequate representation of significant sources of variation that would influence expansion of RE. The key criteria applied in country selection included the following: 1. Global geographical variation among emerging markets, especially since the Bank Group has client countries across developing regions, 2. Availability of a diverse array of renewable natural resources across the selected countries, 3. A mix of public and private markets for RE investments that signify the nature of various barriers facing investment mobilization, 4. A blend of larger and smaller markets where the nature and magnitude of scaling up RE would vary, and 5. Bank Group support for RE activities in the country, since the overall multimethod RE evaluation was designed to evaluate the institution’s performance. The Energy Evaluation Map 4.1 shows the nine countries selected for the in-depth case studies in the RE evaluation. Although budget constraints prevented comparative case studies in countries without a Bank Group program, the institution is supporting RE in an expansive 98 developing countries worldwide. Therefore, the selected countries provided the needed diversity of key characteristics noted in the selection criteria. For example, China, India, and Mexico represented larger markets with substantial populations. They are expected to play prominent roles in the clean energy tran- sition. Nicaragua and Sri Lanka represented medium-scale countries, and Jordan and Morocco represented smaller markets. Similarly, the proportion of public and private participation in the energy sector in general and RE in particular also ranged across the selected countries. This included countries such as Jordan and Turkey and that have transitioned from greater public control to more private participation in RE over the evaluated period. The collective nine countries were also endowed with renewable resources that covered all major technologies. Country case studies for China, India, Kenya, Nicaragua, and Sri Lanka were prepared with in-country visits, and desk research was the primary source of information for the remaining four countries. All case studies included interviews with Bank Group and external Independent Evaluation Group | World Bank Group 27 stakeholders to ensure the accuracy of the findings and to incorporate the views of different stakeholders. The selected case studies were adequately representative of important distinguishing characteristics of the RE sector.  ountry Case Studies Selected for Qualitative Comparative Map 4.1. C Analysis in RenewableSource: Independent Evaluation Group. Morocco Turkey Jordan China Mexico India Nicaragua Sri Lanka Kenya Source: Independent Evaluation Group. Developing a Set of Factors or Preconditions To adequately prepare case studies and carry out QCA, it is essential to clearly define the preconditions, and establish criteria for rating them within each country example. The six preconditions or factors applied in the evaluation are barriers to RE develop- ment, as shown in the figure 4.1, which illustrates the ToC. They include the follow- ing barriers that affect the investment climate for RE: (i) inadequate policies and regulations, (ii) inability to integrate RE into power systems, (iii) insufficient design and technical standards, (iv) weak institutional and human capacity, (v) existence of significant investment risks, and (vi) constraints to mobilizing financing. More detailed definitions of the precondition barriers are presented in table 4.1. The case study specialists were tasked with determining the extent to which the country had addressed these precondition barriers to facilitate RE investments. To make this de- termination consistently across all case studies, a clear set of criteria was established. 28 Qualitative Comparative Analysis: Renewable Energy | Chapter 4 Table 4.1. Major Barriers to Renewable Energy Development Barrier Description Inadequate policies Shortcomings in the policy and regulatory environment estab- and regulations lished by governments can hinder public and private investments in RE, especially when they do not provide adequate opportunities or incentives for investors Inability to integrate Inadequate planning, transmission bottlenecks and insufficient RE into power sys- capacity, limited scope for power trading and pooling, and inability tems to store electricity can all result in inflexibility of power systems to smoothly and efficiently integrate RE, especially as the share of variable RE such as wind power and solar photovoltaic increases. Insufficient design Insufficient design and technical standards can make it difficult and technical stan- to consistently develop high-quality RE infrastructure in line with dards industry practices and international standards. Weak institutional In many developing countries, various institutions involved in the and human capacity development of RE do not have sufficient capabilities to undertake new investments or operate ongoing projects. Existence of signif- Even with improved policies and enhanced institutional capa- icant investment bilities, projects may face residual risks: either transitional risks, risks while reforms are being implemented, or permanent risks that are outside the control of developers and which may discourage investments (these include commercial or offtaker risks, political risks, and RE resource risks). Constraints to mobi- In addition to already identified barriers, the typically high up- lizing financing front investments make it more challenging to mobilize financing for RE. This can occur when RE technologies are new to certain markets, at a scale that exceeds the capacity of domestic capital markets, or in small markets where financial institutions are not well developed. Source: Independent Evaluation Group. Note: RE = renewable energy. QCA requires the relationship between a specific case and each precondition to be established based on the extent to which the barrier was addressed. The case study specialists were tasked with determining whether the country they were assessing for a given RE barrier falls “within group” or “outside group.” For the RE evaluation, the extent to which a precondition barrier was addressed was classified into a fuzzy set of four progressively different relationships to a group. Figure 4.2 illustrates these different relationships to the group, which reflected the different degrees to which a country overcame specific barriers to RE development. This required clear definitions for criteria that case study specialists could apply to screen and score the relationship between the circumstances in each assessed country to the respective Independent Evaluation Group | World Bank Group 29 precondition barrier group. If the precondition met the criteria for fully addressing a specific precondition barrier, then it would be considered fully within the group (in group) and would be given a score between 0.75 and 1.0 (the score within the range also indicated the extent to which it successfully addressed the precondition). Similarly, a country that did not adequately address a specific precondition barrier as per the defined criteria would be considered as falling outside the group (out group) and assigned a score between 0 and 0.25. Countries that fell mostly within group or mostly outside the group based on the defined criteria were scored within the ranges of 0.5 to 0.75 and 0.25 to 0.50, respectively. An example is illustrated in box 4.1, and detailed criteria for classifying and scoring each barrier precondition is provided in appendix A. The example in box 4.1 highlights the significant levels of sectoral nu- ance that are required to create criteria for a fuzzy set, which also requires consider- able subject matter expertise to apply and evaluate. Despite the complexity, a fuzzy set of criteria will lead to results and insights from the analysis that are likely to be richer and more robust with greater differentiation of group relationships than the simpler binary option in a crisp set criteria (that is, in or out of group). Figure 4.2. The Relationship between Country Cases with Precondi- tioned Barrier Groups In group Precondition 0.75–1.0 Country Mostly in group Precondition 0.5–0.75 Country Mostly out group Precondition 0.25–0.5 Country Out group Precondition 0–0.25 Country Source: Independent Evaluation Group. 30 Qualitative Comparative Analysis: Renewable Energy | Chapter 4 Box 4.1. Example of Setting Criteria for Scoring Precondition Barriers to RE Development To ensure consistent coding across the nine countries included in the analysis, the specialists carrying out the in-depth case studies were provided specific scoring crite- ria for measuring the degree to which a given precondition barrier has been overcome in the country. Below is an example describing the criteria for scoring the degree to which a country has improved design and technical standards. Similar criteria were defined for each of the six precondition barriers, which are detailed in appendix A. Table B4.1.1. Example Criteria for Scoring Precondition Barriers Industry and international standards for project design, development, and operation for renewable energy (RE) are A established at the country level covering most of the signifi� In group cantly available RE technologies. Similar international and 0.75–1.0 industry standards are adapted for the country context and adopted for major RE equipment. All of these standards re� lated to RE are vigorously enforced during implementation. Industry and international standards for project design, development, and operation for RE are established at the B country level, covering some specific available RE tech� Mostly in group nologies. Similar international and industry standards are 0.51–0.74 adopted for RE equipment but at times are not fully adapted for the country context. All of these standards related to RE are mostly enforced during implementation. There are few standards affecting RE development that are C consistent with industry and international standards; ones Mostly out group that are established are typically loosely enforced; often 0.25–0.49 higher standards and good practices are applied only when required by development partners or project financiers. D There are no established standards in the country for RE Out group that are consistent with industry practices and international 0–0.24 standards for design and implementation of RE or RE equip� ment; application is inconsistent and ad hoc. Source: Independent Evaluation Group. Independent Evaluation Group | World Bank Group 31 Scoring the Factors or Preconditions Equipped with the ToC, the barriers that should be explored, and the scoring crite- ria, the specialists prepared the case studies for the nine countries. The case studies included a variety of information exploring electricity supply and demand in the country; the power sector structure; the institutional setup, including the roles of the public and private sectors; policies and regulations in place; electricity costs and pricing; RE resources availability; the status of RE development; and plans for scal- ing up in the future, among other aspects. The case studies also included the perfor- mance of the Bank Group’s RE portfolio during the evaluation period for the purpos- es of the broader RE evaluation, although this information was not directly used in the QCA. The case study specialists used the in-depth understanding obtained about the country and sector to translate the qualitative assessment of addressing barriers into quantitative scores based on the established common criteria. To ensure that the scoring was correct and consistently applied across countries, the team overseeing the overall evaluation, including a QCA methodology specialist, met with each case study specialist. It was an opportunity to discuss their findings and ensure that their assessments were calibrated to the established standards and scored appropriately. This effort was the first of two important initiatives to calibrate the assessment across countries for greater consistency. After the individ- ual discussions, the scores for each country were transferred to an overall matrix representing all nine cases, in preparation for a subsequent group discussion with all country case study specialists, the team overseeing the RE evaluation, and two QCA methodology specialists (who carried out the analytical work during the next stage). Convening this broader evaluation team led to clarifying edits in the qualita- tive membership statements and revisions to measures of outcome variables related to energy and environmental benefits. It was a crucial step, since a group discus- sion enabled the sharing of information among the case study specialists, further clarification of scoring criteria, and better calibration of scores across countries for improved consistency.2 The final consensus-based recalibrated scores are shown in table 4.2 (and a more detailed data-collection matrix is presented in appendix B). 32 Qualitative Comparative Analysis: Renewable Energy | Chapter 4 Table 4.2. Scores from Country Case Studies B Barriers C Outputs D Outcomes Integra� Institu� Policies tion into Design and tional and Develop and Reg� Power Technical Human Investment Mobilizing Renewable Energy Environment Country ulations Systems Standards Capacity Risks Financing Energy Benefits Benefits China A (0.80) B (0.70) A (0.90) A (0.90) B (0.70) A (0.90) A (0.84) A (0.79) A (0.94) India A (0.85) A (0.80) B (0.55) A (0.90) B (0.60) B (0.70) B (0.56) C (0.36) B (0.61) Independent Evaluation Group | World Bank Group 33 Jordan A (0.80) B (0.70) B (0.70) C (0.40) B (0.60) C (0.40) C (0.48) C (0.27) C (0.41) Kenya B (0.70) D (0.20) A (0.80) C (0.40) B (0.60) A (0.80) C (0.40) C (0.38) C (0.30) Mexico B (0.70) A (0.90) B (0.70) B (0.70) B (0.60) B (0.60) B (0.69) B (0.60) B (0.69) Morocco B (0.55) B (0.60) C (0.40) C (0.40) B (0.60) C (0.40) B (0.54) C (0.37) B (0.58) Nicaragua B (0.70) B (0.60) C (0.40) C (0.40) C (0.40) C (0.40) B (0.64) B (0.57) B (0.60) Sri Lanka B (0.70) B (0.60) B (0.60) B (0.60) B (0.70) B (0.60) B (0.64) B (0.55) B (0.68) Turkey A (0.80) A (0.90) A (0.90) A (0.80) B (0.70) A (0.80) A (0.86) A (0.80) A (0.90) Source: Independent Evaluation Group. Note: The letters in the first row of headings correspond to the columns in the theory of change in figure 4.1. Output reflects the rounding conventions used by the software. The matrix also included scores reflecting country relationships to groups for the outputs and outcomes in the ToC (columns C and D in figure 4.1). These were based on industry-recognized global data sets that include information for each of the case study countries. The installed capacity of RE (column C, “Output,” in figure 4.1) for each country was based on data from the United States Energy Information Agency. The respective scores were determined by comparing the relationship of each case study country situation with that of the global average for developing countries (re- flecting Part II developing countries that borrow from the Bank Group). If the total installed capacity of RE (in megawatts) in a given case study was above 75 percent of the global average for developing countries, then the country was categorized as A (fully in group). If the RE capacity in a case study was in the second quartile, more than at least 50 percent of countries but less than the top 25 percent, it was classi- fied as B. In countries classified as C in the third quartile, the installed capacity of RE was less than at least 50 percent of developing countries globally but higher than the bottom 25 percent. Those countries where the RE installed capacity was less than 25 percent of the global developing country average were in the bottom quartile, classified as D. The installed capacities of RE were adjusted for country populations so that the results would not skew toward the larger-scale countries. More detailed information on the classification criteria for the installed capacity of RE is presented in appendix A. The matrix also included information reflecting the final causal link in the ToC used for the QCA, reflecting the relationship between installed capacity of RE (column C, figure 4.1) and global energy and environmental benefits (column D, figure 4.1). The information for energy and environmental outcomes was sourced from the Interna- tional Energy Agency. The electricity supplied to the power grid from RE resources (kilowatt hours) were adjusted for their populations, and the results classified into percentile groups between 0.0 and 1.0. A higher percentile score reflected a higher level of electricity produced from RE resources. The global environmental bene- fits were measured as follows: CO2 emissions specifically from electricity generated (grams of CO2 per kilowatt hour) were multiplied by the RE share in total electricity production to calculate the counterfactual avoided emissions, which were then ad- justed for the countries’ populations. The avoided emissions figures were also classi- fied into percentile groups between 0.0 and 1.0 based on their ranking across develop- ing countries. A higher percentile score reflected a higher value of avoided emissions. The final matrix included scores reflecting the status for each country at the end of the evaluation period across the key causal links defined in the ToC as follows: (i) the extent to which barriers to RE development were addressed, (ii) the installed capacity of RE based on the investments that were mobilized, and (iii) the resulting electricity supply and global environmental benefits.3 34 Qualitative Comparative Analysis: Renewable Energy | Chapter 4 Notes 1 Most qualitative comparative analysis projects examine between 15 and 50 cases, relying upon both cross-case and within-case analysis to draw their conclusions. Studies with fewer cases, such as this one, typically lean more heavily on in-depth case knowledge, particular- ly for understanding anomalous or aberrant cases, but studies with more cases place greater emphasis on the cross-case regularities. That is because a greater number of cases naturally limits the degree to which a research team may conduct in-depth case studies. Therefore, those qualitative comparative analysis projects with fewer cases tend to be more qualitative and case oriented, and those with more cases tend to be more quantitative and variable oriented. 2 Further intercoder reliability was not tested, as theory may suggest, because it was not deemed to be necessary nor practical in application to have multiple case study specialists review each country case study for consistency. 3 Based on data availability, the installed capacity of renewable energy was from United States Energy Information Agency data in 2015, and the International Energy Agency was the source for the electricity supply data from renewable energy in 2015 and the carbon dioxide emissions from 2014. 5 APPLICATION OF QUALITATIVE COMPARATIVE ANALYSIS: ANALYZING RESULTS Identifying precondition barriers for renewable energy development Analyzing qualitative comparative results Finding pathways for renewable energy development  37 Analyzing results. QCA was used to analyze the relationships among the precondi- tion barriers, the output (RE installed capacity), and the energy and environmental benefits (electricity supplied and CO2 emissions avoided). The technique is im- plemented through specialized software that uses Boolean algebra to identify the superset or subset relationships present within the data matrix (table 4.2).1 Since QCA is a descriptive technique, its results should be evaluated in the context of the theoretical understanding of the subject matter and the in-depth knowledge of the cases to determine causality. The superset or subset relationships identified by QCA are interpreted in terms of necessity, sufficiency, or both. A near-necessary condition is one that must be present all or almost all of the time for the outcome to occur; the absence of the near-neces- sary condition prevents the outcome. A sufficient condition is one that when present causes the outcome to occur all or most of the time. QCA recognizes causal complex- ity (near-necessary and sufficient conditions may be composed of multiple individual conditions), equifinality (there may be multiple paths to an outcome), and imperfect relationships (a condition may be “almost always” necessary or sufficient). QCA provides two goodness-of-fit measures: consistency and coverage. Ranging be- tween 0.0 and 1.0, the metrics report the degree to which a precondition barrier and output or outcome co-occur within the data set. ▪ Coverage provides a measure of empirical relevance, reporting the degree to which a necessary or sufficient condition explains instances of the occur- rence of the outcome. ▪ Consistency is the more crucial measure and reports the strength of the su- perset or subset relationship. When used to test for the presence of a neces- sity relationship, consistency reports the degree to which cases exhibiting the outcome also exhibit the proposed necessary condition. A score of 1.0 indicates that the necessary condition is present whenever the outcome is present. When used to test for sufficiency, a score of 1.0 indicates that whenever the sufficient condition is present, the outcome is present. Scores less than 1.0 indicate imperfect relationships. The standard necessity con- sistency threshold is 0.9, which permits a small degree of inconsistency to accommodate measurement error and imperfect relationships. The stan- dard sufficiency consistency threshold is 0.8, which indicates that a given condition (or combination of conditions) is generally sufficient to produce the outcome. Scores closer to 1.0 indicates stronger relationships. 38 Qualitative Comparative Analysis: Renewable Energy | Chapter 5  bbreviations Used in the Quantitative Comparative Table 5.1. A Analysis Model Precondition Barriers Outputs or Outcomes QCA Results Pol: Re: Ncon: Policy and regulatory Development of RE capacity Necessity consistency Int: Re.d: Ncov: Integration into power Change in RE capacity Necessity coverage systems (2000–16) Imp: Scon: Improvements to design — Sufficiency consistency and technical standards Str: Eng: Scov: Strengthen institutional Energy benefits Sufficiency coverage capacity Eng.d: Change in energy benefits (2000–16) Mit: Env: — Mitigate investment risks Environmental benefits Env.d: Mob: Change in environmental — Mobilize financing benefits (2000–16) Source: Independent Evaluation Group. Note: — = not available; QCA = qualitative comparative analysis; RE = renewable energy. Multiple QCA analyses were run to test the relationships theorized within the initial ToC based on empirical information translated from the in-depth country case stud- ies. The main variation applied was to model the static output or outcomes (2016) and changes in output or outcomes (2000–16). The remainder of this chapter focuses on the final results of the QCA. The first step in analyzing the QCA results was to confirm whether the empirical evidence substantiated the causal relationship that high levels of RE capacity (out- puts) result in greater energy supply and environmental benefits (outcomes). This would validate, for example, whether development of RE capacity is followed by its adequate integration into power systems and operations that produce the electricity that is consumed. It would also confirm whether the development of RE is displac- ing fossil-based generation, which is necessary to secure environmental benefits through avoided emissions. The QCA found high RE capacity to be necessary and al- Independent Evaluation Group | World Bank Group 39 most always sufficient for the realization of high energy produced and high environ- mental benefits (table 5.2, first and second rows, Static). A substantial increase (that is, change) in RE capacity was found to be almost always necessary and almost al- ways sufficient for substantial increases in energy and environmental benefits (table 5.2, third row, Change). There are two aberrant cases here (the rationale is further described in box 5.1): (i) Mexico experienced a substantial increase in RE capacity yet did not realize a substantial increase in energy benefits (violating sufficiency) primarily because of substantially lower levels of precipitation during the year of data selection (2016) to fuel its significant hydropower capacity, and (ii) Kenya, in contrast, realized a substantial increase in energy benefits despite not experienc- ing a substantial increase in RE capacity, primarily because geothermal power was substituted for hydropower that had become less reliable—a swap between two en- vironmentally friendly technologies. Hence, change in RE capacity is almost always necessary and sufficient for change in energy benefits. A substantial increase in RE capacity was found to be near-necessary but not sufficient for a substantial increase in environmental benefits (table 5.2, fourth row, Change).  ummary of Installed Capacity of Renewable Energy in Rela- Table 5.2. S tion to Energy and Environment Benefits Output or Proposition Outcomes Tested Interpretation of Results Ncon Scon Re is necessary and almost Re → Eng 1.0 0.87 Static always sufficient for Eng (2016) Re is necessary and almost Re → Env 1.0 0.91 always sufficient for Env Re.d is almost always neces- Re.d → Eng.d sary and almost always suffi- 0.96 0.92 Change cient for Eng.d (2000–16) Re.d is necessary but not suffi- Re.d → Env.d 1.0 0.76 cient for Env.d Source: Independent Evaluation Group. Note: Eng = energy benefits; Eng.d = change in energy benefits (2000–16); Env = environmental benefits; Env.d = change in environmental benefits (2000–16); Ncon = necessity consistency; Re = development of renewable energy capacity; Re.d = change in renewable energy capacity (2000–16); Scon = sufficien- cy consistency. These results validate that the theorized output-to-outcomes relationship in the ToC are consistent within these country contexts. High RE capacity is near-neces- 40 Qualitative Comparative Analysis: Renewable Energy | Chapter 5 sary and almost always sufficient for the realization of substantial energy and envi- ronmental benefits. Increased RE development is, in general, both near-necessary and sufficient for increased energy benefits, with the previously noted exceptions. Finally, increased RE development was found to be near-necessary but not sufficient for substantially improved environmental benefits. This latter results was expected, since the environmental benefits depend greatly on (i) the types of energy produc- tion being offset with renewable energies, which can contextually vary according to the generation mix in power systems, and (ii) whether there is concurrent growth of traditional (fossil-based) energy production when RE is also expanding.  oteworthy Anomalies in the Qualitative Comparative Analysis Box 5.1. N Results There were some noteworthy anomalies in the qualitative comparative analysis results, which required revisiting the country case studies and relying on case study specialists’ knowledge to better understand. Kenya: Renewable energy (RE) growth is replacing hydropower with geothermal; this offers far fewer environment benefits than replacing fossil fuel–based production, since it is a swap between two RE technologies. Nevertheless, Kenya is unique in that RE is the most economically viable source of electricity, though only 32 percent of the population has access to electricity. Jordan: The country added all of its wind energy capacity in 2016, but the evaluation did not have access to environmental benefits data for 2016; rather, the evaluation used 2015 data, which does not include the additional 2016 capacity. Mexico: RE production went down in 2016 (especially in terms of hydroelectricity pro- duction), likely brought about by precipitation variations that year; this is a data outlier rather than an indication of a systematic trend. This one-year change in RE production, however, substantially influenced calculations of change in RE capacity for the country. The Kenya and Mexico results were based on data anomalies that can be explained, and therefore, should not alter the conclusions drawn from the results. The Kenya results further validate the robustness of the qualitative comparative analysis results, since it is an outlying circumstance that is fully consistent with the theory of change. Source: Independent Evaluation Group. Independent Evaluation Group | World Bank Group 41 The second step in analyzing the QCA results is to assess the relationships among the six precondition barriers identified in the ToC in relation to RE production. The remainder of this section examines the effect of the preconditions on two measures of RE production: (i) high RE production (static measure) and (ii) substantial in- creases in RE capacity per capita between 2000 and 2016 (change or delta measure). Between these two analyses, Jordan and Nicaragua were the two countries with sub- stantive differences in static and change or delta measures, and variances in analyti- cal results can be attributed to these differences. The relationship between the precondition barriers and QCA was used to identify the combinations of preconditions that are near necessary, sufficient, or both for RE output (both static and change or delta measures). As part of the sufficiency analy- sis, QCA can distinguish among core and contributory causal conditions by applying a counterfactual logic. Core conditions are those that counterfactual analysis iden- tifies as fundamental to understanding the occurrence of the outcome. Contributory conditions are not less important than core conditions but serve a different purpose, providing context and nuance for a full account of the outcome’s occurrence.2 The results of the initial analysis were derived using the static measure for RE development (data from 2016). Consistent with the ToC, the analysis indicates that policy and regulation (POL in table 5.3), and integration of RE in power systems(INT in table 5.3) are each, individually or in combination, near-necessary precondi- tion barriers that need to be addressed to develop RE (table 5.3). Improvements to design and technical standards (IMP in table 5.3) also met the threshold of 0.90 for consistency, though this was not observed in Morocco and Nicaragua . Therefore, it was determined that these two precondition barriers should be considered almost near-necessary rather than near-necessary factors for RE development.  ecessity of Precondition Barriers for Renewable Energy De- Table 5.3. N velopment, Static Measure of Renewable Energy, 2016 Precondition Consistency Coverage Policy and regulations (POL) 0.95 0.84 Integration in power systems (INT) 0.93 0.87 Improvements to design and technical 0.93 0.87 standards (IMP) Source: Independent Evaluation Group. 42 Qualitative Comparative Analysis: Renewable Energy | Chapter 5 Continuing to use the static measure of RE, the QCA reveals two overlapping pathways in the case study sample for RE development (table 5.4). The most com- mon pathway (S1 in table 5.4) is the one followed by China, Mexico, Sri Lanka, and Turkey, which was to address all six precondition barriers. Counterfactual analysis identified integration of RE into power systems and mobilization of financing as core explanatory conditions. Morocco and Nicaragua reveal a second pathway (S2 in ta- ble 5.4) to successful RE development, one that does not require all six barriers to be addressed. Both countries successfully addressed their policies and regulations (POL in table 5.4) and integration into power systems (INT in table 5.4); Morocco addi- tionally mitigated investment risks (MIT in table 5.4) to mobilize the private sector. Nicaragua appears unique in that it successfully achieved RE development with a minimum of advantages. Here, the absence of successfully improving design and technical standards emerges as a core condition, which suggests that understanding how these countries overcame the disadvantage of not having successfully addressed this barrier is important to making sense of their success. It may be due to participa- tion of qualified external private developers, who typically adhere to industry norms and international standards. These results must, of course, be interpreted based on theoretical and substantive knowledge of the RE sector and the country contexts.  athways for Renewable Energy Development Based Table 5.4. P on Sufficiency Analysis, Static Measure of Renewable Energy, 2016 Pathway of Pre� Consis� Raw Unique Observed Pathway conditions Barriers tency Coverage Coverage in Cases Addressed China; In� POL*INT*IMP*STR* dia; Mexico; S1 1.00 0.77 0.35 MIT*MOB Sri Lanka; Turkey Morocco; S2 POL*INT*imp*str*mob+ 0.99 0.47 0.06 Nicaragua Source: Independent Evaluation Group. Note: In qualitative comparative analysis, preconditions in all CAPS are those achieved within the observed case studies; * indicates and; and + indicates or. imp = improvements to design and technical standards; int = integration into power systems; mit = mitigate investment risks; mob = mobilize financing; pol = policy and regulatory; str = strengthen institutional capacity. Independent Evaluation Group | World Bank Group 43 The results of the additional subsequent analysis were derived using the change or delta measure for RE development (data from 2000 to 2016). In this analysis, the QCA results indicated a single condition as consistent with near necessity, successfully addressing policy and regulation barriers (POL in table 5.5). However, the integration of RE into power systems (INT in table 5.5) had a near-necessary score of 0.89. In contrast, under the static scenario, four preconditions (POL, INT, STR, and IMP in ta- ble 5.4) were identified as necessary for achieving the static level of RE development.  ecessity of Precondition Barriers for Renewable Energy De- Table 5.5. N velopment, Change or Delta Measure of Renewable Energy, 2000–16 Precondition Consistency Coverage Policy and regulations (POL) 0.93 0.87 Integration in power systems (INT) 0.89 0.91 Improvements to design and technical 0.87 0.89 standards (IMP) Mitigate investment risks (MIT) 0.85 0.95 Strengthen institutional capacity (STR) 0.83 0.93 Mobilize financing (MOB) 0.83 0.91 Source: Independent Evaluation Group. The findings of the sufficiency analysis for the change or delta in RE development are similar to those of the static analysis (table 5.6). Three pathways to success emerge, the most prominent of which (D1 in table 5.6) includes addressing all six barriers and accounts for five countries (China, India, Mexico, Sri Lanka, and Tur- key). The other two pathways provide routes to RE development that do not address all six barriers. Pathway D2 in table 5.6 represents Morocco and Nicaragua, which substantially increased RE capacity despite addressing just two of the six barriers. Pathway D3 in table 5.6 represents Morocco and Jordan and therefore overlaps with pathway D2. These countries were able to address inadequate policies and regula- tions, overcome challenges integrating RE into power system, and cover residual shortcomings to a degree by mitigating investment risks (MIT in table 5.6). Nic- aragua therefore again stands out as unique because it was able to develop its RE capacity by addressing just the two near-necessary precondition barriers. 44 Qualitative Comparative Analysis: Renewable Energy | Chapter 5  athways for Renewable Energy Development Based on Table 5.6. P Sufficiency Analysis, Change or Delta Measure of Renewable Energy, 2000–16 Pathway of Preconditions Consis� Raw Unique Observed in Barriers Addressed tency Coverage Coverage Cases China; India; Mexico; Sri D1 POL*INT*IMP*STR*MIT*MOB 1.00 0.74 0.33 Lanka; Tur� key Morocco; D2 POL*INT*imp*str*mob+ 1.00 0.43 0.03 Nicaragua Jordan; Mo� D3 POL*INT*str*MIT*mob+ 1.00 0.47 0.03 rocco Source: Independent Evaluation Group. Note: In qualitative comparative analysis, preconditions in all CAPS are those achieved within the observed case studies; * indicates and; and + indicates or. Kenya is an outlier case and is omitted from the list of observed cases. See box 5.1 for more details. imp = improvements to design and technical standards; int = integration into power systems; mit = mitigate investment risks; mob = mobilize financing; pol = policy and regulatory; str = strengthen institutional capacity. Two preconditions are shared by all three pathways. The presence of success- ful improvement of the policy and regulatory environment reflects its status as a near-necessary condition, as discussed above. Successful integration of RE into pow- er systems emerges as the single core condition across all three pathways, indicating that successfully addressing this barrier is essential to improving RE capacity. Ad- dressing this barrier, however, is not sufficient by itself, as all pathways also include the successful reduction of at least one other barrier. The successful improvement of the policy and regulatory environment emerges as a common condition across all three pathways, reflecting its status as a necessary condition. The veracity of the findings was checked with additional comparative analysis (for instance, with and without hydropower energy benefits included), and, although not presented here, these findings further solidified confidence in the QCA results. Notes 1 See http://grundrisse.org/qca/. 2 The sufficiency results for the outcomes analysis do not distinguish between core and con- tributory conditions because that analysis included just a single explanatory condition (renew- able energy installed capacity). Independent Evaluation Group | World Bank Group 45 6 APPLICATION OF QUALITATIVE COMPARATIVE ANALYSIS: INTERPRETATION OF FINDINGS AND CONCLUSIONS Assessing barriers in the theory of change Interpreting findings and conclusions Finding alternative pathways for scaling up renewable energy  47 Interpreting Findings The QCA results provided insights based on the case study analysis into the factors that were critical to scaling up RE. These findings, triangulated with results from other methods, enabled conclusions to be drawn that are generalizable to other countries. The QCA validated the ToC that was initially developed for the RE evaluation. It found potential causal links between the six originally identified barriers in the ToC and mobilizing investments in RE (figure 6.1). The same set of barriers were also independently identified as important obstacles to overcome by a global expert panel on RE (the Delphi panel; see figure B2.1.1 in box 2.1), which was convened by IEG for the RE evaluation. Together, the two methods affirmed the Bank Group’s attempts to help clients address all six of these barriers to different degrees through its investment portfolio. Successfully investing in RE capacity was also found to be near necessary and almost always sufficient for the realization of energy and environmental benefits. The example of Kenya confirmed that the environmental benefits are indeed contingent on RE displacing alternatives that are more polluting (that is, fossil-based generation), and that the causal relationship is weaker if the supplanted technologies are cleaner (that is, other renewables). This outlier finding further substantiated the causal relationships within the ToC. Since the QCA to- gether with other methods fully validated the ToC illustrated in figure 3.1, further adjustments to the theory were not necessary in the case of the RE evaluation.  arriers in the Theory of Change with Causal Link to Figure 6.1. B Scaling Up Renewable Energy 1 Inadequate policies and regulations 2 Inability to integrate RE to power system 3 Insufficient design and technical standards 4 Inadequate institutional capacity 5 Significant investment risks 6 Constraints to mobilizing financing Source: Independent Evaluation Group. Note: RE = renewable energy. 48 Qualitative Comparative Analysis: Renewable Energy | Chapter 6 Several countries mobilized investments in RE by adequately addressing all six barriers. The following five countries expanded RE by adequately addressing all of the identified major barriers: China, India, Mexico, Sri Lanka, and Turkey. Although the QCA did not find it strictly necessary to address all six barriers, the approach was identified to be sufficient given the potential causal relationship between each barrier and the expansion of RE capacity. This approach also highlighted the need for coordination among the primary institutions that make up the Bank Group (In- ternational Bank for Reconstruction and Development, International Development Association, IFC, and Multilateral Investment Guarantee Agency). A portfolio review of RE interventions and country case studies undertaken as a part of the overall RE evaluation found the three institutions to have different comparative advantages helping clients address the various precondition barriers. The need for coordination would also extend to external development partners supporting similar complemen- tary interventions, calling for closer partnerships. Improving policies and regulations and better integrating RE into power systems were identified as two near-necessary conditions for scaling up RE, presenting an additional causal pathway. The QCA results indicate that no country successfully scaled up RE without addressing these two critical precondition barriers (illustrat- ed as 1 and 2 marked by the blue arrow in figure 6.2), implying they are priorities within the overall six. The global expert panel on RE convened by IEG for the RE evaluation also independently corroborated the importance of addressing inade- quate policies and integrating RE into power systems by ranking them within the top three challenges facing a future scale-up in RE. In fact, Morocco and Nicaragua only adequately addressed these two essentials barriers yet managed to mobilize invest- ments in RE by enhancing its investment climate for private participation, partly to develop concentrated solar power resources and wind, respectively. Sector analyses and assessment of the Bank Group portfolio indicate that the comparative advan- tage for addressing these two critical precondition barriers primarily rests with the public sector. The portfolio review also confirmed that the public sector arm of the group (International Bank for Reconstruction and Development and International Development Association) was well positioned to provide extensive support to help clients address inadequate policies and regulations (a third of the 217 projects), al- though it had far less experience assisting with integration of RE into power systems (less than 7 percent of interventions in the portfolio). The latter shortcoming was identified in the overall RE evaluation as a key emerging challenge for the institution, since the precondition barrier is expected to increase in importance because of greater penetration of variable RE in energy mixes as wind power and solar PV expand in client countries according to projections in the clean energy transition. Thus, it can be concluded that the two precondition barriers are Independent Evaluation Group | World Bank Group 49 priority areas to address, and, in some cases, successfully reforming them may be sufficient by themselves, at least for initial and immediate scale-up of RE. Figure 6.2.  Two Additional Alternative Pathways for Scaling Up Renew- able Energy 1 Inadequate policies and regulations 2 Inability to integrate RE to power system 3 Insufficient design and technical standards 4 Inadequate institutional capacity 5 Significant investment risks 6 Constraints to mobilizing financing Source: Independent Evaluation Group. Note: RE = renewable energy. The QCA identified a third pathway to scaling up RE, especially through private participation, by taking additional action to mitigate any residual risks when poli- cy and regulatory shortcomings and integration of RE into power systems are also adequately addressed simultaneously. Jordan and Morocco (in the change or delta scenario), which mobilized private participation in developing its RE resources, combined efforts to reform its policy framework and facilitate grid integration with additional measures to mitigate residual risks (illustrated as 1, 2, 5 marked by the green arrow in figure 6.2). For example, the World Bank helped Jordan publicly assess its solar and wind resources (to mitigate RE resource risks), which helped mo- bilize private investors and develop markets for both technologies; IFC assisted with standardizing project documentation to create a common platform for investing in solar PV, which facilitated the development of seven 10–20 megawatt projects. In Morocco, the World Bank funded a standby facility to ensure continued payments by the offtaker for purchases of electricity generated from concentrated solar power, providing assurances to the private developer against commercial and market risks. The RE evaluation found the Bank Group is well placed to mitigate investment risks, since it can mobilize multiple instruments through its three primary institutions— World Bank (International Bank for Reconstruction and Development and Interna- tional Development Association), IFC, and Multilateral Investment Guarantee Agen- 50 Qualitative Comparative Analysis: Renewable Energy | Chapter 6 cy. This specific third pathway identified by the QCA was recently independently highlighted by the International Energy Agency as “the three main challenges” that need to be addressed to “accelerate significantly” RE growth for meeting the long- term goals established under the SDGs and the Paris Agreement. “Renewable electricity growth still needs to accelerate significantly to meet long-term sustainable energy goals. This growth is possible if governments address the three main challenges to faster deployment: policy and regulatory uncertainty; high investment risks in many developing economies; and system integration of wind and solar PV in some countries.” Source: IEA 2019. Conclusion The application of QCA proved to be useful in triangulating findings with other methods within the broader RE evaluation, helping to draw conclusions related to scaling up RE and meeting global goals established by the international community. In addition to helping validate the ToC underlying the entire evaluation, the QCA also helped identify three specific pathways through which precondition barriers could be addressed to overcome challenges facing the successful achievement of the clean energy transition. Addressing all six precondition barriers was one such pathway, given that they were all found to have a potential causal link to mobilizing investments in RE. A second specific pathway was to address inadequate policies and regulations and grid integration challenges—two precondition barriers that were near necessary in all pathways. A third pathway was to simultaneously address the two near-necessary precondition barriers together with mitigating residual risks—a particularly useful approach for mobilizing private investments in RE. Two preconditions are shared by all three pathways. Improvements to the policy and regulatory environment appear to be a necessary condition, as discussed above. Suc- cessful integration of RE into electrical power systems emerges as a core condition across all three pathways, indicating that overcoming this barrier is essential to im- proving RE capacity. Addressing this barrier, however, is not sufficient by itself, since all pathways also include the reduction of at least one other barrier. The successful improvement of the policy and regulatory environment emerges as a common con- dition across all three pathways, reflecting its status as a necessary condition. Independent Evaluation Group | World Bank Group 51 Although the inclusion of QCA added value to the overall RE evaluation, as with all methodologies it should be applied with rigor, awareness of its limitations, and recognition of the real-world trade-offs that need to be made in its design and ap- plication. Several of these limitations are listed here. First, since QCA relies on the strength of underlying theory and the quality and depth of evidence, and is suscepti- ble to varying sample diversity, size, or both, subject matter specialists were needed for developing the ToC, preparing in-depth country cases, and helping contextualize and interpret the results. Second, it was equally important to have methods experts familiar with QCA to help with experiment design, running the model, and inter- preting the results. Third, it was also important to select an adequately represen- tative group of countries within the available budget, ensuring sufficient variation within the sample for key areas of scrutiny. Fourth, another potential limitation involves the reliability and consistency of scor- ing across participants. Although the team met with a case study specialist to ensure that case assessments were correctly calibrated, the use of intercoder reliability testing (in particular, the blind testing of country scores) would have provided a more objective analysis of the scores. This in turn would have provided additional insights on the range and convergence of results relative to each country case study. Although panel discussions on the results may have reached the same conclusions, they are not immune from the various intergroup dynamics that might inadvertently introduce confirmation bias or satisficing. Finally, the analysis would also have ben- efited from a more nuanced discussion of the timing of reforms, since all three of the discussed pathways are likely to shift at different rates over time. Taken together, findings from the QCA provided vital corroboration of evidence from other methodological approaches used in the evaluation of RE investments. Taken alongside a structured literature review, semistructured interviews, a Delphi-moder- ated expert panel, and the review of the Bank Group’s over 500 investment projects, the approach offered a disciplined and collaborative approach to ensure consistency across case studies. The QCA method provided a means of accurately translating qualitative judgments into quantitative information used in the analysis. It helped overcome other traded-off design features such as multicoder reliability tests, which time and budget did not allow for in this particular case. Ultimately, the robustness of the results and the added value provided in triangulating among various evidence streams made the QCA method a valuable analytical tool for evaluative synthesis. 52 Qualitative Comparative Analysis: Renewable Energy | Chapter 6 BIBLIOGRAPHY Cragun, Deborah, Tuya Pal, Susan T. Vadaparampil, Julie Baldwin, Heather Ham- pel, Rita D. DeBate. 2016. “Qualitative Comparative Analysis: A Hybrid Meth- od for Identifying Factors Associated with Program Effectiveness.” Journal of Mixed Methods Research 10 (3): 251–272. IEA (International Energy Agency). 2014. World Energy Outlook 2014. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2014. IEA (International Energy Agency). 2015. World Energy Outlook 2015. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2015. IEA (International Energy Agency). 2016. World Energy Outlook 2016. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2016. IEA (International Energy Agency). 2017. World Energy Outlook 2017. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2017. IEA (International Energy Agency). 2019. World Energy Outlook 2019. Paris: IEA. https://www.iea.org/reports/world-energy-outlook-2019.IPCC 2019, 2019 Refinement to the 2006 IPCC Guidelines for National Greenhouse Gas In- ventories, Calvo Buendia, E., Tanabe, K., Kranjc, A., Baasansuren, J., Fukuda, M., Ngarize S., Osako, A., Pyrozhenko, Y.,Shermanau, P. and Federici, S. (eds). Published: IPCC, Switzerland. IRENA (International Renewable Energy Agency). 2017. Renewable Energy Statis- tics 2017. Abu Dhabi: IRENA. Jayawardena, Migara, Ryan Watkins, and Maurya West Meiers. 2019. “Unlocking Renewable Energy Development to Address Climate Change: Exploration of Causal Links through Qualitative Comparative Analysis (QCA).” Paper pre- sented at the Evaluation 2019 Conference, American Evaluation Association, Minnesota, November 11–16. Legewie, N. 2013. “An Introduction to Applied Data Analysis with Qualitative Comparative Analysis.” Forum Qualitative Sozialforschung/Forum: Qualitative Social Research 14(3). Marx, A., B. Rihoux, and C. Ragin. 2014. “The Origins, Development, and Appli- cation of Qualitative Comparative Analysis: The First 25 Years.” European Political Science Review: EPSR 6 (1): 115. Independent Evaluation Group | World Bank Group 55 Ragin, Charles C. 1987. The Comparative Method: Moving beyond Qualitative and Quantitative Strategies. University of California Press. Rihoux, B. 2006. “Qualitative Comparative Analysis (QCA) and Related Systematic Comparative Methods: Recent Advances and Remaining Challenges for Social Science Research.” International Sociology 21(5): 679–706. Rihoux, B., and C. C. Ragin. 2008. Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques. Sage Publications. Rubinson, Claude, Lasse Gerrits, Roel Rutten, and Thomas Greckhamer. 2019. Avoid- ing Common Errors in QCA: A Short Guide for New Practitioners. COMPASSS Research Network. Schneider, C. Q., and C. Wagemann. 2010. “Standards of Good Practice in Qualita- tive Comparative Analysis (QCA) and Fuzzy-Sets.” Comparative Sociology 9 (3): 397–418. Schneider, C. Q., and C. Wagemann. 2012. Set-Theoretic Methods for the Social Sci- ences: A Guide to Qualitative Comparative Analysis. Cambridge, UK: Cambridge University Press. Scholz, Vera, Amy Kirbyshire, and Nigel Simister. 2016. Shedding Light on Causal Recipes for Development Research Uptake. Oxford, UK: CDNK. Secretariat, R. 2018. Renewables 2018 Global Status Report. Paris: REN12. Shell International Finance B.V. 2018. Annual Report 2018. The Hague: Shell Interna- tional. Simister, Nigel, and Vera Scholz. 2017. Qualitative Comparative Analysis (QCA). Ox- ford, UK: CDNK. Thomann, E., and M. Maggetti. 2020. “Designing Research with Qualitative Compar- ative Analysis (QCA): Approaches, Challenges, and Tools.” Sociological Methods & Research 49 (2): 356–386. United Nations Economic and Social Commission. 2018. “Regional Preparatory Meeting on Energy Issues.” Outcome Document UN E/ESCWA/RFSD/2018/CRP, United Nations Economic and Social Commission, New York. World Bank, IFC (International Finance Corporation), and MIGA (Multilateral In- vestment Guarantee Agency). 2016. World Bank Group Climate Change Action Plan 2016–2020. Washington DC: World Bank. 56 Qualitative Comparative Analysis: Renewable Energy | Bibliography World Bank. 2020. Renewable Energy: Evaluation of the World Bank Group’s Support for Electricity from Renewable Energy Resources, 2000–2017. Thematic and Sector Evaluation. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/197701602792711871/Renew- able-Energy-Evaluation-of-the-World-Bank-Group-s-Support-for-Electrici- ty-from-Renewable-Energy-Resources-2000-2017. APPENDIX A COUNTRY-LEVEL RATINGS CRITERIA FOR QUALITATIVE COMPARATIVE ANALYSIS Preconditions (Barriers to Developing Renewable Energy) The following ratings criteria and ranges are provided for assessing the barriers to renewable energy (RE) at the country level, which will serve as input to the qualitative comparative analysis (QCA). Please apply the criteria to the country you are reviewing and insert the appropriate rating for each barrier (1.0 being the highest and 0.0 being the lowest score). 1. Assessment of investment climate at the end of a specified time period after attempts to improve policy framework for development of RE (mark the appropriate group based on your knowledge of the country and determine a rating from within the range adjusted for deviations in application based on country-specific evidence). Group Rating Range Description A substantial and adequate legal and policy framework is adopted and is being enforced, with policy measures and regulations in place, A 0.75–1.0 including funding, where required, to incentivize the development of RE in line with goals or targets. A significant legal framework is in place but with noticeable shortcomings in the issuance of regulations and enforcement, and despite lack of B 0.5–0.75 overall policy clarity, there are some policy-based incentives available to incentivize some degree of RE development in general or specifically in a targeted RE technology. Although a legal framework exists, policies and C 0.25–0.5 regulations are unclear enforcement is lacking, and there are limited or no incentives to invest in RE. The legal and policy framework is insufficient, with critical shortcomings, and there are inadequate incen- D 0–0.25 tives to mobilize investments in RE in line with goals or targets. Independent Evaluation Group | World Bank Group 59 2. Assessment of adequate consideration for integration of RE into power systems at the end of a specified time period (mark the appropriate group based on your knowledge of the country and determine a fuzzy number rating from within the range adjusted for deviations in application based on country-specific evidence). Group Rating Range Description Substantial and adequate integrated systems planning is well implemented to connect RE power plants into the grid through a well-coordinated transmission network A 0.75–1.0 and generation nodes, and the electricity from RE sources, especially variable (intermittent) ones, is mostly dispatched with little or no curtailment. RE power plants are largely able to physically connect to the system through an adequately planned and implemented transmission network, but there are significant inefficiencies in dispatch of RE-based electricity B 0.5–0.75 because of technical issues such as inadequate grid code, standards for grid-friendly equipment, requirement for priority dispatch, and curtailment of energy, in particular that from variable (intermittent) resources. There is poor integration into the power system planning and implementation, where transmission and generation planning does not take into consideration linking areas of RE resource availability with load centers. There may be a C 0.25–0.5 limited number of RE projects, which may face technical issues in dispatch such as inadequate grid code, standards for grid-friendly equipment, requirement for priority dispatch, and curtailment of energy, in particular that from variable (intermittent) resources. Lack of integrated power planning results in few RE projects, which face challenges for integrating into the power grid, including a lack of transmission access to key areas with known renewable resources, or the inabil� D 0–0.25 ity of what little scattered RE that has been developed unable to dispatch for technical reasons such as lack of an adequate grid code and standards for grid-friendly equipment. 60 Qualitative Comparative Analysis: Renewable Energy | Appendix A 3. Assessment of establishment and adherence to industry and international standards for good practice in design, development, and operation of RE at the end of a specified time period (mark the appropriate group based on your knowledge of the country and determine a rating from within the range adjusted for deviations in application based on country-specific evidence). Group Rating Range Description Industry and international standards for project design, development, and operation for RE are established at the country level, covering most of the significantly available RE technologies. Similar international and industry A 0.75–1.0 standards are adapted for country context and adopted for major RE equipment. All of these standards related to RE are vigorously enforced during implementation (through checks and balances). Industry and international standards for project design, development, and operation for RE are established at the country level, covering some specific available RE B 0.5–0.75 technologies. Similar international and industry standards are adopted for RE equipment but at times are not fully adapted for country context. All of these standards related to RE are mostly enforced during implementation. There are few standards affecting RE development that are consistent with industry and international standards; those that are established are typically loosely enforced; C 0.25–0.5 often higher standards and good practices are applied only when required by development partners or project financiers. There are no established standards in the country for RE that are consistent with industry practices and interna� D 0–0.25 tional standards for design and implementation of RE or RE equipment; application is inconsistent and ad hoc. 4. Assessment of institutional capacity for developing RE at the end of a specified time period (mark the appropriate group based on your knowledge of the country and determine a rating from within the range adjusted for deviations in application based on country-specific evidence)... Independent Evaluation Group | World Bank Group 61 Rating Group Range Description The key institutions in the power sector have adequate capacity to plan and facilitate the uptake of RE; most investors and developers are capable of designing and sustainably implementing RE projects in line with industry and international standards (including meeting fiduciary, environmental, and social requirements). There is sound A 0.75–1.0 awareness of private sector requirements and good capacity to negotiate arrangements for RE independent power producers (IPPs) and public-private partnerships (PPPs). There is very limited reliance on consultants to prepare and develop RE projects, since there is adequate in-house capacity within domestic institutions to meet needs. There is adequate planning capacity, although there is room for further improvements; there are enough capable developers who can fill capacity gaps by acquiring additional support to undertake a majority of the RE capacity that is targeted in national plans. There is 0.5– B awareness of private sector requirements and sufficient capacity with 0.75 consultant support to negotiate arrangements for RE IPPs and PPPs. There is limited reliance on consultants to prepare and develop RE projects, and some in-house capacity within domestic institutions falls short of needs. The institutional capacity and ability to plan is sufficient for initial development of RE, but planning skills are significantly in short supply for integrating larger volumes of RE (especially variable [intermittent] resources) and limited qualifications across a larger number of key developers and investors needed for scaling up sector development in line with RE development goals. There is limited awareness of 0.25– C private sector requirements, and consultant assistance is required 0.5 to negotiate arrangements for RE IPPs and PPPs. There is significant reliance on consultants to prepare and develop RE projects; the very limited in-house capacity within domestic institutions falls far short of needs. It would take considerable time and continuous support before these institutions are capable of sustainable implementation of RE without additional support. There is are very few qualified developers within the domestic market or interested entrants who have the capacity to sustainably develop RE; there is poor performance in existing RE operations. The key institutions responsible for planning and facilitating RE offtake D 0–0.25 have limited knowledge and experience planning for the integration of RE into power systems. There is poor understanding of private sector requirements and lack of capacity to negotiate arrangements for RE IPPs and PPPs. There is heavy reliance on consultants to pre� pare and develop RE projects. 62 Qualitative Comparative Analysis: Renewable Energy | Appendix A 5. Assessment of efforts to mitigate key risks to development of RE at the end of a specified time period (mark the appropriate group based on your knowledge of the country and determine a rating from within the range adjusted for deviations in application based on country-specific evidence). Rating Group Range Description Major risks faced by RE developers have been systematically and substantially mitigated, with the availability of high-quality resource assessments. Financing and investment risks (offtake, A 0.75–1.0 pricing, timely payments) are at acceptable levels for developers or can be easily hedged. The overall country risk for investments is moderate or low. These are countries with low or modest country risks. RE resource estimates are well developed and of high quality, but there are some unaddressed financing or investment risks B 0.5–0.75 (offtake, pricing, timely payments) that require some financial or other mitigation measures that would provide sufficient assurance to developers for undertaking investments. Although RE resource availability is known, there are substantial financing and investment risks and country-level risks that are a C 0.25–0.5 significant barrier; these require measures such as guarantees, insurance, credit enhancements, and other similar instruments to mobilize investments and meet RE development goals. There are considerable risks to investing in RE, with little invest� ment-grade knowledge about the availability of RE resources. There are high investment and financing risks because of lack of D 0–0.25 credibility with electricity offtakers and nascent or undeveloped financial markets for diversifying risks. There is a high overall country risk faced by investors. Independent Evaluation Group | World Bank Group 63 6. Assessment of leverage and mobilization of financing for RE at the end of a specified time period (mark the appropriate group based on your knowledge of the country and determine a rating from within the range adjusted for deviations in application based on country-specific evidence). Group Rating Range Description There are robust capital markets, including international financing, public funds, or both readily available for A 0.75–1.0 undertaking RE investments, including financiers (especially domestic ones) knowledgeable in carrying out due diligence related to specific RE technologies. There are emerging capital markets where a combination of credit enhancement, and targeted public support B 0.5–0.75 is sufficient for bankability of RE projects and meeting financing needs. Development of capital markets is expected to take considerable time, but there are substantial public C 0.25–0.5 financing efforts to develop RE, including support from development partners, enabling the country to meet some part of its RE development goals. There are weak capital markets and an inability in the domestic market to mobilize financing for RE projects, with limited interest from foreign financiers. There is sig� D 0–0.25 nificant reliance on development aid and concessional financing for the limited RE projects that are undertaken in the country. 64 Qualitative Comparative Analysis: Renewable Energy | Appendix A Output (Investment Leading to Development of Renewable Energy) 7. Assessment of investments in the development of installed generation capacity from renewable sources in a given country at the end of the specified time period (2015) across all developing countries.1 This measure is calculated by dividing the installed generation capacity from renewable sources by each country’s population at the end of the period (2015), and classifying the results in percentile groups between 0.0 (0 percent) and 1.0 (100 percent). A higher mark means a higher value of installed RE capacity. Group Rating Description 0.875 There is a substantial level of installed RE capacity A (midpoint of first (per capita) development (top quartile—better than 75 or top quartile) percent of developing countries) There is well-integrated installed RE capacity (per 0.625 capita), but below the top performers (second quartile— B (midpoint of better than at least 50 percent of countries but worse second quartile) than top 25 percent) There is below-average installed RE capacity (per 0.375 capita), but above the bottom performers (third quartile— C (midpoint of third worse than at least 50 percent of countries, but better quartile) than the bottom 25 percent) 0.125 There is a low level of installed RE capacity (per capita) (midpoint of D development (bottom quartile—worse than 75 percent fourth or bottom of developing countries) quartile) Independent Evaluation Group | World Bank Group 65 Outcomes (Energy and Environment Benefits from RE) 8. Assessment of the energy benefits based on electricity produced from renewable sources in a given country in the final year of the specified time period (2015) across all developing countries.2 This measure is calculated by dividing the electricity production from renewable sources by each country’s population at the end of the period and classifying the results in percentile groups between 0.0 and 1.0. A higher mark means a higher value of electricity production from RE. Group Rating Description 0.875 There is a substantial level of installed RE capacity A (midpoint of first (per capita) development (top quartile—better than 75 or top quartile) percent of developing countries) There is well-integrated installed RE capacity (per 0.625 capita), but below the top performers (second quartile— B (midpoint of better than at least 50 percent of countries but worse second quartile) than top 25 percent) There is below-average installed RE capacity (per 0.375 capita), but above the bottom performers (third quartile— C (midpoint of third worse than at least 50 percent of countries, but better quartile) than the bottom 25 percent) 0.125 There is a low level of installed RE capacity (per capita) (midpoint of D development (bottom quartile—worse than 75 percent fourth or bottom of developing countries) quartile) 66 Qualitative Comparative Analysis: Renewable Energy | Appendix A 9. Assessment of the environmental benefits of carbon dioxide (CO2) that could be reduced because of the amount of electricity produced from renewable sources in the final year of the specified time period (2014) across all developing countries. This measure is based on data for CO2 emissions from electricity generated (grams of CO2 per kilowatt hour)3 at the end of the period multiplied by per capita electricity production from renewable sources, and classifying the results in percentile groups between 0.0 and 1.0 based on their ranking across developing countries. A higher mark means a higher value of avoided emissions. Group Rating Description There are substantially reduced CO2 emissions per capita 0.875 based on electricity produced from renewable sources A (midpoint of first (top quartile—better than 75 percent of developing or top quartile) countries) There are significant levels of reduced CO2 emissions per 0.625 capita based on electricity produced from renewable B (midpoint of sources (second quartile—better than at least 50 percent second quartile) of countries but worse than top 25 percent) There are below-average levels of reduced CO2 0.375 emissions per capita based on electricity produced from C (midpoint of third renewable sources (third quartile—worse than at least quartile) 50 percent of countries, but better than the bottom 25 percent) 0.125 There are low levels of reduced CO2 emissions per (midpoint of capita based on electricity produced from renewable D fourth or bottom sources (bottom quartile—worse than 75 percent of quartile) developing countries) Endnotes 1.  Data for installed capacity is from the United States Energy Information Agency for 2015. Developing countries are those eligible for World Bank Group borrowing (Part II countries). 2.  Data for energy production is from International Energy Agency for 2015. 3.  Data for grams of carbon dioxide per kilowatt hour of electricity is from International Energy Agency for 2014. Independent Evaluation Group | World Bank Group 67 APPENDIX B MATRIX OF CONSOLIDATED PRECONDITION BARRIER SCORES Table B.1. Scores from Country Case Studies Addressing Barriers to Developing Renewable Energy Improvements to Design and Policy and Integration into Technical Strengthen Institu� Mitigate Invest� Mobilize Country Regulatory Power Systems Standards tional Capacity ment Risks Financing Key lines Identify which Identify which inte� Identify which RE Identify which Identify which Identify whether of inquiry policy and reg� gration (connection project design RE development specific invest� funds were ulatory barrier(s) and dispatch) of RE and technical capabilities, at the ments risks out� mobilized, lever� were addressed (especially inter� standards were country, institutional, side the control aged, or both by and indicate how: mittent resources) improved and or project level, were of developers the World Bank » Laws: Energy were addressed indicate how: strengthened to were addressed Group to finance law, electricity and indicate how: » Good industry achieve improved RE and indicate RE projects and law, RE law (1) » Power systems practice for outcomes; indicate how: indicate how: » Policies and planning (G&T) (1) feasibility and whether they were » Enhanced » Grants and regulations » Adequate grid safeguards accomplished through RE resource concessional fi� related to the code (2) studies (1) learning-by-doing assessment (1) nancing mobilized above (2) » Standards for » Enhanced and on-the-job » Financial or to sustain subsidy grid-friendly RE standards for training, knowledge investment schemes (GEF, » Pricing: equipment (3) project design (2) transfer, specialized risks related to CTF, ESMAP/AS� Pass-through, workshops, or expert pricing, offtake, TAE) (1) FIT, sustainable » Strengthening » Development support in following and timely » Mobilize other subsidization of transmission to or introduction areas: payments (that development plans (3) facilitate RE (4) of improved or international » Improve planning is, specific risks partner support » Mandatory » Requirements for RE standards capacity for RE expan� that underlie (Other MDBs, offtake with priority dispatch (countrywide, sion (1) financing con� local banks, DFID, pass-through, with regulation to subsidies, or ensure integrity of institution) (3) » Enhance technical straints) (2) KfW, CAF, other both (4) the system (5) » Other (please capabilities for design » Credit bilateral) (2) list) (4) and implementation of enhancements » Attracted or » Concessioning: » Power trade RE investments (2) for financial in� facilitated private Policies that clari� and pooling that fy approaches to improves RE » Expand capacity to termediaries to financiers for assigning devel� offtake (6) secure financing for extend funding financing RE (3) opment rights for RE (3) to RE (3) » Other (please » Other (please RE (5) list) (7) » Strengthen the gov� » Political risks list) (4) » Other (please ernance, procurement, such as expro� list) (6) and financial manage� priations (4) ment practices and » Other (please capacity (4) list) (5) » Improve the ability to meet environmental and social safeguard needs (5) » Other (please list) (6) China A (0.80) B (0.70) A (0.90) A (0.90) B (0.70) A (0.90) India A (0.85) A (0.80) B (0.55) A (0.90) B (0.60) B (0.70) Jordan A (0.80) B (0.70) B (0.70) C (0.40) B (0.60) C (0.40) Kenya B (0.70) D (0.20) A (0.80) C (0.40) B (0.60) A (0.80) Mexico B (0.70) A (0.90) B (0.70) B (0.70) B (0.60) B (0.60) Morocco B (0.55) B (0.60) C (0.40) C (0.40) B (0.60) C (0.40) Nicaragua B (0.70) B (0.60) C (0.40) C (0.40) C (0.40) C (0.40) Sri Lanka B (0.70) B (0.60) B (0.60) B (0.60) B (0.70) B (0.60) Turkey A (0.80) A (0.90) A (0.90) A (0.80) B (0.70) A (0.80) (continued) Independent Evaluation Group | World Bank Group 69 Table B.1. Scores from Country Case Studies (cont.) Addressing Barriers to Developing Renewable Energy Displacement of Country Development of RE Fossil-Based Alternatives Energy Benefits Environmental Benefits Key lines Identify the development of Identify the displacement of Identify the energy Identify the environmental of inquiry RE due to the project with fossil-based fuels as a result benefits from the project benefits from project, and how the help of the barriers that of RE: and how successfully successfully they were: were addressed: » What fossil-based tech� they were: » Avoided local pollution (SOx, » Total investments made in nology is displaced as result » Increase in access to NOx, PM) (1) RE generation capacity (1) of RE (1) electricity (connections » Avoided global pollution » RE power generation » Is it displacing existing and population) (1) (CO2) (2) capacity (2) or proposed fossil-based » Increase in RE-based » Electricity produced as a capacity? (2) electricity (capacity and result of RE capacity from electricity) (2) project (3) » Reduction in energy insecurity (diversification of generation mix) (3) Score based on reduced CO2 Score based on RE capacity emissions per capita based per capita on electricity produced from renewable sources China A (0.84) — A (0.79) A (0.94) India B (0.56) — C (0.36) B (0.61) Jordan C (0.48) — C (0.27) C (0.41) Kenya C (0.40) — C (0.38) C (0.30) Mexico B (0.69) — B (0.60) B (0.69) Morocco B (0.54) — C (0.37) B (0.58) Nicaragua B (0.64) — B (0.57) B (0.60) Sri Lanka B (0.64) — B (0.55) B (0.68) Turkey A (0.86) — A (0.80) A (0.90) Source: Independent Evaluation Group. Note: ASTAE = Asia Sustainable and Alternative Energy Program; CAF = Development Bank of Latin America (Corporacion Andina de Fomento); CO2 = carbon dioxide; CTF = Clean Technology Fund; DFID = Department for International Development (UK); ESMAP = Energy Sector Management Assistance Program; FIT = feed-in tariff; GEF = Global Environment Facility; G&T = generation and transmission; MDB = multilateral development bank; NOx = nitrogen oxide; PM = particulate matter; RE = renewable energy; SOx = sulfur oxide. Numbers after each item indicate number of items in key lines of inquiry. 70 Qualitative Comparative Analysis: Renewable Energy | Appendix B The World Bank 1818 H Street NW Washington, DC 20433