Doing Business and Country Reforms An Independent Evaluation Group Issues Paper June 29, 2021 Contents Abbreviations ..................................................................................................................................................... v 1. Objectives .......................................................................................................................................................1 2. Introduction ...................................................................................................................................................1 3. Background ....................................................................................................................................................2 4. Stocktaking of Available Evidence.........................................................................................................4 Independent Reviews and Evaluations .................................................................................................4 Mapping the Literature-Based Evidence: Desk and Structured Reviews ................................... 7 Mapping Responses from Expert Interviews..................................................................................... 10 Mapping the Bank Group Doing Business–Informed Portfolio ....................................................11 Doing Business in Evaluated Country Strategies and Private Sector Development Diagnostics ................................................................................................................................................... 13 5. Mapping Doing Business Reforms ...................................................................................................... 14 6. Lines of Inquiry Arising from the Evidence ..................................................................................... 16 Linking Evaluation Tools and Lines of Inquiry ................................................................................. 23 7. Conclusion ................................................................................................................................................... 24 References ........................................................................................................................................................ 27 Boxes Box 6.1. Evaluation Questions from Doing Business and Country Reforms Approach Paper . 17 Figures Figure 3.1. Evolution of Doing Business Indicators ..................................................................................3 Figure 4.1. Identification of Articles by Area in Independent Evaluation Group Desk Literature Review ...............................................................................................................................................8 Figure 4.2. Identification of Articles by Area in Independent Evaluation Group Structured Literature Review ...............................................................................................................................................9 Figure 4.3. Content of Interviews with World Bank Group–Related Expert Practitioners and Users of Doing Business ..................................................................................................................................11 Figure 4.4. Indicators Used to Measure Business Environment Constraints in CPSDs ............. 14 Figure 5.2. Reforms Tracked by Doing Business 2005–20 .................................................................. 15 Figure 6.1. How Doing Business Is Expected to Influence Country Outcomes............................. 17 iii Tables Table 3.1. 2017 Market Share of Doing Business and Other Global Performance Indicators of the Business Environment ...............................................................................................................................3 Table 4.1. Summary of Doing Business–Informed Portfolio Approved FY10–20 ......................... 12 Table 5.1. Country Patterns of Portfolio and Reforms ......................................................................... 15 Table 6.1. Links Between Evidence and Lines of Inquiry ..................................................................... 21 Appendixes Appendix A. Methodological Tools and Evaluation Design .......................................................... 29 Appendix B. Initial Identification of Bank Group Doing Business–Informed Portfolio ......... 38 Appendix C. Summary of Desk-Based Literature Review ............................................................... 46 Appendix D. Analysis of Bank Group Completion and Learning Review of Country Assistance Strategies and Country Partnership Strategies ............................................................ 56 iv Abbreviations CPE Country Program Evaluation CPSD Country Private Sector Diagnostic DB Doing Business DEC Development Economics Unit FY fiscal year GIA Group Internal Audit IEG Independent Evaluation Group IFC International Finance Corporation RAS reimbursable advisory services All dollar amounts are US dollars unless otherwise indicated. v 1. Objectives 1.1 This Issues Paper draws together evidence and initial analysis to propose a foundation and testable lines of inquiry for the Independent Evaluation Group (IEG) evaluation of the relevance and effectiveness of Doing Business (DB) for country reforms. The paper reviews six major sources of evidence on the use and influence of the World Bank Group’s DB indicators and reports and their relevance for client country policy reforms. It then marshals the evidence to formulate six lines of inquiry that will augment evaluation questions laid out in the Approach Paper to guide the evaluation. 2. Introduction 2.1 The Committee on Development Effectiveness of the Bank Group Board of Directors, recognizing the importance of DB to stakeholders and to business environment reforms, requested IEG to assess the way that DB aligns with the Bank Group mandate and a viable development framework for client countries. In response, IEG has committed to examining the relevance and effectiveness of the use of DB indicators in guiding client country business environment reforms—both those supported by the Bank Group and those undertaken without its support. This includes an initial stocktaking of literature and existing evaluative evidence to inform this Issues Paper, to be followed by a focused evaluation to assess DB’s strategic relevance and effectiveness to countries’ reform priorities and to the Bank Group’s strategic agenda. The request from the Committee on Development Effectiveness came just before the August 27, 2020 suspension of the DB report to probe alleged irregularities in the underlying data. 2.2 The August 2020 suspension of the DB report motivated World Bank management to request an assurance review of data integrity to the Bank Group’s Group Internal Audit (GIA) and the launch of an external expert review commissioned by the Development Economics (DEC) unit. GIA’s work examined the data integrity of the production processes related to DB indicators and the DB report, including data collection, data review, and controls related to the end-to-end DB report production life cycle (World Bank 2020a). While unrelated to the specific data irregularities, the World Bank also launched an external review of the DB methodology with the goal of strengthening the product and its usefulness to stakeholders worldwide, with findings expected in mid-2021. DEC (where the team producing DB has been located since 2014) has commissioned a group of independent experts to review many technical aspects of DB, including its selection of topics, the suitability of its case study approach and data- gathering processes, scoring methodologies, and the balance between indicators measuring different regulatory attributes. DEC’s work also includes a special study on 1 the methodology of the total tax contribution rate component of the paying taxes DB indicator. IEG and DEC have established and plan to maintain communication between the IEG team and DEC’s external panel, as well as DEC management, to optimize complementarity. 2.3 IEG’s Approach Paper for its evaluation, Doing Business Indicators and Country Reforms, establishes the objectives, scope, and methods of IEG’s own evaluative work on DB (World Bank 2021). It focuses on the influence of DB on country reforms through IEG’s mandated lens of development effectiveness. It considers questions about the relevance of indicators to the areas they cover, to country reform priorities, and to Bank Group priorities. It also considers questions of effectiveness in the use of DB indicators in achieving improvements in desired outcomes. The Approach Paper describes two sequential outputs: first, this Issues Paper, to identify key issues related to the use and influence of DB indicators and their relevance to country policy reforms by reviewing and taking stock of major sources of evidence; and second, a focused evaluation integrating additional and original evidence sources to respond to the full range of evaluation questions raised in the Approach Paper and in this Issues Paper. 3. Background 3.1 Since 2003, the annual DB project has aimed to provide objective measures of business regulations and their enforcement across a growing number of economies. DB provides comparable, quantitative indicators on business regulations, legal protections, and business processes for 190 economies, which can be tracked over time. The scope of the DB report has grown from 5 to 12 topic areas (figure 3.1), including indicators on the cost and quality of business regulation and on the quality of legal frameworks. Country coverage has expanded from 133 countries in 2003 to 190 in 2021. DB also provides an overall ease of doing business index by aggregating the indicators for 10 areas, which it claims assesses “the absolute level of regulatory performance over time.”1 The aggregate index scores are ranked ordinally to yield an ease of doing business ranking for each country. From the start, DB stated its aims as (i) “motivating reforms through country benchmarking”; (ii) “informing the design of reforms”; (iii) “enriching international initiatives on development effectiveness”; and (iv) “informing theory” (DB 2004, ix-x). 2 Figure 3.1. Evolution of Doing Business Indicators Source: Doing Business website, https://www.doingbusiness.org/. Note: Although Doing Business continues to measure employing workers indicators, they haven’t been ranked or used in the summary ease of doing business ranking since Doing Business 2011. Contracting with government, under pilot, is not yet included in the ease of doing business score. DB = Doing Business. 3.2 DB is recognized as highly influential in business regulatory reform and it is the most used set of indicators on business regulation worldwide. A recent study looking at measures of market share among comparable economic indicator sets suggests that DB is the most popular among global indicators of the business environment based on the number of hits (table 3.1). A recent IEG evaluation, Data for Development IEG, 2018), describes DB as both “influential” and “widely used”; while acknowledging methodological controversies, it finds that “Doing Business attracts unrivaled media and high-level attention.” DB itself has recorded thousands of reforms linked to its indicators and reports that more than 70 countries have formed regulatory reform committees “to ensure the coordination of efforts across agencies” using “the Doing Business indicators as one input to inform their programs for improving the business environment” (World Bank Group 2019a, 29). Table 3.1. 2017 Market Share of Doing Business and Other Global Performance Indicators of the Business Environment, 2019 Indicator Hits (no.) Market Share (%) Ease of doing business index 28,798 65.26 Global Competitiveness Index 7,263 16.46 Heritage Index of Economic Freedom 3,563 8.07 Global Entrepreneurship Monitor 1,901 4.31 Fraser Economic Freedom Index 1,234 2.80 World Competitiveness Rankings 973 2.20 The Enabling Trade Index 272 0.62 Forbes Best Countries for Business 126 0.29 Source: Doshi et al. (2019) attributed to Harvard Berkman Center, "Media Cloud Database," 2017. 3.3 While popular, the DB ratings have also experienced external scrutiny and generated controversy regarding their methodology, accuracy, potential biases, and the 3 way such ratings are used to shape and assess country policy reforms. The IEG evaluation of DB that will investigate the lines of inquiry presented in this Issues Paper is different but complements the GIA and DEC assessments described in the introduction. 4. Stocktaking of Available Evidence 4.1 To validate and enrich the evaluation work described in the Approach Paper, and to identify testable lines of inquiry about the relevance and effectiveness of DB indicators on country policy reforms, the evaluation team took stock of available evidence from six sources: • Past independent evaluations and reviews of DB; • A desk literature review and structured map of literature-based evidence regarding DB; • Interviews with Bank Group experts; • A sample-based mapping of the DB-related portfolio of the Bank Group (carried out as part of the Approach Paper) between fiscal year (FY)10 and FY20; • A preliminary review of World Bank country strategies previously evaluated by IEG; and • A mapping of recorded DB reforms from FY05–20. A summary of key findings from each of these sources is presented in the next sections. Independent Reviews and Evaluations 4.2 Over the past 12 years, IEG and the World Bank have either assessed or commissioned external experts to assess the DB indicators and their use in several reviews. The prior work includes IEG’s evaluations Doing Business: Taking the Measure of the World Bank–IFC Doing Business Indicators (World Bank 2008) and Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations (World Bank 2015). In addition, IEG’s 2017 evaluation Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity (World Bank 2017) noted the influence and wide use of Doing Business, and its 2019 evaluation Grow with the Flow: An Independent Evaluation of World Bank Group Support to Facilitating Trade 2006–17 (World Bank 2019b) reviewed the DB trading across borders indicators.2 In the past decade, the World Bank has commissioned two independent reviews of Doing Business: the 2013 Independent Panel Review of the Doing Business Report (World Bank 2013; 4 produced by a panel selected by the president of the Bank Group) and the 2018 external audit On the Integrity of the “Ease of Doing Business” Indicators: Final Report (Morck and Shou 2018). 4.3 These reviews of Doing Business recommended the continuation of the DB indicators and found they had substantial benefits in motivating country policy reform. Their positive findings include the following: • Validity: IEG’s 2008 evaluation concluded that “Overall, the indicators objectively and reliably measure what they set out to measure, with a few qualifications” (World Bank 2008, xvi). • Influence: IEG finds that “the DB indicators have motivated policy makers to discuss and consider business regulation issues. Its active dissemination in easy- to-understand language permits widespread press coverage and generates interest from businesses, nongovernmental organizations, and senior policy makers” (World Bank 2008, xvi). • Usefulness to guide or monitor reforms: The 2015 investment climate evaluation found “within the limits of the DBs indicators, most investment climate interventions produce positive intermediate outcomes in terms of improvement in time, number of procedures, and cost”(World Bank 2015, 67). IEG’s 2019 trade facilitation evaluation found a positive relationship between support for trade facilitation reform from the World Bank and IFC and improved DB indicators for time to import and export (World Bank 2019b). • Continuation: The 2013 Independent Panel Review recommended that “the Doing Business report be retained as an annual flagship report” (World Bank 2013, 4). The 2018 external audit concluded that “the ease of doing business indicators are one of the World Bank’s most important contributions to research and public policy” (Morck and Shou 2018, 3). 4.4 Each study also identified key areas of attention and action required to strengthen the DB indicators or their use: • DB indicators do not capture the full range of legal and regulatory priorities for business environment reform and may miss some binding constraints. IEG’s 2008 evaluation suggested that the DB team should use World Bank analyses such as enterprise surveys and investment climate assessments to drive the choice of new DB indicators and to periodically reassess the set of indicators being used (World Bank 2008). IEG’s 2015 investment climate evaluation found that DB indicators “are incomplete” because they “do not cover all areas of 5 regulation as identified in the best practice list” (World Bank 2015, xxvii) It also recommended that the Bank Group expand “the coverage of current diagnostic tools and integrate them to produce comparable indicators so that these can capture the areas of the business environment not yet covered by existing tools” (World Bank 2015, xxxviii). Specific missing areas included accounting and auditing, contract laws, competition policy, consumer protection, environmental laws, intellectual property (including privacy, copyright, patent, trademark, and unfair business practices), investment policy and promotion, employment law, labor safety and health, and alternative dispute resolution. By contrast, the 2013 Independent Panel Review recommended that DB change its name to reflect its limited focus on business regulations instead of the entire business environment (World Bank 2013). • With some significant exceptions, DB indicators focus on the compliance costs of business regulations: time, money, or procedural steps. All studies of DB have reached this conclusion, although there are significant exceptions where some indicators document the existence of certain laws or regulations and others have been expanded to include qualitative dimensions of laws, regulations, or the institutions applying them. These exceptions include the extent of protection of property, the protection of minority shareholder rights, an index of building quality control, and the range of assets that can be used as collateral. However, the general implication from the DB indicator scoring is that less regulation is better. The 2015 investment climate evaluation proposed the Bank Group develop “separate measures … to capture a wider range of benefits and costs (social, economic, and environmental) if existing regulations are changed” (World Bank 2015, xl). This recommendation was echoed regarding the trading across borders indicators. The 2019 trade facilitation evaluation states: “Many trade regulations are intended to serve socially beneficial purposes such as enhancing public health, safety, and the environment, or reducing informality and corruption. However, insufficient attention has been paid to such objectives” (World Bank 2019b, ix). On paying taxes, IEG’s 2008 evaluation recommended excluding the tax rate from the indicator, stating that it was more a matter of public policy than compliance burden (World Bank 2008). This finding was echoed in the 2013 Independent Panel report (World Bank 2013). • DB indicators do not track reform impacts well. IEG’s 2008 evaluation recommended that the Bank Group trace the impact of DB reforms at the country level (World Bank 2008). The 2013 Independent Panel Review criticized DB’s implied claim that the reforms it promotes cause economic development and growth, stating “correlations do not justify a causal interpretation…. The 6 evidence in favor of specific country reforms is contingent on many auxiliary factors not captured by DB report topics” (World Bank 2013, 2).3 • The DB practice of aggregating indicators is controversial. The 2013 Independent Panel Review of the DB report found that aggregate rankings are challenging because they involve aggregating across topics—a process that explicitly or implicitly involves a value judgment of what is “better” for doing business and how much better it is. It also finds that “small revisions or inaccuracies in primary data can significantly change a country’s rankings” (World Bank 2013, 3). Therefore, it recommended that the World Bank publish the DB report without the overall aggregate rankings (that is, without the ease of doing business ranking). • DB adjustments to its indicators generate discontinuities. The 2018 external audit found that “frequent methodology changes reduce the value of the indicators to researchers, policy makers, and the media” (Morck and Shou 2018, 1). It recommended that “the World Bank may wish to minimize methodology changes except to fix confirmed problems with existing methodology” (Morck and Shou 2018, 28).4 4.5 These findings point to potential limitations of DB in fully and accurately capturing business environment reform priorities in client countries in ways that speak to their relevance. Mapping the Literature-Based Evidence: Desk and Structured Reviews 4.6 IEG conducted a desk review of 426 articles from 100 leading academic journals as an initial stocktaking of literature-based evidence.5 The DB literature database contains a research literature with abundant coverage of a few DB indicators (starting a business, protecting minority investors, and trading across borders) and much sparser coverage of others (for example, dealing with construction permits and getting electricity; figure 4.1). The desk literature review did not assess the methodological rigor of the published articles. The review found areas with strong evidence on the association of selected indicators with development and economic outcomes for businesses and economies. Yet there were substantial gaps in relation to some indicators, such as construction permits (appendix C). Much of this literature found significant associations of DB indicator measures with outcomes, without necessarily establishing causation. 4.7 The desk review found evidence from three or more academic articles regarding the association of five DB indicators with specific economic outcomes: 7 • Starting a business: Higher entry costs or more steps or documents (as measured by DB) are associated with less firm creation, growth, or profitability. • Employing workers: Developing countries with rigid employment laws tend to have larger informal sectors and higher unemployment, especially among young workers. • Getting credit: Stronger legal rights are associated with more lending, more financial activity, and lower interest rates. • Paying taxes: Higher corporate income tax rates are associated with lower levels of entrepreneurship (new firm formation) and investment. • Trading across borders: Better trade facilitation measured by DB (and the Logistics Performance Index) shows a correlation with increased trade flows. Figure 4.1. Identification of Articles by Area in Independent Evaluation Group Desk Literature Review Source: World Bank Doing Business database. 4.8 Other findings are supported in well-evidenced individual academic articles, but only the five specific findings above are supported by three or more articles. Some claims about the relationship of DB indicators to outcomes are referenced in Doing Business but are not evidenced in the literature. For example, several articles tested the expected significant relationship between the dealing with construction permits indicator and real economic outcomes, but the review found no article establishing a significant relationship. Some other links of indicators to relevant outcomes, such as the link of reduced business entry costs to enhanced productivity or employment, have yet to be firmly established. 8 4.9 In addition to the desk literature review described above, IEG has launched a more in-depth structured literature review to capture a broader range of sources with more systematic selection criteria in terms of materiality and methodology of the evidence to consider. According to its selection criteria, the in-depth structured review only considers evidence on reforms and interventions related to the DB indicators; it focuses on quantitative, empirical studies addressing reforms effects; it includes only studies where allocation to intervention and control groups is random or selection bias has been addressed by design (including natural experiments and nonrandomized designs); and it excludes simple before-and-after comparisons and simulation and forecast models. This is consistent with norms for leading practitioners of systematic review and structured literature reviews, including 3ie and the Campbell Collaboration.6 The in-depth structured review covered more than 8,000 records, including all those in the prior review, and tapped a 3ie database of more than 150,000 records screened from academic databases in health, economics, public policy, and the social sciences, as well as the Google Scholar citation-tracking system. To date, this structured review has identified 150 studies offering rigorous evidence. A first mapping of these studies points to some concentration of the evidence in relation to indicators such as starting a business, getting credit, trading across borders, and resolving insolvency (figure 4.2). By implication, it also points to potential gaps where the relationship of indicators to outcomes has little or no rigorous supporting evidence. The synthesis from this review will inform IEG’s final evaluation. Figure 4.2. Identification of Articles by Area in Independent Evaluation Group Structured Literature Review Source: Independent Evaluation Group. Note: Review as of February 2021. 4.10 The emerging findings from these literature reviews draw attention to the extent and quality of supporting evidence related to the effectiveness of business reforms in the areas covered by the DB indicators. The findings suggest a line of inquiry about the gaps 9 and mixed evidence in some of the business areas covered by the DB indicators. They also invite consideration about how the Bank Group receives and uses information from the academic research literature to strengthen the design or use of its DB indicators. Mapping Responses from Expert Interviews 4.11 IEG carried out semistructured interviews with 18 Bank Group–related expert practitioners and users of the DB indicators. The set of interviewees included Bank Group staff involved in the oversight and production of DB indicators, in their application to country reforms, and in business environment reform activities more broadly. They ranged in rank from current or former economist to vice president and included five current or former directors or senior directors and six current or former managers or senior managers. 4.12 The objective of the interviews was to gather candid feedback based on practical firsthand experience, anecdotal evidence, and perceptions on five key core aspects of the relevance and use of DB indicators for country policy reforms. These five aspects were the following: • Construct validity (the degree to which DB indicators measure what they claim to be measuring); • Relevance to country priorities; • Value to initiate a policy reform dialogue with client countries; • Value to identify and guide policy reforms that address binding constraints; and • Value to monitor and evaluate country policy reforms. 4.13 While the interviews were rich in qualitative detail, figure 4.3 provides a quantitative summary of the favorable and unfavorable evaluative statements shared in the interviews across the five key aspects listed above. Elements of consensus were striking, such as the overwhelmingly favorable responses on the value of the DB indicators and rankings to initiate policy dialogue with governments on business environment reforms. In some cases, this opening led more motivated clients to establish broader frameworks for business environment regulatory and policy reform, extending well beyond the scope of the indicators. By contrast, responses were mostly unfavorable on the value of the DB indicators to monitor and evaluate reforms, with strong cautions on the need to rely on alternative or supplementary evidence. Other aspects had mixed and nuanced discussions that were difficult to quantify, such as discussions of divergent country experiences regarding the client country motivation to engage with DB—for example, some clients took on policy reforms to enhance development outcomes while 10 others “gamed” the DB indicators to gain status and rank. Distinctions were also made according to country institutional capacity. A range of country-specific factors (cultural, political, and socioeconomic) were raised regarding how DB was used in reforms. In addition, several interviewees commented on the relationship of the indicators to academic and empirical research. Therefore, the responses feed hypotheses about both the relevance and effectiveness of the indicators regarding country reforms. Figure 4.3. Content of Interviews with World Bank Group–Related Expert Practitioners and Users of Doing Business Source: Independent Evaluation Group. Note: Independent Evaluation Group count based on frequency of positive and negative responses about Doing Business regarding each characteristic. More than one response was possible in each interview. Interviews conducted virtually in January and February 2021. Mapping the Bank Group DB-Informed Portfolio 4.14 The Bank Group uses DB in several activities: (i) country strategy and policy dialogue; (ii) operations (both financing and advisory); and (iii) research and global knowledge sharing. DB is used by the Bank Group in country strategies and policy dialogue to describe business-enabling conditions in a country, or to motivate future operations in priority areas identified in Country Program Frameworks, for example, or in diagnosing private sector development limitations in Country Private Sector Diagnostics (CPSDs). Alternatively, DB is used sometimes to justify Bank Group lending operations or advisory services, to influence their design, to target certain projects, or to assess the progress of some projects against agreed reform objectives. Finally, the Bank Group also uses DB indicators for research and global knowledge sharing purposes, both as a public good and as the subject of research and knowledge-based activities. 11 4.15 IEG has conducted an initial identification of the investment and advisory projects explicitly informed by DB (table 4.1) using the criteria that (i) the projects have one or more DB indicators in either their objectives or monitoring indicators; or (ii) the project documents use DB indicators to justify the project. Based on the initial identification and review of projects, selecting projects in the operational portfolio using DB, IEG estimates that DB-informed projects make up more than 3 percent of the overall Bank Group portfolio. This comprises a minority of all business environment reform projects in the portfolio. Based on the portfolio review, IEG estimates that there were 683 World Bank and IFC operations over the 10-year period (table 4.1).7 Table 4.1. Summary of Doing Business–Informed Portfolio Approved FY10–20 Projects Interventions Volume/Commitments Institution (no.) (%) (no.) (%) (US$, millions) (%) World Bank lending 269 39 517 43 14,853 96 Subtotal 269 39 517 43 14,853 96 World Bank ASA 165 24 173 14 379 2 World Bank RAS 61 9 86 7 21 0 IFC-AS 188 28 432 36 298 2 Subtotal 414 61 691 57 698 4 Total 683 100 1,208 100 15,550 100 Source: Independent Evaluation Group portfolio review analysis. Note: Figures are projected based on population and sample sizes. Sample was derived from combination of manual and supervised machine learning reviews. Volume/commitment/funds managed information was identified or estimated according to what is allocated to Doing Business interventions. If not explicit in the project document, the amount was estimated based on the number of components, subcomponents, or activities. ASA = advisory services and analytics; IFC- AS = International Finance Corporation advisory services; RAS = reimbursable advisory services. 4.16 The DB-informed portfolio covers advisory and analytic work (table 4.1), delivered as World Bank advisory services and analytics (24 percent of the portfolio), World Bank reimbursable advisory services (RAS; 9 percent), or IFC advisory services (28 percent). It also includes World Bank financing projects providing loans, credits, or grants (39 percent). Based on the portfolio review analysis, 38 percent of DB projects are estimated to be in lower-middle-income countries, whereas 29 percent are in low-income countries, 25 percent are in upper-middle-income countries, and 8 percent are in high- income countries. Most of the assistance to high-income countries is through RAS where the World Bank is reimbursed for the costs of delivering advisory, analytical, or implementation support services. Based on the portfolio analysis, IEG estimates that during FY10–20, the Bank Group provided DB lending support to 97 countries and advisory services and analytics to 127 countries. 4.17 Assessing the relevance and effectiveness of Bank Group interventions informed by DB, across countries and product lines, can contribute to understanding the role that DB plays in supporting client country business environment reforms. This portfolio data 12 may allow IEG to answer questions about the patterns of engagement and relate the findings to country-level indicators that may be useful in explaining those patterns. Combining it with information from projects that were previously evaluated will permit IEG to form a view about the characteristics of internal factors (such as project design, supervision, sequencing, and complementarity) and external factors (such as client country characteristics) that contribute to the success of a DB-informed intervention. Mapping the full portfolio will help inform the selection of case study countries and DB indicator candidates for deep dives in the focused evaluation, as discussed below. Doing Business in Evaluated Country Strategies and Private Sector Development Diagnostics 4.18 The Bank Group also uses DB indicators in its country strategies and diagnostics. IEG reviewed 61 country Completion and Learning Report Reviews with their underlying Country Partnership Framework or Country Assistance Strategy documents. A 64 percent share of these strategy documents substantially referenced DB or proposed a DB-related work program. Planned leading interventions included improving business laws and regulations (38 percent), streamlining procedures (34 percent), using electronic or automated systems (15 percent), and conducting diagnostics (11 percent). Examples include policy development and regulatory streamlining in the Philippines (Country Program Evaluation [CPE], FY09–18); policy dialogue and advice in Mexico (CPE FY08– 17); and policy and project support in in Tunisia (CPE FY05–13) and Rwanda (CPE FY09–17; see annex D.1). The frequent use of DB invites questions on whether the guidance provided is well-targeted and the support envisioned dovetails with the identified issues (relevance). Heavy reliance on RAS in upper-middle and high-income countries and the strong use of lending in lower-middle-income countries suggest a tailoring of DB-informed programs to different client profiles. The interaction of Bank Group supply and client demand can best be explored in case studies. 4.19 IEG reviewed 18 CPSDs, intended as inputs to Systematic Country Diagnostics and as strategic guidance for private sector development engagement of IFC and World Bank. All 18 CPSDs used DB indicators, and 14 of them did so substantially. Yet these diagnostics used multiple indicators to draw their conclusions and formulate recommendations, including both World Bank and external sources (figure 4.4). These assessments point to the potential value of using complementary instruments to develop a deeper and potentially more comprehensive understanding of leading priorities for business environment reform. 13 Figure 4.4. Indicators Used to Measure Business Environment Constraints in CPSDs Source: Independent Evaluation Group review of 18 published CPSDs. Note: CPSD = Country Private Sector Diagnostic; DB = Doing Business; IMF = International Monetary Fund; WEF = World Economic Forum. 5. Mapping Doing Business Reforms DB tracks reforms across the topic areas measured by its indicators. For example, the starting a business indicator set tracks changes related to the ease of incorporating performance in an indicator exceeding a defined threshold8 are classified as reforms and reported by DB. Changes can be classified as positive or negative reforms depending on whether they make it easier or more difficult to do business. Since 2010, DBhas tracked 2,960 business regulatory reforms across the 184 economies where reforms were measured. Among the reforms that DB measures, the most common relate to starting a business; paying taxes; getting credit; registering property; and trading across borders. (figure 5.2). The reforms are notable in the concentration of reforms in a few areas. For example, there have been as many recorded reforms in starting a business as in the bottom three categories combined. DB also celebrates ‘top reformers’—those countries that have achieved the greatest number of measured reforms—partly to encourage competition among countries to achieve such DB reforms. During the evaluation period (DB10–2020), Rwanda, Kazakhstan, Indonesia, United Arab Emirates, India, Ukraine, Azerbaijan, Russian Federation, Uzbekistan, Armenia, Brunei Darussalam, Kenya, and Morocco have been credited with the highest total number of positive national reforms.9 14 Figure 5.1. Reforms Tracked by Doing Business 2010 – 20 Source: Doing Business Reform Database 2020. Note: Includes only positive reforms, excludes subnational reforms. Table 5.1. Country Patterns of Portfolio and Reforms Countries with Large Portfolios Countries with Large Portfolios Countries with Small Portfolios and Many Reforms and Few Reforms and Many Reforms Côte D'Ivoire Bangladesh Armenia Georgia Brazil Azerbaijan Indonesia Cambodia Belarus India Colombia Brunei Darussalam Kazakhstan Ethiopia Mauritius Kenya Jordan Morocco Rwanda Liberia North Macedonia Ukraine Mozambique Russian Federation Vietnam Nigeria Senegal Pakistan Togo Tanzania United Arab Emirates Uzbekistan Source: Doing Business Reform Database 2020. Note: Includes only positive reforms, excludes subnational reforms. Latvia and Lithuania make up an additional category: countries with no identified portfolio and many reforms. 5.1 Sharp deviations between the number of Bank Group DB-informed interventions in a country and the number of reforms captured in DB provide an interesting entry 15 point to select country case studies. Different reform levels in a country may be related to (i) different initial conditions and reform needs; (ii) different levels of motivation for reform; and (iii) different capacity and readiness to achieve measurable reforms. Differences in the size of the Bank Group DB-informed portfolio may be linked to those factors as well. For example, a country that is rebuilding infrastructure after a conflict or natural disaster may not have room in its aid priorities for DB reforms. A country with little reform motivation (due to powerful and resistant vested interests) may record few reforms despite a significant Bank Group DB-informed portfolio. A reform-oriented country with high capacity (or resources to hire capacity) may introduce reforms without Bank Group support. Table 3 lists three categories of countries with potential to be case studies: (i) countries with large DB-informed portfolios and many measured reforms; (ii) countries with large DB-informed portfolios and few measured reforms; and (iii) countries with small (or no) DB-informed portfolio and many measured reforms. These variations in country experiences point to the importance of country motivation and capacity, as well as other characteristics that may influence both the demand for reform and the demand for Bank Group support. 6. Lines of Inquiry Arising from the Evidence 6.1 The Approach Paper for the evaluation offers a theory of change mapping the channels by which the use of DB indicators is expected to influence a country’s legal, regulatory, and procedural reforms. These channels of influence involve several stakeholders, such as the Bank Group and other donors; government leaders and policy makers; domestic civil society and the press; and domestic and international investors. DB influences country reforms through several channels, informing and motivating these stakeholders (figure 6.1). The theory of change hypothesizes that, as a result of the information and influence created through DB, governments introduce reform measures whose implementation progress is then reflected in subsequent DB scores.10 The reforms are expected to reduce the time and cost of compliance with business regulations and improve the quality of regulatory and administrative procedures. This, in turn, is expected to improve market functioning—business entry, exit, and operation—and reduce transaction costs to enterprises arising from regulatory compliance. As a result, there is an expectation of improvements in private enterprise growth in terms of new investment, employment, and business volume turnover. There is no claim that DB- induced reforms are a panacea for social and economic development. However, since its outset, DB has stated the link between the development outcomes associated with the business regulatory reforms it motivates and likely poverty reduction and shared prosperity outcomes.11 16 Figure 6.1. How Doing Business Is Expected to Influence Country Outcomes Sources: Independent Evaluation Group; Doing Business website, https://www.doingbusiness.org/; review of World Bank Group literature and Kelley and Simmons (2020). 6.2 Consistent with this theory of change, the Approach Paper identifies questions on the relevance and effectiveness of DB indicators regarding country policy reforms, which the evaluation will address (World Bank 2021). These evaluation questions apply to country reforms informed or guided by DB, both those initiated by governments on their own without Bank Group support, and those supported by Bank Group projects. They focus on “doing the right things” (relevance) and “doing things right” (effectiveness; see box 6.1). Box 6.1. Evaluation Questions from Doing Business and Country Reforms Approach Paper i. Is Doing Business “doing the right things”? What is the relevance of Doing Business indicators to the following: i. The business environment priorities of the client countries? ii. The substantive dimensions within each indicator’s focus area? iii. World Bank Group strategic and operational priorities? ii. Are the Bank Group and country client stakeholders “doing things right” when using Doing Business indicators to achieve better development outcomes? i. Is Doing Business reform achieving desired outcomes? ii. Are clients and the Bank Group using Doing Business to achieve improved outcomes for business environment reform (for example, entry, investment, and employment) in line with the country’s development policy priorities (for example, public safety, equity, and environment)? Source: Independent Evaluation Group. 17 6.3 While the evaluation will be designed to answer these questions, a central purpose of this Issues Paper is to use the evidence reviewed to identify lines of inquiry and associated hypotheses to guide the evaluation work. Table A.3 in appendix A elaborates the implicit hypotheses embedded in the lines of inquiry detailed below, as well as the links among the lines of inquiry and the evaluation questions. Testing these hypotheses should shed light on the relevance and effectiveness of DB to country reforms, and the factors that shape them, contributing answers to the evaluation questions. 6.4 The review of the evidence gathered through all the sources described above suggests six lines of inquiry to be explored in the evaluation: 1. Whether all DB indicators are relevant to constraints influencing business dynamics and economic contribution of policy reforms. • This issue captures several dimensions of relevance. First, consistent with earlier evaluation work, IEG’s preliminary review of the literature and the observations of interviewed experts suggest that the extent to which indicators capture economically important phenomena varies by indicator area, and the priority of these areas depends substantially on country context. The evaluation will examine the claim to relevance of the indicators through the case studies and country strategy work, structured literature review, stakeholder interviews, and statistical and econometric analysis. 2. Whether (i) client institutional capacity and ability (high versus low) to tailor the use of DB indicators to its development priorities and framework; and (ii) client motivation (for example, achieve policy reform versus improve DB ranking) are key influences on the use and appropriateness of DB indicators. • Interviews of Bank Group experts underscored the important variations among clients, especially in relation to the depth and sophistication with which they can address business environment reform and the motivations driving their engagement in such reform work. The divergent relationships between the level of Bank Group reform support and country DB reform outcomes suggest conditions for success may vary across client countries. o Institutional capacity. While some countries have requested assistance from the Bank Group to launch basic policy reforms, others with sophisticated institutional capacity and commitment have developed their own committees or mechanisms to undertake policy reforms, often involving high-level coordination functions, subnational initiatives, and adaptation and innovation on the indicators and data used to guide reforms. 18 o Engagement motivation. While some countries’ engagement focuses primarily on developmental goals such as how to strengthen productivity or employment growth, others are more narrowly focused on improving their ranking in the DB indicators. • Further, the mapping of the Bank Group portfolio and review of Bank Group country strategies suggest varying patterns of DB-related engagement across income level and other country characteristics, reflecting different mixes of client demand for policy reforms and Bank Group supply of lending, advisory support, or both. In addition, the comparison of DB reforms and portfolio across countries indicate important national differences in the relationship between the level of Bank Group engagement and the number of country reforms undertaken. • Given such variations in experience, the evaluation team will consider the potential to increase the number of case studies to cover a broader variety of country conditions. It will also consider interviewing a broad regional distribution of stakeholders. Finally, the portfolio of Bank Group DB-informed activities can be examined considering variables that capture some of the institutional dimensions of client countries, such as the initial quality of rule of law or regulatory governance. 3. Whether DB indicators are useful to (i) draw attention and resources to business environment reform in areas they measure directly; (ii) draw attention and resources to business environment reform beyond the areas they measure directly; (iii) correctly identify and order priority reform areas; and (iv) monitor and evaluate reform implementation. • Analysis of the Bank Group expert interview responses suggests that there is greater consensus about the value of DB indicators in some functions than in others. For example, there are strong differences of experience between the utility of DB for motivating reform and its utility in monitoring reform progress. DB is praised not only for the reforms it measures but for raising the priority of business environment reform in the policy agenda. However, views are far more mixed about DB as a source of identifying specific reform measures and less favorable about their utility to monitor and evaluate reform progress. Questions about the use of DB to track reforms were also acknowledged in IEG’s 2008 DB evaluation and in the 2013 Independent Panel Review (World Bank 2008, 2013). Thus, evidence from projects and country experience must be considered to assess the relevance and effectiveness of the indicators in these different functions. 19 4. Whether complementary data and guidance enhance the relevance of DB indicators to business environment reforms and the ability to follow with complementary or deepening reforms. • Interview responses of Bank Group experts mostly reflected the view that DB indicators should be supplemented by other data and analysis. The review of the CPSDs shows that these in-depth private sector development diagnostics balance DB data with a variety of other sources of data and analysis. Both interviews and diagnostics suggest that supplementary information and guidance are required to ensure that policy reforms linked to DB indicators (i) are well-targeted to local business and policy priorities; (ii) address a complete range of issues and institutions within a given business environment area; and (iii) adapt to local capacity and commitment. Depending on the legal, regulatory, or institutional reform area and type, remedial actions vary widely in terms of their complexity and the time required to fully implement them, with some requiring long-term commitment, engagement, and application of expertise. Multiple interview respondents remarked that supplementary information is vital to tracking and evaluating reforms and to developing second-generation activities to deepen and sustain initial measures. Country case study experiences in the main evaluation can shed substantial light on what evidence has been used in practice. 5. Whether the feedback loops between available empirical evidence and DB indicator design and use are robust. • The literature reviews point to areas where DB indicators are clearly associated with developmentally important outcomes and areas where there are serious evidence gaps or mixed evidence on the relationship of individual DB indicators to important outcomes. This raises questions not only about the construct validity of some indicators but also about the link between ongoing research and the design and application of the indicators. Initial expert interviews evoked some concerns about the strength of the relationship of the indicators to academic and empirical research. Thus, the feedback paths from empirical evidence about the relevance and effectiveness of DB-guided reforms emerge as an important question. Further, with changes in the global economy and production processes, the relevant elements of the regulatory, legal, and institutional environment may evolve over time. The ability of indicators to reflect learning and changes in real-world circumstances is likely to influence their robustness and sustainability. Counterbalancing this concern is another aspect raised in the 2018 external audit and some interviews—that changes and adjustments to indicators generate discontinuities whose costs may negate their benefits (Morck and Shou 2018). Deep dives will allow a careful probing of 20 evidence on specific indicators and whether and how the indicators or reforms based on those indicators have been shaped by evidence and experience. 6. Whether current institutional arrangements separating indicator generation from project work (including reimbursable assistance) are adequate to prevent a perception of conflict of interest within the Bank Group. • A recurrent theme emerging from Bank Group expert interviews was both a strength and weakness of the Bank Group’s ownership of the DB indicators: Clients seek out assistance from the Bank Group to improve their ranking because of it. Unquestionably a variety of safeguards have been instituted and more are planned after the GIA Audit to separate and protect the function of generating indicators from the support to countries to improve the conditions they measure. Nonetheless, past evaluations and the recent suspension of the indicators suggest that the risk and perception of conflict of interest has persisted. This may pose a challenge to the credibility of the indicators and their acceptance. The evaluation will explore through the stakeholder interviews and case studies why clients are motivated to seek Bank Group support to improve their DB rank, and whether the firewalls in place are understood by clients to vitiate any potential conflict of interest or, conversely, whether a perceived conflict attracts demand for Bank Group technical or financial support. 6.5 Table 6.1 summarizes the lines of inquiry described above and the evidence sources they will draw on to assess the extent of their validity. Table 6.1. Links Between Evidence and Lines of Inquiry Line of Inquiry Evidence Sources 1. (Relevance) Whether all DB indicators Past evaluations: DB indicators do not capture the full range of legal and are relevant to constraints influencing regulatory priorities for business environment reform and may miss some business dynamics and economic binding constraints (World Bank 2008, 2013, 2015). contribution of policy reforms. Past evaluations: Multiple DB indicators focus primarily on compliance costs (in time, money, or procedural steps) with some exceptions (World Bank 2015, 2019) Past evaluations: The DB practice of aggregating indicators is controversial (World Bank 2013) Literature review: There is widely differing strength of evidence for the relationship of individual indicators to outcomes. 2. (Effectiveness) Whether the following factors are key influences on the use and appropriateness of DB indicators include the following: a. Client institutional capacity and World Bank Group expert interviews: Distinctions made according to ability (high versus low) to tailor use of country institutional capacity and motivation result in some countries indicators to its development priorities building on indicator-based dialogue, expanding the agenda of business and framework; and environment reforms. 21 Line of Inquiry Evidence Sources Mapping of portfolio and review of country strategies: Differential patterns of DB-related engagement across income level and other country characteristics reflect different combinations of supply and demand. Mapping of DB reforms: Variations exist in the relationship between the number of country reforms documented and the level of Bank Group engagement. b. Client motivation (for example, Bank Group expert interviews: Some client counterparts took on reform achieve policy reform versus improve to enhance development outcomes and others “gamed” the indicators DB ranking). primarily to gain status and rank. 3. (Effectiveness) Whether DB indicators are useful to do the following: a. Draw attention and resources to Past evaluations: DB indicators have motivated policy makers to consider business environment reform in areas business regulation issues, and DB dissemination attracts interest from they measure directly; businesses, nongovernmental organizations, and senior policy makers (World Bank 2008). Bank Group expert interviews: Overwhelmingly positive responses on the value of DB to initiate reform dialogue on the business environment. b. Draw attention and resources to Bank Group expert interviews: Value of DB to initiate dialogue on business environment reform beyond business reforms beyond the immediate coverage of the indicators. the areas they measure directly; c. Correctly identify and order priority Past evaluations: DB has had less influence on the choice, scope, and reform areas; and design of reforms (World Bank 2008). Bank Group expert interviews: Mixed responses on the value of DB to guide reforms by identifying binding constraints. d. Monitor and evaluate reform Past evaluations: DB indicators do not track reform impacts well (World implementation. Bank 2008, 2013). Bank Group expert interviews: Mostly negative responses on the value of the DB indicators to monitor and evaluate reforms. 4. (Relevance and Effectiveness) Bank Group expert interviews: View that DB indicators should be Whether complementary data and supplemented by other data and analysis. guidance enhance the relevance of DB Country private sector diagnostics: DB indicator data are supplemented indicators to business environment by other data and analysis. reforms and the ability to follow with complementary or deepening reforms. 5. (Effectiveness) Whether the feedback Past evaluations: DB adjustments to its indicators generate discontinuities loops between empirical evidence and (External Audit 2018) indicator design and use are robust. Literature review: There are considerable gaps and mixed evidence on the relationship of individual DB indicators to important outcomes, raising questions about the link between ongoing research and the design and application of the indicators. Bank Group expert interviews: Perspectives on the strength of the relationship of the indicators to academic and empirical research. 22 Line of Inquiry Evidence Sources 6. Whether current institutional Bank Group expert interviews: Reflections of personal and client arrangements separating indicator perceptions of institutional conflict of interest. generation from project work Bank Group internal audit Data Integrity Report: “Pressure from (including reimbursable assistance) are stakeholders as part of the DB report production and publication process adequate to prevent a perception of has not been effectively managed” (World Bank Group 2020a, 6). conflict of interest. Source: Independent Evaluation Group. Note: DB = Doing Business. Linking Evaluation Tools and Lines of Inquiry 6.6 As discussed in the Approach Paper (World Bank 2021), IEG’s focused evaluation will seek to understand the links among DB and country-level reforms, including assessing the relevance and influence of Bank Group interventions. Based on the model presented in the theory of change described in the Approach Paper, and informed by the evidence presented in this Issues Paper, the IEG evaluation will use several evaluation tools and methods to understand and assess the relevance and effectiveness of the use of DB indicators at the global, country, and project or intervention levels. The methods and tools that the evaluation will use to gather data and triangulate evidence are detailed in appendix A. They include the following: • Portfolio review and analysis of DB-informed Bank Group interventions approved or evaluated during FY10–20. Using separate analysis of approved and evaluated projects, the team will examine design features and characteristics, results indicators, and drivers of success and failure. • Structured literature review, discussed above, following IEG’s protocol. After the initial mapping phase, the review will synthesize rigorous evidence to identify findings and knowledge gaps. • Indicator-specific deep dives to assess up to five DB areas, drawing from the literature reviews, portfolio analysis, and case studies. Each deep dive will explore indicator features, construct and content validity, the relevant portfolio’s design and effectiveness characteristics, and the roles of other stakeholders at the country or global level. • Country case studies (with embedded indicator-level case studies) exploring relevance and effectiveness of DB indicators and related reforms in purposively selected country contexts, reflecting different initial conditions, capacity, motivation, levels of Bank Group engagement, and realized outcomes. 23 • Country-level reviews of policy, strategy, and diagnostics, including document reviews for a sample of countries covering Systematic Country Diagnostics, Country Partnership Frameworks, and CPSDs, as well as CPEs and Completion and Learning Report Reviews, to better understand issues of alignment and coherence. • Semistructured interviews of subject-matter experts within IEG, the broader Bank Group, and external stakeholders including governments, donors, nongovernmental agencies, academics, and private sector entities. In addition, an advisory group consisting of external and internal experts will be consulted periodically. • Statistical and econometric analysis examining the validity and robustness of DB indicators and indexes using (i) other measures of the business environment; and (ii) outcome indicators. 6.7 Each line of inquiry and associated evaluation hypothesis will be tested using several methods and tools. Appendix A maps the tools to the hypotheses they should be most helpful in testing. The Approach Paper has highlighted the limitations of each of these methodologies, and the need and intention for triangulation to enhance the robustness of findings (World Bank 2021). Potential error is limited by drawing evidence from multiple sources to test each hypothesis. The focused evaluation is anticipated to be delivered by the end of the first quarter of FY22. 7. Conclusion 7.1 Consistent with the theory of change and the evaluation questions laid out in the Approach Paper (World Bank 2021), this Issues Paper conducted a stocktaking of six threads of evidence, which led to six lines of inquiry and implicit hypotheses. This has allowed the formulation of specific areas that the focused evaluation will shed light on. The lines of inquiry will answer essential questions about the relevance and effectiveness of how the Bank Group and client countries use DB to inform, motivate, guide, and evaluate their business environment reforms. The findings and lessons from the evaluation are expected to inform the Bank Group on how to build on the successful aspects, and learn from challenges and limitations, in applying the DB indicators to addressing critical development challenges. 1For an explanation of the ease of doing business score, see https://www.doingbusiness.org/en/data/doing-business-score. 24 2“Regulations serve both economic purposes and such socially beneficial purposes as enhancing public health, safety, and the environment or reducing informality, corruption, and smuggling. Though some regulations may be unnecessary or purely protectionist, others are legitimate expressions of public policy. Hence trade facilitation reforms should be conceptualized, designed, implemented, and evaluated in the context of policy objectives to protect public health, safety, the environment, good governance, and formality in addition to a compliance cost minimization perspective. The World Bank Group project documents show that such public policy objectives are acknowledged only in a minority of cases, most frequently in terms of collecting public revenues and combatting corruption. Insufficient attention has been paid to other objectives (health, safety, and environment) and only compliance costs are routinely monitored. Recommendation 3: The World Bank Group should systematically apply a differentiated approach to identify and monitor, where relevant, the public policy objectives of trade regulations relating to public health, safety, the environment, good governance, formality, and the rule of law. The World Bank Group should specifically identify the stakeholders potentially impacted by the reforms and the extent of the impact. Wherever relevant, the World Bank Group should apply appropriate indicators to monitor the impact of trade facilitation reforms on affected stakeholders in these dimensions.” (World Bank 2019b, xix). 3The trade facilitation evaluation by the Independent Evaluation Group (IEG) provided a nuanced perspective on the value of Doing Business (DB) indicators for tracking trade facilitation reform outcomes. Applying difference-in-difference and panel analysis, “all the tests performed show no relationship of supported interventions to improvements in indicators of ‘costs’ or ‘number of documents’ required for imports or exports. Instead, a positive relationship is observed with ‘time to export and import’ and with several [Logistics Performance Index] perception-based indicators: ‘efficiency of customs,’ ‘ease of arranging,’ ‘quality of infrastructure,’ and ‘competence of services’” (World Bank 2019b, 32). 4 IEG recognizes the inherent tension between trying to incorporate feedback on the relevance of indicators or the effectiveness of reforms linked to those indicators and the desire to maintain continuity over time. 5 The database of articles was identified and collected by the DB team and shared with IEG. 63ie, a global leader in funding and producing high-quality evidence of what works, how, why and at what cost in international development, funds rigorous impact evaluations using experimental and quasi-experimental designs. 3ie also funds the production of rigorous full systematic reviews, rapid evidence assessments, and evidence gap maps to help fill the gaps in high-quality, timely, policy-relevant evidence to inform policy discussions and research priorities. The Campbell Collaboration is an international social science research network that produces high-quality, open, and policy-relevant evidence syntheses, plain-language summaries, and policy briefs. 25 7Because DB indicators use terms that are in common use, such as “paying taxes,” identification of the relevant portfolio involves manual review to eliminate a large percentage of “false positives”—projects that emerge from a search for relevant terms or codes but do not turn out to qualify as “DB-informed.” However, because it was recognized that traditional search methods missed some portion of the population of qualifying projects, the evaluation team worked with colleagues in IEG’s methods advisory team to use supervised machine learning to ide ntify additional DB-informed projects. This increased the total number of identified projects from 526 to 683. 8DB defines this as “a change of 0.5 points or more in the score and 2 percent or more on the relative score gap, except where the result of indexation” (Doing Business website, methodology for starting a business, methodology for trading across borders, methodology for enforcing contracts, etc. https://www.doingbusiness.org/en/methodology) 9Of course, as noted in IEG’s earlier investment climate evaluation, there can also be sharp gaps between measures of the achievement of reforms that streamline regulatory procedures and measures of economically consequential outcomes such as levels of investment, employment, productivity, and economic growth (World Bank 2015). 10 The theory of change reflects the logic expressed in documentation associated with DB. For example, DB 2019 states: “Doing Business offers policy makers a benchmarking tool useful in stimulating policy debate, both by exposing potential challenges and by identifying good practices and lessons learned” (World Bank 2018, p. 28). DB 2018 states: “[O]ne aspect of DB has remained unchanged: its focus on promoting regulatory reform that strengthens the ability of the private sector to create jobs, lift people out of poverty, and create more opportunities for the economy to prosper” (World Bank 2017, iv). 11 For example, in his introduction to DB 2019, then–Bank Group President Jim Young Kim wrote, “Sound and efficient business regulation is critical for entrepreneurship and a thriving private sector. Without them, we have no chance to end extreme poverty and boost shared prosperity around the world” (World Bank 2019a). He further stated: “The reforms that the report inspires will help people reach their aspirations; drive inclusive, sustainable economic growth; and bring us one step closer to ending poverty on the face of the earth” (World Bank 2018, v). 26 References Doshi, R., J. Kelley, and B. Simmons. 2019. “The Power of Ranking: The Ease of Doing Business Indicator and Global Regulatory Behavior.” International Organization 73 (3): 611–643. doi:10.1017/S0020818319000158. Morck, R., and J. C. Shou. 2018. “On the Integrity of the ‘Ease of Doing Business’ Indicators: Final Report.” World Bank, Washington, DC. https://www.doingbusiness.org/content/dam/doingBusiness/media/Miscellaneous/Audit /Doing-Business-External-Audit-2018_Full-Report.pdf. Kelley, J., and B. Simmons (eds). 2020. The Power of Global Performance Indicators. Cambridge: Cambridge University Press. World Bank. 2008. Doing Business: Taking the Measure of the World Bank–IFC Doing Business Indicators. Independent Evaluation Group. Washington, DC: World Bank. https://hubs.worldbank.org/docs/imagebank/Pages/docProfile.aspx?nodeid = 9780835. World Bank. 2013. Independent Panel Review of the Doing Business Report. Washington, DC: World Bank. http://pubdocs.worldbank.org/en/237121516384849082/doing-business- reviewpanel-report-June-2013.pdf.World Bank. 2015. Investment Climate Reforms: World Bank Group Support to Reforms of Business Regulations. Independent Evaluation Group. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/22724. World Bank. 2016. Doing Business 2016: Measuring Regulatory Quality and Efficiency. Washington, DC: World Bank Group. World Bank. 2017. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Washington, DC: World Bank Group. https://hubs.worldbank.org/docs/imagebank/pages/docprofile.aspx?nodeid = 28015672. World Bank. 2018. Doing Business 2018: Reforming to Create Jobs. Washington, DC: World Bank Group. World Bank. 2019a. Doing Business 2019: Training for Reform. Washington, DC: World Bank Group. https://www.doingbusiness.org/content/dam/doingBusiness/media/Annual- Reports/English/DB2019-report_web-version.pdf World Bank. 2019b. Grow With the Flow: An Independent Evaluation of the World Bank Group’s Support to Facilitating Trade 2006–17. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/facilitatingtrade. pdf.World Bank. 2020a. Data Integrity in the Production Process of the Doing Business. Washington, DC: World Bank Group. http://documents1.worldbank.org/curated/en/134831608154762985/pdf/Data-Integrity-in- Production-Process-of-the-Doing-Business-Report-Assurance-Review.pdf. 27 World Bank. 2020b. Doing Business 2020: Sustaining the Pace of Reforms. Washington, DC: World Bank Group. https://www.worldbank.org/en/news/feature/2019/10/24/doing-business- 2020-sustaining-the-pace-of-reforms.World Bank. 2021. Doing Business Indicators and Country Reforms. Approach Paper. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup.org/sites/default/files/Data/reports/ap_doingbusiness.pdf. World Bank and IFC (International Finance Corporation). 2013. Doing Business 2014: Understanding Regulations for Small and Medium-Size Enterprises. Washington, DC: World Bank Group. https://openknowledge.worldbank.org/handle/10986/16204. 28 Appendix A. Methodological Tools and Evaluation Design Broad Lines of Inquiry The evaluation is inspired by two lines of inquiry that will guide data collection and analysis (box A.1). Evaluation questions were designed to break the topic into answerable components in the areas of Doing Business (DB) influence, relevance, and effectiveness, as well as factors affecting outcomes of DB support (see Evaluation Questions section). Box A.1. Two Lines of Inquiry that Will Guide the Evaluation Doing the Right Things: This line of inquiry examines the relevance of indicators to country contexts and priorities, substantive dimensions of the areas they cover, and World Bank Group strategic and operational priorities. Doing Things Right: This line of inquiry examines whether Doing Business is being used effectively by the Bank Group and client countries to achieve intended immediate and intermediate business environment reform outcomes subject to broader policy priorities. Source: Independent Evaluation Group. Overarching Principles and Methods Three central principles motivated the evaluation design: theory-based evaluation, mixed methods, and multilevel analysis. First, the evaluation will adopt a theory-based approach that seeks to understand the links among DB, World Bank Group or client government–supported interventions, and country-level reforms based on the influence model elaborated. Second, the evaluation will also apply a mixed methods approach that combines an array of complementary methods for data collection and analysis (that is, internal project-level data, external country data sets, project performance data, semistructured interviews, case studies, sector deep dives, and structured literature reviews), then triangulates to ensure robust findings. Third, the evaluation will use three levels of analyses—namely, global, country, and project or intervention levels (figure A.1)—which will provide information on whether and how Bank Group programs and projects related to DB succeeded or not, including factors associated with success (or failure). The global level of analysis will be supported by the structured literature review, analysis of the Bank Group strategies and policies, deep dives into a subset of areas measured by DB, an expert panel discussion, and a quantitative analysis of the DB indicators and indexes. At the country level, the evaluation will collect data and 29 evidence through the following methods: (i) Bank Group country strategy and diagnostics analysis; (ii) desk- and field-based country case studies; (iii) portfolio review analysis for case study countries; (iv) comparative case analysis; and (v) structured interviews with key informants (figure A.1). At the portfolio or intervention level, there will be a portfolio review analysis to understand the breadth of interventions undertaken by the Bank Group to support DB reforms, as well as a chapeau report on relevant Project Performance Assessment Reports. Figure A.1. Products, Methods, and Levels of Analysis Source: Independent Evaluation Group. Note: CLRR = Completion and Learning Report Review; CPE = Country Program Evaluation; DB = Doing Business; PPAR = Project Performance Assessment Report. Design Matrix Table A.1 specifies the evaluation questions established in the Approach Paper and aligns them with methodological tools established in the Approach Paper to answer them. Each check mark— from one to three—represents the anticipated strength of the method to answer the questions. 30 Table A.1. Evaluation Questions and Methods Applied Statistical Portfolio Country- and Econo- Case Review and Deep Literature Level Inter- metric Evaluation Questions Studies Analysis Dives Review Reviews views Analysis 1. Doing the Right Things: Is Doing Business doing the right things in terms of specific indicators and country contexts? What is the relevance of Doing Business indicators to a. The business environment ✓✓✓ ✓✓ ✓✓ ✓✓ ✓✓✓ ✓✓✓ ✓✓✓ priorities of client countries; b. The substantive dimensions ✓✓✓ ✓✓ ✓✓✓ ✓✓✓ ✓✓ ✓✓✓ within each indicator’s focus area and; c. World Bank Group strategic ✓✓✓ ✓✓ ✓✓ ✓ ✓✓✓ ✓✓✓ ✓ and operational priorities. 2. Doing Things Right: Is Doing Business being used by clients and the World Bank Group in ways to achieve the best effect for business environment reform? a. Is Doing Business effectively, ✓✓✓ ✓✓✓ ✓✓ ✓✓ ✓✓ ✓✓ ✓✓ achieving desired outcomes? b. Is Doing Business being ✓✓✓ ✓✓ ✓✓ ✓✓ ✓✓ ✓✓✓ ✓ used by clients and the Bank Group to achieve the best outcomes for business environment reform (for example, entry, investment, employment) subject to broader policy priorities (for example, public safety, equity, environment)? Source: Independent Evaluation Group. The evaluation intends to produce two main outputs: First, this Issues Paper that takes stock of existing evidence on DB relevance, identifies key issues, and elaborates a framework and lines of inquiry for the deeper analysis of a full evaluation to follow. Second, a focused evaluation will apply the theory of change and lines of inquiry (and their implicit hypotheses) generated in the Approach and Issues Papers. Table A.2 delineates the associations among the evaluation questions, lines of inquiry, and hypotheses. This product will answer questions about relevance and effectiveness in terms of both immediate and intermediate outcomes, as described in the theory of change. 31 Table A.2. Evaluation Questions, Lines of Inquiry, and Hypothesis Evaluation Question Line of Inquiry Explicit Hypotheses 1. What is the relevance of Whether all DB indicators are relevant DB indicators vary in their relevance to the DB indicators to the: to constraints influencing business key constraints influencing business a. Business environment dynamics and economic contribution dynamics, hence the economic priorities of client countries? of policy reforms. contribution of reforms, based on the area b. Substantive dimensions and the country context. within each indicator’s focus area? c. World Bank Group strategic and operational priorities? 1a. And Whether (i) client institutional capacity Client institutional capacity and capability 2b. Are clients using DB to and ability (high versus low) to tailor is a key factor influencing how DB achieve improved outcomes the use of DB indicators to its indicators should be used and whether for business environment development priorities and framework; their use is effective. reform in line with the and (ii) client motivation (for example, Client motivation is a key factor influencing country’s development policy achieve policy reform versus improve the how DB indicators should be used and priorities? DB ranking) are key influences on the whether their use is effective. use and appropriateness of DB indicators. 1a, 1b, 2b Whether DB indicators are useful to (i) DB indicators are useful to draw attention draw attention and resources to and resources to legal and regulatory business environment reform in areas reforms in the areas they directly measure. they measure directly; (ii) draw DB indicators are useful to motivate attention and resources to business broader processes of business environment reform beyond the areas environment reform. they measure directly; (iii) correctly DB indicators are useful for identifying identify and order priority reform binding constraints and priority reforms. areas; and (iv) monitor and evaluate DB indicators are useful to monitor and reform implementation. evaluate reform progress. 1a, 1b, 2a, 2b Whether complementary data and DB indicators are more effectively used in guidance enhance the relevance of DB conjunction with complementary data and indicators to business environment guidance to enhance the relevance, reforms and the ability to follow with comprehensiveness, and depth of complementary or deepening reforms. corresponding business environment reforms. 2a. Is Doing Business reform Whether the feedback loops between DB benefits from robust feedback loops achieving desired outcomes? available empirical evidence and DB between empirical evidence (research and And from feedback loop in indicator design and use are robust. evaluative data) and indicator design. theory of change. 1c. and derives from Whether current institutional Current institutional arrangements challenges regarding data arrangements separating indicator separating DB indicator generation from integrity. generation from project work DB reform project work (including (including reimbursable assistance) are reimbursable assistance) are adequate to adequate to prevent a perception of prevent a perception of conflict of interest conflict of interest within the World within the World Bank Group. Bank Group. Source: Independent Evaluation Group analysis and Approach Paper. Note: DB = Doing Business. 32 Expected Evaluation Tools Used to Pursue Six Main Lines of Inquiry As noted, the Independent Evaluation Group (IEG) evaluation will use several evaluation tools and methods to understand and assess the relevance and effectiveness of the use of DB indicators at the global, country, and project or intervention levels. Each line of inquiry identified in the Issues Paper and its subordinate evaluation hypothesis can be explored applying a constellation of tools to provide evidence. Table A.3 maps the tools to the lines of inquiry they should be most helpful in illuminating. The Approach Paper has highlighted the limitations of each of these methodologies, and the need and intention for triangulation to enhance the robustness of findings. Potential error is limited by drawing evidence from multiple sources to test each hypothesis. Table A.3. Lines of Inquiry and Evaluation Tools Expected Evaluation Tools Six Main Lines of Inquiry PRA SLR IDD CCS CLR SSI SEA 1. All DB indicators are relevant or not relevant to constraints influencing X X X X X X business dynamics and economic contribution of policy reforms. 2. The following factors are or are not key in influencing the use and appropriateness of DB indicators: a. Client institutional capacity and ability (high versus low) to tailor use of X X X X X indicators to development priorities and framework; and b. Client motivation (for example, achieve policy reform versus improve DB X X X X X ranking). 3. DB indicators are useful or not useful for: a. Drawing attention and resources to business environment reform in areas X X X X they measure directly; b. Drawing attention and resources to business environment reform beyond X X X X the areas they measure directly; c. Correctly identifying and ordering priority reform areas; and X X X X d. Monitoring and evaluating reform implementation. 4. Complementary data and guidance enhance or do not enhance the X X X X relevance of DB indicators to business environment reforms and the ability to follow with complementary or deepening reforms. 5. The feedback loops between empirical evidence and indicator design and X X X use are robust or weak. 6. Current institutional arrangements separating indicator generation from X X project work are adequate or inadequate to prevent a perception of conflict of interest. Source: Independent Evaluation Group. Note: CCS = country case studies; CLR = country-level reviews of policy, strategy, and diagnostics; DB = Doing Business; IDD = indicator-specific deep dives; PRA = portfolio review and analysis; SEA = statistical and econometric analysis; SLR = structured literature review; SSI = semistructured interviews. 33 Portfolio Review and Analysis The evaluation will conduct a systematic desk review and assessment of projects approved and those evaluated to identify design features and characteristics, drivers of success and failure, and achievement of reform objectives and social objectives. A combination of manual identification, manual codification, and supervised learning methods will be used to identify all Bank Group projects approved between fiscal year (FY)10 and FY20 that were informed by the DB report or indicators in their objectives, components, and result matrices, and that have relevant influence in their motivation (figure A.2). Figure A.2. Identification Methodologies in Place Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; DB = Doing Business; IFC = International Finance Corporation. Structured Literature Review IEG has commissioned and is supervising a structured literature review, following IEG’s protocol covering academic and Bank Group literature on the DB indicators. The structured literature review focuses on reviewing empirical studies directly examining regulatory reforms and interventions related to the DB project. The structured literature review is expected to provide empirical evidence on the links between the DB indicators and reform outcomes and impact. The review will follow IEG’s structured literature 34 review framework for major evaluations, with a clearly defined search strategy, inclusion and exclusion criteria, and extraction and synthesis. Before this broad and rigorous review, the team conducted a desk review of relevant literature identified by the DB team from top journals to understand the pattern of evidence. Country Case Studies The evaluation will conduct up to 10 country case studies in purposively selected countries. Case selection will reflect a diversity of country conditions and contexts (including regional diversity, income level, and International Development Association and fragility, conflict, and violence classifications). The country-selection criteria will include the presence of DB reforms, whether supported by Bank Group or government, and the presence of evaluation evidence (Implementation Completion and Results Reports, Expanded Project Supervision Reports, Project Completion Reports, Project Evaluation Reports) in countries with Bank Group support. The evaluation will adopt a template for data collection and follow standard protocols to facilitate comparison across case studies. These methods include the following: (i) a review of literature on the DB indicators and related reforms in the country; (ii) a review of Bank Group country strategies, diagnostics, and analytical works; (iii) a review of the Bank Group’s portfolio to support doing business in the country; (iv) structured interviews with stakeholders (government, multilateral development banks, private sector, nongovernmental organizations, academics, and so on); (v) analysis of the effectiveness of DB in motivating reforms and informing the design of reform; and (vi) an assessment of the extent to which Bank Group DB interventions are achieving their desired outcomes and have contributed to improving economic and social outcomes. The evaluation team will also consult with internal and external experts to identify the countries that offer the richest opportunities for learning. Cases will involve both desk- and (remote) field-based assessments and they will aim to identify to what extent the Bank Group’s efforts were effective and “how” and “why” specific reforms or reform interventions were or were not successful in delivering the intended results. Case studies will attempt to capture variables that allow an assessment of the contributions of DB and DB-informed activities to observed outcomes, accounting for contextual factors and parallel and complementary activities that may help explain observed outcomes. Case studies will take into account any potential alternative explanations (besides the role and influence of DB) for observed outcomes, additional explanatory factors, and unintended results. Indicator-Specific Deep Dives Indicator-specific deep dives will provide an opportunity for the evaluation to study DB areas in a structured and focused manner. The deep dive methodology, guided by a common template, will include (i) a focused structured literature review and profound 35 analysis on five of the areas covered by DB, and (ii) a review of indicator-relevant portfolios and full consideration of evidence (from multiple sources) describing their relevance and effectiveness. It is anticipated that the selection of deep dives will be based on the prevalence of areas in recorded projects, country reforms, and the literature. Each deep dive into an area of DB will discuss indicator and subindicator features, and will examine construct and content validity (the degree to which the index or indicators measure the concept they intend to and the extent to which the index or indicators are representative of the area they aim to measure). For each area, the deep dives will provide detail on the relevant portfolio’s design features and draw on evidence from the portfolio review and analysis (PRA) on relevance and effectiveness, including factors that are associated with success and failure. Each deep dive will draw on a structured literature review for that area. Deep dives will also draw from the draft case studies and will consider the role of stakeholders (other than the Bank Group) at the country and global levels. Semistructured Interviews An initial set of 18 semistructured interviews, primarily of experienced current and former Bank Group stakeholders and practitioners, inform this Issues Paper. Further semistructured interviews will be conducted with subject-matter experts and practitioners within IEG, the broader World Bank Group, and external bodies including governments, donors, nongovernmental agencies, academics, and private sector entities. It is anticipated that, at an early stage, the evaluation will convene an expert panel to assist in identifying key issues and challenges, better understanding the underlying theory, and developing a set of preliminary hypotheses. The interviews with Bank Group staff engaged in DB projects will help clarify institutional priorities, program features, achievement of objectives, and lessons derived from the experiences. They will also attempt to gauge stakeholders’ perspectives on the accuracy and usefulness of DB for identifying reform priorities, guiding the design of reforms, and monitoring reform outcomes. The evaluation team will develop an interview guide to ensure key questions are asked consistently across interviews while maintaining the flexibility needed to follow response trajectories that may not adhere to the guide. Country-Level Reviews of Policy, Strategy, and Diagnostics The evaluation will conduct a series of systematic document reviews for a sample of countries to complement the evaluation’s portfolio review. The evaluation carried out a systematic review of all the Bank Group Country Partnership Frameworks approved after FY15 to better understand the level of alignment and coherence of Bank Group country-level strategies and DB-related concerns. A similar review was carried out for a random sample stratified by income level, Region, and fragility, conflict, and violence 36 status for countries that have been subject to Systematic Country Diagnostics and all published Country Private Sector Diagnostics. A categorical array will be developed to systematically assess evaluation questions across strategy documents and diagnostics. The Issues Paper has already drawn on an analysis of the Completion and Learning Report Reviews of the same countries that were reviewed in the Country Partnership Framework analysis. Statistical and Econometric Analysis The evaluation will apply statistical and econometric methods to examine the robustness of DB indicators and indexes. Statistical and econometric analysis of DB and external indicators would compare the DB indicators with other measures of the business- enabling environment that are “not based primarily on DB” (including those based on enterprise surveys) and would examine the link of movements in DB indicators (“reforms”) to outcome indicators, such as country-level data on business entry and productivity, private investment (domestic and foreign) and employment, and private sector growth. Design Limitations IEG anticipates multiple limitations to its proposed mix of methods and will seek to mitigate the risks they impose (including by triangulating from multiple evidence sources to test hypotheses). These include gaps and inconsistencies in the literature and country social and economic data sets; the limited number of relevant evaluated projects and country programs with DB relevance (especially in light of the lack of IEG-validated evaluations of advisory services and analytics); the limited ability of a small number of case studies and deep dives to represent the universe; limited data on intermediate outcomes and sustainability of reforms and difficulties in establishing attribution to specific reforms; and difficulties in conducting fieldwork (even remotely) imposed by the coronavirus pandemic (COVID-19). There may also be some false negatives or false positives in identifying relevant literature, project activities, and reforms due to the multifaceted nature of areas covered by some DB indicators, lack of standard terminology, and the sometimes-indirect nature of influence. Establishing causal connections between DB indicators and observed reforms, and between reforms and observed outcomes, is complicated by the likelihood of multiple causal factors. 37 Appendix B. Initial Identification of Bank Group Doing Business–Informed Portfolio The initial review identified 1,669 potential Doing Business (DB) projects (372 World Bank lending projects, 916 World Bank advisory services and analytics [ASA] projects, 380 International Finance Corporation [IFC] advisory services projects, and 1 IFC investment services project) that were approved between fiscal year (FY)10 and FY20 (table B.1). Additionally, the team identified three false negatives that were added. From the 1,672 projects, the Independent Evaluation Group (IEG) reviewed 1,334 projects, composed of 373 World Bank lending (100 percent of their identified portfolio), 451 World Bank ASA (57 percent, random sample), 127 World Bank reimbursable advisory services (100 percent) and 382 IFC advisory services (100 percent). IEG also conducted an automated process using supervised machine learning methods to identify World Bank lending projects that included DB in their rationale but that were not possible to identify under the manual identification methodology. This process identified 226 additional potential DB World Bank lending projects, of which IEG reviewed 107 (47 percent, a random sample stratified by income level and Region). This portfolio review aimed to provide an overview of the World Bank Group projects approved between FY10 and FY20 that reference DB indicators or subindicators or relevant dimensions in their rationale, objectives, or result matrices. This support is channeled through the World Bank and IFC. The World Bank portfolio includes lending and ASA, while the IFC portfolio is composed exclusively of advisory services. Based on a manual (human) review of a random sample, IEG estimates there were 269 World Bank DB-informed lending projects and 414 World Bank and IFC advisory or analytic projects approved during FY10–20. By volume, most of the support was provided through advisory work (61 percent), of which the World Bank provided 55 percent while IFC delivered 45 percent. By commitment or expenditure value, World Bank lending predominated (table B.1). Organizationally, most of the projects were delivered by World Bank and IFC staff operating within the Macroeconomics, Trade, and Investment and the Finance, Competitiveness, and Innovation Global Practices within the Equitable Growth, Finance and Institutions Vice Presidency (table B.2). For lending, World Bank policy-based instruments are by far the most popular vehicle (table B.3). 38 Table B.1. Summary of Doing Business Portfolio Approved During FY10–20 Projects Interventions Volume/Commitments Institution (no.) (%) (no.) (%) (US$, millions) (%) World Bank lending 269 39 517 43 14,853 96 Subtotal 269 39 517 43 14,853 96 World Bank ASA 165 24 173 14 379 2 World Bank RAS 61 9 86 7 21 0 IFC-AS 188 28 432 36 298 2 Subtotal 414 61 691 57 698 4 Grand total 683 100 1,208 100 15,550 100 Source: Independent Evaluation Group portfolio review analysis. Note: All information is preliminary and projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by matching learning, and 789/452 for World Bank ASA projects, representing the weight of each institution in the population. Volume/commitment/funds managed were identified or estimated according to what is allocated to DB-related interventions. If the amount was not explicitly stated, it was estimated based on the number of components, subcomponents, or activities. ASA = advisory services and analytics; FY = fiscal year; IFC-AS = International Finance Corporation advisory services; RAS = reimbursable advisory services. Table B.2. Doing Business–Informed Portfolio by Operational Unit, FY10–20 Projects Interventions Estimated Amount World Bank Group Operational Unit (no.) (%) (no.) (%) (US$, millions) (%) World Bank lending Macroeconomics, Trade, and 157 23 277 23 5,419 35 Investment Finance, Competitiveness, and 51 7 113 9 5,202 33 Innovation Energy and Extractive Industries 18 3 32 3 2,765 18 Other 42 6 94 8 1,467 9 Total World Bank lending 269 39 517 43 14,853 96 Subtotal of financing 269 39 517 43 14,853 96 World Bank ASA Macroeconomics, Trade, and 59 9 62 5 352.1 2.26 Investment Finance, Competitiveness, and 34 5 36 3 10.1 0.06 Innovation Other 73 11 74 6 16.86 0.11 Total World Bank ASA 165 24 173 14 379.03 2 World Bank RAS Finance, Competitiveness, and 24 4 38 3 6.69 0.04 Innovation Macroeconomics, Trade, and 13 2 20 2 5.97 0.04 Investment Other 24 4 28 2 7.95 0.05 Total World Bank ASA 61 9 86 13 20.61 0.13 39 Projects Interventions Estimated Amount World Bank Group Operational Unit (no.) (%) (no.) (%) (US$, millions) (%) IFC-AS Equitable Growth, Finance, and 129 19 312 26 219.21 1 Institutions Macroeconomics, Trade, and 24 4 57 5 44.54 0.29 Competitiveness Investment Climate 16 2 29 2 17.74 0.11 Other 19 3 34 3 16.50 0.11 Total IFC-AS 188 28 432 36 298.00 2 Subtotal of advisory 414 61 691 57 698 4 Total 683 100 1,208 100 15,550 100 Source: Independent Evaluation Group portfolio review analysis. Note: All information is preliminary, and projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by matching learning, and 789/452 for World Bank ASA projects, representing the weight of each institution in the population. Volume/commitment/funds managed were identified or estimated according to what is allocated to Doing Business–related interventions. If the amount was not explicitly stated, it was estimated based on the number of components, subcomponents, or activities. ASA = advisory services and analytics; FY = fiscal year; IFC-AS = International Finance Corporation advisory services; RAS = reimbursable advisory services. Table B.3. Doing Business–Informed Lending Portfolio by Instrument, FY10–20 Projects Estimated amount (n) (%) (US$, millions) (%) Policy Based 175 65 11,674 79 (Adjustment) Investment 88 33 2,644 18 Program-for-Results 6 2 535 4 World Bank lending 269 100 14,853 100 Source: Independent Evaluation Group portfolio review analysis. Note: All information is preliminary, and projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by matching learning, and 789/452 for World Bank ASA projects, representing the weight of each institution in the population. Volume/commitment/funds managed were identified or estimated according to what is allocated to Doing Business–related interventions. If the amount was not explicitly stated, it was estimated based on the number of components, subcomponents, or activities. ASA = advisory services and analytics; FY = fiscal year. 40 Figure B.1. Evolution of the Doing Business–Informed Portfolio by Institution, FY10–20 a. Financing support b. Advisory support Source: Independent Evaluation Group. Note: All information is preliminary and projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by matching learning, and 789/452 for World Bank ASA projects, representing the weight of each institution in the population. ASA = advisory services and analytics; FY = fiscal year; IFC-AS = International Finance Corporation advisory services; Mov Avg = Moving Average; RAS = reimbursable advisory services. 41 Distribution by Income Level Figure B.2. Distribution of Doing Business Projects by Income Level, FY10–20 a. Bank Group projects by income level and institution b. Share of projects by institution and income level Source: Independent Evaluation Group portfolio review analysis. Note: All information is preliminary and projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by matching learning, and 789/452 for World Bank ASA projects, representing the weight of each institution in the population. ASA = advisory services and analytics; FY = fiscal year; IFC-AS = International Finance Corporation advisory services; RAS = reimbursable advisory services. L = low income; LM = lower-middle income; UM = upper-middle income; H = high income. 42 Distribution by Region Figure B.3. Distribution of Doing Business Projects by Region, FY10–20 a. Overall b. By institution Source: Independent Evaluation Group portfolio review analysis. Note: All information is preliminary and projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by matching learning, and 789/452 for World Bank ASA projects, representing the weight of each institution in the population. ASA = advisory services and analytics; EAP = East Asia and Pacific; ECA = Europe and Central Asia; FY = fiscal year; IFC-AS = International Finance Corporation advisory services; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; OTH: Other; RAS = reimbursable advisory services; RGN = Regional; SAR = South Asia; SSA = Sub-Saharan Africa. 43 Doing Business Areas Figure B.4. Type of Doing Business Areas of Business Regulation Supported, FY10–20 a. Overall (n = 683) b. Distribution by institution Source: Independent Evaluation Group portfolio review analysis. Note: All information is preliminary and projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by matching learning, and 789/452 for World Bank ASA projects, representing the weight of each institution in the population. ASA = advisory services and analytics; FY = fiscal year; IFC-AS = International Finance Corporation advisory services; RAS = reimbursable advisory services. 44 Figure B.5. Use of Doing Business Report in Projects, FY10–20 Source: Independent Evaluation Group portfolio review analysis. Note: n = 683. All information is preliminary and projected based on the population and sample sizes. Specifically, the results were multiplied by a factor of 226/107 for World Bank lending projects identified by matching learning, and 789/452 for World Bank ASA projects, representing the weight of each institution in the population. ASA = advisory services and analytics; DB = Doing Business; FY = fiscal year; IFC-AS = International Finance Corporation advisory services; RAS = reimbursable advisory services. 45 Appendix C. Summary of Desk-Based Literature Review Table C.1 summarizes the desk-based literature review of 426 articles collected by the Doing Business team. Table C.1. Summary of Desk-Based Literature Review Indicator Papers (no.) Strong Evidence Modest Evidence Mixed Evidence Starting a 79 Higher entry costs or more steps or Lower entry costs are associated with larger small and medium enterprise Lower entry costs encourage business documents are associated with less firm sector, trade.2 formalization. creation, growth, or profitability.1 Reduced complexity encourages entry. High entry costs increase or reduce productivity and employment.3 Getting credit 29 Stronger legal rights are associated More credit information is associated with more private credit, better lending Credit information sharing with more lending, more financial terms, and reduced defaults and delinquencies.5 reduces the likelihood of activity, and lower interest rates.4 banking crises.6 Paying taxes 28 Higher tax rates constrain Tax complexity and number of payments reduce formal entry, investment, Progressivity of tax rates entrepreneurship and investment.7 and innovation.8 encourages entry. 9 Enforcing 41 Better contract enforcement encourages investment in asset-specific contracts (customized) inputs.10 Greater costs and procedures in enforcing contracts are linked to lower economic growth.11 Lower contract enforcement costs reduce chance of early liquidation.12 Countries with more foreign direct investment and debt finance have lower contract enforcement costs.13 Trading across 69 Better trade facilitation measured by Countries with lengthy export times are less likely to export more time- borders DB (and LPI) shows correlation with sensitive goods.15 increased trade flows.14 Faster customs clearance can offset the impact of being a landlocked or smaller country.16 Export gains from better trade facilitation outweigh gains of tariff cuts in importing African countries.17 Longer trade times are associated with higher levels of trade-related corruption.18 46 Indicator Papers (no.) Strong Evidence Modest Evidence Mixed Evidence Construction 17 Time and cost of getting construction permits are not related to economic permits growth.19 No relationship between DB construction permit indicators and the presence of corruption or bribery.20 Construction permit indicators do not permit identification of policies that matter systematically.21 Indicators do not align with the experience of surveyed firms, which report huge variation within countries.21 Registering 22 Higher burden of registering property is associated with reduced benefit of property trade opening on national income.23 Registering property indicators have no relation to foreign direct investment inflows into Sub-Saharan African countries.24 Registering property indicators are inconsistent with actual experiences of firms reported in enterprise surveys.25 Protecting 60 Country-level minority investors’ protection policies matter more than firm- minority investors level decisions for improving corporate governance.26 Lower levels of disclosure discourage investment and affect corporate governance.27 Getting electricity 16 Getting electricity indicators have little predictive power for a country's electrification rate and the level of technical and nontechnical losses in the system.28 A better rank on ease of getting electricity is associated with greater perceived quality of electricity supply.29 Electricity connection processes tend to be lengthier and more cumbersome in countries where other administrative processes are also burdensome.30 Administrative reforms to reduce the number of days to get electricity can improve the country’s rankings.31 DB measures of electricity constraints are negatively associated with measures of energy consumption.32 Frequency of outages is correlated with the difficulty of connection process, while electricity tariffs are not associated with either measure. 33 47 Indicator Papers (no.) Strong Evidence Modest Evidence Mixed Evidence Resolving 34 Longer time to resolve insolvency in some sectors is negatively associated to insolvency net job contribution by start-ups and, to a much lesser extent, by incumbents.34 Inefficient bankruptcy is associated with less bond issuance by risky, but not by safe, borrowers.35 Legal infrastructure facilitating the resolution of insolvencies mitigates the negative effect that limited supply of long-term finance has on economic volatility.36 DB’s recovery rate indicator correlated with availability of domestic credit from a country’s banking sector.36 Good insolvency regimes are negatively associated with stock of nonperforming loans and their rate of increase.38 Good insolvency regimes are associated with a lower frequency of insolvencies.39 Employing 31 Developing countries with rigid Greater likelihood of positive impact of trade liberalization on employment workers employment laws tend to have larger and wages in countries with flexible labor markets and vice versa.41 informal sectors and higher More regulated labor markets tend to have higher wages at the expense of unemployment, especially among sector wide employment.42 young workers.40 There is an economically significant association between digital technology use by businesses and a country’s statutory minimum wage and its employment protection regulations.43 There is a negative correlation between the stringency of labor regulation and the intensity of its enforcement.44 Strong negative consequences of labor regulation on labor market outcomes are based entirely on measures of de jure stringency of regulations.45 There is a significant association between digital technology use by businesses and a country’s statutory minimum wage and employment protection regulations.46 Dismissal laws can prevent employers from arbitrarily discharging employees and thereby enhance employees’ innovative efforts and encourage firms to invest in technology and innovation.47 Source: Independent Evaluation Group review of Doing Business team’s database of relevant articles from 100 leading journals. Note: “Strong Evidence” denotes at least three articles with convergent findings on the relation of Doing Business indicators to outcomes. “Modest Evidence” indicates only one supporting article. “Mixed evidence” indicates articles with findings both confirming and contradicting the finding. For construction permits, there are no confirming findings, only findings of its lack of relationship to economically important outcomes. DB = Doing Business; LPI = Logistics Performance Index. 48 1 Klapper et al. (2006), Prantl (2010), Branstetter et al. (2013), Bruhn (2008), Divanbeigi and Ramalho (2015), Freund and Bolaky (2008). 2 Branstetter et al. (2013), Klapper and Love (2010), Ayyagari et al. (2007). 3 Formalization: Williams (2014), Campos et al (2015), Andrade et al. (2013), Bruhn and McKenzie (2014), Mel et al. (2013), Pinotti (2012), Van Stel et al. (2007). Productivity: Boedo and Mukoyama (2012), Chari (2011) vs. Poschke (2010), Schivardi and Viviano (2011), Westmore (2016). Labor: Bruhn (2008), Schivardi and Viviano (2011), vs. Janiak (2013), Fang and Rogerson (2011). 4 Haselmann et al. (2010), Campello et al. (2016), Ibrahim et al. (2017). 5 Demirgüç-Kunt et al. (2017). 6 Houston F. et al. (2010). 7 Block (2016), Belitski et al. (2016), Andersson and Henrekson (2015), Lee and Gordon (2005), Becker et al. (2012), Lawless (2013), Fisman and Svensson (2007), Chowdhury et al (2014) vs. Di Giovanni and Levchenko (2013). 8 Block (2016), Block (2016), Baliamoune-Lutz and Garallo (2014), La Porta and Shleifer (2008), Jerbashian and Kochanova (2016). 9 Block (2016), Baliamoune-Lutz and Garallo (2014). 10 Nunn (2007). 11 Kovač (2016). 12 Demirgüç-Kunt, et al. (2017). 13 Ahlquist, and Prakash, A. (2010). 14 Sourdin and Korinek (2011), Iwanow and Kirkpatrick (2009), Moïsé and Sorescu (2013), Portugal-Perez and Wilson (2009), Djankov et al. (2010), Volpe Martincus et al. (2015). 15 Li (2019). 16 Hillberry, et al. (2018). 17 Portugal-Perez, A., et al. (2009). 18 Shepherd, B. (2009). 19 Poel, et al. (2014). 20 Freund, et al. (2014) and Kenny (2007). 21 Kraay, et al. (2013). 22 Hallward-Driemeier, and Pritchett (2015). 23 Freund, and Bolaky (2008). 24 Nangpiire, et al. (2018). 25 Hallward-Driemeier, and Pritchett (2015). 26 Doidge, et al. (2007). 27 Fernandes et al. (2010), Athari et al. (2016) and Iliev, et al. (2015). 28 Geginat and Ramalho (2018). 29 Geginat and Ramalho (2018). 30 Geginat and Ramalho (2018). 31 Haidar I., Hoshi and Shorenstein (2015). 32 Haidar I., Hoshi and Shorenstein (2015). 33 Arlet (2017). 34 Calvino et al. (2016). 35 Becker, B., and Josephson (2016). 36 Demirgüç-Kunt et al. (2017). 37 Menezes (2014). 49 38 Bricongne C., Demertzis, Pontuch and Turrini (2016). 39 Bricongne C., Demertzis, Pontuch and Turrini (2016). 40 Djankov and Ramalho (2009), Jales (2017), Acar et al. (2019). 41 Selwaness and Zaki (2015). 42 Selwaness and Zaki (2015). 43 Packard et al. (2017). 44 Kanbur, et al. (2016). 45 Botero J., et al. (2004). 46 Truman, and Montenegro (2017). 47 Acharya et al. (2013). 50 References Acar, A., L. Bossavie, and M. Makovec. 2019. “Do Firms Exit the Formal Economy after a Minimum Wage Hike? Quasi-Experimental Evidence from Turkey.” Policy Research Working Paper 8749, World Bank Group, Washington, DC. Acharya, V. V., R. P. Baghai, and K. V. Subramanian. 2013. “Labor Laws and Innovation.” Journal of Law and Economics 56 (4): 997–1037. Ahlquist, J. S., and A. Prakash. 2010. “FDI and the Costs of Contract Enforcement in Developing Countries.” Policy Sciences 43 (2): 181–200. Andersson, M., and M. Henrekson. 2015. “Local Competitiveness Fostered through Local Institutions for Entrepreneurship.” In The Oxford Handbook of Local Competitiveness, edited by David B. Audretsch, Albert N. Link, and Mary Walshok, 145–190. New York: Oxford University Press. Arlet, J. N. 2017. “Electricity Sector Constraints for Firms Across Economies: A Comparative Analysis.” Doing Business Research Notes, Report 117320. World Bank, Washington, DC. Athari, S. A., C. Adaoglu, and E. Bektas. 2016. “Investor Protection and Dividend Policy: The Case of Islamic and Conventional Banks.” Emerging Markets Review 27: 100–117. Ayyagari, M., T. Beck, and A. Demirgüç-Kunt. 2007. “Small and Medium Enterprises Across the Globe.” Small Business Economics 29 (4): 415–434. Baliamoune-Lutz, M., and P. Garello. 2014. “Tax Structure and Entrepreneurship.” Small Business Economics 42 (1): 165–190. Becker, B., and J. Josephson. 2016. “Insolvency Resolution and the Missing High-Yield Bond Markets.” The Review of Financial Studies 29 (10): 2814–2849. Belitski, M., F. Chowdhury, and S. Desai. 2016. “Taxes, Corruption, and Entry.” Small Business Economics 47 (1): 201–216. Block, J. 2016. “Corporate Income Taxes and Entrepreneurship.” IZA World of Labor: 257. Boedo, H. J. M., and T. Mukoyama. 2012. “Evaluating the Effects of Entry Regulations and Firing Costs on International Income Differences.” Journal of Economic Growth 17 (2): 143–170. Botero, J., S. Djankov, R. Porta, and F. C. Lopez-De-Silanes. 2004. “The Regulation of Labor.” The Quarterly Journal of Economics 119 (4): 1339–1382. Branstetter, L., F. Lima, L. J. Taylor, and A. Venâncio. 2014. “Do Entry Regulations Deter Entrepreneurship and Job Creation? Evidence from Recent Reforms in Portugal.” The Economic Journal 124 (577): 805–832. 51 Bricongne, J. C., M. Demertzis, P. Pontuch, and A. Turrini. 2016. “Macroeconomic Relevance of Insolvency Frameworks in a High-Debt Context: An EU Perspective.” Discussion Paper 32. Directorate General Economic and Financial Affairs (DG ECFIN), European Commission, Brussels. Bruhn, M. 2008. “License to Sell: The Effect of Business Registration Reform on Entrepreneurial Activity in Mexico.” Policy Research Working Paper 4538, World Bank, Washington, DC. Bruhn, M., and D. McKenzie. 2014. “Entry Regulation and the Formalization of Microenterprises In Developing Countries.” The World Bank Research Observer 29 (2): 186–201. Calvino, F., C. Criscuolo, and C. Menon. 2016. “No Country for Young Firms?: Start-Up Dynamics and National Policies.” OECD Science, Technology and Industry Policy Papers 29, OECD Publishing, Paris. https://doi.org/10.1787/5jm22p40c8mw-en. Campello, M., and M. Larrain. 2016. “Enlarging the Contracting Space: Collateral Menus, Access to Credit, and Economic Activity.” The Review of Financial Studies 29 (2): 349–383. Campos, F., M. Goldstein, and D. McKenzie. 2015. “Short-Term Impacts of Formalization Assistance and a Bank Information Session on Business Registration and Access to Finance in Malawi.” Policy Research Working Paper 7183, World Bank, Washington, DC. Chari, A. V. 2011. “Identifying the Aggregate Productivity Effects of Entry and Size Restrictions: An Empirical Analysis of License Reform in India.” American Economic Journal: Economic Policy 3 (2): 66–96. Chowdhury, F., M. Belitski, and S. Desai. 2014. “Taxes, Corruption and Entrepreneurship.” Academy of Management Proceedings 1: 10559. De Andrade, G. H., M. Bruhn, and D. McKenzie. 2013. “A Helping Hand or the Long Arm of the Law? Experimental Evidence on What Governments Can Do to Formalize Firms.” World Bank Economic Review 30 (1): 24–54. De Mel, S., D. McKenzie, and C. Woodruff. 2013. “The Demand for, and Consequences of, Formalization among Informal Firms in Sri Lanka.” American Economic Journal: Applied Economics 5 (2): 122–50. Demirgüç-Kunt, A., B. L. Horváth, and H. Huizinga. 2017. “How Does Long-Term Finance Affect Economic Volatility?” Journal of Financial Stability 33: 41–59. Di Giovanni, J., and A. A. Levchenko.2013. “Firm Entry, Trade, and Welfare in Zipf's World.” Journal of International Economics 89 (2): 283–296. Divanbeigi, R., and R. Ramalho. 2015. “Business Regulations and Growth.” Policy Research Working Paper 7299, World Bank Group, Washington, DC. Djankov, S., C. Freund, and C. S. Pham. 2010. “Trading on Time.” The Review of Economics and Statistics 92 (1): 166–173. 52 Djankov, S., and R. Ramalho. 2009. “Employment Laws in Developing Countries.” Journal of Comparative Economics 37 (1): 3–13. Doidge, C., G. A. Karolyi, and R. M. Stulz. 2007. “Why Do Countries Matter So Much for Corporate Governance?” Journal of Financial Economics 86 (1): 1–39. Fang, L., and R. Rogerson. 2011. “Product Market Regulation and Market Work: A Benchmark Analysis.” American Economic Journal: Macroeconomics 3 (2): 163–88. Fernandes, N., U. Lel, and D. P. Miller. 2010. “Escape from New York: The Market Impact of Loosening Disclosure Requirements.” Journal of Financial Economics 95 (2): 129–147. Fisman, R., and J. Svensson. 2007. “Are Corruption and Taxation Really Harmful to Growth? Firm Level Evidence.” Journal of Development Economics 83 (1): 63–75. Freund, C., and B. Bolaky. 2008. “Trade, Regulations, and Income.” Journal of Development Economics 87 (2): 309–321. Freund, C., M. Hallward-Driemeier, and B. Rijkers. 2014. “Deals and Delays: Firm-Level Evidence on Corruption and Policy Implementation Times.” World Bank Policy Research Working Paper 6949, World Bank, Washington, DC. Geginat, C., and R. Ramalho. 2018. “Electricity Connections and Firm Performance in 183 Countries.” Energy Economics 76: 344–366. Haidar, J. I., T. Hoshi, and W. H. Shorenstein. 2015. “Implementing Structural Reforms in Abenomics: How to Reduce the Cost of Doing Business In Japan.” NBER Working Paper 21507, National Bureau of Economic Research, Cambridge, MA. Hallward-Driemeier, M., and L. Pritchett. 2015. “How Business Is Done in the Developing World: Deals Versus Rules.” Journal of Economic Perspectives 29 (3): 121–40. Haselmann, R., K. Pistor, and V. Vig. 2010. “How Law Affects Lending.” The Review of Financial Studies 23 (2): 549–580. Hillberry, R., and X. Zhang. 2018. “Policy and Performance in Customs: Evaluating the Trade Facilitation Agreement.” Review of International Economics 26 (2): 438–480. Houston, J. F., C. Lin, P. Lin, and Y. Ma. 2010. “Creditor Rights, Information Sharing, and Bank Risk Taking.” Journal of Financial Economics 96 (3): 485–512. Ibrahim, M., and P. Alagidede. 2017. “Financial Development, Growth Volatility, and Information Asymmetry in Sub‐Saharan Africa: Does Law Matter?” South African Journal of Economics 85 (4): 570–588. Iliev, P., K. V. Lins, D. P. Miller, and L. Roth. “Shareholder Voting and Corporate Governance around the World.” The Review of Financial Studies 28 (8): 2167–2202. Iwanow, T., and C. Kirkpatrick. 2009. “Trade Facilitation and Manufactured Exports: Is Africa Different?” World Development 37 (6): 1039–1050. 53 Jales, H. 2017. “Estimating the Effects of the Minimum Wage in a Developing Country: A Discontinuity Design.” Journal of Applied Econometrics 33: 29–51. Janiak, A. 2013. “Structural Unemployment and the Costs of Firm Entry and Exit.” Labour Economics 23: 1–19. Jerbashian, V., and A. Kochanova. 2016. “The Impact of Doing Business Regulations on Investments in ICT.” Empirical Economics 50 (3): 991–1008. Kanbur, R., and L. Ronconi. 2016. “Enforcement Matters: The Effective Regulation of Labor.” International Labor Review 157 (3): 331–356. Kant, C. 2016. “Are Institutions in Developing Countries Malleable?” Journal of Policy Modeling 38 (2): 272–289. Kenny, C. 2007. “Construction, Corruption, and Developing Countries.” Policy Research Working Paper No. 4271, World Bank Group, Washington, DC. Klapper, L. F., and I. Love. 2010. “The Impact of Business Environment Reforms on New Firm Registration.” Policy Research Working Paper No. 5493, World Bank Group, Washington, DC Klapper, L., L. Laeven, and R. Rajan. 2006. “Entry Regulation as a Barrier to Entrepreneurship.” Journal of Financial Economics 82 (3): 591–629. Korinek, J., and P. Sourdin. 2011. “To What Extent Are High-Quality Logistics Services Trade Facilitating?” OECD Trade Policy Papers, No. 108, OECD Publishing, Paris. https://doi.org/10.1787/5kggdthrj1zn-en. Kovač, M. 2016. “Institutional Development, Transaction Costs and Economic Growth: Evidence from a Cross-Country Investigation.” Journal of Institutional Economics 12 (1): 129–159. Kraay, A., and N. Tawara, N. 2013. “Can Specific Policy Indicators Identify Reform Priorities?” Journal of Economic Growth 18 (3): 253–283. Lee, Y., and R. H. Gordon. 2005. “Tax Structure and Economic Growth.” Journal of Public Economics 89 (5–6): 1027–1043. Li, W. 2019. “Time Barrier to Export for OECD Countries.” Economics Letters 175: 106–112. Martincus, C. V., J. Carballo, and A. Graziano. 2015. “Customs.” Journal of International Economics 96 (1): 119–137. Menezes, A. P. 2014. “Debt Resolution and Business Exit: Insolvency Reform for Credit, Entrepreneurship, and Growth.” Public Policy for the Private Sector, Note 343, World Bank, Washington, DC. Moïsé, E., and S. Sorescu. 2013. “Trade Facilitation Indicators: The Potential Impact of Trade Facilitation on Developing Countries’ Trade.” OECD Trade Policy Paper 144, OECD Publishing, Paris. https://doi.org/10.1787/5k4bw6kg6ws2-en. 54 Nangpiire, C., R. G. Rodrigues, and I. O. Adam. 2018. “Ease of Doing Business and Foreign Direct Investment Inflow Among Sub-Sahara African Countries. International Journal of Business and Emerging Markets 10 (3): 289–303. Nunn, N. 2007. “Relationship-Specificity, Incomplete Contracts, and the Pattern of Trade.” The Quarterly Journal of Economics 122 (2): 569–600. Packard, T. G., and C. E. Montenegro. 2017. “Labor Policy and Digital Technology Use: Indicative Evidence from Cross-Country Correlations.” Policy Research Working Paper 8221, World Bank Group, Washington, DC. Pinotti, P. 2012. “Trust, Regulation and Market Failures.” Review of Economics and Statistics 94 (3): 650–658. Poel, K., W. Marneffe, S. Bielen, B. Van Aarle, and L. Vereeck. 2014. “Administrative Simplification and Economic Growth: A Cross-Country Empirical Study.” Journal of Business Administration Research 3 (1): 45. Porta, R. L., and A. Shleifer. 2008. “The Unofficial Economy and Economic Development.”. National Bureau of Economic Research Working Paper No. w14520, Cambridge. Portugal-Perez, A., and J. S. Wilson. 2009. “Why Trade Facilitation Matters to Africa.” World Trade Review 8: 379. Poschke, M. 2010. “The Regulation of Entry and Aggregate Productivity.” The Economic Journal 120 (549): 1175–1200. Schivardi, F., and E. Viviano. 2011. “Entry Barriers in Retail Trade.” The Economic Journal 121 (551): 145–170. Selwaness, I., and Z. Chahir. 2015. “On the Interaction Between Trade Reforms and Labor Market Regulation: Evidence from the MENA Countries’ Labor Markets.” Working Paper 970, Economic Research Forum, Cairo. Shepherd, B. 2009. “Speed Money: Time, Corruption, and Trade.” MPRA Paper 17337, University Library of Munich, Germany. Van Stel, A., D. J. Storey, and A. R. Thurik. 2007. “The Effect of Business Regulations on Nascent and Young Business Entrepreneurship.” Small Business Economics 28 (2): 171–186. Westmore, B. 2016. “Scaling New Heights: Achievements and Future Challenges for Productivity Convergence in Lithuania.” OECD Economics Department Working Paper No. 1307, OECD Publishing, Paris. Williams, C. C. 2014. “Tackling Enterprises Operating in the Informal Sector in Developing and Transition Economies: A Critical Evaluation of the Neo-Liberal Policy Approach.” Journal of Global Entrepreneurship Research 4 (1): 1–17. 55 Appendix D. Analysis of Bank Group Completion and Learning Review of Country Assistance Strategies and Country Partnership Strategies Description of the Approach and Data Set The Independent Evaluation Group (IEG) analyzed the most recent country strategy Completion and Learning Report Reviews (CLRRs) for 65 countries to learn the extent to which the World Bank’s country strategy objectives were informed or motivated by Doing Business (DB), the extent to which they achieved the desired business environment reform outcomes, and the relevant lessons from the reviews. This review complemented a review of 61 Country Partnership Framework documents (CPFs) also conducted by IEG for the purpose of the DB evaluation. To make it comparable, the team reviewed the latest CLRRs with their corresponding Country Assistance Strategy (CAS), Country Partnership Strategy (CPS) for the same countries as in the CPF analysis. The final review included 61 countries, those with available CLRRs. The review looked at how each country incorporated the DB report or indicators into its CAS or CPS using the rating methodology summarized in table D.1, identifying the type of DB interventions proposed in the strategy documents and the level of achievement of these interventions. Table D.1. Rating Methodology for Country Assistance Strategy or Country Partnership Strategy Coverage of Doing Business Issues Rating Description None or minor CAS or CPS does not mention DB reform/indicator issue at all CAS or CPS briefly mentions country’s DB ranking/indicators Substantial or priority CAS or CPS substantially discusses DB rankings/indicators (one or two paragraphs of context with for reform multiple DB indicators or areas cited), but DB issues are not a top priority in the proposed reforms CAS or CPS lists DB reform priorities without discussion in the document CAS or CPS lists DB-related reforms and brief discussion of the DB country rankings and indicators CAS or CPS lists DB-related reforms and substantial discussion of the DB country rankings and indicators Source: Independent Evaluation Group review. Note: CAS = Country Assistance Strategy; CPS = Country Partnership Strategy; DB = Doing Business. Of the 61 reviewed CAS and CPS documents, the largest number correspond to countries in the Sub-Saharan Africa Region (30 percent), followed by East Asia and Pacific (25 percent), Latin America and the Caribbean (21 percent), Europe and Central Asia (10 percent), South Asia (8 percent), and Middle East and North Africa (7 percent). By income level, lower-middle-income countries account for 38 percent of the reviewed 56 CAS and CPS documents, followed by upper-middle-income countries with 35 percent; approximately one-fifth of the countries are in the low-income category and 10 percent are high-income countries. More than half of the strategies reviewed correspond to International Development Association or blended support–eligible countries, and 15 percent correspond to countries with fragility, conflict, and violence (FCV) status (figure D.1). Nine of the CAS or CPS documents reviewed were prepared before fiscal year (FY)10, while the rest were prepared between FY10 and FY15. Figure D.1. Characteristics of the Countries in the Sample a. By Region b. By income level c. By FCV status Source: Independent Evaluation Group review. Note: Panels a and b exclude one strategy in an income-unclassified country. CAS = Country Assistance Strategy; CPS = Country Partnership Strategy; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; FCV = fragility, conflict, and violence; MNA = Middle East and North Africa; SAR = South Asia; SSA = Sub- Saharan Africa. Summary of Findings Overall Description The review revealed that 64 percent of the country strategies (39 countries) substantially discuss or propose DB-related reform priorities (figure D.2). In only seven countries’ strategies (Belize, Brazil, Guatemala, Lao People’s Democratic Republic, Maldives, Nepal, and Vietnam) are DB issues not mentioned, while DB ratings, indicators, or both are briefly discussed in 15 countries’ documents. In one country, DB issues are substantially discussed but not taken into consideration for reform proposals. The 57 strategies of 38 countries considered reforms measured or justified using the DB report regardless of the level of discussion of DB in the document. Figure D.2. Degree of Discussion of Doing Business Reform Issues in Country Strategies Source: Independent Evaluation Group review. Note: CAS = Country Assistance Strategy; CPS = Country Partnership Strategy. Just over half of the reviewed CAS and CPS documents used DB as an indicator of reform progress. In 28 percent of these country strategies, DB was used as a justification for the proposed work program, and in 10 percent it was used to describe the business environment (figure D.4, panel a). Different country strategies use DB in different ways. For example, the 2014–2017 country strategy for Poland proposed that the International Bank for Reconstruction and Development would focus on reducing regulatory burdens for private sector development in the areas where the country was lagging the most in DB. (World Bank Group, 2018a) This CPS measured results for this objective using several DB indicators, such as reducing the time to start a business from 32 days to 21 days, the time to obtain construction permits from 32 to 21 days, and the time to get electricity from 186 to 123 days. Different DB statistics were used in different contexts, with the absolute score cited most often (52 percent of documents), the DB ranking cited in 33 percent of country strategies, and the “distance to frontier” cited in only 8 percent of the strategies (figure D.4, panel b). 58 Figure D.3. Use of Doing Business Report in in Country Assistance Strategies and Country Partnership Strategies a. Overall use (n = 61) b. Most common DB indicators (percentage of country strategies, n = 61) Source: Independent Evaluation Group review. Note: Numbers do not add up to 100, as a single CAS/CPFs can use the DB in multiple ways. CAS = Country Assistance Strategy; CPS = Country Partnership Strategy; DB = Doing Business. By area, country strategies most commonly referenced the overarching ease of doing business indicator (44 percent), while 43 percent of them mentioned the starting a business area, followed in frequency by registering property, trading across borders, and dealing with construction permits. The area of employing workers was not referred to in any country strategy (figure D.5). 59 Figure D.4. Discussion of Doing Business Reform areas in Country Assistance Strategies and Country Partnership Strategies Source: Independent Evaluation Group review. Note: Numbers do not add up to 100 percent because a single Country Assistance Strategy or Country Partnership Strategy can include interventions related to multiple areas. n = 61. Ease of doing business is the most covered category in four Regions (figure D.6, panel a):1 the Middle East and North Africa, East Asia and Pacific, Latin America and the Caribbean and South Asia. Regarding specific business areas, starting a business is the predominant theme in Sub-Saharan Africa, while in Europe and Central Asia, it is dealing with construction permits. In Latin America and the Caribbean and East Asia and Pacific, trading across borders is the second-most commonly cited area. In the East Asia and Pacific, South Asia, Europe and Central Asia, and Sub-Saharan Africa Regions, getting credit is an important topic, although the rest of the Regions do not address this issue. In lower-income countries (figure D.6, panel b), starting a business and paying taxes were the most predominantly discussed business areas (accounting for 37 percent of the DB mentions). In lower-middle and upper-income countries, discussions of the different DB areas are similarly distributed: Starting a business, getting credit, and registering property were the top-mentioned business areas in lower-middle-income countries, and starting a business, getting credit, and dealing with construction permits were the most common areas in upper-middle-income countries. The shares of resolving insolvency 1This category was coded when the Country Assistance Strategy or Country Partnership Strategy referred to promoting a general improvement of the private sector or business climate as measured by the Doing Business indicator. 60 and getting electricity are higher in high-income countries compared with the other income group levels, at 12 percent and 6 percent, respectively. Almost half of the DB references focus on three business areas in FCV countries (figure D.6, panel c). The main theme is starting a business, followed by trading across borders and enforcing contracts, both at 12 percent. By contrast, in non-FCV countries, frequently mentioned areas were starting a business, registering property, and trading across borders. In Haiti, the Prime Minister created a working group on Doing Business reform that includes members from the public administration and representatives from the private sector and has appointed a coordinator. However, progress on the objective of reforming the business environment has been limited (CLR Review, World Bank Group, 2015). 61 Figure D.5. Discussion of Doing Business Topics a. By Region b. By income level c. By FCV status 62 Source: Independent Evaluation Group review. Note: Figures show the distribution of the 129 Doing Business mentions. FCV = fragility, conflict, and violence. Of the 61 reviewed CAS and CPS documents, 38 proposed a work plan that discussed DB reforms and one briefly mentioned the DB rankings and that the International Finance Corporation (IFC) and the World Bank were working on the topic. Among these 38 CASs and CPSs, 37 included advisory services and 20 lending support. Overall, 48 percent of the revised CASs and CPSs planned an IFC advisory services DB project and 25 percent a DB World Bank advisory services and analytics (among those three were reimbursable advisory services). On the investment side, 21 percent of the revised CASs and CPSs mentioned a World Bank DB investment project financing, 10 percent a DB development policy operation, and 8 percent an IFC investment services DB-related project (figure D.7). Significantly, some investment projects might have arisen from an earlier advisory intervention, but a deeper analysis would be needed to identify them. For example, in Cabo Verde, the analysis of the interaction between institutions shows only limited evidence of cooperation. The Completion and Learning Review (CLR) notes that IFC worked closely with the World Bank on the investment climate and DB indicators, which led to a DB task force and the adoption of a national action plan for investment climate reforms. The CLR also notes that information sharing between the World Bank, IFC, and the Multilateral Investment Guarantee Agency was adequate, but coordination on the micro, small, and medium enterprises agenda and access to financing was weak (CLRR, World Bank 2019a). 63 Figure D.6. Doing Business Interventions by Institution Source: Independent Evaluation Group review. Note: Numbers do not add up to 100, as a single CAS or CPS can include interventions supported by multiple institutions. n = 61. ASA = advisory services and analytics; CAS = Country Assistance Strategy; CPS = Country Partnership Strategy; IFC- AS = International Finance Corporation advisory services; IFC-IS = IFC investment services. The type of intervention identified in DB-related projects included in CASs and CPSs is heavily focused on two matters: improving business laws (38 percent) and streamlining procedures (34 percent; figure D.8). Thus, DB is used to influence at an upstream level through laws and regulations and also at a downstream level through the efficiency of business procedures and requirements. Unexpectedly, interventions related primarily to capacity building account for just 3 percent. The Seychelles is a good example of a country where a combined upstream and downstream strategy was implemented. An evaluation of the development policy loan series reports that the government implemented an online system to register companies and a virtual one-stop shop for starting a business, permitting the automation of business processes. It also introduced a flat-fee structure for services associated with company registration. Revisions to the 1972 Companies Act allowed for the creation of a virtual one-stop shop. The CLR reports that reforms led to a decrease in the costs involved in registering a business. Data from DB confirm that the cost to start a business decreased in the Seychelles, while the Seychelles’ overall ranking in DB improved from 103/182 and 93/190 between 2011 (see 2012 report) and 2016 (Seychelles CLR Review, World Bank Group 2018). 64 Figure D.7. Intervention Type in Country Strategy Documents Source: Independent Evaluation Group review Note: Numbers do not add up 100, as a single CAS or CPS can include multiple interventions. n = 61. CAS = Country Assistance Strategy; CPS = Country Partnership Strategy. Effectiveness The review of the CLRRs reveals that among the 38 CASs and CPSs that proposed a DB- related work program, 74 percent of them achieved or mostly achieved the DB corresponding objectives (figure D.9). Disaggregated by income level, DB-informed interventions in low-income countries were more successful, with 86 percent of interventions achieving their objective to some degree; this number decreases to 79 percent for lower-middle-income countries, 64 percent in upper-middle-income countries, and 80 percent in high-income countries (although low in number). The success rate shows a significant disparity across Regions. Although Regions such as Latin America and the Caribbean, Sub-Saharan Africa, and Europe and Central Asia achieved the objectives of more than 75 percent of their interventions, this number decreases to 67 percent in South Asia, 50 percent in the Middle East and North Africa, and 33 percent in East Asia and Pacific. Country examples include Georgia, where the relationship of governance challenges to business environment reform emerged as a central concern. Success has arguably been augmented by putting better governance front and center in the country’s program. The various international indexes of governance, transparency, DB, investment climate, and competitiveness all served as important metrics for government achievements and its communication to citizens of the value of the progress achieved (Georgia CPSPR; World Bank Group 2014). 65 Figure D.8. Achievement of Objectives for the Doing Business Reform Interventions Proposed in Country Strategies a. Achievement of objectives b. Success rate by income level c. Success rate by Region Source: Independent Evaluation Group review. Note: n = 38. CAS = Country Assistance Strategy; CPS = Country Partnership Strategy; DB = Doing Business; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia; SSA = Sub-Saharan Africa. 66 Achievement of declared objectives does not necessarily translate into improvements in DB indicators. This could be because some business environment reforms were outside the scope of what DB measures or reflect limits to the value of DB as a reform- monitoring tool. Although 74 percent of the 38 countries with a DB-related program achieved or mostly achieved their business environment reform objectives, only 45 percent also showed improvements in DB indicators (fig. D.10). This result suggests the need for a deeper analysis. Box D.1. De Facto versus De Jure According to the Tunisia Country Program Evaluation for 2005–13, the Doing Business survey may be an accurate reflection of business regulations de jure, but it does not accurately reflect the very difficult de facto business environment. In both World Bank Doing Business and the World Economic Forum’s Index of Competitiveness, Tunisia’s overall ranking with respect to the ease of doing business improved throughout 2014. However, the evaluation raised questions about the accuracy of Doing Business indicators in assessing the investment climate in the circumstances prevailing in Tunisia before the revolution. Apparently the government, directly or indirectly, guided interviewers toward informants representing offshore private companies. Consequently, the indicators largely reflect the conditions in the offshore regime. Source: World Bank 2014. Figure D.9. Improvement in the Doing Business Indicators Source: Independent Evaluation Group review. Note: n = 38. CAS = Country Assistance Strategy; CPS = Country Partnership Strategy; DB = Doing Business. Finally, a review of a sample of 28 of the 61 CASs and CPSs shows that DB is the leading source of information to measure the business environment (figure D.11). A full 60 percent of the strategy documents referenced DB to measure progress on the business environment, while 40 percent used alternative sources such as International Monetary Fund Country Reports, national records used by governments, the World Bank Enterprise Survey, the Legatum Prosperity Index, World Bank indicator-based reform, the World Economic Forum Global Competitiveness Index, and the Fraser Institute’s Economic Freedom of the World Report. 67 Figure D.10. Data Sources Used to Measure Business Environment Source: Independent Evaluation Group review. Note: n = 28. DB = Doing Business; GCI = Global Competitiveness Index; IMF = International Monetary Fund; WEF = World Economic Forum. 68 Annex D.1. Additional Examples Philippines, 2009–18 Country Program Evaluation: Some indicators show improvements in the business environment in some local government units, but results are inconclusive and were not replicated at the national level (World Bank Group, 2019a). Recently enacted laws on competition (2015) and doing business (2018) are expected to reduce red tape. Similarly, the passage of bills on land and valuation should ease land transactions. The World Bank Group contributed to policy development and streamlining processes (mainly in selected subnational units). This included International Finance Corporation (IFC) advisories on shipping, cabotage, agricultural imports, and drafting laws on competition and doing business through the Trade and Competitiveness program. IFC contributed to improving quarantine procedures to reduce the costs of complying with import. The Republic Act No. 11032, known as the “Ease of Doing Business and Efficient Government Service Delivery Act of 2018,” amends Republic Act No. 9485, otherwise known as the Anti-Red Tape Act of 2007. Rwanda, 2019 Country Program Evaluation: The country has implemented several regulatory reforms, leading to positive developments in the business environment in recent years (World Bank Group, 2019c). In global terms, Rwanda ranked 41 in Doing Business (DB) 2018 (World Bank 2018), up from 62 in 2016 and 150 in 2008. Yet foreign direct investment levels remain low (3 percent of gross domestic product in 2016) and the private sector’s contribution to the economy is still small. A complex political economy environment has been cited as limiting fair competition and effective implementation of regulatory reforms, with holding companies closely affiliated with the government, the ruling party, and the military playing a dominant role in the private sector. The private sector remains underdeveloped with a narrow manufacturing base, limited diversification of exports, a small financial sector, and an underdeveloped services sector. Sustaining growth and poverty reduction—from already impressive achievements—will require significant structural change in the economy. Most importantly, in the face of declining foreign aid, limited public resources, and increasing government debt, the sustainability of the public investment–driven growth model of the past two decades is in doubt. Tanzania, CLR Review FY12–16: Moreover, in the 2018 DB report (World Bank 2018d), Tanzania ranks 137 out of 190 countries, which compares less favorably with its Sub- Saharan Africa neighbors and reveals weak private sector competitiveness. … In the 2018 DB report, Tanzania’s ranking of 137 is significantly below its Sub-Saharan Africa neighbors—Kenya (80), Rwanda (41) and Uganda (122). 69 However, some interventions did not directly contribute to the achievement of the Country Assistance Strategy objectives and outcomes. For instance, the CLR lists eight International Development Association operations and four IFC operations supporting the first program objective, on constraints for DB and financial intermediation. In fact, only one of these interventions directly focuses on the business environment (the Business Environment and Industry Development for Jobs development policy operation), and it was approved at the end of fiscal year 2016—too late to have a meaningful impact on the Country Assistance Strategy objective. Albania, Country Program Evaluation (2021): The initial reforms have also begun improving the investment climate and tackling corruption in public service delivery (World Bank Group 2021). Albania was ranked for the first time in the top half of the global ranking in DB 2014 (World Bank 2014) and has continued to improve its business regulations, narrowing the gap with the global frontier in DB 2018 (World Bank 2018). Albania’s World Governance Indicators ranking in control of corruption improved from 27 in 2012 to 41 in 2016. Nevertheless, the Country Partnership Framework objective with respect to reducing Albania’s distance to frontier (Doing Business) was not achieved in 2018. The absence of a reliable and affordable power supply is a top constraint to doing business in Albania and inefficiencies in the energy sector are a major source of fiscal imbalance. Albania’s competitiveness is also impaired by inadequate road infrastructure. Moldova, Country Partnership Strategy Review (2013): Regarding improved competitiveness of Moldova’s enterprise sector, there is evidence that good progress was made in improving Moldova’s Metrology, Standards, Testing, and Quality system to European Union standards and reducing the regulatory compliance costs, with Moldova’s overall DB ranking improving from 103 out of 181 countries in 2008 to 83 out of 185 countries in 2012 (World Bank Group, 2013). However, data are not available on the share of enterprises with access to formal sources of investment finance, and the share of overall credit going to the private sector has not increased. The objective was for regulatory compliance costs for enterprises to decrease from 16 percent in 2007 to less than 12 percent in 2012 (as measured by Cost of DB survey). Actual results reported were that regulatory compliance costs stabilized at 10 percent over the prior three years (2010–2012). Mexico, Country Program Evaluation (2018): Many subnational governments adopted recommended reforms with the notable impact of nationwide replication by the Mexican government, by decree, in 2015–16 (World Bank Group, 2018b). IFC’s subnational DB reports (initiated in 2005, repeated in 2007, and since rolled out to all 31 states) proved a 70 catalyst for reforms, with 9 of 12 pilot states initiating reforms.2 DB in Mexico is now a key reference point for policy makers, and its reports are prepared with federal and state budgets. IFC’s diagnostics were useful, the pilot interventions were extended and adopted in Mexico City and the state of Guerrero, and rollout in the state of Oaxaca is underway. Although attribution is difficult, a time-series analysis suggests a trend in improvement in the overall DB indicator. References World Bank Group. 2013. Moldova CPSCR Review FY09–13. Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2014. Georgia CPSCR Review. Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2014. Tunisia—Country Program Evaluation FY2005–13. Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2015. Haiti CLR Review FY09–14. Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2017. Doing Business 2018: Reforming to Create Jobs. Washington, DC: World Bank Group. World Bank Group. 2017b. Moldova CLR Review FY14–17 Independent Evaluation Group. Washington, DC: World Bank Group. World Bank Group. 2018a. Poland – CLR Review FY12–17. Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2018b. An Independent Evaluation of the World Bank Group’s Support to Mexico (2008–17). Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2018c. Seychelles CLR Review FY12–16. Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2018d. Tanzania CLR Review FY12–16. Independent Evaluation Group. Washington, DC: World Bank. 2World Bank comments that “the Subnational Doing Business … were a joint IBRD [International Bank for Reconstruction and Development / IFC [International Finance Corporation] effort. … IBRD/IFC have produced 6 subnational reports (2005, 2007, 2009, 2009, 2012, 2014, and 2016). The Subnational Doing Business report in 2016 covered 32 states (including Mexico City)…. The phrase ‘improvement in the Overall Doing Business indicator’ is not clear: it would be better to use instead ‘improvement in the Ease of Doing Business index’” (World Bank Management comment on Issues Paper). 71 World Bank Group. 2019a. Cabo Verde CLR Review FY 15–17. Independent Evaluation Group. Washington, DC: World Bank World Bank Group. 2019b. The World Bank Group Partnership with the Philippines, 2009 –18 – Country Program Evaluation. Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2019c. An Evaluation of the World Bank Group’s Support to Rwanda (2009–17) Independent Evaluation Group. Washington, DC: World Bank. World Bank Group. 2021. Albania Country Program Evaluation FY 2011–19 Independent Evaluation Group. Washington, DC: World Bank. 72