49249 MAY 2009 Show Me the Money II: From Concept to ABOUT THE AUTHORS Practice SANDA LIEPINA is a senior operations manager More than a year and a half has passed since IFC Advisory Services in Eastern and oversees IFC Advisory Services in the Eastern Europe Europe and Central Asia (AS ECA) introduced a standard methodology for and Central Asia (AS ECA) assessing aggregate cost savings (ACS) for businesses resulting from IFC- regional Business Enabling supported Business Enabling Environment (BEE) reforms. Within the region, Environment (BEE) portfolio. She has managed regulatory 13 regulatory reforms in six countries have been assessed with this reform initiatives from the side methodology, showing an estimated $301 million in cost savings for the of government (the successful private sector with an average impact of $29 for every dollar spent on these Business Environment Improve- ment reform program in Latvia, advisory projects. These results look impressive, but they also raise questions 1998­2002) and the side of on the concept and application of the methodology: How do we know this advisory services providers is adequate impact for the resources invested? What else needs to be done (IFC, EC, USAID, etc.) in various Central and Eastern Europe to refine the approach? Can these types of impact measures be applied as countries, Commonwealth of decision tools at the program level? Independent States countries, and China and Latin America. On a global level, a review of project supervision reports from the current SANWAREE SETHI active BEE portfolio reveals that 15 projects are reporting on the corporate- is a Monitoring and Evaluation wide standard ACS indicator. Thus, another question emerges: What does it Specialist. She oversees results measurement for IFC's take to implement a consistent approach across regions? Advisory Services in Eastern Europe and Central Asia. Background A summary of the methodology is available in CHRISTOPHER MILLER the annex, and a more detailed version can be joined IFC in 2008 as an operations officer for the BEE The ACS methodology was adapted from the found in the SmartLesson "Show me the portfolio in Eastern Europe and extensive work done in the area of impact Money: Quantifying the Impact of Regulatory Central Asia. evaluation for regulatory reforms by various Simplification Projects," published in June researchers in the European Union, the 2007. Organisation for Economic Cooperation and Development, and the United States, as well The following lessons learned highlight the as analytical work conducted by Foreign key issues that were considered while Investment Advisory Service (FIAS). The operationalizing the impact evaluation methodology assesses aspects of the business methodology. environment before and after the IFC- supported reforms are enacted by the Lessons Learned government in order to quantify the changed costs for SMEs undertaking the given 1) Implementation: Simplicity and a procedure. Two types of costs are considered: conservative approach are key Direct costs include the economic costs (labor Success factors: or administrative costs) to an enterprise related to the regulatory procedure before · Provide a simple, easy-to-use Excel and after the reform. template for the project teams. Indirect (opportunity) costs include the impact · Rely on conservative, data-supported on profitability due to the productive use of approach focusing on direct impact. time that would otherwise have been dedicated to completing the regulatory After we completed the first "Show Me the procedure. For example, if registration time is Money" SmartLesson, we discovered that the reduced, then the SME can begin operation conceptual framework was not sufficient to and collect revenues earlier. allow easy data collection of impact results on IFC SMARTLESSONS -- MAY 2009 1 the ground. Individual projects operate in widely different · designing the related administrative reforms at the environments with varying priorities and resources. The agency level. project teams could do little with the formulas and the overall description of data points that were needed, as These dimensions are critical for solid attribution of the outlined in the methodology note. Explaining how every impact to the IFC project. Impacts of reforms where IFC's calculation has to be made to every team member was a input can be questioned are not included. For example, the daunting task. Therefore, the initial focus in 2007 was on region has been tracking but not including in the impact turning the conceptual methodology into a practical, self- calculations the reforms that have been implemented based explanatory Excel-based tool1, which allows for the entry of on the general recommendations provided in policy papers very clear and predefined data points. This simplified Excel- or survey reports that the projects publish. Since 2005, these based tool can be implemented at the project level in projects have contributed to amendments or revisions of a diverse environments with minimal support and training total of 53 regulations enacted by the governments, but we required. Once it was shown to work in the field, the have quantified the impact of only 13 reforms that have regional monitoring and evaluation (M&E) team focused met the criteria for significant contribution from IFC and on ensuring that the methodology was applied uniformly lend themselves to this type of quantification (See Lesson across projects and products to ensure consistent results. Two). To gather input for the Excel-based template, public data Relying on easily accessed data sources, using a simple are used whenever possible, supported by surveys and other template, and maintaining a conservative approach of only credible sources.2 To increase the simplicity and reliability of measuring what can be supported by the data and is clearly the methodology, the template focuses only on immediate attributabletoIFC,havebeenkeytocreatingtheconsistency impacts that can be attributed to the regulation, quantified that allows for aggregation and comparison of data across and supported with data. By avoiding attempts to measure different projects and countries. secondary impacts, such as social benefits, effects on rate of formalization, etc., the resulting tool is very simple for 2) Consistency: Impact measurement starts with project projects to use with impact results that are easily verifiable design. and that can be aggregated across different projects. The downside to this approach is that the methodology cannot Success factors: be applied to every type of reform and will only collect a portion of the overall impact achieved. Although various · Incorporate the M&E template into project design. approaches to measuring some secondary benefits of regulatory change have been considered, thus far no · Be aware of the limitations in quantifying impact for standard solution has been found that can be rolled out different regulatory reforms. across multiple projects. This is our key challenge going forward: how to expand the scope of what gets quantified At project design stage, setting impact targets becomes an while still maintaining the simplicity of the tool, which has easier and more meaningful task with this measurement been a key success factor so far. For the time being the tool because it requires structured thinking about the secondary benefits can still be captured through qualitative potential impact of regulatory changes. This helps to assessments. structure projects around regulatory issues that are likely to yield the greatest impact. For example, a project design Implementation also involves a number of judgments about team evaluating various options for reforming business what to include in the impact calculation. For example, a licensing can use the tool to conduct scenario testing on common issue is how much involvement a project must various regulatory reforms and determine which will have have in a given regulatory reform in order to count and the greatest impact and affect the largest population. In attribute its impact. The ECA region has maintained a addition, previous results quantified through the conservativeguidelinethatrequiresprojectstodemonstrate methodology can guide future targets. This data-driven, significant contribution to a regulatory change in order to impact-oriented approach has also been welcomed by the count the impact. Significant in this case means direct and donors funding the BEE programs in the region such as the verifiable involvement in the design of the specific reform Swiss State Secretariat for Economic Affairs (SECO), the UK as well as a close and sustained relationship with the Department for International Development (DFID), the government partner designing and implementing the Canadian International Development Agency (CIDA), the reform. This has to be demonstrated via evidence of Swedish International Development Corporation Agency assistance provided in-- (SIDA), and the Agency for International Business and Cooperation (EVD). · drafting the required policy papers to agree on the reform Finally, the ability to measure impact with this methodology is largely shaped by product and the ease of quantifying · drafting the legislation enacting the reforms; and/or various reforms. ECA experience has been that the current methodology is very effective for business operations and 1 Please see the related documents section on the SmartLessons Web site, below the business entry or exit and likely for tax simplification3 but abstract for this paper. 2 In most cases, publicly available data sources are sufficient to calculate the impact 3 Tax product in ECA is in the early stages of development and is yet to show its full of reforms. These data sources include state statistics agencies, various government potential. Preliminary estimates based on tax reform projects elsewhere suggest that ministries and agencies, enterprise surveys, and Doing Business reports. Other sources tax simplification reforms have the potential to yield significant benefits for the pri- of data that are used include focus groups, expert estimates, and targeted surveys vate sector and affect essentially all businesses. FIAS experience from the late 1990s conducted as part of the BEE project. in Latvia showed tax simplification to be the area with largest impact by far (156 2 IFC SMARTLESSONS -- MAY 2009 perhaps may be somewhat less effective for other products, metrics normalized for country size have been introduced. such as legal reforms aimed at investor protection or These include the cost savings as a percentage of the private- substantive provisions of the tax code. The reason for this is sector share of GDP and the cost savings per registered that the nature of changes enacted--procedural changes business entity in the country. and changes to the requirements that the business has to satisfy to engage in a certain activity--lend themselves to quantification relatively easily. (All of these requirements Table 2. Results by country type take time and resources to comply. Once we know the time, we can express it in monetary values.) At the same time, Country $ Impact/ Impact as a % of Impact per improvements in legal provisions for minority shareholders Type $ spent private sector shared registered of GDP entity or improvements in cost-deduction rules for tax purposes cannot be quantified this way. Quantifying these MIC 122 Ops Entry/Exit Tax Ops Entry/ Tax Exit improvements would require a different set of data that is not as easily available in our countries of operation. The 0.2% 0.02% NA 67 7 NA response on a product level should be to expand the IDA 35 0.1% 0.06% 0.01% 44 24 5 methodology where feasible to capture the benefits of reforms and contribute to project design decisions but, at the same time, to ensure that the cost savings indicator is We also looked at the impact by the income group. The table not prioritized to a degree that would result in the exclusion 2 shows that while middle-income countries (MIC) do very of reforms that can't be easily quantified. well in absolute dollar terms, when the same results are put into perspective, based on GDP and the number of registered 3) Evaluation: Interpreting the results is as important as entities, the trends and possible impact lessons that could be collecting them. derived from the data look quite different. Success factors: Even with normalized data for countries within the region, we still do not know if the level of impact generated in this · Normalize impact on an aggregate level. region is adequate, compared to what IFC could achieve with the same resources and products elsewhere. In order to do · Standardization of impact calculation methodology IFC- this, we need a standard impact measurement methodology wide is a key next step. across all regions. The current practice where each region assesses impact by its own methodology weakens the aggregate results and makes it more difficult Table 1. Results by country4 to tell a compelling and credible story about the Country Key Reforms Reforms by Impact $ Impact/ $ Impact/# work being done by the business line and what Product (MUSD) $ spent registered impact we have on private-sector development businesses5 globally.Discussionswithdifferentregionsindicate Azerbaijan Registration Entry/ Exit $8.3 24 47.5 that there is interest in pursuing this, assuming Belarus Registration Entry/ Exit $21.5 9 86.3 endorsement from the BEE business line and the Georgia Mining Regulation Operations $1.3 1 1.6 IFC Results Measurement Unit. Tajikistan Inspections, tax admin Operations $11.2 6 79.8 To introduce a uniform approach across all regions Uzbekistan Inspections, permits, Operations, $47.1 35 95.1 to allow for meaningful aggregation and registration, entry/ exit, comparison of impact data, we need to address liquidation, tax admin taxation several issues: Ukraine Inspections, permits Operations $212 63 70.1 · Attribution: There is no standard for attribution among products such as the public- Some of the impact data generated in ECA are shown in the private dialog (PPD), the Doing Business reform following tables. advisory, and business operations. PPD projects in the Mekong region have addressed this issue in their Initially, we looked only at the total impact per country. From impact assessment,6 and some business operations these results, it became apparent that we needed to projects have regional standards; but these are not normalize the impact numbers so that there could be unified across different regions. meaningful comparison across countries. Larger economies produce larger absolute impacts simply due to the greater · Timeline: Some impact results are calculated over a number of businesses affected by the changes. As a result it is multiyear timeline while others are not, which leads to difficult to compare $200 million in cost savings in Ukraine inconsistent reporting of impact. In ECA, the impact is with $11 million in Tajikistan. To address the issue, additional calculated on a one-year horizon to maintain a MUSD of 171.5 MUSD of total cost savings). FIAS Occasional Paper #18 by Sanda consistently conservative approach; some regions include Liepina, Jacqueline Coolidge, and Lars Grava (November 2007). http://www.ifc.org/ a three- or five-year outlook. ifcext/fias.nsf/Content/FIAS_OccasionalPapers 4 For 13 ECA BEE supported specific regulatory reforms for which impact was quanti- fied with this methodology. 5 Including legal entities (companies), sole proprietors, agricultural enterprises 6 Impact Assessment of the Public-Private Dialogue Initiatives in Cambodia, Lao PDR, (farms). Vietnam. Accessed through iDESK for Capacity Building for Public Private Dialogue project (539685). IFC SMARTLESSONS -- MAY 2009 3 Table 3.Aggregate Cost Savings by Region and Product globally7 REGION # OF PROJECTS USING TOTAL VALUE (USD) ACS INDICATOR OF REPORTED ACS SSA 2 114,001,932 EAP 5 279,679,769 ECA 4 33,955,917 LAC 3 768,372 CSA 1 8,000 Total 15 428,413,990 · Documentation: Very few projects provide supporting information in the Project Supervision reports on the impact being counted, the nature of the reforms, or the methodology used in the calculation of the value. As a result, while technically we do have aggregate data for the business line (Table 3), we cannot rely on it for meaningful analysis. Conclusion The strengths of this methodology are ease of use at the project level and the ability to meaningfully aggregate the results across DISCLAIMER projects. The tradeoff for this simplicity is that IFC SmartLessons is an awards the methodology provides a conservative program to share lessons learned estimate without evaluating most of the in development-oriented advisory secondary impacts. There are several services and investment operations. The findings, opportunities to continue to develop the interpretations, and conclusions methodology to address some of the current expressed in this paper are those limitations, but the immediate need is for a of the author(s) and do not standard system to be implemented across all necessarily reflect the views of IFC BEE projects globally. Deciding on a common or its partner organizations, the Executive Directors of The World approach will increase the data available for Bank or the governments they analysis and the resources available to help represent. IFC does not assume develop the methodology. any responsibility for the completeness or accuracy of the 7 Active portfolio, December 31, 2008. Data collected from PSRs. information contained in this Data from global projects reflected in regional portfolios to avoid document. Please see the terms double counting.The total sum of aggregate cost savings reported and conditions at www.ifc.org/ through PSRs as of FY09Q2 is $836.5 million, however when cor- smartlessons or contact the rected for double counting with "world projects" and visible data entry mistakes, the aggregate results, shown below, fall to $428.4 program at smartlessons@ifc.org. million IFC SMARTLESSONS -- MAY 2009 4 ANNEX 1: Quantifying the Impact of Regulatory Simplification Projects Bycomparingspecificaspectsofthebusinessenvironmentbeforeandafter1IFC-supportedreformsareenactedbythegovernment, it is possible to quantify the benefits accruing to the target population, i.e., the aggregate private-sector cost savings. The methodology described below can be used to assess the economic cost for businesses at four distinct stages along the reform cycle: Project establishes Project identifies Once the reform is Once the reform Is implemented, the a baseline in order to recommendations enacted, the project identify need for to be made and calculates ex-ante project collects data regulatory change sets impact target impact based on the a gain to verify the ex- reform specifics post impact achieved Detailed methodology: It distinguishes between two types of costs on businesses: · Direct costs: direct impact on economic cost (labor or administrative costs) of an enterprise resulting from the reform of regulatory procedure · Indirect (opportunity) costs: impact on revenues or costs, due the different use of time formerly dedicated to administrative procedures. To the extent possible, all of the data below should be specific to the affected sample of firms. For example, if the reform affects small farmers differently than it does individual entrepreneurs, the calculations below should be made for each subsector of the economy and then summed at the end for a total impact. If the data are not available at the subsector level, then countrywide data can be used as a substitute. Direct Costs Direct costs can be estimated at firm level, at a specific economic sector level, or for the SME sector as a whole, depending on the set of data available for each. AC = (PO + PU) * N The calculation of direct costs makes use of basic indicators and, in particular, PO = Cost of official payments related to the leverages IFC experience with surveys in the region. Direct costs can be completing the given procedure (data available differentiated between administrative costs (AC) and labor costs (LC). from official sources) PU = Cost of unofficial payments related to the completing the given procedure (data available Administrative costs (AC) can be calculated for each procedure by multiplying from surveys or estimated as a % on top of the the cost of the procedure2 by the number of times the procedure is undertaken official costs) by a representative firm per year: N = Number of times the given procedure is undertaken by a representative firm per year Labor costs (LC) can be calculated for each procedure by multiplying the cost of employees directly dedicated3 to the given procedure by the number of times the LC = DE * W * N DE = Number of full-time employee working days procedure is undertaken by a representative firm per year: dedicated to the given procedure W = Average daily employee wage Total direct cost is calculated as the impact on net profits of administrative and N = Number of times the given procedure is labor costs (AC + LC). A variation in administrative and labor costs increases gross undertaken by a representative firm per year profits but, in turn, will imply higher profit tax, i.e. reducing the total impact. For this reason, in order to calculate direct costs effects, an estimated average profit tax rate5 will be needed. The average tax rate can be estimated for the overall economy or be differentiated according to the typology of businesses (i.e. individual entrepreneurs versus legal entities) or economic sectors. Total Direct Cost = (AC + LC) * (1 ­ t) * F t = Average profit tax rate F = Number of firms affected by the given procedure 1 IFC ECA understands that, under ideal circumstances, impact assessments would involve the use of experimental analysis to compare the counterfactual of an IFC intervention rather than a before-after comparison. However, given that in our region the relevant legislation exists at the national level, it is not possible (or advisable) to construct municipal-level compari- sons for the sake of impact assessment. We believe that this methodology provides a sound alternative in cases where project intervention occurs at the national level, i.e., cases where it is virtually impossible to assess impacts using experimental methodology. 2 IFC BEE surveys collect data on total costs of the procedures, which are represented by the sum of official and unofficial payments. 3 These estimates assume that the employee time can be disposed of or dedicated to other administrative activities (versus revenue-enhancing ones) 4 Including all social benefits costs. 5 Estimated taking into account, among other factors, the share of revenues officially reported. IFC SMARTLESSONS -- MAY 2009 5 Indirect Costs DE = P * (DF ÷ DS) Indirect costs require a more detailed approach to calculations and greater use P = Average annual net profit for start-up firms, of assumptions. Overall, we distinguish between two main categories of for each industry or average for the specific opportunity costs: those related to delay of entry (DE) and those resulting sector affected by the given procedure from temporary closure (TC) of a firm's activities: DF = Average number of working days spent by firms in order to complete the given procedure DS = Average number of working days per year Delay of entry (DE) costs are those resulting from a new firm's entrance to the in the economy/sector14 market, i.e., costs from the deferral of launching profit-generating activities. Examples of these types of procedures are business registration, the first time a firm applies for a specific permit or license, or any other entry controls. The cost of this delay can then be measured as the proportionofprofits"lost"duetodelayedentryintothemarket,i.e.bymultiplyingtheannualnetprofitforstart-upcompanies by the proportion of working days spent on the given procedure:6 Temporary closure (TC) costs result from suspension of a firm's activity, i.e., the loss of productive activities for existing companies. Typical examples of procedures stopping economic activity are inspections, repeated licenses, repeated permits, and the suspension of activity due to the absence of licenses or permits. These costs are typically faced by existing companies and should only be calculated in cases where the closure or suspension was unjustified, or only for that subpopulation of firms that appealed and overturned the closure or suspension.7 The cost of this delay can be calculated by multiplying the average annual loss of a company whose activities are stopped by the proportion of working days that a company is stopped by the number of times the procedures is undertaken by a representative firm per year: TC = L * (DC÷ DS) * (1- t) * N L = Average annual losses for an active firm whose activity is stopped but which remains active, i.e. which retains all its production factors DC = Average number of working days a firm is closed due to the given procedure DS = Average number of working days per year in the economy/sector t = Average profit tax rate N = Number of times the given procedure is undertaken by a representative firm per year Total Indirect Cost = (DE * F) + (TC * F) F = Number of firms affected by the given procedure Total Costs If we sum to total direct and indirect costs from the baseline and compare them to the summed costs for our target ex-ante- or ex-post calculations, we are able to calculate the one-off savings resulting from the passed legislation. As seen in the attached template, in order to ensure conservative numbers, wherever applicable, data from the baseline year (e.g. number of firms affected, wage levels, etc.) are used in calculating impact. 6 Note that in the case of barrier to entry there is no need to multiply by the number of times the given procedure is undertaken in a given year; it is assumed that it is undertaken at most once a year. 7 Please review Excel templates with calculations to see how this has been done, it can be achieved based on results of overturned cases of firm closure. IFC SMARTLESSONS -- MAY 2009 6