62240 May 2011. Number 40 Ex-Post Impact Evaluation of an Export Promotion Matching Grant: Tunisia’s EMAF II 1 J Gourdon, JM Marchat, S Sharma, T Vishwanath (Bruhn 2008). This Quick Note is another addition to this body of work and presents results from Introduction: Among the root causes of the one of the first ex-post IE of an active export current political turmoil in the MENA region are promotion MG in a middle income country, the large number of unemployed but increasingly Tunisia. This IE was undertaken in preparation of educated youth. For the region to achieve a lending operation. stability, it will have to ensure a more inclusive and faster growth path and find enough jobs for Tunisia’s Export Development Projects: Two this cohort. Exports will play a key role in moving Export Development projects have been in this direction and this Fast Brief examines the implemented in Tunisia since 2000. Aside from impact of Matching Grants (MG) in supporting components on trade facilitation and on exports. establishing a pre-shipment export guarantee facility, both had an MG scheme, the Export Matching Grants: MGs are short-term, temporary Market Access Fund (EMAF). This IE, for reasons mechanisms that partially finance activities of data availability, was on EMAF II which started promoting improvements in the private sector. in 2005 and closes in mid-2012. Operating on a Since 1994, the World Bank has financed a total of budget of 18.4 million Euros, the team includes 3 37 PSD projects with MG components with 22 senior experts who select and advise beneficiaries. active in 2008. Matching grants do face criticism It provides non-reimbursable co-financing to focusing on "additionality" (funding activities that assist investments in market research and firms would have financed anyway) and programs to increase export market access at the “selectivity” issues (failure to distinguish between firm level. Its performance targets are set in terms private benefits and broader economic benefits) of value of exports, destination markets and (Biggs 1999), and on sustainability (Phillips, 2001). product diversification. This IE tries to estimate the impact of EMAF II on firms’ exporting MGs and Impact Assessments: Although there performance over 2004-08. have been MG assessments, few have used recent impact evaluation (IE) techniques 2. This is The Approach: In terms of Beneficiaries, EMAF II changing (McKenzie 2009) with recent IEs in SME has been well received by the Tunisian private support programs (Tang 2009), rainfall insurance sector and demand for support was high. By (Giné and Yang 2009), and regulatory reforms December 2009, 1710 firms (mostly SMEs) had applied, 72 % of these were accepted. Of all firms benefitting from EMAF II, a third had been in 1 - We are grateful to participantsinf the December 2010 workshop EMAF I while 167 dropped out without on “Impact Evaluation of Trade Interventions: Paving the Way” in Washington DC for comments. The survey was supported by the disbursements. Among the 1231 firms accepted MENA Region Impact Evaluation Initiative of the World Bank. since 2005, many were still implementing their Results using customs data come from joint work with DECTI (A activities at the end of 2009 and were not selected Mattoo & A Fernandes) and PRMTR (O Cadot). Simon Bell, Sector Manager, Finance and Private Sector Development, MENA Region, for the IE exercise. the World Bank cleared this Quick Note. 2 - Impact evaluation studies whether the changes in well-being are indeed due to a program intervention and not to other factors. Methodology - The performance of EMAF II was exporting status. The subsequent survey yielded a evaluated by comparing changes in relevant usable dataset of 428 firms (196 treated and 232 outcomes before and after across firms that had untreated) presenting similar distribution on key MG support (the “Treatment” group) and similar observables. firms but did not get EMAF II support (a “Control” group). Hence, it was needed to define PSM: First, the propensity score4 (PSM) for a control group similar to the treatment group, so receiving EMAF II support (i.e. to be treated) that changes in outcomes in the control group needs to be computed based on key firm were a good measure of what would have characteristics such as location, the age of the happened to EMAF II firms without such firm, sector, number of employees, sales, support3. exporting status in 2004, share of domestic capital, and number of years the current owner has run Given that an ideal ex-ante identical control group the firm. These variables were carefully selected was lacking, the best evaluation strategy was: i) to to ensure proper ex-post comparison group randomly select as control group Tunisian firms creation. The distribution of propensity scores is “similar” to the treatment group, and ii) compare shown next. As expected (since on average they the treatment and control groups after controlling are more likely to be in EMAF II) the P-scores of for differences along dimensions of size, age, firms in the treated group are more on the right sector, and prior exporting status of firms. side of the graph. Critically, the figure shows that Implicit assumed was that once these observables we will be able to compare a treated firm with an were controlled for, all other differences in identical untreated firm for most of the outcomes across these two groups before and distribution. Specifically, 404 firms (221 untreated after EMAF II are due to the MG. This approach and 183 treated) have common support. was applied to two datasets. One is an in-depth survey of 196 EMAF II firms and 232 similar Chart 1. PMS. control firms while the second dataset is based on detailed customs data and compares 400 EMAF II firms to 2600 Tunisians firms. IE on a Firm Level Survey: On Sampling, data show that EMAF II recipients are different from average Tunisian firms. They are more likely to 0 .2 .4 .6 .8 1 have been exporting before linking up with Propensity Score Untreated Treated: On support EMAF II, and their distribution across industries Treated: Off support is somewhat different from most Tunisian firms. Source: Authors’ calculation based on survey data. Such differences can be addressed through a random stratified sample of the control group Finally, it must be noted that with the matching which allows for selecting firms with similar process, the untreated group with high propensity profile to the recipients. Hence, the universe of score will be more used for comparison than those non-recipient firms was grouped into strata based with low propensity score so this will give more on three key observable characteristics - size, prior weight to firms that were “targeted” by EMAF II. exporting status and sector. Then, each stratum Hence it is key to see if the firms especially was assigned a size proportional to the number of targeted by EMAF II would have done better or EMAF II recipients in the corresponding stratum. worse without the support (additionality Finally, included firms were sampled randomly concern). within each stratum. This ensured that the Results: In contrast with the raw comparison distribution of treatment and control groups (without PSM), PSM gives higher and statistically across strata was identical in size, sector and significant differences for growth in export 3 - Ideally, these groups should have been “identical” prior to EMAF II. A way to ensure that such ideal treatment and control groups were 4 - In the statistical analysis of observational data, propensity available would have been a randomized acceptance of applicants to score matching (PSM) attempts to provide unbiased estimation EMAF II. This best-case strategy is ruled out since acceptance was of treatment-effects. PSM employs a predicted probability of not random: the most promising applicants were more likely to be group membership based on observed predictors. accepted. April 2011 · Number · 2 volume and growth in number of destinations exporter ID, transaction value, country of (light shades mean no significant differences in destination, and product code, were obtained Chart 2). This indicates that EMAF II made a real from Tunisian Customs for 2000-2008 for 3000 difference among firms with high propensity firms (including 400 EMAF II firms). The dataset score (the targeted firms), suggesting EMAF II’s was completed with additional data from the successful targeting of firms that really needed National Statistic Office and the Industrial support. Promotion Agencies for important observables. This sample represents roughly 55 % of export of Chart 2. EMAF II key outcomes goods (excluding oil) in Tunisia. The limitation here is that firms in services (30 % of EMAF II firms) are not taken into account as they are not reported in custom’s data; this is a manufacturing sample. This however allows i) for a more robust matching and disaggregation of results by types of treated firms and ii) to test for the duration of the impact of EMAF II. Here we consider that the impact of the MG for a firm receiving support in 2005 should be measurable Source: Authors’ calculation based on survey data. on export transactions in 2005 and 2006. Hence, estimates suggest that participation in Chart 3. PSM. EMAF II is associated with an increased growth in firms’ total exports. The annual export growth is 38.9 % higher for EMAF II assisted firms compared to controls firm with similar propensity scores in 2004-2008. Similarly, annual growth in destination markets reached is 4 % higher for firms assisted by EMAF. Finally, the impact of EMAF II is positive but not significant on another 0 .2 .4 Propensity Score .6 .8 output that captures the extensive margins in Untreated Treated exports: the number of products. Source: Authors’ calculation based on custom’s data. The same dataset can also assess the employment Results: The probability of getting EMAF II impact of EMAF II. Although not a stated support based on location, date of creation, sector, objective of this MG, the employment impact is employment, exporter status, exports total value, important given Tunisia’s high unemployment number countries and number of products in 2004 and recent events. Employment in EMAF II firms is recomputed. Propensity distribution (Chart 3) is grew annually by 5.5 % in 2004-2008 and by 4.6 % different because of the large sample of control for firms in the control group. The PSM approach firm (the probability among the entire sample to yields similar results though the gap is larger: be treated is smaller). Here, due to a much larger annual growth of 10.2 % for EMAF II firms sample, all treated firm have the same p-score. against 5.1 % for the control group. This suggests a positive MG impact on employment, though it Using PSM, the results are positive - EMAF II must be noted that the small sample size implies support has increased the difference in growth results are sensitive to the specification used and rate in the 2 years of treatment for export not always statistically significant. outcomes, i.e. volume, number of destinations and number of products. Increase in growth is IE Based on Customs Data: Given the relatively slightly higher than with the survey, but the limited size of the surveyed sample (428 firms) impact is for two years instead of four and the and thus to get a better grasp of the effect of impact on extensive margins is higher5. EMAF II, another ex-post impact evaluation exercise based on a larger sample was also 5 - The exclusion of services could explain those differences, conducted. Transaction-level export data with since dropping these firms in the survey showed lower impact April 2011 · Number · 3 Chart 4. EMAF II key outcomes with PMS. destinations and is likely to have had a positive impact on employment. On practical implications, first, incorporating an ex ante evaluation strategy is needed at preparation. It can have large returns for policy learning as data quality is likely to improve by access to a baseline survey and estimation biases reduced. Second, it confirm the relevance of an Source: Authors’ calculation based on custom’s data. EMAF and underlines the need for improving the design, most notably through a better tailoring of Based on the above, EMAF II has been successful the scheme which could be open only to new on key targets but data show the impact varies exporters and firms seeking market diversification widely among types of firms. New exporters (i.e. to reinforce additionality) and ii) further benefited the most (30 % of beneficiaries); one develop training activities linked to the support might say that EMAF II was key in helping offered (to increase the duration of the impact). generate a new class of exporters. Firms which expanded their markets (50 %) also had positive References results but those who expanded their product line Biggs, T. (1999). A Microeconometric Evaluation of the Mauritius (20 %) did not do better than control firms. Technology Diffusion Scheme (TDS), RPED Paper 108, World Bank, Washington DC. EMAF II supported firms have good export growth in 2005 and 2006 but afterwards outcomes Bruhn, M. (2008). License to sell: The effect of business registration do not differ much from random non-EMAF II reform on entrepreneurial activity in Mexico, Policy Research Working Paper 4538, World Bank, Washington DC. firms. Perhaps the duration of support by EMAF II was not sufficient to allow managers to be on Giné, X. and Yang, D. (2009). Insurance, credit, and technology their own on export markets, and limits to adoption: Field experimental evidence from Malawi, Journal of production expansion may have been reached Development Economics, Volume 89, May 2009, Pages 1-11. after a first large increase. Phillips, A. (2001). Implementing the Market Approach to Enterprise Support: An Evaluation of Ten Matching Grant Schemes, Chart 5. EMAF II outcomes over time. Policy Research Working Paper 2589, The World Bank, Washington DC. McKenzie D. (2009). Impact Assessments in Finance and Private Sector Development: What Have We Learned and What Should We Learn? Policy Research Working Paper 4944, The World Bank, Washington DC. Mills, G. (2006). Matching Grant Schemes: What they are, why they exist and how they work, ITC Position paper. Tang, H. (2009). Evaluating SME Support Programs in Chile Using Panel Firm Data, Policy Research Working Paper 5082, Source: Authors’ calculation based on custom’s data. The World Bank, Washington DC. Conclusion: Although the short duration of the Contact MNA K&L: impact and limited additionality for a specific Emmanuel Mbi, Director, Strategy and Operations. MNA Region, The World Bank class of manufacturing firms joins earlier criticisms of MGs, overall results from the PSM Regional Quick Notes Team: based on different data sources suggest that Omer Karasapan, , Roby Fields, and Hafed Al-Ghwell EMAF II was successful. It had a statistically Tel #: (202) 473 8177 significant, positive impact on firm performance The MNA Quick Notes are intended to summarize lessons along targeted dimensions of total exports, learned from MNA and other Bank Knowledge and Learning number of export products and export activities. The Notes do not necessarily reflect the views of the World Bank, its board or its member countries. on export volume but a higher impact on number of destinations or products. April 2011 · Number · 4