53903 Number 5 FAST TRACK BRIEF Independent Evaluation of IFC's May 2009 About IEG-IFC IFC's Independent Evaluation Group (IEG-IFC) independently Development Results 2009 evaluates IFC's investment and advisory services and reports its Knowledge for Private Sector Development findings to IFC's management Independent Evaluation Group T and Board of Directors. IEG- he Independent Evaluation of IFC's Development Results 2009 assesses the 1 IFC is a resource for helping staff development outcomes and additionality (unique role and contribution) of IFC understand what IFC has learned interventions. It analyzes factors driving results, and reviews performance patterns and how IFC can do better on a thematic topic. This year's thematic is IFC's Advisory Services (AS), knowledge services business in the future. that IFC provides to either private companies or governments in support of private sector About Fast Track Briefs development. The report's main findings are: Fast Track Briefs help inform the World Bank Group (WBG) Investment Services (IS) results improved overall. Seventy-two percent of IS operations managers and staff about maturing between 2006-08 achieved high development outcome ratings, compared to 63 new evaluation findings and percent between 2005-07 (Fig. 1). Performance was especially strong in Europe and Central recommendations. The views Asia (ECA) and Latin America and the Caribbean (LAC). But there was no improvement, expressed here are those of IEG from a low base of around 50 percent, in East Asia and Pacific (EAP), Middle East and North and should not be attributed Africa (MENA), and Sub-Saharan Africa (SSA). Most operations were implemented prior to to the WBG or its affiliated the crisis and thus results do not yet reflect the deterioration in global economic conditions. organizations. The findings here do not support any general The rapid growth of AS has changed the nature of IFC. The 1,262 AS staff make up nearly half of all inferences beyond the scope of IFC's operational staff, and AS teams now dominate IFC's presence in the field. Quality trade-offs the evaluation, including any are possible, given the high reliance on relatively new staff, outsourcing work through some 1,300 International Finance Corporation inferences about the WBG's past, current or prospective overall short­term consultants, and considerable organizational change. An overall strategy has been lacking. performance. In terms of results, 70 percent of reviewed AS operations achieved high development ratings. Online access Performance has been strongest in Southern Europe and Central Asia (SECA) and weakest in LAC. http://www.ifc.org/IEG As shown in Figure 2, results were significantly better for infrastructure (INF), business enabling environment (BEE), and corporate advice (CA) operations and weaker in the case of environmental and social sustainability (ESS). Key drivers of AS performance have been: client commitment; programmatic approaches; IFC's additionality including through some, though not all, AS-IS links; and local presence and ownership and monitoring and evaluation (M&E) quality. Charging for AS has been associated with better performance. The Independent Evaluation of IFC's Development Results 2009 was discussed at a Committee on Development Effectiveness meeting on March 11, 2009. IndEpEndEnT EvAluATIOn OF IFC'S dEvElOpmEnT RESulTS 2009 To enhance development impact, the report recommends that IFC: i) Effectively manage the tension between protecting the portfolio and responding to opportunities during crisis; ii) Set out an overall strategy for IFC advisory services, addressing the need for a clear vision and business framework and more closely linked with IFC's global corporate strategy; iii) Pursue more programmatic AS interventions; iv) Improve execution of the AS pricing policy; and v) Strengthen AS performance measurement and internal knowledge management. Figure 1: Development outcomes of IFC investment operations improved in Figure 2: In Advisory Services, environment and social sustainability project 2 the last three years ratings have lagged those of other business lines 100% 100% Percentage of projects rated high 90% 90% High Development Effectiveness Rating 77% 80% 80% 74% 71% 70% 70% 65% 60% 58% 60% 50% 50% 40% 40% 30% 20% 30% 10% 20% 0% 10% 1996-98 1997-99 1998-2000 1999-2001 2000-02 2001-03 2002-04 2003-05 2004-06 2005-07 2006-08 0% Development Outcome (3 year moving average) Investment Outcome (3 year moving average) INF (n=30) BEE (n=98) CA (n=66) A2F (n=55) ESS (n=36) Business Line Note: Three year moving averages; `High' = met or exceeded benchmarks Note: Excludes cases where it was too early to rate, or where data was insufficient Background In times like these, IFC's dual role as a financier and as a provider of knowledge (together with the World Bank) assumes particular Over the last decade, many developing countries have experienced importance. Concerning the first role, IFC's founding articles state strong economic growth, typically accompanied by falling levels that it should invest in viable private sector projects in developing of poverty. The private sector has been a key contributor to countries for which `sufficient private capital is not available on this growth, through new capital investment, but also through 3 reasonable terms'. In such crisis times, the onus is on IFC to ramp innovation and entrepreneurship, helping create jobs and opening up its financing efforts. But IFC's role as a knowledge provider up new markets. Developing countries with the highest levels of (together with the World Bank) is also important, particularly as private investment and that have made the widest strides in bridging policymakers and administrators focus on business regulations, knowledge and technology gaps with the developed world--from good governance, and the environmental and social sustainability India to the Baltic states--have generally grown the quickest. of growth. The role entails providing advice that helps shape the conditions for sustainable private sector development --through, The current global financial crisis places many of these hard won for example, promoting more effective regulation--and to enhance gains under severe threat. The crisis began in the developed world, the capacity, skills and behavior of actors involved with private but has since spread to the developing world, and has particularly sector enterprise on the ground (including effective management affected countries with economies more connected to global markets. of the social and environmental effects of private activities). Import demand from developed countries is falling, and companies in developing countries, both large and small (particularly small), This year's Independent Evaluation of IFC's Development Results have also found that funds for new investment have dried up, or have (IEDR) looks at each of these roles in turn: IFC's effectiveness in become much more expensive and more difficult to obtain. Private financing development, through its growing portfolio of investment capital flows to developing countries in 2009 are expected to be, at operations, with an emphasis on IFC's experience during previous best, around half their level in 2007 (of $1 trillion). Past crises suggest crises and helping clients mitigate investment risks (Part I); and, for that it may take some years for these flows to return to their pre­crisis the first time and thus the main theme of the report, IFC's experience levels. More generally, the crisis has led policy makers and analysts organizing and delivering its Advisory Services (AS) interventions-- to reevaluate the role of markets and the private sector, particularly knowledge services that IFC provides to either private companies or where the value of effective regulation, prudential oversight and governments in support of private sector development (Part II). In fiduciary management was wrongly de­emphasized or ignored. IndEpEndEnT EvAluATIOn OF IFC'S dEvElOpmEnT RESulTS 2009 terms of results, the report focuses on IFC investment operations regions, business conditions were most supportive and IFC work that reached early operating maturity between 2006 and 2008 and quality strongest. South Asia (SA) exhibited improving performance, IFC AS projects with Project Completion Reports during the same with higher IFC work quality than in the past. period. The review of AS development effectiveness comes with Performance lagged considerably in EAP, and in the mainly low certain caveats, given the M&E system was only introduced in income MENA, and SSA--with barely a half of the projects in 2006, and considering the often intangible nature of knowledge these regions meeting or exceeding specified benchmarks and transmission. Nonetheless, the report for the first time provides a standards. External conditions were partly responsible--projects in combined account of both arms of IFC's business (i.e. investments SSA and MENA generally featured high levels of country, sponsor and AS), including situations where these instruments have been and product competitiveness risks--but the quality of IFC's work offered to the same client. The report also complements a recent IEG and contribution to the project tended to have a larger impact. This evaluation of the effectiveness of World Bank Economic and Sector 4 was especially the case in EAP, where nearly 40 percent of projects Work and Technical Assistance, which was completed in 2008. exhibited low quality IFC additionality. There is evidence of better Financing Development screening and appraisal work in MENA and improved supervision quality in SSA. IFC's portfolio of investment operations (loans, equity, and other financial products) continued to grow in the last year. The Among IFC's strategic sectors, project performance showed continued cumulative volume of active investment activities increased by improvement in health and education, was better in agribusiness, around a quarter, from $32.7 billion in fiscal year (FY) 2007 to $40 and remained strong in infrastructure and financial markets. At billion in FY 2008. The number of investments rose by a lesser order the same time, performance lagged in non­telecommunications 6 (8 percent), reflecting a general preference for larger investment information technology (software and internet). In other sectors, oil, operations (increasingly involving corporate finance rather than gas, mining and chemicals projects achieved relatively poor ratings. project finance), and a more wholesale approach to reaching Risk exposure was clearly a factor in weak non­telecommunications SMEs, i.e. through financial intermediaries and larger companies. information technology projects, most of which were small operations involving inexperienced sponsors and unclear product A growing portfolio provides opportunities to extend IFC's competitiveness. However, work quality in this sector was also well development reach. IEG's evaluations of investment operations 5 below par: high in just 40 percent of cases. Improved IFC work that reached early operating maturity between 2006­08, quality was in evidence in the health sector, where IFC showed that show that IFC's project development results (along with IFC's it had learned lessons from past experience, but the portfolio has not financial returns) improved overall. More specifically, 72 percent achieved much diversity. Oil, gas, mining and chemicals projects did of evaluated projects (85 percent by volume) achieved outcomes not meet benchmarks for a number of reasons: sponsor technical that, on balance, met or exceeded project financial, economic, weaknesses; higher than expected asset acquisition cost, and one environmental and social benchmarks and standards, and made case of unsatisfactory environmental compliance. Environmental positive contributions to private sector development beyond and social effects ratings were stable for real sector projects, but the project. This compares with 63 percent of projects (75 remained weak in financial intermediary (FI) operations, reflecting percent by volume) achieving high outcomes in 2005­07. On a the need to strengthen client capacity and securing commitment, cumulative basis, since independent evaluation started in 1996 as well as addressing shortfalls in IFC supervision and additionality. up to and including 2008, 62 percent of projects (70 percent by volume) have achieved high development outcome ratings. The development results reported above do not yet reflect the sharp deterioration in global economic conditions, which has Stronger overall results in the outer years reflected several factors: just now begun to affect investment returns in most developing i) the exit of a particularly weak performing cohort of projects that countries. The development results reported here largely reflect matured in 2005 (51 percent of projects maturing in 2005 realized project experience during 2003­08, a period of unprecedented high development outcomes, compared to 75 percent maturing growth in emerging markets. Most evaluated projects had been in 2008); ii) more favorable economic conditions in much of the substantially implemented, and some had been closed, by late developing world (until late 2008, by which time most evaluated 2008 when the crisis started to affect the developing world. projects had been substantially implemented); iii) improving IFC project appraisal and structuring quality; iv) the conscious move The development results of maturing operations are, however, by IFC toward larger projects, which have been more likely to expected to decline in the coming years. Past evaluation shows achieve high ratings than smaller projects, due in part to greater that projects approved in the years prior to the crisis (and being internal scrutiny; and v) especially strong performance in ECA and implemented during the downturn) are most at risk from a LAC, where the majority of mature operations are located. In these development perspective. Approximately 40 percent of IFC's IndEpEndEnT EvAluATIOn OF IFC'S dEvElOpmEnT RESulTS 2009 portfolio (62 percent by volume) falls into this category, thus IFC Important strategic questions follow. These include whether, in is exposed to considerable downside development risk. At the same grafting such a substantial knowledge business onto a financing time, IFC has strengthened considerably its internal risk management institution, IFC has the appropriate balance of effort between AS processes and its capacity to bear and manage financial risks appears to and investment services (IS) to ensure greatest development impact. have improved significantly in recent years. Importantly, evaluation Quality trade­offs are also possible, given substantial organizational suggests that investments approved in the wake of the crisis, i.e. at the change, a high reliance on relatively new staff (60 percent have been bottom of the business cycle, will tend to have better development with IFC less than three years), and outsourcing work through some results. Thus, there are also upside opportunities to be grasped. 1,300 short­term consultants each year. There is also increased possibility of conflict of interest or market distortion--where AS is The experience of past crises underlines two key responses by offered together with financing, or is provided at less than market value. IFC: first, careful portfolio risk management, particularly for projects in early implementation; and second, IFC additionality. IFC deploys its AS in the pursuit of general objectives that are common The latter is particularly important in two respects: i) acting as an with those for IFC investments. These objectives include focusing on honest broker in restructurings; and, ii) in pursuing a well timed frontier markets including International Development Association and targeted approach to new operations particularly through the (IDA) countries and frontier regions of non­IDA countries, as well signaling effect IFC interventions can provide to other investors. as small and medium-sized enterprises (SMEs) and agribusiness, the strategic sectors of finance, infrastructure, health and education, Knowledge for Development and support for environmental and social sustainability (including IFC AS have been growing rapidly, with an active portfolio climate change). The allocation of AS resources has been largely approaching $1 billion and employing 1,262 staff, a sevenfold aligned with these priorities. That is, IFC AS have generally targeted increase in the last seven years (figure 3). The nature and face of high need destinations such as IDA countries and Africa in particular. IFC has changed significantly: AS staff now make up the majority Relevance, however, does not guarantee impact. Fifty­two percent of 7 of IFC's presence on the ground in developing countries. The rapid IFC's AS projects, where ratings could be assigned, were rated high growth of AS has happened in a largely unchecked manner. This is on achieved development impact. Projects rated substantially higher well illustrated in the emergence of more than 50 AS products, 18 on other dimensions of performance such as strategic relevance, regional facilities covering seven regions, 13 global business units, output, and outcome achievement with an overall development and around a half of AS work being contracted out to consultants. effectiveness success rate of 70 percent. Ratings did not change Figure 3: Advisory Services and AS staff have grown rapidly Figure 4: Stronger AS performance when clients make cash contributions 1262 250 1300 90% 1126 225 1100 Share of projects with high development 80% 200 863 175 900 70% Expenditure ($ millions) 686 245 effectiveness rating Number of staff 150 60% 576 700 125 497 191 500 50% 100 342 149 75 119 40% 300 168 98 50 30% 68 100 25 45 24 20% 0 -100 2001 2002 2003 2004 2005 2006 2007 2008 10% Fiscal year 0% EXPENDITURE ($m) AS STAFF No Client Contribution In Kind/Parallel Client Contribution ($) Contribution Sources: IFC Human Resources; donor-funded operations quarterly AS Source: Independent Evaluation Group financial reports. Note: Includes activities of the foreign Investment Advisory Services, which are partly funded by the World Bank and Multilateral Investment Guarantee Agency (MIGA) (IFC is the main funds provider and manager). In FY08, IFC provided $11.8 million, MIGA provided $4 million (for investment policy and promotion activities), and the World Bank provided $2 million. IndEpEndEnT EvAluATIOn OF IFC'S dEvElOpmEnT RESulTS 2009 significantly for projects that began before, as opposed to projects The professionalization of AS, however, remains a work in progress initiated after, the major organizational changes in 2005-06. By and significant organizational issues still persist: overlapping region, ratings have been substantially better in SECA, and weaker and parallel implementation structures in several regions (SSA, in LAC. Evaluated global projects also did not perform well. By EAP and SA); few well established products outside of finance business line, infrastructure (INF), business enabling environment and infrastructure; lack of clarity about how AS and IS are best (BEE), corporate advice (CA) and access to finance (A2F) tend to integrated in different contexts; limited consideration of IFC's perform better than environmental and social sustainability (ESS). comparative advantages relative to other knowledge service providers at the strategic and project levels; and no umbrella Looking at the main drivers of results, the review found a strong AS strategic framework to weave different strands together. connection between client commitment (evidenced by contribution to project costs and especially so for ESS projects), (figure 4) strong There are also substantial gaps in delivery that need to be addressed project design and implementation, IFC's proximity to the client --particularly in matching corporate intent with consistent as defined by IFC's local presence and involvement, programmatic implementation on the ground. This applies with respect to the (rather than one off) interventions, and effective monitoring and execution of the pricing policy, as well as ensuring good quality project evaluation (M&E)--and the achievement of development objectives. design and implementation, and effective collaboration with other Strong additionality is fundamental for better performance. High actors, including the World Bank. Getting the right staffing mix has additionality has been particularly noticeable among BEE operations been a particular challenge, with a heavy reliance on external, short­ in IDA countries with high business climate risk, and has been term consultants, and relatively new staff (compared to those involved evident through some packages of services, such as SME linkage with investment operations). The chosen mix has major implications projects in agribusiness, manufacturing and extractive sectors. for the quality and continuity of IFC's AS, and preservation of global Such packaging raises potential conflicts of interest, which need knowledge leadership. At all stages of delivery, M&E data provided to be tackled effectively, and needs appropriate pricing. Intrinsic by staff and consultants (in particular) has remained unreliable. constraints in capturing the impact of AS are compounded by the Relatedly, IFC­commissioned reviews of AS facilities, products and relatively weak application of M&E guidelines to date by IFC staff. projects, while offering insights on the organization and delivery of AS, have exhibited shortcomings in independence and design. Over the last five years, IFC's management has taken actions to enhance its AS effectiveness, through efforts aimed at strengthening Charging effectively for IFC's advisory services is perhaps the their organizational alignment and delivery processes. Efforts to bring most important step going forward. Effectively charging clients greater structure and clarity to IFC's AS in the last few years include: for services will introduce a market test for AS and is likely to categorizing AS activities into five business lines; consolidating have a positive impact on all aspects of the business: in creating some global and regional facilities; categorizing products by level incentives for greater client buy in, stronger project design and of maturity; developing AS staff competencies; AS training; and implementation, stronger M&E, the development of products establishing an AS Vice Presidency. IFC's attention to the delivery that best meet demand, and ensuring IFC additionality. In the of advisory services has focused on establishing mechanisms and immediate term, IFC would need to strictly implement the current systems to ensure: adequate, sustainable funding; client commitment; pricing policy, which is largely cost­based, i.e. the price the client is sound project design and implementation, and robust M&E of expected to pay is a proportion of the cost of the project. Over time, performance. IFC's efforts in these areas appear to compare favorably efforts should be made to move to a market value­based approach with measures taken by other multilateral development banks, for to pricing, such that IFC does not run the risk of crowding out example in the introduction of a pricing policy (that broadly seeks other knowledge providers. IFC investments are priced according to build client commitment and reduce possible market distortion to this principle for the same reason. The current economic crisis, by limiting any subsidies to public goods), and an M&E system and its likely effects on donor and IFC funding, is an opportunity that seeks to capture outcomes and impacts, as opposed to just for IFC to push harder in the direction of value­based pricing, and outputs. The momentum of transformation continues with the to encourage other development institutions to do likewise. recent introduction of new policies, procedures and guidelines related to pricing, conflict of interest, funding and governance. IndEpEndEnT EvAluATIOn OF IFC'S dEvElOpmEnT RESulTS 2009 EndnOTES 1 The full-text IEDR, which includes IFC's management response, Chairperson's Summary: Committee on Development Effectiveness, and Advisory Panel statement can be accessed at http://ifc.org/ieg/reports. 2 Business Lines: INF=Infrastructure; BEE=Business Enabling Environment; CA= Corporate Advice; A2F=Access to Finance; ESS=Environmental and Social Sustainability. 3 IFC Articles of Agreement: Article 1 - Purpose. 4 Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000-06, Independent Evaluation Group, 2008. 5 At early operating maturity, operations have generally recorded at least eighteen months of operating revenue, which is typically five years after approval; 6 These results patterns across regions and sectors are broadly consistent with IFC's own self-assessments, although with some optimism bias in self-ratings, which were on average 5 percent higher than those assigned by IEG. 7 In many country offices, outside regional hubs, IFC Advisory Services staff significantly outnumber investment officers and are the face of IFC in the country. IndEpEndEnT EvAluATIOn OF IFC'S dEvElOpmEnT RESulTS 2009 RECOmmEndATIOnS FOR IFC The review comes at a time of deep distress in financial markets and a severe downsizing in private economic activities. It reminds us of the critical importance of sustainable development in the private sector, for which regulatory frameworks are important and excessive deregulation costly. In these circumstances, the review provides further findings on what IFC might do to enhance development effectiveness and additionality: IFC's operations during the crisis: Effectively manage the tension between protecting the portfolio and responding to opportunities during crisis. In the past, this tension has not always been managed adequately and IFC has missed opportunities to have a deeper impact. Experience suggests the importance of arrangements to isolate portfolio problems from new business development, to mitigate conflicts of interest that may impede effective collaboration with the World Bank and the International Monetary Fund (IMF), and to establish clear rules of engagement in crisis response, particularly for staff in the field. Experience also indicates the important role IFC and the World Bank Group (WBG) must play in promoting sound frameworks for prudent financial risk management and safeguards to ensure sustainable private sector development. This is especially relevant today, as the world reexamines the roles of governments and markets in the wake of the financial crisis. IFC's role in Advisory Services: Set out an overall strategy for IFC advisory services, that addresses the need for a clear vision and business framework, and is closely linked with IFC's global corporate strategy. Following years of unchecked growth and recent organizational changes, the role of AS in IFC's business model needs to be addressed. The strategy would need to better articulate IFC's comparative advantages in AS, objectives and goals for AS in different contexts (a source of confusion among staff), and to consider the best staffing combinations (with respect to internal or external as well as global or local staff), delivery unit organization, incentives and performance measures to help realize these objectives and goals. Pursue more programmatic AS interventions. Evaluation shows that IFC has achieved better results in AS projects that have been carried out in conjunction with other AS interventions. One­off activities have been less effective. However, programmatic efforts of this kind have been in the minority (about a fifth of all AS projects), and IFC should accordingly seek to expand this type of intervention. Improve execution of the AS pricing policy, through greater client contributions. Over the longer term, it would be important to seek client contributions that reflect value and impact, i.e. not just cost, to create a true test of client demand, incentives for better AS delivery, and ensure IFC is being additional. Strengthen AS performance measurement and internal knowledge management. In the short term, it would be important to have more hands­on M&E support in the field, post­project completion follow up, lessons-capture (including from dropped or terminated projects), and more arms­length facility, product and project reviews. In the medium term, it would pay to introduce an expanded project completion report (XPCR) system (akin to the expanded project supervision report (XPSR) system for investment operations, and carried out later than the PCR to better capture impacts), more programmatic impact evaluation and impact research, setting results­based targets for AS in its corporate scorecard, and regular benchmarking of IFC AS activities and systems with other providers of knowledge services, including other multilateral development banks and commercial providers. In the longer term, the aim could be to establish a specialized research unit focusing on generating and bringing together private sector development knowledge work. The IEDR was reviewed by a Panel of international experts in the area of knowledge and development. Panel members were: Professor Carl Dahlman, Luce Associate Professor of International Relations and Information Technology, Georgetown University School of Foreign Service; Acha Leke, Partner, McKinsey & Company; and Laurence Prusak, founder and former Director, Institute for Knowledge Management. In a joint statement, which was submitted to the Board under separate cover, the panel agreed with the above recommendations, and suggested additional steps IFC may take in the same direction. The Panel's report can found at http://www:ifc.org/ieg/reports. IndEpEndEnT EvAluATIOn OF IFC'S dEvElOpmEnT RESulTS 2009 IEG pROduCTS Studies The World Bank Group Guarantee Instruments 1990 - 2007: An Independent Evaluation Biennial Report on Operations Evaluation in IFC 2008: Enhancing Monitoring and Evaluation for Better Results IFC in Nigeria: An Independent Country Impact Review IFC in Indonesia: An Independent Country Impact Review Independent Evaluation of IFC's Development Results 2008: IFC's Additionality in Supporting Private Sector Development Supporting Environmental Sustainability: An Evaluation of World Bank Group Experience Doing Business, An Independent Evaluation: Taking the Measure of the World Bank-IFC Doing Business Indicators Financing Micro, Small, and Medium Enterprises: An Independent Evaluation of IFC's Experience with Financial Intermediaries in Frontier Countries IFC in Ukraine: 1993-2006 - An Independent Country Impact Review IFC Advisory Services in Eastern Europe and Central Asia: An Independent Evaluation of the Private Enterprise Partnership Program Evaluation Briefs, and Evaluation Notes Lessons from World Bank Group Responses to Past Financial Crises Improving Results in Sub-Saharan Africa IFC's Experience and Additionality in Middle-Income Countries IFC's Experience in the Transport Sector IFC Operations in Romania An Evaluation of IFC's Frontier Country Strategy IFC and the Millennium Development Goals Resources Task Manager Director-General, Director, IEG-IFC: Head of Knowledge, IEG-IFC Help Desk: Daniel J. Crabtree Evaluation: Marvin Taylor-Dormond Dissemination, and (202) 458-2299 Vinod Thomas Quality, IEG-IFC: AskIEG@ifc.org Sid Edelmann