53902 Number 4 FAST TRACK BRIEF November 2008 About IEG-IFC IFC's Independent Evaluation Biennial Report on Operations Evaluation in IFC 2008 Group (IEG-IFC) independently evaluates IFC's investment and advisory services operations and reports its findings to IFC's Enhancing Monitoring and Evaluation for Better Results T management and Board of he Biennial Report on Operations Evaluation (BROE) in IFC assesses the adequacy, Independent Evaluation Group Directors. IEG-IFC is a resource for helping staff understand what coverage, and quality of monitoring and evaluation (M&E) in IFC and makes IFC has learned and how IFC can recommendations for improvements going forward. It covers IFC management's do better business in the future. M&E of its performance and also IEG's independent evaluation activities. About Fast Track Briefs The report's main findings are: Fast Track Briefs help inform the World Bank Group (WBG) In the last three years, IFC has made advances in monitoring and evaluating its managers and staff about performance: (i) new M&E tools have been introduced to cover all active investments new evaluation findings and and advisory services operations; and (ii) the use of M&E results in informing IFC recommendations. The views strategy and in linking incentives to project results has grown. expressed here are those of IEG and should not be attributed IFC's Development Outcome Tracking (DOTS) and Expanded Project Supervision to the WBG or its affiliated Report (XPSR) systems complement each other: DOTS is a monitoring system and organizations. Management's XPSR is an evaluation system. As such, DOTS has more breadth while the XPSR system Response to IEG is included in has more depth of coverage. the published IEG report. The findings here do not support any Some important gaps remain in IFC's M&E activities, including: (i) shortcomings in the general inferences beyond the quality of advisory services self-evaluations and IFC's development reach indicators; (ii) scope of the evaluation, including lack of systematic monitoring of IFC's additionality; and (iii) underdeveloped systems International Finance Corporation any inferences about the WBG's past, current or prospective overall and methods for assessing IFC's effectiveness beyond the project level, particularly at the performance. country or sector level. In light of IFC's growing and changing business, IEG has: (i) increased the scope of its Online access evaluation activities; (ii) enhanced the transparency of IFC's performance through the http://www.ifc.org/IEG disclosure of all IEG corporate, thematic, and sector evaluations; and (iii) made progress in assessing IFC's additionality. IEG needs to work in closing some gaps going forward by: (i) better disseminating evaluation findings among staff in the field; (ii) lowering the transaction costs of joint evaluations; and (iii) completing its own results framework and related targets and benchmarks over the coming years. The report recommends that IFC management and IEG develop more robust and consistent metrics for monitoring and assessing IFC's additionality and development impact at the sector and country levels, as IFC transitions to a more programmatic approach in its interventions. The Biennial Report on Operations Evaluation in IFC 2008 was discussed at a Committee on Development Effectiveness Subcommittee meeting on October 1, 2008 BIEnnIAl REpORT On OpERATIOnS EvAluATIOn In IFC 2008 The M&E Landscape of IFC's Results already shows a relatively high degree of consistency--about 80 percent of the time--with the ratings of the well-established The demand for M&E of IFC's results has grown since March XPSR system. 2006 when IEG last reviewed M&E in IFC. IFC's activities have continued to expand--in volume, geographic coverage IFC has piloted a self-evaluation system for advisory services ,and product range. This has made IFC a more prominent player operations--the Project Completion Report system (PCR)--and within the World Bank Group (WBG) and more generally in supplemented this with external evaluations of particular advisory the development community. Greater visibility invites greater services projects and programs. It has begun to carry out country- scrutiny and a greater interest in assessment of IFC's development level assessments of performance through participation in WBG performance. As commercial capital flows to private businesses Country Assistance Strategy Completion Reports (CASCRs). in developing countries have expanded (particularly in middle- IFC has also used aggregated DOTS data to track and report income countries such as China, Brazil and South Africa), interest development results and reach by region, sector, and for IFC as in understanding IFC's additionality--or unique value added-- a whole. has grown. The demand for M&E of IFC's results should also be As M&E activities have grown, so has their use within IFC. M&E seen in the context of the call for greater accountability of public and independent evaluations of IFC's development performance institutions both national and international and for how taxpayers' are now used to inform IFC's strategy development, as well money is spent. as progress with corporate initiatives such as decentralization. M&E in IFC has two parts: M&E, carried out by IFC Of particular significance are the mechanisms that have been management, and independent evaluation, conducted by IEG. introduced to link incentives (through performance awards) at Each part of the evaluation system has responded to the changing different levels, including at the level of individual staff, to project demand dynamics with new activities. Concerning M&E, IFC results. In doing so, IFC has been at the forefront of performance has introduced new instruments to track the project results of its measurement related to private sector development among investment operations and to self-evaluate its advisory services MDBs. interventions and is implementing a new disclosure policy. In Important gaps remain in IFC's M&E activities. At the project independent evaluation, IEG discloses its corporate, thematic, level, IEG's review indicates a number of areas for improvement. and sector evaluations and has increased the range of its evaluation First, while DOTS data quality (covering investment operations) activities to cover IFC Advisory Services, more IFC investment has improved over the second half of fiscal year 2008, it will need operations, a new framework for assessing additionality, more to be sustained if IFC is to achieve consistent accuracy throughout country evaluations, and joint evaluations with other multilateral the reporting cycle. For advisory services self-evaluations, development banks (MDBs). quality shortcomings have been more substantial. However, the M&E of IFC's Results evaluation system which is relatively new and is based on a new methodology (to a large extent DOTS is based on the existing In the last three years, IFC has made advances in tracking the XPSR framework, whereas, there was no approach for evaluating development performance of its operations. There is commitment advisory services prior to the PCR system) can be expected to at the top to strengthen the measurement and reporting of the take time to get established. Second, IFC has started reporting its development impact and additionality of IFC operations. IFC development reach with reference to benefits that have accrued has rolled out DOTS, a new monitoring system that tracks the from its investments, such as numbers of people employed and development performance of all active IFC investments. The number of patients served. At this stage, reach indicators cover monitoring system complements the existing XPSR system less than 60 percent of IFC's portfolio companies and do not have in providing development results data on a wider number of clear links with IFC's contributions to a project. Third, while IFC active investment operations, in offering a leading indication of has taken steps to better articulate additionality in new operations, development performance with respect to XPSR assessments, and IFC has yet to follow through with systematic monitoring of in tracking results through to project closure. The XPSR system, its additionality--crucial for understanding how IFC can best for its part, provides deeper analysis of project development complement commercial finance and optimize its contributions performance, derives lessons, estimates the contribution of the to project development effectiveness. operation to IFC's profitability, and assesses the quality of IFC's work in structuring, appraising, supervising, and administering More broadly and importantly, at the country, region, and the operation, including the quality of IFC's unique role and sector levels, IFC needs to develop systems for monitoring contribution to the project. DOTS is a work in progress, but it and evaluating its additionality--including its catalytic role in BIEnnIAl REpORT On OpERATIOnS EvAluATIOn In IFC 2008 facilitating private investors in making good investments through IFC shows strong compliance--with over 90 percent adherence financial, operational, and institutional contributions--and to MDB good practice standards. Most areas of current non- development impact--including effects on incomes, poverty, compliance relate to experimental standards--those that are not and the environmental and social dimensions--beyond simply essential in ensuring comparability among the MDBs or for aggregating some of the benefits from individual interventions having a satisfactory evaluation system. and reporting the volume of new investments. IFC employs As with M&E, independent evaluation needs to be made more corporate and departmental (by region and sector) scorecards effective going forward: (i) Since 2005, IFC staff have become of performance. However, they tend to emphasize new business increasingly decentralized and reaching out to them with lessons activity in certain areas as rough estimates of expected development from evaluation has become more challenging. IEG has not been results and additionality. The CASCRs, conducted jointly with the able to provide staff with full access to its lessons database during World Bank, are one instrument for assessing IFC's performance this period, but is now finalizing a Web-based lessons database at the country level. However, IFC's involvement in CASCRs has and is intensifying efforts to disseminate evaluation findings been largely superficial to date. To some extent, the situation with among staff in the field; (ii) IEG has explored the connection assessing IFC's development impacts and additionality beyond between risk, development performance, and IFC work quality, the project level reflects the nature of the institution, with process but IEG and IFC need to make further advances in this area; and accountability focused around individual investments and (iii) IEG has taken the opportunity to extend its participation in advisory services operations. IFC is in the process of transitioning joint evaluations, particularly with IEG-World Bank, but faces a towards a more programmatic approach to interventions, and will number of challenges in maximizing its value added, for instance, need to establish systems and methods for assessing its effectiveness in developing a consistent methodology for assessing IFC's country beyond the project level, particularly at the country and sector level and sector-level contributions and in minimizing the transaction of operations. costs involved in joint evaluations; (iv) IEG is yet to extend its Independent Evaluation of IFC's Results reach to offer an in-depth coverage of topics that are emerging as areas of strategic importance and interest to IFC and its Board of A core function of independent evaluation is to provide checks and Directors, such as process and impact evaluation; (v) gaps remain balances on the M&E systems by objectively validating the results in IEG's methodology for evaluating IFC's development impact they produce. Self-evaluation of IFC investment operations shows and additionality in advisory services operations, and at the broad consistency with IEG's assessments over the last three years, country and sector levels; and (vi) IEG will also need to develop with a annual average rating discrepancies of between two to five its own results framework and related targets and benchmarks over percentage points. These gaps have been stable in the similar range the coming years. throughout the history of the system and there are no distinct patterns across departments, regions or sectors. Through the pilot Following the introduction of its new Disclosure Policy, IEG phases of IFC's M&E program for advisory services, IEG has began disclosing its macro studies. IEG's evaluation outputs are reviewed self-assessment of IFC's performance and has highlighted now increasingly available to external audiences. This development initial data quality concerns that IFC management is now seeking helps inform external stakeholders about IFC's performance, and to address. Going forward, IEG will provide validation through stimulates a debate about private sector development in general. the introduction of evaluation notes for advisory services. Judged The initial reaction to IEG's macro studies has been largely positive, against international good practice standards, which cover both but IEG has yet to develop hard measures to assess the uptake of self and independent evaluation, the project evaluation system in and satisfaction with its disclosed products. BIEnnIAl REpORT On OpERATIOnS EvAluATIOn In IFC 2008 COnCluSIOnS And RECOmmEndATIOnS Recommendations for IFC Management IFC should consider developing more robust and consistent metrics for monitoring and assessing its additionality and development impact at the sector and country level as IFC transitions to a more programmatic approach to interventions. To start with, IFC may wish to focus on sectors and countries in which it has taken a programmatic approach, and where there has been a critical mass of IFC activities. IFC should aim to expand the coverage of its portfolio in reporting on results. It should improve the portfolio coverage of reach indicator and report the development results of its entire mature portfolio. IFC should strengthen managerial oversight of M&E for data quality and ratings for both investments and advisory services with proper record-keeping functions. For advisory services in particular, in order to improve quality, IFC may need to introduce some minimal acceptance standards for PCRs. Recommendations for IEG IEG should further develop its system for evaluating IFC's additionality and development impact at the sector and country levels. IEG needs to complete its own results framework and develop a scorecard with related targets and benchmarks. IEG should continue its efforts to harmonize evaluation methodologies in the context of the Evaluation Cooperation Group's Working Group on Private Sector Evaluation to enable comparability among MDBs and for enhanced benefits of joint evaluations, including with IEG-World Bank. Resources Task Manager: Director-General, Director, IEG-IFC: Head of Knowledge, IEG-IFC Help Desk: Hiroyuki Hatashima Evaluation: Marvin Taylor-Dormond Dissemination, and (202) 458-2299 Daniel J. Crabtree Vinod Thomas Quality, IEG-IFC: AskIEG@ifc.org Sid Edelmann