45972 RESULTS-BASED MONITORING AND EVALUATION SYSTEM: A SUCCESSFUL PILOT INITIATIVE OF IFC-SEDF CAPTURES IMPACT MUSTAHID HOSSAIN IFC-SEDF's Monitoring and Evaluation (M&E) team developed a results-based performance measurement system in order to assess the effectiveness of Value Addition to Firms (VA2F) interventions. The system is based on a planning matrix derived from causal chains (Figure 1) to make intended impacts within the framework of IFC development objectives and strategies. We began testing this M&E system on our VA2F interventions in three sectors: Readymade Garments (RMG), Light Engineering (LES), and Agribusiness. We have enriched the system with our own experience of implementation, and now it covers the whole spectrum of IFC-SEDF's business lines: VA2F, Access to Finance, Business Enabling Environment (BEE), and Sustainability. In this paper we briefly share how we developed our M&E system, and the lessons we picked up along the way. Figure 1: M&E Causal Chain Project Plan of Log frame Planning Operation Outcome Impact Matrix Based on Program Logic Model Objectives for IFC-SEDF's M&E System: · To gather and consolidate data, and measure the effectiveness of IFC-SEDF interventions by establishing clear links between past, present, and future interventions and results. · To help IFC-SEDF extract, from past and ongoing activities, relevant information that can subsequently be used as the basis for periodic fine-tuning, reorientation, and planning. · To periodically review if IFC-SEDF work is going in the right direction, whether progress M&E team engaging with the clients can be measured, and how future efforts might be improved. · To help improve IFC-SEDF performance and achieve better results leading to the desired impact. SmartLessons, May 2007 1 How the System Works The application gathers monitoring data on various vital business and development indicators (variables), and gives a snapshot of the overall performance of the VA2F interventions by covering 2005 and 2006 periodic cycles compared to the baseline values in terms of sales, assets, employment, profit, production, gender, bank credit, clients' cost sharing, leverage value, etc., as prescribed by IFC's Results Measurement Unit and other donors. The system captures information on both assisted and non- assisted groups. Non-assisted group members are selected based on the consideration of similar attributes of the assisted SMEs. Lessons from the Data Collection 1. Know the people in industry: It is excruciatingly difficult to get verifiable data. Most clients become circumspective when asked for performance-level data such as sales, assets, loan balance, etc. Therefore, it is critical to select collectable and verifiable data points and/or develop relationships with clients so that they are comfortable sharing their financial and business performance details. Basically, trying to know the people in the industry and maintaining a solid and constant rapport with them should be an M&E priority. Example: Our cold-calling strategy to get data from the RMG clients failed. They simply hung up the phone on us. Many clients did not keep appointments. Since all RMG companies are registered members in a local association, the Bangladesh Garments Manufacturers and Exporters Association (BGMEA), we leveraged our relationship with BGMEA to meet key people at specific RMG companies. We invested significant time and energy to build a rapport. So far we have visited and collected information over a hundred RMG companies, and we try to sustain the business relationship. 2. Go to the field: Sometimes clients provide us with inconsistent data, intentionally or unintentionally. This can skew M&E results drastically. We learned the importance of going to the field regularly to verify data ourselves instead of blindly believing what we received. The SME mindset is such that they feel that revealing their strengths can put them at a disadvantage. Example: After reporting successful growth in the first year, one of our Agribusiness clients reported lower sales figures in the second monitoring year. We got suspicious and went to the field with the owner ­ to speak to his dealers, his retailers, and the farmers that buy his products. We discovered that the products were still selling like hotcakes, and the manufacturer could not produce enough to meet demand. Our Operations Officer in charge of the project received an updated (and true!) sales figure the following week. 3. Agreement on selecting comparison group: The criteria for randomly selecting non-assisted group members are extremely important for making justified comparisons. The team and management should be unanimous in terms of selection of comparison group members. Example: We picked up our comparison group members from the same geographic clusters. If our assisted RMG company was in Dhaka city and produced woven materials, we picked up a non-assisted company that also was from Dhaka and produced woven goods. Plus, we made SmartLessons, May 2007 2 sure they had similar products lines, capacity, assets, and manpower. Such agreement on selecting comparison group members helped us to make justified comparisons. Lessons on Institutional Perception Since M&E plays a support function role, it is often perceived as a lesser priority for the Operations Officers. Plus, the Officers are extremely preoccupied with projects and their M&E conceptualization varies widely. For synergy to occur, the M&E team must be vigilant and take the initiative to make Operations Officers understand M&E objectives and concepts and remain friendly with and supportive of the operational colleagues who get them all the data. The M&E team has to clearly demonstrate that it is in the best interest of the other departments that they facilitate the M&E team to measure and verify their success. Example: Working with the M&E team paid off for task managers at IFC-SEDF to measure progress in terms of total program cost leverage. We used total program cost to total sales difference of monitored assisted companies along with operational overhead expenses and IFC fees in order to calculate total program cost leverage. In this case, the leverage is 1:25. It means that out of every dollar of VA2F support, clients generated 25 dollars of sales revenue. Likewise, our system generates subsector and client-specific leverage, and Operations Officers at IFC- SEDF started to use such values for reporting purposes. Lessons from the Pilot IT System · The team expects M&E to function automatically, flawlessly, like a machine. Therefore, a user- friendly system that delivers the most current intervention status and generates as customized a report as possible based on any given input is a must. The key here is constantly updating the data. · IFC-SEDF interventions consist of several interlinked projects. Cross-cutting components like IT, gender, and poverty work with primary VA2F pillars like Agri, LES, and RMG. Such program activities keep changing, and so must the indicators. This requires constant updating of the computer application system to capture new results. Therefore, indicators need to be flexible in order to be measured over a period of time, and the computer application system needs to be adjusted accordingly. · The system also has to be very robust ­ do a periodic backup of the system, and never shred any collected and filled-out data forms from the field. Going Forward Considering the significance of benchmark values, the M&E team is collecting national and regional benchmark data on IFC-SEDF assisted sectors. This should give IFC-SEDF crucial information on how to develop industrial competitiveness in different sectors of Bangladesh. Going forward, we may suggest greater utilization of benchmarks within our M&E system. Example: The rejection rate per thousand finished items at the final quality control level is 8 percent among the IFC-SEDF assisted RMG companies, whereas in Bangladeshi RMG industry, the rate is 10 percent. Surprisingly, the rejection rate ranges between 0.5 and 3 percent in the regional RMG companies, and globally the rate is only 1 percent. Most of our clients companies SmartLessons, May 2007 3 are not aware of these benchmarks, and are inward looking. With the use of these benchmarks, we will be able to make them realize how far they lag behind, and make a solid case for improving competitiveness. Integrating Technical Assistance and Advisory Services (TAAS) and IFC corporate indicators with the local M&E has become critical for the smooth functioning of Advisory Services. The existing TAAS system is good for initiating projects and capturing their status. However, the M&E aspect of TAAS could be enriched by supplementing them with more appropriate business-specific performance-level indicators. Example: Actual machine capacity is crucial in the RMG sector, reduced mortality rate is a key indicator in the poultry sector, and environmental-related benefits like reduced natural gas consumption is significant in the LES sector. Many outcome and impact level indicators require subjective interpretation and more in-depth study to arrive at meaningful results for the indicators. The SEDF M&E team is working with IFC's Result Measurement Unit in Headquarters to test the robustness of our results using randomization and experimental design in order to derive further significance of the effectiveness and impact generated from the IFC-SEDF interventions. Leverage IFC resources! Conclusion. Clearly, M&E does not just provide additionality to other departments. M&E does much more. Systems like this can help M&E take a leading role in program design to reflect the increasing demand for proof of impact from both IFC and our donors. This Smart Lesson details the challenges faced by this resource-intensive effort, and recognizes the investment that should be made in M&E as an indispensable component in justifying IFC Advisory Services. Special thanks to Amrit Rai, Sunil Pradhan, Monira Firdouse, Joseph Amick, Shazia Ahmed, Sarah Karim, and Wendy Werner. About the Author Mustahid Hossain joined IFC-SEDF in November 2004. His main interests are Experimental Design and Impact Monitoring. Prior to IFC-SEDF, he was with General Electric Power Systems, Dell Computers, and Merrill Lynch Investment Bank. Approving Manager: Deepak Adhikary, Deputy General Manager, IFC-SEDF. DISCLAIMER IFC SmartLessons is an awards program to share lessons learned in development-oriented advisory services and investment operations. The findings, interpretations, and conclusions expressed in this paper are those of the author(s) and do not necessarily reflect the views of IFC or its partner organizations, the Executive Directors of The World Bank or the governments they represent. IFC does not assume any responsibility for the completeness or accuracy of the information contained in this document. Please see the terms and conditions at www.ifc.org/smartlessons or contact the program at smartlessons@ifc.org. SmartLessons, May 2007 4