48325 APRIL 2009 ABOUT THE AUTHORS Capturing Results a Better Way: MUSTAHID HOSSAIN Eight Lessons in Productive Data Collection Mustahid is the task manager for M&E at IFC-SEDF. Prior to joining IFC-SEDF in November Although many development partners have been discussing impact 2004, Mustahid was with the and evaluation for the last 30 years of international development, General Electric Company, Merrill Lynch Investment there still seems to be disagreement on indicators, impact, and how Bank, and Dell Computers. to collect the data, and discussions are ongoing. Evaluations and MONIRA FIRDOUSE impact assessments are crucial for IFC advisory services projects. They is the program assistant for the Monitoring and help identify issues in implementing strategies that ultimately stand Evaluation Unit. She joined IFC-SEDF in February 2005. in the way of realizing the project's goals and provide vital lessons for Earlier, she worked for American Express Bank Ltd. in the success of future endeavors. Based on our experiences at the IFC the Credit and Marketing Department South Asia Enterprise Development Facility (SEDF), we want to share some lessons on how to collect monitoring and evaluation (M&E) data DEEPAK P. ADHIKARY is the principal operations in the most effective way and what data to look for. officer. He was the head of IFC-SEDF in Bangladesh. Deepak was involved in many Background Anecdotal feedback and actual application of program evaluations while he the piloted performance measurement tool worked with the GTZ, the During the first cycle of the project, our M&E revealed several other issues; addressing them International Labour team, as a pilot, adopted a results-based taught us some valuable lessons: Organization, and Swisscon- performance measurement tool (hereafter tact before joining IFC. referred to as the "tool") to assess the Lessons Learned Special thanks to David Martz, effectiveness of our interventions. The tool Abdullah Mamun, Farhana was based on a planning matrix derived from 1) Develop the relationship with the client Gaffar, Shazia Ahmed, and causal chains (Figure 1) to make intended or the information provider before Mohammad Rehan Rashid. impacts within the framework of IFC requesting any data. development objectives and strategies. A major hurdle was the enterprises' reluctance We wanted this tool to help us assess the to share data, such as the status of their impacts of interventions and guide us in business performance. In countries like improving our future efforts. However, due to Bangladesh, information is spread by word of various challenges, not all staff members were mouth, according to oral tradition. This means able to use the tool to its full potential, and that the transmission of knowledge, thus our M&E results fell short of their high information, data, etc., occurs not via books aspirations. Although everyone wanted a andWebpages,butthroughpeopleinteracting "good" and useful system, few of them with one another in informal settings. Owners welcomed the work involved in updating files of small and medium enterprises (SMEs) prefer unless there was some kind of urgency. Often to get to know the people they are dealing their input came too late in the process to be with before they give out business or finance- useful. related data. Local businessmen feel Figure 1. M&E Causal Chain IFC SMARTLESSONS -- APRIL 2009 1 comfortable providing information in a subjective manner by talking to peers and colleagues, rather than by doing lonely research on facts and evidence. In other words: first, the relationship; then, the data-sharing. In the future, we plan to employ the oral tradition approach to build up rapport and strong networks with associations and business chambers. IFC staff engaged directly with farmers in the field to assess their needs. To ensure success, stakeholders must be part of the process. 2) Secure stakeholder consultation and buy-in at the outset. 3)IncludeaclauseintheMemorandumofUnderstanding Previously M&E was perceived as a data collection unit that (MOU) that ensures data sharing related to the project worked on numbers in an isolated way. Now we have come from every stakeholder involved. to know that stakeholders must be part of the process to ensurethetool'ssuccess.Stakeholdersincludetaskmanagers Previously, we collected data from multiple sources using and transaction leaders, clients, service providers/ various methods but were often left with numerous blanks associations, management, development agencies, and or conflicting information. It is not that clients did not want donors (Figure 2). It is important to get feedback from the to cooperate; they just lacked the foresight to provide the stakeholders regarding the project's interventions and the necessary data in the language we wanted. By coaching the expected impact they will have. Every stakeholder needs to clients on how data can be collected and how the clients understand how the data affect and illuminate their themselves could leverage the information, we started project's success, and hence take ownership over this collecting required data right at the outset through the dimension of the project. Teamwork and group MOU, and we were able to see significant improvements in brainstorming generate the kind of innovative approach data compliance. that is necessary. Because we kept the data to ourselves, the clients didn't see much value in their contributing. We learned that we Figure 2. Stakeholder Consultation should also share company data and our diagnosis with the clients so that they could see the value of the information. Consultant Report TAAS Client 4) Add an independent quality control unit. Donors Service Providers/ Even with more consistent data sharing, we encountered Association Information System Annual Report problemswithsustainingdataqualityintermsofconsistency Transaction / Case Study and comparability. We learned that an independent quality Leader/ Task Manager control unit would be a useful addition (Figure 3) so as to Management Impact Assessment provide a neutral view on data validation and process PSD M&E compliance. 5) Educate users about the value of constant feedback. We found that involving donors is a useful strategy. Because donor staff members keep changing, we also realized that It took time to educate users on M&E. Many of them simply it is important for them to understand the past, the present, did not see beyond data collection. As a result, the tool and the future outcome of plans. missed out on receiving constant feedback, which would 2 IFC SMARTLESSONS -- APRIL 2009 have given diverse users' perspectives. Ultimately, we the outcome and impact level only on the primary leveraged corporate-wide demand and an M&E network to beneficiaries. We then learned that it is critical to focus on educate users on the importance of M&E by circulating the changes in the business practices of those affected by information from organized M&E discussions among our interventions directly and indirectly, because that colleagues who were not originally a part of the process. provides greater insight into the strengths and weaknesses That said, it is still a most difficult process. of our interventions. Figure 3. M&E Framework 6) Customize tools for collecting baseline and monitoring For example, most of the access-to-finance work involved data. building partner financial institutions' competency in lending to SMEs by introducing new SME-friendly products We have multiple demands for our information, We need and training. We helped these institutions build and to adhere to the indicators and guidelines provided by the maintain a good tracking system that would look at changes Results Measurement Unit of headquarters to summarize in their SME loan portfolio, nonperforming loans, the the global data from the field (more generic/standard in number of borrowers and outreach, and the growth of new nature); provide sectoral benchmarks to create an industry- products. However, when the external evaluators saw that specific baseline so that we can do some comparisons after the banks were doing well with our assistance, their first the interventions; give information and feedback to the question was, "So what?" They wanted to see precisely task managers; and use the data as our own tool for "value- how the SMEs benefited by taking a loan from the bank added marketing" of our advisory services through sharing and how they improved their business performance. The industry-level information with the clients so that they are same concerns were raised in all the business lines. The aware of the changes taking place within an industry. biggest lesson we learned is that, in designing interventions, we really need to be specific about what kind of changes Initially, we followed the generic indicators suggested by we want to see in the ultimate beneficiaries and how we the Results Measurement Unit. But we soon learned that it can attribute our interventions. is crucial to design exclusive, user-friendly templates to address the specific issues of each relevant sector. For example, if we are working on the garment sector, it is important to look at the export volume, labor margin, and unit price of the garments to see how growth has been achieved. On the other hand, it is important to benchmark the cost of raw materials and inputs like electricity and gas for the light engineering sector, the focus of which is on reducing costs of inputs to other industries that use light engineering products as their raw materials. The biggest lesson we learned is that it is really important for the results measurement team based in the field to understand the causal relationship between interventions and certain results so that they can tailor the indicators accordingly. The M&E team should be involved at the project design stage. Doing so increases the workload, but The ready-made garments industry requires customized then there is no shortcut. tools for collecting data. 7) Go beyond measuring the impact on primary beneficiaries. 8) Follow through in the short and medium term. Previously, we tracked the effects of our interventions at We also need to look beyond short-term M&E. For example, IFC SMARTLESSONS -- APRIL 2009 3 after facilitating the organization of an electrical fair to promote local products, at year's end we measured the participants' sales and job growth. It was only later that we realized that we should have continued our M&E efforts and tracked how many future contracts or businesses the participants secured as a result of participating in the fair. Conclusion Our experience from the pilot demonstrates that a revised Management Information System(MIS)-basedperformancemeasurement tool that addresses the aforementioned bottlenecks is the best way to ensure that the desired goals of the project are achieved. We also plan to use our comprehensive M&E data base for all kinds of reporting, such as semiannual IFC reporting, corporate Advisory Services project management templates, semiannual donor reporting, management reporting, press releases, and case studies. DISCLAIMER IFC SmartLessons is an awards program to share lessons learned in development-oriented advisory services and investment operations. The findings, interpretations, and conclusions expressed in this paper are those of the author(s) and do not necessarily reflect the views of IFC or its partner organizations, the Executive Directors of The World Bank or the governments they represent. IFC does not assume any responsibility for the completeness or accuracy of the information contained in this document. Please see the terms and conditions at www.ifc.org/ smartlessons or contact the program at smartlessons@ifc.org. IFC SMARTLESSONS -- APRIL 2009 4