Knowledge Brief Health, Nutrition and Population Global Practice RWANDA: VERIFICATION OF PERFORMANCE IN COMMUNITY RBF Mr. Adrien Renaud and Dr. Jean-Paul Semasaka June 2014 KEY MESSAGES:  The supply-side and demand-side Results-Based Financing interventions contributed to improving the performance of Community Health Workers: o Although the frequency of over reporting errors did not change significantly, the errors declined substantially from over 140 percent to seven percent, indicating a clear improvement in performance.  Average quality scores indicate progress in reporting with a 9 percent increase in timeliness and completeness of reports, an 11 percent improvement in report accuracy, and an eight percent enhancement in the quality of management. estimated at 340 per 100,000 live births. Context To improve these maternal and child health (MCH) Rwanda is a low-income country with approximately 10.9 outcomes, the Government of Rwanda has been million inhabitants. Despite significant progress being implementing Results-Based Financing (RBF) nationwide, made since 1994, the country’s economic and with almost every administrative unit operating under a development indicators remain low. In 2011, the Gross performance agreement. In 2010, Rwanda implemented Domestic Product (GDP) per capita in Rwanda was two pilot RBF interventions at the community level: approximately US$ 580, and 58 percent of the population lived under the national poverty line. With a 0.429 score on 1. A supply-side scheme, using Performance-Based the Human Development Index, Rwanda ranked 166th out Financing (PBF) to incentivize cooperatives of of 187 countries in the 2011 Human Development Report Community Health Workers (CHW) to provide for the published by the United Nations Development Program (i) provision of 10 MCH services; (ii) quality reporting; (UNDP 2011). and (iii) the quality of cooperative management. Similarly, health indicators perform poorly: the total health 2. A demand-side scheme, using in-kind incentives to expenditure is approximately US$55 per capita, and life reward women who utilize three selected MCH expectancy at birth is 55 years; under-five mortality is at 54 services, including (i) antenatal care; (ii) delivery in a per 1,000 live births; and the maternal mortality ratio is health facility; and (iii) mother and child postnatal care Page 1 HNPGP Knowledge Brief  within three days of birth. inconsistencies identified. Data is checked against referral forms submitted by clients referred by CHWs. Introduction The second method pertains to the assessment of the This HNP Knowledge Brief focuses on the key findings of quality of CHW cooperatives, using a quality checklist. It the World Bank Case Study entitled: “Verification of entails the health center assessing the quality of Performance in Results-Based Financing (RBF): the Case reporting (i.e. timeliness, completeness, accuracy) of Community and Demand-Side RBF in Rwanda” (2014). monthly and the local steering committee validating this information every quarterly. It also involves the district This case study sought to shed light on the extent to which hospital evaluating the quality of cooperatives’ the RBF instigated positive change with regard to CHW management and the district steering committee validating performance in Rwanda. this evaluation. The third method concerns the verification of the METHODOLOGY quantity of in-kind incentives distributed, which Data for eight indicators were collected in four sectors in three provinces during the fourth quarters of 2010, 2011 performed by the district hospital (optional) during monthly and 2012. Data sources included key informants, registries routine monitoring visits to health centers. The demand- and Health Management Information System (HMIS). side scheme encourages women to use MCH services offered by health centers. When a woman comes to the To assess progress in the quantity of services provided, health center for any of three pre-defined services, she pre-verification and post-verification data were compared, receives an in-kind incentive. The health center is respectively using data on performance as self-assessed responsible for distributing the incentives and for renewing by CHWs in four cooperatives at cell level and verified data stock when appropriate. available at national level. In parallel, the analysis linked to reporting quality and cooperative management quality The fourth method involves the counter-verification of builds on data contained in quality checklists. the above-mentioned mechanisms, carried out by the health center, the sector and the district hospital or by the Reports developed after RBF counter-verification visits Ministry of Health (MoH) on either a purposive or were completed in the community (MoH, 2012a; MoH, systematic basis. 2012b) were also used to verify the distribution of in-kind incentives as well as trace patients in the community. Study Findings These reports were used to counter-balance the fact that purposive counter-verifications at the health center, sector The case study concentrates on the quantity of services and district steering committee levels were unavailable for provided, generating results through the comparison of analysis. self-assessed performance prior to verification and performance formally assessed after verification. It considers changes which occurred between the fourth RBF VERIFICATION MECHANISMS quarters of 2010, 2011 and 2012. The community RBF interventions implemented in Rwanda use four methods to verify the performance of CHW While this analysis did not highlight significant changes in cooperatives (i.e. supply-side scheme) as well as to verify the frequency of reporting the rates of errors during the the distribution of in-kind incentives (i.e. demand-side periods under review, it revealed significant decreases in scheme). the size of over-estimations – even though under- estimations remained stable: The first method concerns the verification of the quantity of services provided by CHWs, which is performed  48 percent of indicators were accurately self-assessed monthly for every cooperative (no sampling) by the by CHWs (i.e. where verification detected no error); affiliated health center and validated quarterly by a steering  24 percent were over-estimated; and committee headed by the local government administration.  28 percent were under-estimated with average under- estimation of 8 percent. Each month CHWs fill in the routine community HMIS form which includes the 10 quantity indicators. This form is then Interviews conducted for the case study indicate that errors compiled by CHWs at cell level (i.e. group of villages). At are mainly unintended mistakes rather than fraudulent the end of each month, a meeting is held with health center attempts to increase CHWs’ income. These mistakes are representatives and cooperative leaders to prepare a generally caused by (i) misunderstandings with regard to sector (i.e. group of cells) report. For each indicator, the definition of used indicators, leading CHWs to report summations are checked and potential data patients who do not meet the criteria used for incentivized indicators; and by (ii) compilation errors made when Page 2 HNPGP Knowledge Brief  consolidating reports at different levels of the scheme. Lessons Learned Reporting quality scores were only considered for 2011, as A high degree of variability was observed in the way tools they were unavailable for 2012. These scores are high for were used from one health center to another, from one completeness and timeliness, while indicating less district to another. For instance, different health centers progress with regard to reporting accuracy. On average, use different criteria to assess the quality of community among the 435 CHW cooperatives included in the supply- reports. side scheme and during the five quarters between the  A higher degree of homogeneity is desirable to enable fourth quarter of 2010 and the fourth quarter of 2011, comparisons. scores evolved from: Lack of incentives/penalties and lack of motivation in  85 to 94 percent for report timeliness and reporting accuracy/inaccuracy: Although most completeness; misreported data can be attributed to unintended mistakes,  81 to 89 percent for management quality; and there are no incentives to reduce the frequency of these  68 to 79 percent for report accuracy. errors – there are no penalties attached to misreporting. It should be noted here that the payment for quality reporting Among the sample of hospitals visited for the purpose of only takes into account the internal consistency of their the case study, there was no evidence that a verification of reports, not whether it triangulates with referral reports at in-kind incentives was carried out. the health facility.  The introduction of financial incentives/penalties linked There are two types of counter-verifications performed: to reporting accuracy could help reduce both the frequency and the size of reporting errors.  Purposive counter-verifications at the health center, and sector and district steering committee levels. The focus on curative care – rather than on preventive and promotional care – stems from the strong link between  Studies performed at national level including, among the HMIS and easily verifiable PBF indicators. other things, patient tracing in the community. Two  The incentivization of CHWs will have to focus both on studies have been conducted: increasing awareness-raising activities and on the The results of the purposive counter-verifications are not mechanisms required to verified results inherent to recorded and, thus, no results were presented by the case these activities. study. The supply-side study reviewed data for six indicators for the second quarter of 2012, comparing cell Consistent use of counter-verification: interviewed reports with (i) data compiled at sector level and (ii) results respondents indicated that sector steering committees – entered in the national database (i.e. HMIS on which who currently perform many verification activities on behalf payments are based). The study found a relatively high of health facilities – were strained by their added level of discrepancies between: responsibilities, stressing the need to further strengthen the  Cell and sector reports in 24 to 70 percent of the counter-verification mechanism. sectors (depending on the indicators),  Although results of one counter-verification study show  Sector reports and the national database in 17 to 67 that the in-kind incentives reach intended beneficiaries, percent of the sectors. more primary evidence is required. The study also checked whether clients reported having Pre-verification data submitted by CHWs is not registered been referred to a health center by a CHW to confirm the into a database. provision of care. This counter-verification established that  The analysis of the differences between reported and 97 percent of clients sampled could be identified in the verified data is critical to learning. It is also important in community. Conversely, the demand-side study tracked informing changes toward a less systematic and less patients in the community. The period assessed was April costly verification system (for example, sampling). to September 2011. A total of 107 patients reported by health facilities to have received in-kind incentives were Overall documentation and use of data: the case study randomly selected in the 55 health centers of the study: 97 was limited by the lack of recorded data. For instance, as percent of patients confirmed having been treated at the data pertaining to in-kind incentives was not consistently facility. Among them, 97 percent reported having received registered, an analysis of data linked to the demand-side treatment for the relevant service, and 98 percent of intervention was difficult. women confirmed receiving in-kind incentives  Better documentation and better data use are required. Page 3 HNPGP Knowledge Brief  Recommendations References Basinga, P. et al. 2010. Paying Primary Health Care Centers for 1. Develop a system capable of systematically comparing CHWs reported data and verified data to guide the Performance in Rwanda. Washington DC: World Bank, Policy Research provision of incentives/penalties. This could be Working Paper Series. realized by modifying the mechanism used for the Rusa, L., et al. 2009. Performance-Based Financing for Better Quality of verification of the quality of CHW cooperatives, which Services in Rwandan Health Centers: 3-Year Experience. Tropical currently focuses on the internal consistency of reports (for example, data accuracy) as well as on components Medicine & International Health 14: 830–37. that are not expected to change over time such as Ministry of Health of Rwanda. 2008. National Community Health Policy. district authorization and legal status. Kigali. 2. Select indicators that reflect the key objectives and _____. 2009. Financement Basé sur la Performance au Niveau goals of the project while also being measurable and Communautaire : Guide de l’Utilisateur. Kigali. verifiable, potentially including more preventive measures. _____. 2011.. The National Community Performance-Based Financing 3. Implement systematic counter-verifications: the first and In-Kind Conditional Cash Transfers: Program Implementation task should be to provide sector steering committees Manual. Kigali. with standardized guidelines and tools to facilitate the _____. July 2012a. Evaluation du Processus du Modèle d’Incitations du adequate sampling of households as well as the Côté de la Demande au Rwanda. Kigali. preparation of reports that allow consolidation at sector, district, and national levels. At a later stage, _____. .December 2012b. Rapport de la Deuxième Contre-Vérification risk-based sampling could be introduced. du Financement Basé sur la Performance au Niveau Communautaire au 4. Increase both the frequency and the rigor of counter- Rwanda. Kigali. verification activities to better assess the performance UNDP. 2011. World Development Report 2011: Sustainability and of the schemes as well as identify areas for Equity: A Better Future for All. New York: UNDP. improvement. World Bank. 2011. World Development Indicators Database. 5. Ensure that health facilities and CHW cooperatives across the country use standardized tools and criteria. Washington DC: World Bank. 6. Train key actors of the community RBF intervention to ensure that verification criteria are used uniformly. This HNP Knowledge Note highlights the key findings from a study by the World Bank on the “Verification of Performance in Results -Based Financing (RBF): the Case of Community and Demand-Side RBF in Rwanda” by Mr. Adrien Renaud and Dr. Jean-Paul Semasaka, 2014. The Health, Nutrition and Population Knowledge Briefs of the World Bank are a quick reference on the essentials of specific HNP-related topics summarizing new findings and information. These may highlight an issue and key interventions proven to be effective in improving health, or disseminate new findings and lessons learned from the regions. For more information on this topic, go to: www.worldbank.org/health. Page 4