91772 H N P D i s c u s s i o n P a p e R VERIFICATION OF PERFORMANCE IN RESULTS- BASED FINANCING (RBF) The Case of Community and Demand-Side RBF in Rwanda Adrien Renaud and Dr. Jean-Paul Semasaka June 2014 VERIFICATION OF PERFORMANCE IN RESULTS-BASED FINANCING (RBF): The Case of Community and Demand-Side RBF in Rwanda Adrien Renaud and Dr. Jean-Paul Semasaka Consultants June 2014 Health, Nutrition, and Population (HNP) Discussion Paper This series is produced by the Health, Nutrition, and Population (HNP) Family of the World Bank's Human Development Network (HDN). The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations, or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For information regarding the HNP Discussion Paper Series, please contact the Editor, Martin Lutalo at mlutalo@worldbank.org or Erika Yanick at Eyanick@worldbank.org. © 2014 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved. Health, Nutrition, and Population (HNP) Discussion Paper Verification of Performance in Results-Based Financing (RBF): The Case of Community and Demand-Side RBF in Rwanda Adrien Renauda and Dr. Jean-Paul Semasakab a Freelance Health Economist b Consultant With thanks to funding from the Health Results Innovation Trust Fund (HRITF) Abstract: Rwanda, led by its Ministry of Health, was the first African country to implement Performance- Based Financing (PBF) nationwide in its health centers and hospitals. The country then went on to pilot RBF interventions at the community level. Rewarding community members who provide and use Maternal and Child Health (MCH) services is an innovative form of Results-Based Financing (RBF). The supply-side of this community PBF scheme focuses on cooperatives of Community Health Workers (CHWs). It pays them to provide selected MCH services and rewards them for the quality of their reporting as well as for good management. Conversely, the demand-side of this community PBF scheme provides women with in-kind incentives when they utilize three selected MCH services in health centers. Verifying the performance of these interventions is an integral part of RBF program implementation. This case study aims to describe the verification mechanisms used in Rwanda, which include monthly verification of the quantity of services provided by the CHWs, quarterly assessment of the quality of the functioning of the CHW cooperatives (including its reporting), verification of the quantity of in-kind incentives distributed in a less systematic way, as well as counter-verification of these three verification processes. This paper presents results of these verification methods, and discusses the obstacles faced, the way they were addressed, and the challenges that are still ahead. This case study is part of a broader analysis, involving multiple country case examples. It endeavors to expand knowledge about verification processes and practices and to address the design and implementation needs of RBF programs. Keywords: Community PBF, Results-Based Financing, Maternal and Child Health, Verification, Demand-Side. Disclaimer: The findings, interpretations, and conclusions expressed in the paper are entirely those of the authors, and do not represent the views of the World Bank, its Executive Directors, or the countries they represent. Correspondence Details: Petra Vergeer, The World Bank, 1818 H Street, NW Washington, DC 20433, USA; e-mail: pvergeer@worldbank.org 3 4 Table of Contents ACKNOWLEDGMENTS .......................................................................................................... 6 LIST OF ACRONYMS .............................................................................................................. 7 EXECUTIVE SUMMARY ........................................................................................................ 8 INTRODUCTION .................................................................................................................... 11 1. METHODOLOGY............................................................................................................ 13 2. CONTEXT: THE COMMUNITY RBF INTERVENTIONS IN RWANDA .................. 15 3. MAJOR CHARACTERISTICS OF THE VERIFICATION METHODS ....................... 19 3.1. HOW IS THE QUANTITY OF SERVICES PROVIDED BY CHWS VERIFIED?...................... 20 3.2. HOW IS THE QUALITY OF THE CHW COOPERATIVES ASSESSED? ............................... 22 3.3. HOW IS THE QUANTITY OF DISTRIBUTED IN-KIND INCENTIVES ASSESSED? ............... 23 3.4. HOW IS DATA COUNTER-VERIFIED? ........................................................................... 24 4. FINDINGS FROM THE APPLICATION OF THE VERIFICATION METHODS ....... 27 4.1. WHAT WERE THE RESULTS OF THE VERIFICATION OF THE QUANTITY OF SERVICES PROVIDED BY CHWS? ............................................................................................................................ 27 4.2. WHAT WERE THE RESULTS OF THE ASSESSMENT OF THE QUALITY OF CHW COOPERATIVES? 32 4.3. WHAT WERE THE RESULTS OF THE VERIFICATION OF THE DISTRIBUTED QUANTITY OF IN- KIND INCENTIVES? ................................................................................................................ 34 4.4. WHAT WERE THE RESULTS OF COUNTER-VERIFICATIONS PERFORMED AT VARIOUS LEVELS? 35 4.5. USE OF VERIFICATION FINDINGS ................................................................................ 36 5. VERIFICATION COSTS ................................................................................................. 39 6. LESSONS LEARNED ...................................................................................................... 40 REFERENCES ..................................................................................................................... 43 ANNEX 1: LIST OF PEOPLE MET ............................................................................................ 44 ANNEX 2: ANALYSIS FRAMEWORK FOR THE CASE STUDIES .................................................. 44 ANNEX 3: CHW MONTHLY REPORT ..................................................................................... 44 ANNEX 4: CHW COOPERATIVE QUALITY ASSESSMENT REPORT .......................................... 44 ANNEX 5: DEMAND-SIDE SCHEME HEALTH CENTER REPORT ............................................... 44 ANNEX 6: CHW SUPERVISION REPORT (ESTABLISHED BY HEALTH CENTER) ........................ 44 5 Acknowledgments The authors are grateful to the World Bank for publishing this report as an HNP Discussion Paper. The authors of this study would like to thank the Ministry of Health of Rwanda for its cooperation during data collection. Special gratitude goes to the Maternal and Child Health (MCH) Directorate and its head, Dr. Fidèle Ngabo, and to the community health desk and its head, Cathy Mugeni. The authors also would like to thank Management Sciences for Health (MSH) and especially Dr. Cedric Ndiziye, who helped with data collection and provided valuable advice on an earlier stage of this document. To ensure the accuracy of the data presented, the document was also reviewed by the MCH Directorate. Finally, this study would not have been possible without the help, support, advice, and input of the World Bank staff. Special thanks go to Petra Vergeer and Anna Heard. 6 LIST OF ACRONYMS CAAC Contracting unit of the Ministry of Health (Cellule d’Appui à l’Approche Contractuelle) CHW Community Health Worker DOTS Directly Observed Therapy GDP Gross Domestic Product GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria HMIS Health Management Information System MCH Maternal and Child Health MoH Ministry of Health MSH Management Sciences for Health NGO Nongovernmental Organization NURSPH National University of Rwanda, School of Public Health PBF Performance-Based Financing PMTCT Prevention of Mother to Child Transmission RBF Results-Based Financing TB Tuberculosis VCT Voluntary Counseling and Testing The exchange rate used in this case study is from February 15, 2013: US$1 = RF 624.101 7 EXECUTIVE SUMMARY Rwanda is a low-income country located in the African Great Lakes region. Since the genocide in 1994 it has achieved impressive results in the field of development, but indicators in the country remain low. Still, health indicators, while low, are generally better than for neighboring countries. Results-Based Financing (RBF) is highly developed in Rwanda, and almost every administrative unit operates under a performance agreement. Rwanda was the first African country to implement Performance-Based Financing (PBF) nationwide. The health center and hospital schemes have been taken as a model by many followers on the continent. In 2010, Rwanda implemented two pilot RBF interventions at community level: • A supply-side scheme, the community PBF, which rewards Community Health Worker (CHW) cooperatives for providing 10 selected Maternal and Child Health (MCH) services, for the quality of their reporting, and for good management. • A demand-side scheme, which encourages women to utilize three selected MCH services by providing them in-kind incentives when they do so. This study aims to describe the verification systems involved in both of these schemes, but also to identify the results they achieved, the obstacles they faced, the way they addressed them, and the challenges that are still ahead. The Rwanda community RBF interventions involve four types of verification (see figure 1.1): • Verification of the quantity of the services provided by CHWs performed monthly for every cooperative by the affiliated health center and validated quarterly by a steering committee headed by the local government administration. • Assessment of the quality of CHW cooperatives, including the quality of the reports (timeliness, completeness, accuracy), assessed monthly by the health center, and validated quarterly by the local steering committee; and of the quality of the cooperatives’ management, assessed quarterly by the district hospital, and validated by a district steering committee. • Verification of the quantity of in-kind incentives distributed, performed by the district hospital (optionally) during monthly routine monitoring visits to health centers. • Counter-verification of the information provided by these three mechanisms, by the health center, the sector, the district hospital, or the Ministry of Health (MoH) on a purposive or on a systematic basis. 8 Figure 1.1 Verification of the Community RBF Scheme in Rwanda Quarterly cooperative Quarterly validation District management quality District of cooperative District Steering level assessment Hospital management quality Committee Monthly verification of in- kind incentives quantity Contract Monthly quantity Quarterly quantity verification validation Health facility Sector CHW Steering Sector Monthly report quality Health Quarterly report quality cooperative Committee level assessment facility in- validation charge of ~100 CHWs CHWs Health facility in- Contract charge of CHWs Source: Authors 2014. This study identified the following results of these verification mechanisms: • Verification of the quantity of services provided by CHWs: during the fourth quarter of 2010, the fourth quarter of 2011, and the fourth quarter of 2012, in the four sectors visited and for the eight indicators for which data could be collected in the framework of this case study, 48 percent of indicators were accurately assessed by CHWs (that is, verification detected no error). Further, 24 percent were overestimated (average overestimation 23 percent), and 28 percent were underestimated (average underestimation 8 percent). Although error rates did not really diminish during the period, the level of errors dramatically fell (for example, average overestimation fell from 147 to 7 percent). • Assessment of the quality of CHW cooperatives: between the fourth quarter of 2010 and the fourth quarter of 2011, the 435 CHW cooperatives received, on average, between 85 and 94 percent of the points available for report timeliness and completeness, between 68 and 79 percent for report accuracy, and between 81 and 89 percent for management quality. • Verification of the in-kind incentives distributed: the results of the verification of the in-kind incentives distributed could not be analyzed in the framework of this study because they are not recorded. • Counter-verification: the results of purposive counter-verifications at health center, and sector and district steering committee levels also could not be analyzed because they are not recorded. A counter-verification study based on a random sample of 60 sectors included in the supply-side scheme found that 97 percent of the patients reported by the CHW cooperative to have received their services were found back in the community. In another 9 study on the demand-side scheme, a similar proportion of patients reported by health centers to have received in-kind incentives confirmed that they did. The errors detected by the verification systems appear to be unintended mistakes rather than fraud attempts intended to increase the income of CHWs. The major causes of these mistakes were as follows: • Recording errors: CHWs can misinterpret definitions of the indicators, and declare they referred patients who did not in fact match the criteria for receiving subsidies. • Compilation errors: calculation mistakes can be made when compiling reports between different levels of the scheme. The verification system in the Rwanda community RBF interventions is highly integrated with the rest of the health system: verification is done by actors who also supervise those who are verified. This allowed a satisfactory level of fraud detection, kept costs relatively low, created ownership, and strengthened the Health Management Information System (HMIS). However, this strong degree of integration, combined with a high level of decentralization, resulted in decentralized results and variable standards: the processes that are implemented can differ from one district or one health center to another. It might be useful to develop more standardized tools and procedures for documentation and evaluation. Moreover, the integrated nature of the verification system in the Rwanda community RBF interventions pushes attention toward verified results, which trigger payment. The difference between reported and verified results is overlooked: there is no real incentive for CHWs to report results accurately, and no sanctions for reporting results inaccurately. Finally, the level of documentation and registration of the results is low, especially for the demand- side scheme. Better documentation and use of data are necessary. It seems symptomatic that much of the difficulty in performing this study was due to lack of data documentation. Moving forward, the Rwandan government should also consider increasing both the frequency and the rigor of counter-verification activities to better assess performance of the schemes and areas for improvement. In spite of these challenges, the verification system in the community RBF interventions has yielded great achievements in a very difficult environment characterized by limited physical accessibility of CHWs and beneficiaries, and by low reporting capacities of cooperatives. Improvements to the scheme are necessary, but these can only be slow and gradual. 10 INTRODUCTION Rwanda is a low-income country located in the African Great Lakes region, with approximately 10.9 million inhabitants. Its recent history is marked by the 1994 genocide, since which it has achieved impressive results in the field of development (economic growth, infrastructure) In spite of these accomplishments, the economic and development indicators in Rwanda remain low. In 2011, the Gross Domestic Product (GDP) per capita was approximately US$580, 1 and 58 percent of the population lived under the national poverty line. With a 0.429 score on the Human Development Index, it ranked 166th out of 187 countries in the 2011 Human Development Report published by the United Nations Development Program (UNDP 2011). Health indicators are quite low, although better on average than in neighboring countries: the total health expenditure is approximately US$55 per capita; life expectancy at birth is 55 years; under-five mortality rate is 54 per 1,000 live births; and the maternal mortality ratio is estimated at 340 per 100,000 live births. Table 1.1 Basic Facts about Rwanda Basic Facts about Rwanda Population (millions) 10.9 GDP per capita (current US$) 580 Percentage of population under the national poverty line 58 (%) Total health expenditure per capita (current US$) 55 Life expectancy at birth (years) 55 Under-five mortality (per 1,000 live births) 54 Maternal mortality ratio (per 100,000 live births) 340 Source: World Bank 2011. Results-Based Financing (RBF) is very developed in Rwanda: from the village level to the ministries, almost every administrative unit signs a performance contract within its hierarchy. In the health sector, Rwanda was the first African country to implement Performance-Based Financing (PBF) nationwide. The health center and hospital schemes have been taken as a model by many followers on the continent, and have been widely documented (Rusa et al. 2009; Basinga et al. 2010). More recently, Rwanda has been implementing two pilot RBF interventions at community level in part of the country: • A supply-side scheme, the community PBF, focused on Community Health Workers (CHWs) • A demand-side scheme at community level Both schemes are focused on Maternal and Child Health (MCH). They are innovative in the sense that most low- and middle-income country RBF schemes are focused on clinical services and leave the community aside or limit its interventions to specific activities (such as the household survey in the Burundi scheme). These community RBF interventions directly reward the members of the community as actors for their own health. 1. In 2011 US dollars. This figure, as well as all other figures presented in this section, is taken from World Development Indicators Database 2011 (unless otherwise indicated). 11 As for verification, the community RBF interventions in Rwanda raise specific issues linked to the nature of their geographic dispersion. First, they involve a large number of providers compared to the target population: for example, in Rwanda, while a health center typically serves a geographic area covering 10,000 people, 3 CHWs serve only 750 people on average. At the same time, the educational level of these CHWs is often very low, which presents a challenge in their reporting abilities. Physical accessibility matters often complicate the relationship of CHWs (or with beneficiaries in the case of the demand-side scheme) with providers. When the verification systems were set up and implemented, all these constraints had to be addressed by the community PBF program and the demand-side scheme. This study aims at describing these verification systems, but also at identifying the results they achieved, the obstacles they faced, the way these were solved, and the challenges that remain. Figure 1.2 Rwanda in Africa Source: World Bank 2013. 12 1. METHODOLOGY Data collection for this study was carried out in two steps. First, relevant documents (see references) were analyzed, and telephone interviews were conducted with key informants. Second, quantitative and qualitative data were gathered during a two-week visit to Rwanda in 2013. On the qualitative side, key actors at central level and relevant field actors were interviewed. A set of four districts was selected in consultation with the Ministry of Health (MoH). Random sampling was not feasible, and we tried to choose a sample that is representative of the geographical diversity by selecting districts in three of the five provinces of the country, while taking into account physical accessibility constraints. In each of the selected districts, interviews were conducted with the district administration staff in charge of the health sector and with the district hospital staff in charge of PBF. One health center per district was also selected in consultation with the MoH (once again taking into account physical accessibility constraints), and interviews were conducted with the head of the health center, the person in charge of CHWs at health center level, the CHW cooperative president, and the person in charge of social affairs (including health) at sector level (that is, local government). Nothing was known about the performance or reputation of the cooperatives prior to selection. The list of people met is presented in annex 1, and table 1.2 summarizes the areas and facilities that were visited. Table 1.2 Districts and Facilities Visited District Hospital visited Health center visited Kamonyi Remera Rukoma Kamonyi (Gacurabwenge sector) Gicumbi Byumba Mulindi (Kaniga sector) Rwamagana Rwamagana Rubona (Rubona sector) Kirehe Kirehe Kirehe (Kirehe, Gatore, and Kigina sectors) Source: Authors 2013. On the quantitative side, the data sources differed according to the various verification mechanisms. The Rwandan community RBF interventions involve four types of verifications: • The verification of the quantity of services provided by CHWs: as explained later (see sections 3.1 and 4.1), only the data on verified services is registered in the national database. To assess the difference between the data before and after verification, we had to use the paper reports of the CHW cooperatives we visited. In the absence of an alternate solution, we compared the data on the initial performance as self-assessed by the CHWs of this sample with the verified data that is available at national level. There are only four CHW cooperatives in this sample, and they have not been randomly selected; consequently, the results cannot be considered representative. However, we considered them relevant for this case study since they are corroborated by other studies (see sections 4.1 and 4.4). • The quality of CHW reports and of CHW cooperatives management: the data could be retrieved from the national database, but they were available only until the end of 2011, due to a change in the software used by the community Health Management Information System (HMIS). • The quantity of in-kind incentives distributed: as will be explained later (see sections 3.3 and 4.3), the demand-side scheme does not register the data before verification is made. This made the quantitative study of the results of verification impossible. 13 • The counter-verification: two counter-verification reports were produced in the framework of RBF community interventions, one on the demand-side scheme (MoH 2012a) and one on the community PBF scheme (MoH 2012b). Data from these reports were analyzed. Moreover, sector steering committees began counter-verification visits in the community in the last quarter of 2012; reports of these visits in the four sectors visited in the framework of this study were also analyzed. Additionally, data on the cost of verification were collected through the accountancy service of the MCH directorate of the MoH. The analysis of the quantitative and qualitative data was done according to a framework that has been used in several other case studies on the verification mechanisms in various countries (annex 2). This framework was designed to allow comparisons between the systems. It comprises five major elements that determined the major sections of this case study: overview of the RBF interventions (section 2), description of the verification systems at stake (section 3), findings of the verification methods (section 4), cost of verification (section 5), and finally lessons that can be learned from implementing the verification methods (section 6). 14 2. CONTEXT: THE COMMUNITY RBF INTERVENTIONS IN RWANDA The origins of the current community RBF interventions go back to 2005, when a project was implemented with support from the World Bank, but did not give satisfactory results. According to the national community health strategy (MoH 2008), this project contracted CHWs for achieving five targets: • Reduction of maternal mortality through increased health facility deliveries • Reduction of deaths due to malaria through increased use of treated mosquito nets • Reduction of under-five deaths due to dehydration through increased use of oral rehydration solution • Improved personal hygiene • Accurate and timely reporting by CHWs This strategy was implemented in 30 districts, but the results were not judged satisfactory. According to people interviewed in the framework of this study, the following reasons were identified: • The scheme was implemented at district level to support the decentralization process, but part of the funds were channeled to finance other district priorities. • The indicators were not sufficiently precise and verifiable. • The verification system and tools were not adequate. This relative failure could have led to discontinuation of RBF interventions at community level in Rwanda. However, the impact evaluation of the “health facility” PBF scheme (as opposed to the “community” PBF scheme), conducted between 2006 and 2008, identified gaps in community activities, which led to continuing community RBF interventions instead of stopping them. Indeed, the study showed especially that the impact of the “health facility” PBF on health was good, except for certain MCH services such as antenatal care or family planning, for which behavior change and involvement at community level was necessary (Basinga et al. 2010). The design and management of a new intervention involved a variety of actors at different levels, among these were the following: • Government: the MoH is involved through its MCH directorate, which runs the scheme, and through its PBF unit (Cellule d’Appui à l’Approche Contractuelle, CAAC), which ensures consistency with the national PBF policy. • Donors: the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) and the World Bank fund the scheme together with the government. • Technical partners: nongovernmental organizations (NGOs) were involved, such as Management Sciences for Health (MSH), which gave technical assistance. • Academia: the National University of Rwanda, School of Public Health (NURSPH) is involved in the evaluation of the impact of the scheme, in collaboration with the World Bank. As mentioned earlier, the Rwandan community RBF interventions actually include two schemes, both focused on MCH: • One supply-side scheme, named community PBF, that gives financial incentives to CHWs • One demand-side scheme that gives in-kind incentives to women seeking some MCH services at health center level 15 This decision to implement two schemes at the same time was part of a strategy aimed at determining which intervention would have the best impact on results. The schemes were indeed implemented in the framework of an impact evaluation, which influenced its design: 200 sectors were selected, and each of these was randomly assigned to treatment or control groups. There are five types of sectors in the country: • Sectors included in the supply-side scheme • Sectors included in the demand-side scheme • Sectors included in both the supply- and demand-side schemes • Sectors included in the control group • Sectors not included in the study Participation is mandatory for facilities included in one of the treatment groups of the impact evaluation. Table 1.3 summarizes the population in each of these five groups. Table 1.3 Population in Each of the Study Arms of the Study Study arm Number of Population Population (%) sectors Supply 50 1,276,024 12 Demand 50 1,123,608 11 Supply and demand 50 1,248,060 12 Control 50 1,153,748 11 Sectors not included in the 216 5,735,782 54 study Total 416 10,537,222 100 Source: 2012 Census of the Population. CHWs are volunteer workers living in the community and providing selected community health services. According to the national community health policy (MoH 2008), they “form the link between the health centers and the community, serving as the mouthpiece and ears of the health service at the community level. They know the communities well as they live and work among them, and they are often respected individuals.” CHWs are not contracted individually. They are organized in cooperatives, which have three members per village. The cooperative signs a contract with the sector (local government, which plays the role of purchaser), through which it is rewarded for two activities: • It receives a quarterly fee-for-services amount for a set of 10 MCH output indicators (table 1.4), selected for their direct impact on the activities of the health center on the one hand, and for their easy verification on the other hand. This aspect was key, because the activities are performed at community level, which renders verification difficult. This is why referral indicators were preferred over awareness raising indicators, for example: the former leave a trace at the health center (referred patients are seen at the facility), and are thus easier to verify. A subsidy is attached to every indicator: the total amount is not capped, and the more patients a cooperative refers to the health center, the more money it gets. • It receives a quarterly grant for the cooperative quality. The amount is a percentage of a lump sum (proportional to population size), and the percentage is determined by an evaluation made by the sector and district steering committees based on a quality assessment 16 checklist with nine indicators (table 1.4) on the one hand, and on their accurate, timely, and complete participation in the community HMIS on the other hand. Payments are not made to individual CHWs, but to cooperatives, (there are approximately 100 CHWs in one cooperative). In the four cooperatives that were met for the purpose of this study, PBF payment ranged from RF 1,800,000 to RF 6,000,000 (US$2,795 to US$9,317) per quarter. This incentive payment is only one part of the income of the cooperatives: they are supposed to implement income-generating activities, so that they are not entirely dependent on PBF revenue. The cooperatives that were met for the purpose of this study declared that income-generating activities accounted for between 6 and 13 percent of their total financial resources, the remainder came from PBF. The demand-side scheme aims at encouraging women to utilize MCH services at health centers. When a woman comes to the health center for any of the three services that have been selected for the scheme (the indicators are listed in table 1.4 and were selected because of their low utilization rates), she receives a predefined in-kind incentive (such as adult or baby clothes, water treatment tablets, soap, shawl, bed sheets, umbrellas). The health center is responsible for distributing the incentives, and for renewing the stock, which it manages through a special account funded by the MoH. The indicators that are rewarded in the Rwanda community PBF scheme and demand-side scheme are summarized in table 1.4. 17 Table 1.4 Indicators In Rwanda Community PBF scheme and Demand-Side Scheme Supply-side Pay-for- Pay-for-reporting: Demand-side Pay-for-indicators reporting: Cooperative quality Report quality 1. Children (6 to 59 1. Timeliness 1. Contract signed 1. Pregnant women months) monitored for 2. Completeness 2. District consulting a health nutritional status 3. Accuracy authorization center for prenatal 2. Women accompanied obtained care visits (in-kind to the health center for 3. Appropriate stock incentive given only antenatal care before or of community HMIS once per pregnancy) during the fourth month and reporting forms 2. Women delivering of pregnancy 4. Presence of a cash in health facilities 3. Women accompanied book and evidence of 3. Mother-child pairs for delivery at health at least three receiving care at a facility transactions in the health center within 4. New family planning period 10 days of birth users referred by CHWs 5. Cooperative cooperatives to the health members’ meeting center held this quarter and 5. Regular users of minutes available modern contraceptives at 6. Cooperative the health center president named 6. TB 2 — cases followed 7. Legal status per month at home in the obtained community — DOTS 3 8. Full bank program information present 7. “Real” 4 suspected TB 9. Business plan cases referred to the present for the period health center 8. Women accompanied for postnatal care 9. Women accompanied for PMTCT 5 10. Patients referred for VCT 6 Source: MoH 2011. 2. Tuberculosis. 3. Directly Observed Therapy. 4. “Real” means that health care provider at health facility performed lab examinations for TB since the case was considered legitimately suspicious; data are retrieved from the health center data. 5. Prevention of Mother To Child Transmission. 6. Voluntary Counseling and Testing. 18 3. MAJOR CHARACTERISTICS OF THE VERIFICATION METHODS The verification method, as well as the whole scheme, was designed to work within the Rwandan administrative and health system and to avoid creating new structures. To understand this verification system, it is necessary to first understand the Rwandan administrative system: • The country is divided in five provinces. • Each province is divided in districts (30 districts in the country, approximately 6 districts per province): this is the level of the district hospital. • Each district is divided in sectors (416 sectors in the country, approximately 14 sectors per district): this is the level of the health center and of the CHW cooperative. • Each sector is divided into cells (2,148 cells in the country, approximately 5 per sector). • Each cell is divided into villages (14,842 villages in the country, approximately 7 per cell): this is the CHW level (3 CHWs per village). The verification system climbs this administrative pyramid step by step, only omitting provincial level, which is not yet fully operational; for verification activities as well as for other activities, CHWs are monitored by the health center; health centers are monitored by the district hospital; and district hospitals are monitored by central level. The Rwanda community RBF interventions involve four types of verification (see figure 1.1): • Verification of the quantity of services provided by CHWs performed monthly for every cooperative by the affiliated health center and validated quarterly by a steering committee headed by the local government administration. • Assessment of the quality of CHW cooperatives including the quality of the reports (timeliness, completeness, accuracy), assessed monthly by the health center and validated quarterly by the local steering committee, and assessment of the quality of the cooperative’s management, assessed quarterly by the district hospital and validated by a district steering committee. • Verification of the quantity of in-kind incentives distributed, performed by the district hospital, optionally, during monthly routine monitoring visits to health centers. • Counter-verification of the information provided by these three mechanisms by the health center, the sector, the district hospital, or the MoH on a purposive or on a systematic basis. 19 3.1. HOW IS THE QUANTITY OF SERVICES PROVIDED BY CHWS VERIFIED? Verification of the quantity of services provided by CHWs is performed every month for every cooperative, without any sampling. First, at village level, each month the CHWs fill in the routine community HMIS form (paper-based, see annex 3) that comprises approximately 50 indicators, including the 10 quantity indicators of the community PBF scheme. It must be emphasized that the form is for HMIS purposes, and that it is used for PBF only to avoid unnecessary duplication. Since the size of the population served by one CHW is very low (250 people per CHW on average), it is not necessary to include names of the patients or unique identification numbers in the forms or in the databases. In case one patient needs to be tracked, information can easily be found in the CHW’s register. This community HMIS form is then compiled by CHWs at cell level (at least one CHW per village participates in this compilation), and a monthly cell community HMIS report is established. It is based on the same form as the village community HMIS report. The health center, which is the verifier, only steps in after this cell-level compilation is done. A meeting is held at the health center at the end of the month. It can be attended by all CHWs; however, in the sectors visited in the framework of this case study, it is only attended by the board of the CHW cooperative and the health center person in charge of CHWs, 7 assisted by the health center data manager. A compilation of the cell HMIS reports is done, and a sector community HMIS report is established using the same form. For every indicator, the calculations are checked, and possible data inconsistency is tracked. For the 10 indicators that are linked to PBF payment, the cooperative members and the health center staff also check whether the number of services compiled through this exercise matches the number of services that has been recorded by the health center. The sources that are used for this check are the referral forms that are given to the patients by CHWs and the registries of the health center. At the end of the meeting, the health center makes recommendations to the cooperative if necessary. One should note the relationship between CHWs and verifier goes well beyond the verification process: the health center is also responsible for planning and monitoring the activities of CHWs, for training, and the like. The verification of the quantity is just another activity that they perform together. This daily cooperation, combined with the fact that the health center can, in a certain sense, be considered the superior of the CHW cooperative and is evaluated on its ability to monitor its activities, illustrates the very integrated and supportive nature of the community PBF verification system. Health centers were given a large amount of autonomy in how to implement verification of the quantity of services provided by CHWs. Hence, the exact methodology and time taken for this activity varied widely in the four facilities visited for this case study: in one of the facilities, the staff in charge of CHW and the data manager said that they were comparing the cell reports with data sources on their own, while the other facilities said they undertook this task together with CHWs. In one facility, the staff in-charge of the health center said that he sometimes helped with the quantity verification, while in other facilities the person that sometimes assisted was the data 7. The health center person in charge of CHWs is usually a highly qualified nurse or equivalent, such as an environmental health specialist. The staff person is always a school graduate. 20 manager. As a result of this great variability, the duration of the meeting could vary from three hours to two entire days according to the people that were interviewed for the purpose of this case study. Once the meeting between CHWs and health center staff is concluded, the monthly sector community HMIS report is stored in an online database by the health center data manager. This task was previously performed by the district hospital, but has been transferred to the health center in the course of the project to alleviate the workload of hospital data managers. Data is then analyzed at national level, and feedback is given to district hospitals if needed. Every three months, a sector steering committee meets to validate the data and allow payment. The main reason for establishing this committee was to include third-party actors between the health center and the CHW cooperative. The sector steering committee is composed of eight members: • The administrative sector in-charge of health and social affairs (chairperson) • The head of the health center (vice chairperson) • The president of the CHW cooperative (nonvoting member) • The health center person in charge of CHWs (secretary) • The accountant of the health center (added in the course of the project because he is the manager of the health center subaccount dedicated to the CHW cooperative; it is important that he has a sound understanding of the fund flows) • One community member (not member of any of the represented institutions) • One representative of the NGOs supporting CHW cooperatives • The environmental health officer of the health center (nonvoting member) It is apparent that the composition of the committee is dominated by health center staff. The capacity of the steering committee to fulfill its role of external oversight between the health center and the CHW cooperative depends on the dynamism and interest of its president, the administrative sector in-charge of health and social affairs. The president of the steering committee signs the contract with CHW cooperatives and acts as purchaser in the community PBF schemes, but the human resources that he/she relies on to implement decisions of the steering committee are members of the health center staff. This once again exemplifies that the verification process of the Rwanda community PBF is very integrated: not only is the CHW performance verified by a “superior;” but the validation of this verification is performed by a committee dominated by the same stakeholder. No conflict of interest related to this situation has been reported to the authors of this study, but it must be noted that such proximity between the verifier and the actor that is verified could pose problems. During the meeting of the steering committee, participants check that the compilations done in the three monthly sector community HMIS reports are correct, and analyze the evolution of performance to detect possible incoherence. As will be explained later, they also analyze the results of the quality assessment of the HMIS reports (see section 3.2). If some corrections have to be made in the online database after the meeting, they must be done by the health center data manager. If specific data quality problems are identified, the steering committee can decide to perform counter- verification visits in one or several villages (see section 3.4). In visited facilities, the minutes of the meeting show that their average duration is three hours. Only after the sector steering committee meeting is held can payment be released. The amount that is due to the cooperative (depending on performance) is first paid by the MoH to a health center subaccount dedicated to the community PBF. The health center then transfers this amount to the 21 cooperative account. The average duration between the meeting and the actual payment to the cooperative is two months on average according to the health facilities visited for the purpose of this case study; this relatively long duration can be explained by the two-step payment system. 3.2. HOW IS THE QUALITY OF THE CHW COOPERATIVES ASSESSED? Two aspects of the quality of the CHW cooperatives are assessed: • The quality of the community HMIS reports • The quality of cooperative management The quality of the CHW cooperatives’ monthly community HMIS reports is assessed quarterly by the sector steering committee. As in the case of the verification of the quantity of services, the assessment of report quality is performed every quarter for each cooperative, and there is no sampling. It must be noted that the whole community HMIS report, and not just the indicators linked to a quantity incentive, is assessed. The assessment is prepared by the health center staff in charge of CHWs (sometimes with the assistance of the president of the cooperative) who checks the following: • All reports are complete (completeness) • Their date of submission complies with the deadlines (timeliness) • They are internally consistent (accuracy), meaning that the accuracy of sums is checked; for example, it is ascertained that the total number of children seen for malnutrition matches the sum of treated, cured, and referred children The community PBF user guide (MoH 2009) offers guidelines for the criteria to be used to assess whether the report is complete, timely, and accurate. For example, it states that if one of the three sector monthly reports is late, the cooperative loses half of the points for timeliness. If two reports are late, the cooperative loses all the points for this indicator. These guidelines were applied with a certain degree of variability among the four facilities visited for this case study. For example, some of these facilities applied a stricter rule, by considering cell or even village reports instead of sector reports. The threshold for losing half of the points (one report late) stayed the same, but given that many more reports were assessed, this made it more difficult for cooperatives to receive high scores for report quality. The time devoted to this activity also varied among the facilities visited: some prepared the assessment of the quality of the reports during the meeting as they elaborated on them; others performed it during a separate meeting attended only by the president of the cooperative and the health center staff in charge of CHWs; and in one case, the assessment was prepared by the staff in charge of CHWs alone, without any assistance. Although the steering committee is responsible for the assessment, most of the work is performed during the preparation for the meeting by the health center staff in charge of CHWs. The health center is therefore charged with assessing a report to which it contributed: as explained earlier (see section 3.1), the sector community HMIS report is elaborated in collaboration between the CHW cooperative and the health center. Again, no conflict of interest related to this situation has been reported to the authors of this study, but this does not mean that this never happens or that it could not happen in other settings. Once preparation is done by the health center staff in charge of CHWs, the steering committee only has to validate the results and to make recommendations if a particular problem is detected in a 22 specific village or cell. The results of the assessment are also captured at health center level by the data manager. The quality of the cooperative management is not assessed by the sector steering committee, but by the district hospital. As in the case of quantity verification and of assessment of report quality, there is no sampling; the cooperative management quality assessment is performed every quarter for each CHW cooperative. There is one staff member in charge of community health activities at the district hospital. 8 That person is a member of the monitoring and evaluation team of the hospital and is responsible for this activity. He or she can be assisted by an NGO monitoring the CHW cooperative (if any); and in some of the hospitals visited, he or she could be assisted by other members of the monitoring and evaluation team. The assessment of the cooperative management quality is done during a visit at the health center that can occur during other monitoring activities performed by the hospital at the health center, but is always done by a staff member who performs no other activities during the visit. The assessment schedule is defined by the hospital, and cooperatives are warned in advance. Unannounced visits are not possible because CHWs are not considered health staff. An appointment is necessary because they are volunteers, have other activities, and are not required to be present at the facility all the time. During the assessment, the evaluator holds a meeting with the board of the cooperative and the health center staff in charge of CHWs, and uses a checklist with nine indicators (annex 4). Together, they make sure that the documents are present and up-to-date, and they score each indicator. At the end of the visit, the evaluator gives the score to the cooperative and makes necessary recommendations. One can note that some indicators considered for the assessment of the quality of management of cooperatives (table 1.4) are not expected to vary over time — for example, contract signed, district authorization obtained, and legal status obtained. The presence of these indicators was justified at the start of the scheme, when the cooperatives were still in the process of building themselves. They should now evolve to match the growing capacities of CHW cooperatives. In the hospitals visited, the staff in charge of this assessment declared that the process could take between one and three hours. The results of the assessment are validated by the district steering committee, and are captured in the web-based database by the hospital data manager. 3.3. HOW IS THE QUANTITY OF DISTRIBUTED IN-KIND INCENTIVES ASSESSED? The verification system linked to in-kind incentives distributed to women seeking MCH services (pregnant women consulting for ANC at the health center, women delivering in the health center, and mother-child pairs receiving care at the health center within ten days of birth) is less developed than the supply-side verification system. This can partly be explained by the fact that demand-side schemes generally require less verification: their main purpose is not to detect potential overestimation by providers, but to confirm that goods meant to be distributed are not embezzled, are distributed correctly, and to the targeted group conditional on the activity agreed. The physical presence of these goods can be checked by the hierarchy at any time, which eases the verification process. 8. As in health centers, this staff member is a highly qualified nurse or equivalent, such as an environmental health specialist. 23 The verification of in-kind incentives is integrated into the monitoring of health centers by the district hospital. The monitoring and evaluation team of the hospital performs monthly routine visits to health centers, during which a wide range of aspects of health centers’ activity is discussed. The main purpose of these visits is not to control or to sanction, but to strengthen the capacities of health centers. The hospital team uses a predefined checklist, in which it selects the themes of the monitoring visit according to problems or weaknesses detected through the HMIS or during former visits. Thus, the verification of the quantity of in-kind incentives distributed to women seeking MCH care is not the main purpose of these visits, and it is possible that this subject is not discussed at all during a visit. When it is, members of the hospital monitoring team check the registers where names, addresses, and personal details of the women who benefited from the incentives are recorded, and verify whether their findings match physical stocks found at the facility. They also make sure that these women are recorded in the other registries (birth, antenatal visit) and that all the women matching the criteria for eligibility for incentives actually received them. They do this by randomly selecting five women in the facilities’ registries and checking whether they are also in the scheme’s registry. The results of this verification (for example, number of women that were in the facilities’ registries but were not tracked back in the scheme’s registry) are to be recorded in a paper-based form (annex 5) that was given to the authors of this study by the MoH at central level, but that was not found on the field in any of the four hospitals visited. The result is that no study of the results of the verification process was possible. 3.4. HOW IS DATA COUNTER-VERIFIED? None of the verification processes that have been described so far involves physical encounters with patients, who are the actual beneficiaries of the schemes. All processes are limited to the registries of the facilities. For this reason counter-verifications — which include meeting patients and asking them to confirm they actually received care as stated in facilities’ registries — are performed at various levels. Two types of counter-verifications are done in the Rwandan community RBF interventions: • Counter-verifications performed on a purposive basis (for example, when a specific problem is detected) • Counter-verification performed on a systematic basis (for example, with random sampling) Purposive counter-verifications can be performed by health centers, sector steering committees, and district hospitals. At health center level, the person in charge of CHWs regularly meets individual CHWs at village level during supervision or household visits. These visits are conducted using a predefined guide (annex 6). The health center staff in charge of CHWs met in the framework of this study said they can request the CHWs show them some of the patients who they declared to have referred, and can have an interview with the patient to check that the declaration was accurate, or to see the in-kind incentives distributed. However, such questions are not defined in the supervision guide, and no record can be made of the results of these checks made by the health center. The sector steering committee can also perform counter-verification visits. Every quarter, it purposively selects one or more villages in which specific problems have been detected, and some of its members (two to four according to the counter-verification reports of the facilities visited in the framework of this case study) visit individual CHWs in their community. In the sectors visited, 24 it was noted that the sector administration sometimes used counter-verification as a tool to perform activities linked to their own objectives (maintaining public order and public health in the sector) rather than for the quality of the data. The members who go to the field check the accuracy of the reports and can ask to see one or more patients registered in these reports (for paid indicators or for other indicators). The team then makes recommendations to improve the quality of data, and a report is written (no standard format exists and no compilation of the data is done). In the reports that could be studied, no mention was made of the duration of the visits. This activity was supposed to be performed by sector steering committees since the beginning of the project, but after a few quarters, it was noticed that these committees performed no or few counter- verifications. This is one of the reasons the performance assessment of the steering committees was changed. Until the second quarter of 2012 their payment was a lump sum based on the number of health centers in the district, but it was changed to a performance-based payment to improve incentives, especially for counter-verifications. The sector steering committee’s performance is assessed by the district steering committee; this assessment triggers a quarterly payment (RF 100,000, to pay for meeting organization, counter-verification, supervision, and stationery and other materials). Counter-verifications are now an indicator that is part of the sector steering committee assessment. But this is still a new process; in each of the four sectors visited for the purpose of this study, only one counter-verification visit had been conducted at the time of the visit. District steering committees can also perform counter-verification visits. In the four districts visited, two district steering committees mentioned this activity. One declared that two visits had been conducted in one year in two sectors selected for specific data reliability problems. Each visit included a random sampling of 20 patients who were interviewed to check whether they actually used the services of the CHWs as declared in the monthly community HMIS reports. The other district that mentioned counter-verification activities performed more systematic visits with the support of an international NGO whose purpose was to strengthen the quality of health data districtwide. Once again, there is no compilation at national level of this counter-verification data. These counter-verification visits performed by health centers, and sector and district steering committees must be considered an educational tool aimed at improving CHWs’ capacities rather than data quality for several reasons: • They are part of a larger monitoring process; counter-verification is only a small part of them. • They can be made on any indicator in the monthly community HMIS report, not just on paid indicators. • Few or no record is made of the results. • Patients or villages might be selected because they are easy to reach rather than at random or based on risk. • Some health centers visited indicated that if time was short, counter-verification could be limited to the checking of CHW documents, which would not involve physical encounters with the patients. In addition to the purposive counter-verifications at local level, the Rwanda community RBF interventions encompassed two scheme-level studies that involved physically meeting patients. The MoH and its partners have performed the following: • One schemewide counter-verification study on supply-side: although this study was called counter-verification, it was very different from the purposive counter-verifications described 25 above. In addition to tracking back patients in the community, it checked the accuracy of the verifications of the quantity of services provided by CHWs made by health centers, and it involved a general evaluation of the scheme. • An assessment of the demand-side scheme that involved, among many other components, tracking back the in-kind incentives given to patients in the community. Both studies included random sampling of sectors (40 for demand-side, 53 for supply-side). The scope of these national counter-verifications was wider than just checking the accuracy of verification, and aimed at assessing the strengths and weaknesses of the entire schemes. However contrary to the counter-verifications performed at local level, they involved a detailed study and reporting of data. 26 4. FINDINGS FROM THE APPLICATION OF THE VERIFICATION METHODS Data on the results of the verification processes could not be collected in the same way for every process. • The results for the verification of the quantity of services provided by CHWs were assessed through the sample of four sectors visited in the framework of this study. • The results of the quality assessment of CHWs’ cooperative reports and management could be assessed through the national database. • The analysis of the results of the verification of the quantity of in-kind incentives provided to patients through the demand-side model was not possible because of the lack of records for this verification. • The results of counter-verification at sector level could be collected through the counter- verification reports of the sectors visited. • The results of the schemewide counter-verifications were assessed through their respective reports (MoH 2012a and MoH 2012b). 4.1. WHAT WERE THE RESULTS OF THE VERIFICATION OF THE QUANTITY OF SERVICES PROVIDED BY CHWS? As mentioned above (see section 3.1), only the data that are validated quarterly by the sector steering committee are entered in the national database. The performance, as it is self-assessed by CHWs at village and cell levels, is recorded in paper reports that are not compiled at district or national level. Hence, it was not feasible for this study to assess the difference between self- assessed and verified performance comprehensively (that is, for the whole scheme) — that would have meant collecting all the village or cell paper reports, capturing them in a database, and comparing them with the national database. Hence, we decided to focus our study of the verification results of the quantity of services provided by CHWs on the four sectors that could be visited. It bears repeating that this sample cannot be considered representative. Although the conclusions that follow were corroborated by national studies (MoH 2012b), they still must be interpreted bearing in mind that they are based on a small sample. We collected the cell reports (that is, reports established before the health center begins the verification process) and captured the performance for paid indicators as self-assessed by CHWs in an Excel worksheet. We compared them with the results recorded in the national database (that is, after the verification process is complete). To detect learning effects, we did so at three points in time: fourth quarter of 2010, fourth quarter of 2011, and fourth quarter of 2012. Unfortunately, two indicators had to be excluded: “post-natal care” and “regular family planning users referred” because they appear to be documented not through the reports of CHWs, but directly through the registers of health centers. The results of the comparison between the performance as self-assessed by CHWs at cell level and performance after the entire verification process is completed (national database results) are presented in table 1.5. 27 Table 1.5 Differences between Self-Assessed and Recorded Performance of CHWs in Four Sectors in the Fourth Quarter of 2010, 2011, and 2012, Respectively (aggregated by indicator) Number Average Accurately Over-estimated Average Average over- Underestimated Indicator of reports difference assessed by by CHWs 11 underestimation 9 estimation 12 (%) by CHWs 13 (%) available (%) CHWs 10 (%) (%) 14 (%) Woman accompanied 35 2 49 23 6 29 13 for delivery Woman accompanied 35 1 54 20 14 26 9 for antenatal care Patients accompanied 35 0 60 14 6 26 35 for VCT Children monitored 34 -2 26 26 24 47 6 for nutrition status New family planning 35 8 60 20 246 20 244 users referred TB — cases followed 35 -13 54 34 143 11 93 per month (DOTS) TB suspects referred 34 3 38 35 76 26 44 Women referred for 33 40 42 18 150 39 150 PMTCT Total 276 -1 48 24 23 28 8 Source: Cell reports and national database, MoH 2013. Note: Differences between performance as self-assessed by CHWs at cell level and performance as recorded in the national database for eight paid indicators in the four sectors visited in the framework of the case study for the fourth quarter of 2010, the fourth quarter of 2011, and the fourth quarter of 2012 (aggregated by indicator). 9. Differences between performance as self-assessed by CHWs at cell level and performances as recorded in the national database. A positive difference means that the CHW cooperative on average underestimated its performance, and a negative difference means that the CHW cooperative overestimated its performance. 10. Percentage of reports where for this indicator the performance as self-assessed by CHWs at cell level was the same as the performance recorded in the national database. 11. Percentage of reports where for this indicator the performance as self-assessed by CHWs at cell level was greater than the performance recorded in the national database. 12. For indicators that were overestimated by CHWs, average difference between performance as self-assessed by CHWs at cell level and performance recorded in the national database. 13. Percentage of reports where for this indicator the performance as self-assessed by CHWs at cell level was smaller than the performance recorded in the national database. 14. For indicators that were underestimated by CHWs, average difference between performance as self-assessed by CHWs at cell level and performance recorded in the national database. 28 During the fourth quarter of 2010, the fourth quarter of 2011, and the fourth quarter of 2012, in the four sectors that were visited and for the eight indicators for which data could be collected in the framework of this case study, the performance self-assessed by CHWs was overestimated by 1 percent compared to the performance recorded in the national database (that is, after the verification process is complete). This figure aggregates underestimations and overestimations, which is why it is preferable to study the results separately. We therefore distinguished between accurately assessed performance, overestimated performance, and underestimated performance. During the period, 48 percent of the indicators were accurately assessed by CHWs at cell level; 24 percent were overestimated (average overestimation was 23 percent); and 28 percent were underestimated (average underestimation was 8 percent). The indicator for which the lowest rate of accurate estimation was found was malnutrition monitoring (26 percent of assessments were accurate) — explained by the high number of children monitored, which multiplies the risk of counting and compilation errors. Other indicators for which the accuracy rate is low are referrals for suspected TB and for Prevention of Mother to Child Transmission (PMTCT). This can be explained by the fact that the definitions for these indicators are slightly more complicated than for other indicators; for example, suspected TB cases must be referred if certain symptoms are observed (the patient must have been coughing for more than two weeks). These relatively more complicated definitions can explain why CHWs make more mistakes for these indicators. To detect possible learning effects, the analysis of the discrepancies between the performance as self- assessed by CHWs at cell level and the performance after the verification process is complete has also been made over time. It was not feasible to perform a continuous analysis of these discrepancies, but we were able to compare the discrepancies at various points in time — the fourth quarter of 2010, the fourth quarter of 2011, and the fourth quarter of 2012. The results are presented in table 1.6. Table 1.6 Differences between Self-Assessed and Recorded Performance of CHWs in Four Sectors in the Fourth Quarter of 2010, 2011, and 2012, Respectively (aggregated by quarter) Accurate Overesti Underest Average ly Average Average mated imated Indicat differenc assessed overesti underesti n 15 by by or e by mation mation CHWs CHWs (%) CHWs (%) (%) (%) (%) (%) 2010 86 -1 49 22 147 29 16 Q4 2011 95 -6 45 26 23 28 4 Q4 2012 95 3 51 23 7 26 12 Q4 Total 276 -1 48 24 23 28 8 Source: Cell reports and national database, MoH 2013. 15. Number of indicators included in the calculation. To simplify, the eight indicators for which payment is made have been averaged. 29 Note: Differences between performance as self-assessed by CHWs at cell level and performance as recorded in the national database for eight paid indicators in the four sectors visited in the framework of the case study for the fourth quarter of 2010, the fourth quarter of 2011, and the fourth quarter of 2012 (aggregated by quarter). Q4 stands for “fourth quarter.” Contrary to expectation, error rates between performance as self-assessed by CHWs and performance after the verification process is complete did not diminish greatly during the observation period in the four sectors visited in the framework of this case study; the accuracy rate in these sectors was 49 percent in the fourth quarter of 2010, 45 percent in the fourth quarter of 2011, and 51 percent in the fourth quarter of 2012. However, the average size of the errors diminished (from 147 percent to 7 percent for overestimations and from 16 percent to 12 percent for underestimations). An explanatory hypothesis for this contradictory evolution might be that the learning effects also concerned the verifier (the health center): the verification process might have gradually become more successful in detecting smaller errors, which could have negatively impacted the accuracy rate, but that might also have made the average size of errors smaller. As the verification process is manual, it would be very difficult — if not impossible — to reduce the error rate to zero. For this reason assessment of the level of error requires looking at both error rate and size of error. As explained earlier (see section 3.1), the verification of the quantity of services provided by CHWs is a twofold process: a first step is conducted by the health center together with the CHW cooperative, and a second step is conducted by the sector steering committee. Table 1.7 presents, for each indicator, the percentage of inaccurate reports detected for each of these steps. 30 Table 1.7 Percentage of Inaccurate Reports in Four Sectors in the Fourth Quarter of 2010, 2011, and 2012, Respectively Errors detected by the Errors detected by the sector steering health center committee (comparison (comparison between between sector reports Indicator cell and sector reports) and national database) Inaccurate Inaccurate Number of Number of indicators indicators reports reports (%) (%) Woman accompanied for delivery 51 35 14 35 Woman accompanied for antenatal 43 35 14 35 care Patients accompanied for VCT 49 35 20 35 Children monitored for nutrition 59 34 29 34 status Family planning users referred 23 35 23 35 TB — cases followed per month 23 35 37 35 TB suspects referred 37 35 41 34 Women referred for PMTCT 52 33 36 33 Total 42 277 27 276 Source: Cell and sector reports and national database, MoH 2013. Note: Percentage of inaccurate reports detected by the health centers and by the sector steering committees in the four sectors visited in the framework of this case study for eight paid indicators, during the fourth quarter of 2010, the fourth quarter of 2011, and the fourth quarter of 2012. In the four sectors visited in the framework of this case study, for the eight indicators and for the three quarters for which data were collected, 42 percent of the indicators were considered inaccurate by the health center in the first step of the verification process; 27 percent of these indicators were subsequently corrected by the sector steering committee. Although most errors are detected by the health center, the second step seems to be indispensable to ensure the quality of data. According to stakeholders who were interviewed, the type of errors corrected by the steering committee are generally mistakes in calculation and compilation; the committee sometimes “redo” what the health center reports by reestablishing the first performance assessment done by CHWs. These results must be interpreted bearing in mind at least two limitations of the above analysis. First, the sample size does not allow generalization. Second, there is no real “declaration” of performance in the community PBF scheme: village or cell reports are established by CHWs knowing that they will be corrected when they are compiled at sector level, and that no sanction against them is defined if data are inaccurate. This might be one of the reasons for the relatively high level of error presented in tables 1.5 to 1.7. 31 According to the people interviewed for the purpose of this case study, the errors are caused by two main factors: • Recording errors: CHWs misinterpret the definitions of the indicators, and declare they referred patients who did not in fact match the criteria for receiving subsidies (for example, a patient who coughs and whom they refer as a TB suspect, but who happens not to have all the symptoms). Another source of recording error can be that the health center does not properly register referred patients. • Compilation errors: calculation mistakes can be made when compiling village reports into the cell report, and then cells reports into the sector report. All people interviewed insisted that fraud is extremely rare and that almost all errors are unintended. This assertion can be justified because the indicators have been selected according to their relative ease of verification by the health center: the facility receives patients that are referred, and it has all the documents (records and referral letters) that can deter fraud. 4.2. WHAT WERE THE RESULTS OF THE ASSESSMENT OF THE QUALITY OF CHW COOPERATIVES? As explained earlier (see section 3.2), the assessment of the quality of CHW cooperatives is twofold. It includes the following: • The assessment of the quality of the CHW cooperative’s monthly community HMIS report • The assessment of the quality of the management of the cooperatives Figure 1.4 shows the evolution of the average score of CHW cooperatives for the first of these two items, which includes report completeness, accuracy, and timeliness. It encompasses the scores for all facilities included in the supply-side scheme. However, unfortunately data for 2012 could not be collected as the database in which the results of the quality assessment are compiled was changed at the beginning of 2012, and the MoH was still working on finalizing the new database at the time of data collection for this study. As a consequence, only 2011 data could be studied. 32 Figure 1.4 Average Score of CHW Cooperatives Included in the Supply-Side Scheme (n=435) for Report Timeliness, Completeness, and Accuracy between the Fourth Quarter of 2010 and the Fourth Quarter of 2011 100% 95% 90% 85% 80% Completeness 75% Accuracy Timeliness 70% 65% 60% 55% 50% 2010 Q4 2011 Q1 2011 Q2 2011 Q3 2011 Q4 Source: Authors 2013. Note: Evolution of the average score (defined as the percentage of the total number of points available) of CHW cooperatives included in the supply-side scheme (n=435) for report timeliness, completeness, and accuracy between the fourth quarter of 2010 and the fourth quarter of 2011. Q4 stands for “fourth quarter.” The very high scores achieved by CHW cooperatives (they get between 85 and 94 percent of the points available for timeliness and completeness, between 68 and 79 percent for accuracy) show that the standards of the evaluation could be strengthened. The criteria for measuring report quality currently leave some room for improvement. They have not changed since the beginning of the program, while the capacities of CHW cooperatives have improved. Quality measurement should be adjusted to the capacities of the actors assessed, and this seems not to be the case for report quality assessment. The same conclusions can be drawn from the analysis of the evolution of the assessment of the quality of CHW cooperative management, presented in figure 1.5. 33 Figure 1.5 Average Score of CHW Cooperatives Included in the Supply-Side Scheme (n=435) for Cooperative Management between the Fourth Quarter of 2010 and the Fourth Quarter of 2011 95% 90% 85% 80% 75% 70% 65% 60% 55% 50% 2010 Q4 2011 Q1 2011 Q2 2011 Q3 2011 Q4 Source: Authors 2013. Note: Evolution of the average score (defined as the percentage of the total number of points available) of CHW cooperatives included in the supply-side scheme (n=435) for cooperative management between the fourth quarter of 2010 and the fourth quarter of 2011. Q4 stands for “fourth quarter.” During the observation period, the 435 cooperatives included in the supply-side scheme managed to get between 81 and 89 percent of the points available in the assessment of their management. As seen in the quality of the reports, and even more so, the assessment indicators should be adapted to the capacities of the cooperatives. The fact that the average score was never less than 80 percent shows that this assessment has room for improvement. Indeed, as noted above (see section 3.2), many indicators included in the cooperative management assessment cannot vary (for example, contract signed, district authorization obtained, legal status obtained). The assessment indicators should be aligned with the improved capacities of CHW cooperatives. 4.3. WHAT WERE THE RESULTS OF THE VERIFICATION OF THE DISTRIBUTED QUANTITY OF IN- KIND INCENTIVES? As explained above (see section 3.3), district hospitals verify the quantity of in-kind incentives distributed to women seeking MCH care in the sectors included in the demand-side scheme by randomly selecting five women in the facilities’ registers and checking whether they are also in the scheme’s registry. It was explained that no record of the results of these specific tasks could be found in the four hospitals visited for this study. Thus, it was not possible to study the results of this verification, and to compare the number of incentives distributed before and after verification. It is understandable that the system pays less attention to the 34 verification of the distribution of in-kind incentives than to the verification of supply-side performance. However, in the current verification system, determining whether the incentives actually reach the intended beneficiaries relies only on counter-verification. Given that only one counter-verification study was conducted on the demand-side scheme (see section 4.4), it can be concluded that this aspect deserves greater attention. As far as this study is concerned, there is also a real missed opportunity in analysis and system improvement, since there is much to be learned from the comparisons a proper recording of verification data would allow. 4.4. WHAT WERE THE RESULTS OF COUNTER-VERIFICATIONS PERFORMED AT VARIOUS LEVELS? As explained earlier (see section 3. 4), two types of counter-verification are performed in the framework of the Rwandan community RBF interventions: • Purposive counter-verifications at health center, and sector and district steering committee levels • Studies performed at national level and including, among other things, patient tracing in the community The results of the purposive counter-verifications are not always recorded, and are never compiled at national level. The four reports of counter-verification conducted by sector steering committees in sectors that were available in the facilities visited for this study show that counter-verification by sector steering committees is used as a tool to support CHWs, to check that they are correctly using the registers, and to improve their capacities rather than as a fraud detection instrument. The main reason for selecting villages was overutilization (for example, Voluntary Counseling and Testing [VCT]) or underutilization (for example, family planning) of services. The reports studied always found an explanation for overutilization (for example, sensitization campaigns) and gave recommendations in cases of underutilization, but did not detect a single case of fraud. The existence of these purposive counter-verifications is a program strength; they utilize some risk-based sampling, which is a positive experience that should be expanded. However, the lack of compilation and recording of results is an aspect of the program that needs to be improved. At national level, two studies were conducted and included, among other things, patient tracing in the community: • One for the supply-side scheme • One for the demand-side scheme The supply-side study selected 60 sectors, that is, 60 percent of the sectors in the supply-side scheme. It was not limited to tracking back patients in the community. It performed a thorough assessment of the scheme. Moreover, for six indicators and for the second quarter of 2012, it compared the cell reports with the compilations established at sector level on the one hand, and with the results registered in the national database (on which payments are based) on the other hand. The results of these comparisons are presented in table 1.8. 35 Table 1.8 Percentage of Sectors with Discrepancies between the Sum of Cell Reports, Sector Reports, and the National Database according to the 2012 Counter-Verification (second quarter 2012) Sectors with discrepancies Sectors with discrepancies Indicator between cell and sector report between sector report and (%) national database (%) TB suspect 60 57 TB care 59 67 Delivery 27 17 Antenatal care 24 18 Family planning 33 24 Nutrition 70 47 monitoring Source: MoH 2012b. The study found discrepancies between cell and sector reports in 24 to 70 percent of the sectors (depending on the indicators), and discrepancies between sector reports and the national database in 17 to 67 percent of the sectors. Indicators for which higher levels of discrepancy were found were nutrition monitoring and TB. This relatively high level of discrepancy is consistent with the results presented in section 4.1, and only bear on the four sectors visited for the purpose of this study. Slightly less than half of the indicators were assessed accurately by CHWs at village level; the indicators for which the accuracy rate was lower were the ones with very high numbers of patients (nutrition) or had more complicated definitions (TB). The study also checked whether patients cooperatives claimed to have referred to the health center existed, and confirmed that they had been treated (counter-verification). A total of 240 patients were selected (but the selection could not be made randomly because of accessibility matters), of which 97 percent could be identified in the community (MoH 2012b). The study on the demand-side community scheme also tracked back patients in the community. The period assessed was April to September 2011. A total of 107 patients reported by health facilities to have received in-kind incentives were randomly selected in the 55 health centers of the study. Among the 107 patients, 97 percent confirmed having been treated at the facility during the relevant period. Of those, 97 percent claimed to have been treated for the relevant service, and 98 percent of women confirmed they received in-kind incentives (MoH 2012a). These studies suggest that fraud is relatively rare, but for a stronger deterrent effect on CHWs, the patient tracing should be more frequent and include a larger number of CHW cooperatives. However, the cost- effectiveness of such an activity is doubtful, given the low number of patients not found back in the studies carried out so far. 4.5. USE OF VERIFICATION FINDINGS The findings of the verification system in the supply-side scheme are used primarily for paying CHW cooperatives; the results of both quantity and quality verification result in a quarterly subsidy paid to the cooperative. During the seven quarters of the project, approximately RF 9.5 billion (US$15.2 million) has been paid to cooperatives. 36 Based on the findings of our study of the differences between the CHW cooperative’s reports and the data taken from the verification system, we can compare (for the eight indicators for which data could be collected) the amounts that were actually paid to CHWs for the quantity of services provided in the four health facilities visited with the amounts that would have been paid in the absence of any verification system (amounts based on CHW self-assessed performance). It would have been desirable to perform these analyses on a wider scale, at national level, and for all indicators; enabling the system to provide such analysis is an area where improvement is still needed in the community PBF scheme. The results of our comparison are presented in table 1.9. Table 1.9 Difference between Amounts That Would Have Been Earned by CHW Cooperatives If Self-Assessed Performance Had Been the Basis of Payment and Amounts Actually Earned in Four CHW Cooperatives in the Fourth Quarter of 2010, 2011, and 2012, Respectively Amount that would have been earned based Amount earned Unit price Difference Indicator on self-assessed based on database performance RF US$ RF US$ RF US$ % Woman accompanied for 600 1.0 907,200 1,452 929,400 1,487 2 delivery Woman accompanied for 480 0.8 914,880 1,464 928,320 1,485 1 antenatal care Patients accompanied for 1,080 0.2 6,757,560 10,812 6,779,160 10,847 0 VCT Children monitored for 40 0.1 4,349,440 6,959 4,251,960 6,803 -2 nutrition status Family planning 630 1.0 1,217,160 1,947 1,327,410 2,124 9 users referred TB — cases followed per 5,048 8.1 837,968 1,341 742,056 1,187 -11 month TB suspects 12,620 2.0 8,076,800 12,923 8,316,580 13,307 3 referred Women referred 840 1.3 559,440 895 929,040 1,486 66 for PMTCT Total 23,620,448 37,793 24,203,926 38,726 2 Source: Authors 2013. Note: Difference between the amounts that would have been earned by CHW cooperatives if self-assessed performance had been the basis of payment and the amounts actually earned (based on the national database) in the four CHW cooperatives visited in the framework of this study, for the eight indicators for which data could be 37 collected and for the three quarters that were studied (fourth quarter 2010, fourth quarter 2011, fourth quarter 2012). In the four sectors visited and for the eight indicators and three quarters for which data could be collected, the total amount that would have been paid in the absence of verification was RF 23.6 million (US$37,793), compared with RF 24.2 million (US$38,726) based on the results of the verification system. This means that the verification system actually improved the income of CHWs by 2 percent; on average, it corrected more underestimations than overestimations. This seems to contradict the results presented earlier (see section 4.1) that showed that self-assessed performance was on average overestimated by 1 percent compared to verified performance. This apparent contradiction can easily be explained by the fact that indicators are not all paid the same rate. Besides the payment of CHW cooperatives, the main use of the findings of the verification system is to enhance the knowledge and capacities of CHWs toward improvement of the HMIS; it is expected that CHWs will report data more frequently and more accurately thanks to the support of the health center during verification meetings. However, the results of the comparisons between data as self-assessed by CHWs and data after the verification is complete (see sections 4.1 and 4.2) do not allow us to determine whether this has actually been the case. But the areas in which the findings of the verification system are not used may in fact be more informative than the areas where they are used. For example, they are not used to sanction CHWs or CHW cooperatives that do not accurately report their performance. Accuracy is defined in the Rwanda community RBF scheme as the accuracy of internal calculations of the report. Accuracy in that case does not mean that verification supports what has been reported for the indicator; that is, it does not mean that verification does not lead to a modification of the reported indicator. Accuracy is an area that is rewarded in the framework of the assessment of the quality of CHW cooperatives, but because of the way it is defined in the Rwanda community RBF scheme, it is not possible to use the assessment of the quality of the CHW cooperative as an incentive to avoid under- or over-reporting of HMIS data. The Rwanda community PBF scheme tends to support CHWs and to help them improve themselves, rather than to control or punish them. The implementation manual does not define any sanction, and problems linked to the accuracy of CHWs’ self-assessment of performance are to be addressed at local level. No facility visited for this case study mentioned any sanction against a CHW for fraud. Some sanctions have, however, been mentioned for inappropriate behavior (a CHW lacking commitment to his/her job or disobeying rules of the cooperative). Another area where results of the verification process could be used, but are not, is the provision of analysis at national level. As explained earlier, there is no recording of the difference between performance as self-assessed by CHWs and performance after verification. Consequently, there can be no compilation of these differences, and no study can be made on a large scale without a very burdensome data collection process undertaken beforehand. These differences are an important indicator of the capacities of the CHWs, and finding a way to record and compile them is one of the challenges that lie ahead for the Rwanda community PBF scheme. As for the demand-side scheme, it has already been mentioned that verification results are insufficiently documented, and that no analysis could be made for this study, except for the results of one counter- verification report. 38 5. VERIFICATION COSTS The study of the cost of verification in the Rwanda community PBF interventions is made difficult by the fact that verification activities are integrated with other activities of the health system at health center, and at district hospital and central levels. To estimate the real cost of the verification, a detailed study of the time allocated by health staff to each of their tasks would be necessary; such an analysis was not possible in the framework of this case study. For this reason we decided to study the costs of verification from a programmatic perspective. We only analyzed the funds allocated by donors and government to the program, which funded four types of activities: • Sector steering committees • Counter-verifications performed at national level • Salaries of the staff in charge of CHW at health center level • Subsidies to the CHW cooperatives and provision cost of in-kind incentives Two of these four activities are directly related to the verification process: the sector steering committees and the counter-verifications. Part of the work performed by the staff in charge of CHWs at health center level is related to verification, and part of it to strengthening the capacities of CHWs. Table 1.10 presents the results of the cost analysis from a programmatic perspective. Table 1.10: Main Cost Items of the Rwanda Community PBF Interventions during Implementation Amount for seven quarters Item % RF US$ Steering committees 259,625,258 415,400 2 Counter-verification 141,812,500 226,900 1 Salaries of the staff in charge of CHWs at health 1,341,995,072 2,147,192 10 center level Subsidies to the CHW cooperatives and cost of in- 11,071,913,826 17,715,062 86 kind incentives Total 12,815,346,656 20,504,555 100 Source: MoH 2013. Of the programmatic costs, 13 percent of the funds are used to support verification or CHW capacity building. But to have an exact idea of the costs of verification, more information is needed, such as the time spent by the in-charges of CHWs at health center level on activities that are not related to verification. They are likely to spend more time supporting CHWs in their daily activities (hygiene and the like) than in verifying their performance. Moreover, to provide an exact assessment of the costs of verification, it would be necessary to know the following: • How much time is dedicated to the verification of CHW quality by district hospital monitoring and evaluation teams • How much time is dedicated to verification by the MCH directorate of the MoH 39 6. LESSONS LEARNED The verification system in the Rwanda community RBF interventions deliberately uses existing actors. The findings of this case study tend to show that this internal verification system has kept costs low and created ownership and integration. Verification costs are relatively low (at about 13 percent) compared to other RBF projects in the Africa region where administrative costs can be around 30 percent (but as stated in section 5, the study of costs could not give a satisfactory image of verification costs in the Rwanda community RBF interventions). Additionally, the system is very integrated into the services of the MoH; in sectors included in the project, verification has become a routine exercise for health centers and for hospitals. Counter-verification is also tending to become a routine exercise at sector and district levels (but this is still not the case at national level). The result of using existing actors is that for both supply- and demand-side schemes, the verification activities and the supervision or monitoring activities are very similar; they are sometimes merged and performed by the same actors. This somewhat contradicts the separation required between the financing function (in which verification activities are included) and the regulation function (in which monitoring and supervision activities are included). Despite this low level of separation of functions, the system appears to have a satisfactory capacity to detect and correct errors (more than half of the indicators that are self-assessed by CHWs are corrected by the verification systems). One positive side effect of the project is that integration has strengthened the HMIS. Integration with HMIS has actually been a key feature of the project design. Paid indicators have been selected from the HMIS forms, and the forms are used as the first verification tool. Their verification occurs in the framework of the verification of the whole community HMIS form. However, the very high level of integration of the verification system with the rest of the health system also has negative effects. In a decentralized system, integration may result in decentralized and variable standards and processes. With everything done and decided at a decentralized level, the degree of variability in the way tools are used from one health center to the other and from one district to the other can be relatively high. It was, for example, noted that visited health centers had different understandings of the criteria used to assess the quality of community HMIS reports. A higher degree of homogeneity might be desirable; although this might be a concern more for researchers for comparison purposes, than for beneficiaries. On a more important note, the high degree of integration focuses attention on verified data, which determines payment and which is entered in the community HMIS. Little attention is paid by MoH actors in charge of the verification process to data before verification. The information on the data before verification is not compiled; it stays at the local (sector) level. The lessons that this information could yield on the capacities of CHW cooperatives are therefore lost. While verified data is used to determine payment, reported data and the difference between reported and verified data should not be overlooked as it is a critical learning tool. Moreover, lack of attention to data before verification makes any change toward a less systematic and less costly verification system (for example, sampling) impossible. If, ultimately, data before verification are not used, it is unclear why CHWs should have to spend so much time filling in community HMIS forms; after all, their job will be reperformed by the health center and by the steering committee. As a result of this lack of attention to the difference between performance before and after verification, CHWs are not motivated to report data accurately. It is true that the pay-for-reporting formula rewards them for accuracy, but this only concerns the “internal consistency” of the report (meaning that sums must be accurately calculated) and not the results of the verification. An indicator should be considered 40 accurate if verification does not lead to a modification in the reported indicator, which is not the definition of accuracy used in the scheme. In addition, criteria from the community PBF user guide (MoH 2009), which should be used to assess whether the report is timely, complete, and accurate, are not applied in a uniform manner. Moreover, this accuracy assessment of the report is actually done by the health center staff in charge of CHWs, although he or she has also participated in the elaboration of the report. One reason for the relatively high proportion of inaccurately reported indicators highlighted by this study might well be this lack of incentive for accuracy, as well as the lack of sanctions for inaccuracy. Although most misreporting can be attributed to unintended mistakes rather than fraud (overreporting is as frequent as underreporting), there is no incentive to reduce these errors. For this reason the MoH should set up a uniform system to collect and compile the data as reported by CHWs and systematically compare it to verified data to identify trends and outliers and create incentives for improvement. This incentive system could be put in effect to financially reward CHWs to provide accurate data, or financially punish them in case of too large or frequent inaccuracies. This could be done by transforming the current quality assessment, which is too strongly focused on internal consistency of the reports (for example, data accuracy) and on items that are not supposed to change over time (for example, district authorization and legal status obtained). The high degree of integration also has consequences on the selection of indicators. The strong link with HMIS, and in particular the fact that PBF indicators have been selected to be easily verifiable, has pushed the focus of the project toward the curative part of the activity of CHWs, neglecting their health promotion role. Most indicators are meant to help health centers increase their activity; little attention is paid to the role that CHWs should play in communication, sensitization, and the like. For example, CHWs are paid for accompanying patients to the health center for VCT or for PMTCT, not for educating the community and informing people on how HIV is transmitted. One of the challenges ahead for the community PBF scheme will be to find ways to financially incentivize CHWs to increase their awareness-raising activities. This will not be easy because many of these activities are hard to verify (they consist of meetings held at village level or visits at household level). Selection of indicators should consider both the ease of implementation and the consequences of the new incentives. Indicators should reflect the key objectives and goals of the project while also being measurable and verifiable. One of the major requests of many people interviewed for this study was that the number of counter- verifications should be increased. So far, counter-verification has been underutilized. The responsibility of checking verification tasks performed by health centers falls largely to the sector steering committee, which has other tasks to perform and whose human resources available to go to the field are mostly health center staff. Only two nationwide studies involving counter-verification activities have been performed since the beginning of the project (one for the supply-side scheme and one for the demand- side scheme). These were aimed not only at checking the accuracy of the data, but also at assessing the whole scheme; their objectives were closer to the ones of midterm reviews than to those of a counter- verification. Moreover, they did study the quality of the reports and of cooperative management, but without comparing their results to those of the district and sector steering committees. A more systematic way of performing counter-verifications should be introduced in the scheme. The first task should be to provide to sector steering committees standardized guidelines and tools that allows them to correctly sample the households visited, and to report their findings in a way that allows compilations at sector, district, and national levels. At a later stage, risk-based sampling could be introduced. 41 More generally, for verification as well as for counter-verification, for supply-side as well as for demand- side schemes, better documentation and use of data are necessary. Much of the difficulty in performing this study was due to lack of data documentation. This is exemplified by the fact that no real analysis of the verification results could be made in the demand-side scheme due to lack of data. Although the results of one counter-verification study tend to show that the incentives actually reach their beneficiaries, more evidence of this would be welcome. In the current state of the verification system, analysis for learning purposes is difficult. It is necessary to measure and document results to know what can be improved and how best to achieve that. In spite of the challenges highlighted in the preceding paragraphs, the verification system in the community RBF interventions has yielded great achievements in a very difficult environment characterized by limited physical accessibility of CHWs and beneficiaries, and by low reporting capacities of cooperatives. Improvements to the scheme are necessary, but they can only be slow and gradual. 42 REFERENCES Basinga, Paulin, Paul J. Gertler, Agnes Binagwaho, Agnes L.B. Soucat, Jennifer R. Sturdy, and Christel Vermeersch. 2010. Paying Primary Health Care Centers for Performance in Rwanda. World Bank Policy Research Working Paper Series, Washington, DC. Rusa, L., Ngirabega, J. d. D., Janssen, W., Van Bastelaere, S., Porignon, D. and Vandenbulcke, W. 2009. “Performance-Based Financing for Better Quality of Services in Rwandan Health Centres: 3-Year Experience.” Tropical Medicine & International Health 14: 830–37. Rwanda. Ministry of Health. 2008. National Community Health Policy. Kigali. ———. 2009. Financement Basé sur la Performance au Niveau Communautaire — Guide de l’Utilisateur, January. Kigali. ———. 2011. The National Community Performance-Based Financing and In-Kind Conditional Cash Transfers — Program Implementation Manual, June. Kigali. ———. July 2012a. Evaluation du Processus du Modèle d’Incitations du Côté de la Demande au Rwanda. Kigali. ———. 2012b. Rapport de la Deuxième Contre-Vérification du Financement Basé sur la Performance au Niveau Communautaire au Rwanda, December. Kigali. United Nations Development Programme (UNDP). 2011. World Development Report 2011 — Sustainability and Equity: A Better Future for All. New York: UNDP. World Bank. 2011. World Development Indicators Database. Washington, DC: World Bank. 43 ANNEXES ANNEX 1: LIST OF PEOPLE MET people met.docx ANNEX 2: ANALYSIS FRAMEWORK FOR THE CASE STUDIES Analysis framework ANNEX 3: CHW MONTHLY REPORT CHWs monthly report ANNEX 4: CHW COOPERATIVE QUALITY ASSESSMENT REPORT ANNEX 5: DEMAND-SIDE SCHEME HEALTH CENTER REPORT In French demand side scheme supervision ANNEX 6: CHW SUPERVISION REPORT (ESTABLISHED BY HEALTH CENTER) CHW supervision report 44 About this series... This series is produced by the Health, Nutrition, and Population Family (HNP) of the World Bank’s Human Development Network. The papers in this series aim to provide a vehicle for publishing preliminary and unpolished results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For free copies of papers in this series please contact the individual authors whose name appears on the paper. Enquiries about the series and submissions should be made directly to the Editor Martin Lutalo (mlutalo@worldbank.org) or HNP Advisory Ser- vice (healthpop@worldbank.org, tel 202 473-2256, fax 202 522-3234). For more information, see also www.worldbank.org/hnppublications. The world bank 1818 H Street, NW Washington, DC USA 20433 Telephone: 202 473 1000 Facsimile: 202 477 6391 Internet: www.worldbank.org E-mail: feedback@worldbank.org