Knowledge Brief Health, Nutrition and Population Global Practice VERIFICATION OF PERFORMANCE IN RESULTS-BASED FINANCING: THE CASE OF AFGHANISTAN August 2015 KEY MESSAGES:  Verification of the quantity of services delivered at facility level improved significantly, with the error rate decreasing from 17 percent in 2010 (Q3) to five percent in 2012 (Q3).  The percentage of “missing patients” plummeted from 33 to 7 percent between 2010 and 2013; more than 98 percent of those found (both at health center and hospital levels) confirmed receiving the services recorded.  Qualitative evidence suggests that the Results-Based Financing verification has system helped to improve the credibility and reliability of HMIS data.  Evidence also indicates a positive effect on coordination, communication and capacity. Introduction care (PNC), delivery care, nutrition, immunization coverage, tuberculosis (TB), as well as quality of care. This Since the end of the Taliban period, Afghanistan made RBF program contracts international NGOs, incentivizing remarkable progress, particularly in the health sector. them for both the quantity and the quality of select services. Between 1990 and 2013, infant mortality declined from 121.3 to 70.2 deaths per 1’000 live births; under 5 mortality Context diminished from 179.1 to 97.3 deaths per 1’000 live births; and maternal mortality dropped from 1’300 to 460 maternal This HNP Knowledge Brief focuses on the key findings of deaths per 100’000 live births. the World Bank case study entitled “Verification of Performance in Results-Based Financing: the Case of To further improve the coverage and utilization of Maternal Afghanistan” (2015). and Child Health (MCH) services, the Government of Afghanistan – under the stewardship of the Ministry of Data for this case study was collected through a review of Public Health (MOPH) – launched a new supply-side RBF-related documents and through 13 in-depth semi- Results-Based Financing (RBF) scheme, covering 11 structured interviews. An initial group of three interview provinces and providing a standardized basic package of respondents was purposively sampled based on health services (BPHS). This package focuses on priority recommendations from the World Bank’s Task Team. MCH services such as antenatal care (ANC), post-natal Obtained data was triangulated across and between Page 1 HNPGP Knowledge Brief  respondents and with information obtained from the desk The allowable margin of error for health facilities is 20 review to ensure validity. Snowball sampling was then used percent for quantity indicators identified through patient to identify seven respondents and interview them on the tracing and 10 percent for facility-verified quantity quantity of services delivered. In addition, six respondents indicators. Discrepancies between the HMIS and verified were interviewed on the verification of quality. They were data for a specific indicator greater than these thresholds identified using purposive sampling based on the expertise results in non-payment to the health facility for that of the World Bank’s Task Team as well as that of the local indicator. consultant. Study Findings RBF VERIFICATION MECHANISMS This RBF intervention uses four methods to verify The verification of the quantity of services at facility performance. These include: level shows an increasing level of agreement between 1. The verification of the quantity of services delivered health facility registries and data contained in the HMIS. at facility level, conducted quarterly by an independent third party firm and validated by the The rate of discrepancy fell from 17 percent during the third MOPH. This verification is applied quarterly to 25 quarter of 2010 to 5 percent during the third quarter of 2012 percent of all facilities, as well as to a 10 percent (Figure 1). sample of facilities verified in previous quarters to Figure 1. Trends in Level of Agreement between HMIS and Facility- determine the level of agreement between data from Level Verification Data for the Quantity of Services Delivered the Health Management Information System (HMIS) and data recorded at health facility level in registries. 2. The verification of the quantity of services received by the community, performed quarterly by the same third party firm to ensure patients actually received services as reported by facilities. Patient tracing is carried out by community monitors (composed of two individuals: one male, one female) on 5 households for each of the 9 indicators for each health facility. This community level verification consists of community Source: Health Results Innovation Trust Fund Annual Report: Afghanistan. monitors visiting households to ascertain services March 18, 2013. provided, ideally based on reviewing treatment cards held by the client, as well as asking clients based on The verification of the quantity of services at the their memory. community level also shows significant progress with 33 3. The verification of the quality of services delivered percent of errors (i.e. divergences between services at facility level and carried out every quarter by the reported in the HMIS and patients traced in the community) Provincial Health Office (PHO) and the third party, during the third quarter of 2010 decreasing into a 5 percent using the National Monitoring Checklist (NMC). error rate during the third quarter of 2012 (Figure 2). Difficult-to reach facilities may have been visited less than every quarter, but most facilities were visited at Figure 2. Trends in Level of Agreement between HMIS and least twice a year. There was confusion regarding the Community-Level Verification Data for the Quantity of Services use of the NMC with some respondents using the Delivered whole NMC, and others a subset of it. 4. The counter verification of the quality of services, carried out first by central level officials periodically visiting health facilities in the provinces and using the opportunity to complete the NMC, which is then cross- checked with the facilities’ previous NMC scores. In addition, the MOPH cross-checked on an annual basis similar fields between the NMC and the balanced scorecard (used by national-level managers to track health sector performance) to identify any significant Source: Health Results Innovation Trust Fund Annual Report: Afghanistan. March 18, 2013. differences. Page 2 HNPGP Knowledge Brief  Interviews also underscored the critical role played by the in Afghanistan as the RBF program continues to be verification process in building institutional capacity. entirely and solely financed by the World Bank. The Respondents stated that the verification of the quality of verification process is perceived to be too costly and services, including the implementation of the NMC, too complex to be taken over by the Government in the successfully enabled PHOs and NGOs to closely supervise foreseeable future. This sustainability concern is health facilities, helping them improve their performance compounded by the cost and intensity of the through performance analyses, problem resolution community level verification of the quantity of services, exercises and coaching. Respondents also acknowledged which requires visiting 25 percent of households as that the verification system as a whole contributed in well as selecting and training new community monitors strengthening the Health Management Information System each quarter. (HMIS) in terms of the credibility and reliability of its data.  Respondents also indicated that more provisions should be made to protect patient privacy and Moreover, the elaborate and labor intensive nature of the uphold confidentiality. This was particularly verification process implemented in Afghanistan underlined with regard to community monitors, who are strengthened coordination and communication among selected within the community of interviewed patients. implementing partners, thereby improving the overall efficiency of the RBF program as well as its verification  It was difficult in some communities to recruit system. female community monitors. This problem was likely related to insecurity, continued poor education of The MOPH used the verification data to do further analysis, women, cultural barriers among conservative and every two months a meeting was held to share the communities, or other discomfort with the role of results of the verification with all implementers and other community monitors. This shortage in female stakeholders. There were presentations and discussions community monitors is particularly common in rural about how to improve performance and data quality in areas where, as a result, the number of sampled provinces where there was a higher degree of discrepancy households had to be sometimes reduced. Lessons and Challenges  Concerns were also raised by several interview respondents about the reliability and validity of the A number of constructive lessons were learned, and patient information used to verify the receipt of challenges identified. These include: services (NB: patient satisfaction is not measured). In  The involvement of the respected third-party many cases, patient treatment cards were either not evaluator was perceived to have lent credibility to available at the household level or were incomplete. the RBF program in Afghanistan. The process Furthermore, although the recall period was intended consistently reached nearly all health facilities included to be two weeks, it was often longer, thereby in the RBF program and actively involved community generating recall bias, which in turn may have resulted members. It was also noted by one respondent that the in an under-estimation of the number of services participation of the international third-party evaluator received and a higher discrepancy rate between the may have contributed to building institutional capacity HMIS and health facility registries. for verification within Afghanistan.  The study respondents indicated that the time demand  A noted success of the verification process was of implementing the NMC was a key challenge. They that it contributed to strengthening the HMIS, both explained that there was an opportunity cost to NMC specifically for the indicators that were linked to implementation, namely that it usually limited the incentive payments and for the system as a whole. opportunity for supportive supervision. In addition, Verification reports were being used not only for some study respondents noted that staff capacity for payment but also for actively improving the routine NMC implementation might be a challenge. reporting system and overall quality of the data. There was consensus among the interview respondents that the credibility and trust in the data generated by the Recommendations HMIS has improved. The verification mechanisms used in the RBF program in  Movement restrictions have hampered the Afghanistan show explicit gains in health facility capacity of field teams to carry out quantity and performance and in the delivery of the BPHS. Additional quality verifications. Respondents indicated that too improvements are nevertheless required to further attain many people were put at risk to carry out the health results and further improve the performance of the verification process. health system. The study thus recommends the following:  Long term sustainability is another major concern Page 3 HNPGP Knowledge Brief  References 1. Streamline the verification process to improve Arur, T., Peters, D., Hansen, P., Mashkoor, M., Steinhardt, L. and cost-effectiveness and sustainability. Targeted, Burnham, G. 2010.Contracting for health and curative care use in risk-based verifications combined with a paper-based Afghanistan between 2004 and 2005. Health Policy and Planning review of automated data, and random verification have been seen as more cost-effective approaches 25:135-144. now that data quality has improved and discrepancy Belay, Tekabe 2010. Building on early gains in Afghanistan’s health, rates are low. Afghanistan may consider moving in this nutrition, and population sector: challenges and options. Washington, direction to streamline and ensure the sustainability of its verification process without compromising the D.C.: The World Bank. validity. Cashin, C., Chi, Y., Borowitz, M., Smith, P. and Thompson, S. 2. Change the role of communities in the RBF (eds.).Forthcoming. Pay for Performance in Health Care: Implications program from direct involvement in verification to for Health System Performance and Accountability. Open University monitoring and supervision. The involvement of Press. communities has been seen as a strength of Afghanistan’s RBF verification process, but there are Cashin, C. and Vergeer, P. 2013. Verification of performance in results- also numerous concerns. One recommendation was to based financing: the case of the United Kingdom Quality and Outcomes retain community participation in monitoring health Framework (QOF). Washington, D.C.: The World Bank. service delivery given the important role communities play in accountability, but to also scale back its role in Edward, A. et al. 2011. Configuring Balanced Scorecards for Measuring the verification of the quantity of services delivered. Health System Performance: Evidence from 5 Years’ Evaluation in Furthermore, it was suggested that community-level Afghanistan. PLoS Medicine. Volume 8, Issue 7. Online. Accessed July monitoring should rely on existing community 9, 2013. structures, such as community health assemblies. 3. Strengthen the strategic purchasing role of the Peters, David et al. 2007. A balanced scorecard for health services in MOPH and the PHOs, including monitoring and Afghanistan. WHO Bulletin 85: 146 to 151. evaluation, and RBF verification functions: In any case, continued efforts should be made to strengthen Rahmiza, M., Amiri, M., Burhani, N. Leatherman, S., Hiltebeitel, S., and the capacity of the MOPH and PHOs to become Rahmanzai, A. 2013. Afghanistan’s national strategy for improving strategic health purchasers and fully exploit the quality in health care. International Journal for Quality in Health Care verification process to continue to improve the 25(3): 270-276. coverage and quality of priority health services in Afghanistan. Sondorp, E., Palmer, N., Strong, L. and Wali, A. 2009. Afghanistan: paying NGOs for performance in a post-conflict setting. In Eichler, R. and Levine, R.Performance Incentives for Global Health: Potential and Pitfalls. Washington, D.C.: Brookings Institution Press. This HNP Knowledge Brief highlights the key findings from a study by the World Bank on the “Verification of Performance in Results-Based Financing: the Case of Afghanistan” (2015). The Health, Nutrition and Population Knowledge Briefs of the World Bank are a quick reference on the essentials of specific HNP-related topics summarizing new findings and information. These may highlight an issue and key interventions proven to be effective in improving health, or disseminate new findings and lessons learned from the regions. For more information on this topic, go to: www.worldbank.org/health. Page 4