38685 H N P D I S C U S S I O N P A P E R How are Health Services, Financing and Status Evaluated? An Analysis of Implementation Completion Reports of World Bank Assistance in Health Savitha Subramanian, David Peters, and Jeffrey Willis November 2006 How are Health Services, Financing and Status Evaluated? An Analysis of Implementation Completion Reports of World Bank Assistance in Health Savitha Subramanian, David Peters, and Jeffrey Willis November 2006 Health, Nutrition and Population (HNP) Discussion Paper This series is produced by the Health, Nutrition, and Population Family (HNP) of the World Bank's Human Development Network. The papers in this series aim to provide a vehicle for publishing preliminary and unpolished results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For free copies of papers in this series please contact the individual author(s) whose name appears on the paper. Enquiries about the series and submissions should be made directly to the Managing Editor, Janet Nassim (jnassim@worldbank.org). Submissions should have been previously reviewed and cleared by the sponsoring department, which will bear the cost of publication. No additional reviews will be undertaken after submission. The sponsoring department and author(s) bear full responsibility for the quality of the technical contents and presentation of material in the series. Since the material will be published as presented, authors should submit an electronic copy in a predefined format (available at www.worldbank.org/hnppublications on the Guide for Authors page). Drafts that do not meet minimum presentational standards may be returned to authors for more work before being accepted. For information regarding this and other World Bank publications, please contact the HNP Advisory Services at healthpop@worldbank.org (email), 202-473-2256 (telephone), or 202-522-3234 (fax). © 2006 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved. ii Health, Nutrition and Population (HNP) Discussion Paper How are Health Services, Financing and Status Evaluated? An Analysis of Implementation Completion Reports of World Bank Assistance in Health Savitha Subramaniana David Petersb Jeffrey Willisc aJunior Professional Associate, Health, Nutrition and Population, Human Development Network, The World Bank, Washington, DC, USA bSenior Public Health Specialist, Health, Nutrition and Population, Human Development Network, The World Bank, Washington, DC, USA cConsultant, Health, Nutrition an Population, Human Development Network, The World Bank, Washington, DC, USA Paper prepared for the Health, Nutrition and Population Unit, Human Development Network, The World Bank, September 2006 Abstract: This paper reports on an analysis of how World Bank assistance at the country level has influenced health services, health financing, and status on peoples' health. The Implementation Completion Reports used to evaluate all 118 projects involving health services completed between fiscal years 2003-2005 were systematically analyzed to determine how they measured changes in health services, health financing, and health status outcomes. The results showed that few Bank-assisted projects in the health sector evaluated changes in health services (42%), health financing (17%), or health status (33%), with nearly all those measuring change demonstrating improvements. In multivariate models including the type of project organization (e.g. disease program, sector wide approach), project inputs, key project activities, and contextual factors including per capita income level, geographic region, and Country Policy and Institutional Assessment ratings, there was a statistically significant association between use of a sector-wide approach (SWAp) and measurement of improvements in health services and improvements in health status. Projects that used contracting mechanisms were also more likely to show an improvement in health services. No other type of organization of project support, project input or project activity was statistically associated with measurement of improvements in health services in the multivariate analysis. The results from this analysis show that the three strategic priorities outlined in the 1997 HNP Strategy-- (i) to improve health, nutrition, and population outcomes of the poor; (ii) to enhance the performance of health care systems; and to (iii) secure sustainable health care financing, were not well measured in the evaluation frameworks of Bank assistance in health. With the development of the new HNP Strategy, the World Bank should iii encourage policy makers to demand more rigorous monitoring and evaluation of health sector investments with Bank funds. Keywords: Health services delivery; Health status; Implementation Completion Reports Disclaimer: The findings, interpretations and conclusions expressed in the paper are entirely those of the authors, and do not represent the views of the World Bank, its Executive Directors, or the countries they represent. Correspondence Details: David Peters, Mail Stop G7-701, 1818 H Street, N.W. Washington, DC 20433; Tel: (202) 458-8113; Email: dpeters1@worldbank.org iv Table of Contents ACKNOWLEDGEMENTS ...........................................................................................vii EXECUTIVE SUMMARY ............................................................................................. ix INTRODUCTION............................................................................................................. 1 BACKGROUND AND METHODOLOGY ................................................................................ 1 RESULTS .......................................................................................................................... 8 A. DESCRIPTIVE STATISTICS............................................................................................ 8 B. FACTORS ASSOCIATED WITH IMPROVEMENTS IN HEALTH SERVICES, HEALTH FINANCING, AND HEALTH STATUS............................................................................... 12 C. ORGANIZATIONAL APPROACHES ............................................................................... 14 D. PROJECT INPUTS & ACTIVITIES................................................................................. 15 E. COUNTRY CONTEXT.................................................................................................. 15 F. MULTIVARIATE ANALYSIS ........................................................................................ 15 DISCUSSION AND CONCLUSIONS.......................................................................... 17 ANNEX1 .......................................................................................................................... 20 ANNEX 2 ......................................................................................................................... 23 ANNEX 3 ......................................................................................................................... 24 List of Figures Figure 1: Most Common Health Services Indicators Used in Project ICRs....................... 8 Figure 2: Most Common Health Financing Indicators Used in Project ICRs .................... 9 Figure 3: Most Common Health Status Indicators Used in Project ICRs ........................ 10 Figure 4: Percent of HNP Project ICRs Showing Improvements in Health Services, Health Financing, or Health Status According to Primary Organizational Approach ................................................................................................................................... 14 List of Tables Table 1: Definition of Study Outcome Variables ............................................................... 3 Table 2: Definitions of Approaches to Organizing Development Assistance in the Health Sector........................................................................................................................... 5 Table 3: Types of Project Inputs and Activities Identified................................................. 6 Table 4: Project Characteristics According to World Bank Region (Percent) ................. 11 Table 5: Crude Association between Project Characteristics and Improvement in Health Services, Health Financing, and Health Status.......................................................... 13 Table 6: Logistic Regression Results for Improvement in Health Services and Health Status ......................................................................................................................... 16 v vi ACKNOWLEDGEMENTS This paper was commissioned by the Health, Nutrition and Population Department of the World Bank. Thanks are due to colleagues who have provided suggestions and comments, including Martha Ainsworth, Cristian Baeza, Peter Berman, Jan Bultman, and Benjamin Loevinsohn The authors are grateful to the World Bank for publishing this report as an HNP Discussion Paper. The work was supported in part by a generous contribution from the Government of the Netherlands through the Bank-Netherlands Partnership Program grant to the World Bank. vii viii EXECUTIVE SUMMARY The question of how to improve the delivery of health services in developing countries is of central importance to the achievement of the Millennium Development Goals, and to the World Bank's assistance to the health sector. In 1997, the Bank wrote a Health, Nutrition, and Population (HNP) Strategy that identified three priorities to guide the 1 direction of the Bank's work in the HNP Sector, specifically to work with client countries to: (i) improve health, nutrition, and population outcomes of the poor; (ii) enhance the performance of health care systems; and (iii) secure sustainable health care financing. Although the Bank's HNP strategy did not provide metrics for how it would pursue its strategy, it is still pertinent to ask how well Bank assistance has contributed to these objectives. This paper reports on an analysis of how World Bank assistance at the country level has influenced health services, health financing, and status on peoples' health. The Implementation Completion Reports used to evaluate all 118 projects involving health services completed between fiscal years 2003-2005 were systematically analyzed to determine how they measured changes in health services, health financing, and health status outcomes. In this study, measurements of health services involve any change in quality, efficiency or use of health services; health financing indicators include any measure of the financing of health services; and health status includes any measurement of the health, nutrition, or fertility status of people. Key characteristics of the country context and project design were also assessed to identify their relationships to improvements in health services, financing, and health status. This study is primarily interested in how projects measured change in the three priority areas of the Bank's HNP strategy, so unlike other evaluations, it did not evaluate each project based on their individual development objectives, nor of the relevance of the indicators used to the project's development objectives. The results showed that few Bank-assisted projects in the health sector evaluated changes in health services (42%), health financing (17%), or health status (33%), with nearly all those measuring change demonstrating improvements. Projects measuring any improvements in health services delivery were five times more likely to measure improvements in health status than those that did not demonstrate improvements. In multivariate models including the type of project organization (e.g. disease program, sector wide approach), project inputs, key project activities, and contextual factors including per capita income level, geographic region, and Country Policy and Institutional Assessment ratings, there was a statistically significant association between use of a sector-wide approach (SWAp) and measurement of improvements in health services (Adjusted Odds Ratio 14.3; 95% Confidence Interval 1.6-130.0) and improvements in health status (Adjusted OR 8.9, 95% CI 1.2-65.4). Projects that used contracting mechanisms were also more likely to measure an improvement in health 1The World Bank. (1997) Health Nutrition, & Population Sector Strategy. Washington:, The World Bank. ix services (Adjusted OR 4.1, 95% CI 1.1-15.9). No other type of organization of project support, project input or project activity was statistically associated with improvements in health services in the multivariate analysis. Projects that had activities involving logistics systems were also statistically associated with measurement of improvement in health status, though no particular type of project input was significantly associated with improvements in health status. None of the contextual factors examined were statistically associated with improvements in health services or status in these projects. The results from this analysis show that the three strategic priorities outlined in the 1997 HNP Strategy were not well measured in the evaluation frameworks of Bank assistance in health. With the development of the new HNP Strategy, the World Bank should encourage policy makers to demand more rigorous monitoring and evaluation of health sector investments with Bank funds. Bank management should also find incentives for better measurement of health services, financing, and status in their operational work. One option is to establish a dedicated unit to enhance monitoring and evaluation, similar to what is being done with the Global AIDS Monitoring and Evaluation Team (GAMET). Without better use of information on health services, financing, and status, it will remain difficult to determine which strategies are most effective in improving health services and health status across the countries where the World Bank is providing assistance. x INTRODUCTION BACKGROUND AND METHODOLOGY It is widely recognized in the international community that weak health systems are a critical barrier to the achievement of the health related Millennium Development Goals (MDGs). Yet relatively few systematic attempts have been made to evaluate health systems performance across countries. The most recent Operations Evaluation Department evaluation (now called the Independent Evaluation Group) of World Bank assistance in 1999 recognized the positive contributions of the World Bank in health, nutrition, and population policies and services, but noted that "the Bank typically focuses on providing inputs rather than on clearly defining and monitoring progress toward HNP development objectives".2 The World Bank's 1997 HNP Sector Strategy focused on assisting client countries to improve HNP outcomes, enhance performance of health care systems, and secure sustainable health care financing, but did not provide metrics for evaluation or benchmarking progress towards these objectives. This study provides one way to look at the question of how well World Bank assistance at the country level has measured performance in the three priority areas of health services, health financing, and impact on peoples' health status. It is part of a larger set of sector work intended to better understand how to improve health services in developing countries. Much of the debate concerning international assistance in health over the last few decades has been characterized by tension over whether to concentrate on priority programs for specific diseases or on strengthening health systems.3 Although sustained success using either organizational approach is clearly dependent on the other, practical and ideological considerations have led program designers and politicians to favor one or the other. Yet there is remarkably little evidence to suggest one type of approach works better than another.4 Aside from limited numbers of case studies, there has not been a systematic analysis of the organizational approaches taken, or of the influence of the types of project investments. This study hopes to contribute to the debate by examining how health services, health financing, and health status have been measured and evaluated in World Bank projects, and how these are related to the organizational approach and types of investments made. This evaluation is limited in scope, with the evidence coming from an inventory and systematic review of results that are measured and reported in the official project evaluation reports of completed Bank-assisted projects, the Implementation Completion 2Johnston T, Stout S. (1999) Investing in Health: Development Effectiveness in the Health, Nutrition, and Population Sector. Washington: The World Bank. 3Mills A. (2005) Mass Campaigns versus General Health Services: What Have We Learnt in 40 Years about Vertical versus Horizontal Approaches. Bulletin of the World Health Organization 83(4) 325-30. 4Ovretveit J. (2006) Strengthening Health Services in Low Income Countries: A review of research on implementation and results. Karolinska Institutet Medical Management Centre, Draft paper, June 1, 2006. 1 Reports (ICRs), and should be considered in the context of other evaluations of Bank assistance. The specific aims of this study are to: 1. Identify how World Bank HNP projects measured changes in health services, health financing, and impact on peoples' health 2. Identify project characteristics associated with improvements in health services, health financing, and health status. Methods: Sample The study reviewed all World Bank projects, containing a health component and completed during the period of three fiscal years: FY03-FY05. Project information was extracted from the ICRs, which are prepared by the units of the Bank responsible for designing and supervising the projects, with input from the Borrower, to document their achievements. The Bank's "Business Warehouse" was used to obtain a list of all projects with health themes both managed by HNP and non-HNP Sector Boards from FY03- FY05. There are eight relevant themes under Human Development which were chosen as selection values: Child Health, Health System Performance, Nutrition and Food Security, Population and Reproductive Health, HIV/AIDS, Other Communicable Diseases, Injuries and Non-communicable diseases, and Other Human Development. Of the 119 projects initially identified, four projects were excluded because they had health codes as a secondary theme but did not have identifiable health, nutrition, population (HNP) inputs or activities. There was also one project in India that was comprised of three separate state projects (Karnataka, Punjab, and West Bengal), and another project in India comprised of two state projects (Bihar and Orissa). Given that these were large projects (ranging $80 to $130 million per state) that were treated as separate state projects with their own project logical frameworks, we also assessed them as separate projects, so that the final sample included 118 projects. A complete list of the projects reviewed is shown in Appendix 1. Data Collection In gathering data for this study, each project ICR was reviewed in full using a checklist to assess what indicators were used for measuring health services, health financing, and health outcomes, as well as the primary organizational approach and types of inputs and activities involved. Three data collectors were trained on the checklist and definitions of health services, financing, and health outcomes indicators. In order to achieve consistency in the data entry process between data collectors, data were double-entered. When data was inconsistent between two entries, the cause was investigated and the problem was resolved through discussion between the two data collectors. Two other reviewers were used to assess the data entered, and reconcile any remaining discrepancies between the different reviewers. 2 Definition of Study Outcome Variables The three main outcome variables used in this study were indicators of change in health services, health financing, and health status indicators (Table 1). Health services outcomes included only those indicators that measured aspects of the delivery or use of health services. Many projects report physical outputs (e.g. supply of drugs, construction of materials), capacity building outputs (e.g. numbers of people trained), or institutional outputs (e.g. development of new organizations, new rules) that are useful indicators of project accomplishments, but are not considered in this analysis. Health financing indicators were similarly restricted to changes in level or type of financing of health. Health status measures are those assessing changes in health, nutrition, or fertility status of people. Table 1: Definition of Study Outcome Variables Study Outcome Description Variable Health Services Any measurement of an indicator that measures the utilization, coverage, quality, efficiency, or distribution of health services, including changes in health behaviors of individuals. These types of indicators are usually defined as project outcomes, but may also be defined as health service outputs in the evaluation frameworks. Health Financing Any measurement of an indicator that describes the financing dimensions of health, whether for individuals, government, or other organizations. Health financing indicators can be impact level measures (e.g. income protection for individuals), or output and outcome levels (e.g. enrollment levels in health financing schemes, contributions by government or individuals, cost of services). Health Status Any measurement of an indicator that describes the long-term effect on the health, nutrition, and population status of the population. Mortality rates, disease incidence and prevalence rates, case fatality rates, cure rates, and rates of malnutrition and fertility are all examples of health status measures. These indicators can be considered as project outcomes, though they tended to be identified as project impact or project goal level indicators. Note: The study examined any measurement of health services, health financing, and health status in the projects, and focused on those projects demonstrating any change in these indicators taken from at least two measurements at different points in time. To determine whether there was a change in any of the study variables, the data collectors first assessed whether the project reported any indicators measuring health services, health financing, or health status. Each indicator where a measurement was made was recorded. Each indicator was then assessed for the number of measurements made, and whether any positive change occurred during the life of the project. A minimum of two measurements was required to demonstrate a change. If three or more measurements were taken, then a linear slope was estimated. Indicators for health services, health financing, and health status were assessed individually, and then as a group for each type of outcome. In cases where there were multiple indicators for one of 3 the groups, the indicators were combined to assess the proportion that demonstrated any positive change. When this proportion was greater than or equal to 50%, the project was recorded as demonstrating a positive change. In doing this we made the general assumption of weighting each of the indicators equally and did not look specifically at the individual effect of each indicator. Estimates for the indicators were taken as recorded in the ICR, and no attempt was made to assess whether the reported change in each indicator was statistically significant.5 If there were no health services, health financing or health status indicators with actual measurement levels reported, the project was recorded as not demonstrating a measured changed in that outcome. Definitions of Explanatory Variables In order to identify factors associated with any positive changes in health services, financing, or status, the study examined the organizational approaches taken. Table 2 describes the approaches to organizing development assistance that were considered, which consisted of budgetary support and a set of sector specific approaches including disease control priorities and sector wide approaches (SWAps). Projects were classified under the dominant/primary organizational approach that was used in each project and this judgment was made based on how the majority of health funding was used as a guide. Where this was not possible, the assessment was made based on the majority of activities. 5Estimating whether the reported changes were statistically significant would not have been possible from the ICRs, as they rarely reported the source, sample size, or standard errors of the estimates. 4 Table 2: Definitions of Approaches to Organizing Development Assistance in the Health Sector Approach Descriptions A. General Budget Support Development assistance provided as general support to the government which are not targeted to the health sector, sometimes considered an extra-sectoral approach. These projects rely on government systems to manage the funds based on policy agreements to achieve broad objectives that include the health sector. These projects provide general budget support without earmarking of funds to a sector (e.g. Poverty Reduction Support Credits, Structural Adjustment Loans). B. Health Sector Approaches Disease Program Strengthening a priority program organized around control of a disease (e.g. HIV/AIDS, tuberculosis, malaria) or specific set of health services and conditions (e.g. immunization, family planning) Geographic Priority Strengthening management and delivery systems in a particular geographic area of a country, often chosen because of characteristics of a particular population. Management Support System Focus on a particular management support system (e.g. systems to provide key inputs such as essential drugs, buildings, or equipment). Sector Wide Approach (SWAp) Investments and actions based on common sectoral policy and objectives, expenditure framework, and systems for implementation and review. In some cases, funding is provided as budget support but with specified allocation to the health sector, and in some cases, the projects were labeled Adaptable Program loans. Others Projects organized around other approaches, such as health providers, financing mechanism, community engagement approaches, or accountability approaches Table 3 outlines another set of project inputs and activities that were identified in each of the ICRs. Types of project inputs were largely identified from ICR tables on project expenditures and supplemented by the text of the ICRs, whereas the project activities were largely derived from text description of the ICR. 5 Table 3: Types of Project Inputs and Activities Identified Variable Description Project Inputs Salaries Whether the project provided funding for salaries of health workers Training Inputs Whether the project provided funding for training of health workers Technical Assistance Inputs Whether the project provided funding for technical assistance Pharmaceuticals Inputs Whether the project provided funding for pharmaceuticals, including vaccines and contraceptives Equipment Inputs Whether the project provided funding for the purchase, repair or maintenance of equipment Buildings Inputs Whether the project provided funding for the building or repair and maintenance of buildings Software Inputs Whether the project provided funding for software, such as computer information systems Project Activities Human Resources Systems Project activities that involved changing human resource systems, including increasing the number of health workers, improving health worker capacity, or systems to manage human resources (e.g. performance review, development of career structures, staff recruitment procedures, etc.) Health Services Management Project activities concerned with management of health Systems services, including quality of care approaches, health management information systems, etc. Logistics Systems Project activities involved in supply chain management (i.e., selection, procurement, distribution, rational use of drugs, commodities, etc.). Regulation of Health Providers Whether policies or activities to regulate health providers was reported Contracting of Health Providers Whether explicit contracts were made with health organizations to deliver services Public Planning Involvement Whether civil society organizations or members of the public were involved in planning of the project Three types of variables were used to describe the country context. Geographic region was assigned according to the World Bank regional classification. Countries were also categorized into low income, lower middle income, and upper middle income according to the Bank's classification of the gross national income during the project period based on data from the World Development Indicators.6 To consider how the policy and governance environment of each country is related to changes in health services, financing, and status, the Country Policy and Institutional Assessment (CPIA) cumulative ranking was used for each country from 1995-2004, using the six point scale as assessed by World Bank staff. CPIA ratings prior to 2000 were reported on a five point scale, and so were converted to a six point scale by multiplying the rating by 6/5. For each project, 6The one exception is Slovenia, a high income country that was coded with the upper middle income countries for this analysis. 6 the average CPIA rating was determined from project effectiveness to project closing dates. For those projects starting before 1995 or extending into 2004, the fullest set of years was used. Analytical Methods The study initially used exploratory data analysis to describe central tendencies, ranges and distributions of each of the variables. Subsequently, the study looked at the statistical association between the outcome and explanatory variables. The magnitude of the statistical association was first explored through the calculation of unadjusted odds ratios (UOR). The statistical significance of the association was guided by use of p- values <0.05 and 95% confidence intervals (95% C.I.). Fisher's exact test was utilized to calculate p-values when there were very small sample sizes in the bivariate analysis (i.e. less than 5 measurements in a cell). Multiple logistic regression analysis was used to control for potentially confounding variables and derive adjusted odds ratios (AOR) for two outcome variables: improvements in health services and improvements in health status. The number of projects measuring health financing outcomes was too small (n=20) to conduct meaningful multivariate analysis. Measurement of health services and health status are likely endogenous, but the data limitations did not warrant more sophisticated modeling to simultaneously assess these outcomes. Inclusion of some or all of the contextual variables did not substantively change the results, so we show the model with the full set of variables. Collinearity of the independent variables was tested through the variance inflation factor, and found insignificant for the models used. Since general budget support projects were qualitatively different from other projects (i.e. they tended to be shorter operations and were less likely to have the types of project inputs and activities identified), analysis was conducted for the full set of projects as well as separately for the 88 projects that did not involve general budget support. Since the results of the multivariate models were similar, the full sample is shown here. 7 RESULTS A. DESCRIPTIVE STATISTICS A wide variety of indicators relevant to this study were used in the evaluation of World Bank assisted projects (see Annex 2 for details). Figure 1 shows that immunization coverage indicators were the most common types of health services indicators used (32% of all projects), followed by measures outpatient utilization (25%) and delivery care coverage (21%). There were no measures of the equity of health services use. Figure 1: Most Common Health Services Indicators Used in Project ICRs 35% 30% 25% ects 20% Proj oft 15% Percen 10% 5% 0% Child Antenatal care Delivery care Outpatient Hospital services Technical quality Family planning immunization coverage coverage services use use or efficiency of health care use coverage 8 The most common health financing indicator used in the evaluations involved public sector allocation (13% of all projects), followed by indicators about financial processes (6%). Figure 2: Most Common Health Financing Indicators Used in Project ICRs 35% 30% 25% 20% Projects of 15% Percent 10% 5% 0% Public sector health Public financing Household health financing Health insurance program allocations procedures Among the health status indicators used in project ICRs, disease prevalence or incidence measures were the most commonly used set of indicators (25% of all projects), followed by rates of infant or neonatal mortality (20%) and maternal mortality ratios (16%). 9 Figure 3: Most Common Health Status Indicators Used in Project ICRs 35% 30% 25% s ect oj 20% Pr of 15% cent Per 10% 5% 0% Infant or Child mortality Maternal Life Cause- Disease Malnutrition Fertility rates neonatal rates mortality ratios expectancy specific prevalence or prevalence mortality rates mortality rates incidence Table 4 outlines the project characteristics according to their World Bank region. Across all regions, 75% (89/118) of the projects had any measurement of a health service indicator. Measurement of health service indicators ranged from a low of 60% in Europe and Central Asia (ECA) to a high of 91% in the South Asia Region (SAR). There was not a statistically significant difference between regions in any measurement of health service indicators in the bivariate analysis (2= 5.92, p=0.31). Only a minority of projects 42% took more than one measurement of any health services indicator. However, of those projects that measured any change in health services, all demonstrated an improvement. The proportion of projects showing improvement in health services was lowest in MENA (17%) and highest in South Asia (55%), though there was no statistically significant variation across regions (2=9.55, p=0.09). There was less measurement of health financing indicators in the projects reviewed. Only 21% of projects measured any kind of health financing indicator, most of which were in the Latin America & Caribbean (LAC) Region (9 projects). 10 Table 4: Project Characteristics According to World Bank Region (Percent) AFR EAP ECA LAC MENA SAR Total (N=38 ) (N=10) (N=20 ) (N=27) (N=12 ) (N=11 ) (N=118) Main Outcomes Health Services Indicator ­ 76.3 90.0 60.0 70.4 83.3 90.9 75.4 Any Measurement Health Services Indicator(s) ­ 50.0 50.0 20.0 48.2 16.7 54.6 41.5 Improvement Shown Health Financing Indicator ­ 21.1 0.0 20.0 33.3 8.3 45.5 21.2 Any Measurement Health Financing Indicator(s) ­ 18.4 0.0 10.0 22.2 0.0 36.4 17.0 Improvement Shown Health Status Indicator ­ 50.0 70.0 30.0 37.0 41.7 63.6 45.8 Any Measurement Health Status Indicator ­ 44.7 40.0 20.0 33.3 16.7 18.2 32.2 Improvement Shown Type of Development Assistance Budget Support (Extra- 42.1 10.0 25.0 25.9 0.0 27.3 27.1 sectoral) Health Sector Approach 57.9 90.0 75.0 74.1 100 72.7 72.8 Organizational Approach Disease Program 29.0 30.0 5.0 22.2 25.0 27.3 22.9 Geographic Priority 5.3 10.0 20.0 7.4 16.7 36.4 12.7 Management Support System 7.9 30.0 20.0 14.8 58.3 0.0 17.8 Sector Wide Approach 10.5 10.0 0.0 7.4 0.0 9.1 6.8 Other Approaches 5.3 10.0 30.0 22.2 0.0 0.0 12.7 Project Inputs Salaries 26.3 20.0 20.0 11.1 8.3 45.5 21.2 Training 71.1 90.0 70.0 70.4 83.3 90.9 75.4 Technical Assistance 84.2 90.0 90.0 81.5 100 100 88.1 Pharmaceuticals 68.4 80.0 45.0 44.4 58.3 81.8 60.2 Equipment 60.5 90.0 60.0 59.3 91.7 63.6 66.1 Buildings 60.5 60.0 55.0 40.7 91.7 63.6 58.5 Software 36.8 50.0 60.0 59.3 50.0 45.5 49.2 Project Activities Human Resources Systems 79.0 90.0 75.0 77.8 83.3 90.9 80.5 Health Services Management 63.2 90.0 80.0 70.4 58.3 90.9 72.0 Systems Logistics Systems 76.3 80.0 70.0 59.3 75.0 36.4 67.8 Contracting with Organizations 34.2 40.0 5.0 25.9 16.7 45.5 27.1 Regulation of health providers 31.6 30.0 65.0 44.4 16.7 36.4 39.0 Public involvement in planning 15.8 0.0 10.0 22.2 16.7 27.3 16.1 Note: Improvement is based on a positive change from at least two measurements Measurement of any health status indicator occurred in about one-half of all projects (46%). The lowest rate of any measurement of health status was in East and Central Asia (ECA) (30%), and the highest in South Asia Region (SAR) (64%). There was no statistically significant difference across the regions (2= 6.97, p=0.22). Positive changes in health status measurement were found in only 32% of all projects, ranging from 17% in Middle East North Africa Region (MENA) to 45% in the Africa Region (AFR), with the difference across regions not reaching statistical significance (2= 6.71, p=0.24). Nearly all projects (97%) that had at least two measurements of any health status indicator showed a positive change. 11 About one quarter of the projects evaluated involved general budget support, with the remainder focusing their assistance to the health sector. Out of the projects that focused on health sector approaches, 23% of projects primarily focused on programmatic priorities, 13% of projects were organized around geographic areas, 18% of projects focused on management support systems, 7% of projects used a SWAp, and 13% of the projects used other organizational approaches as their primary approach. The vast majority of projects provided technical assistance (88%) and training (75%), whereas salary support was provided in only about one-fifth of all projects, mostly in SAR. About two-thirds of projects provided equipment, most commonly in the MENA (92%) and EAP (90%) regions, whereas provision of pharmaceuticals was more common in SAR (82%) and EAP (80%) compared to a Bank average of 60%. Most of the projects involved activities supporting human resources (81%) and management of health services (72%), while about two-thirds supported logistics systems. More than one third of the projects involved regulatory interventions with health providers, an approach particularly common in ECA (65%), and least common in MENA (17%). Contracting was found in about one-quarter of the projects, most commonly in EAP and SAR, whereas engagement of the public in planning was found in only 16% of the projects. B. FACTORS ASSOCIATED WITH IMPROVEMENTS IN HEALTH SERVICES, HEALTH FINANCING, AND HEALTH STATUS Table 6 demonstrates the relationship between project variables and improvements in health services, health financing, and health status measurements. In this analysis, projects are considered to have failed to demonstrate an improvement if they did not measure change or did not show improvement when two or more measurements were made. There are several large and statistically significant associations. Notably, projects showing improvements in health services were five times more likely to show improvements in health status (UOR 4.56, 95% C.I. 1.97-10.5) and three times more likely to show improvements in health financing (UOR 3.2, 95% CI 1.18-8.75). 12 Table 5: Crude Association between Project Characteristics and Improvement in Health Services, Health Financing, and Health Status Improvements in Health Improvements in Health Improvements in Health Status Services Financing N Frequency Unadjusted OR Frequency Unadjusted OR Frequency Unadjusted OR (%) (95% CI) (%) (95% CI) (%) (95% CI) Project Outcomes Measurement of Health Service 89 - - 19.1 2.05 (0.55-7.56) 39.3 18.1 (2.4-139.5) Improvement in Health Services 49 - - 26.5 3.20 (1.17-8.75) 49.0 4.56 (1.97-10.5) Measurement of Health Financing 25 44.0 1.14 (0.47-2.77) - - 40.0 1.72 (0.69-4.31) Improvement in Health Financing 20 65.0 3.20 (1.17-8.75) - - 45.0 2.15 (0.80-5.77) Type of Development Assistance General Budget Support 32 31.3 0.55 (0.23-1.29) 25.0 2.06 (0.75-5.62) 18.8 0.43 (0.16-1.16) Health Sector Support 86 45.4 1.83 (0.77-4.31) 14.0 0.49 (0.18-1.33) 34.9 2.32 (0.86-6.26) Organizational Approach Disease Program 27 51.8 1.72 (0.72-4.09) 3.7 0.15 (0.02-1.14) 40.7 1.82 (0.74-4.44) Geographic Priority 15 46.7 1.27 (0.43-3.77) 26.7 1.98 (0.56-6.99) 20 0.53 (0.14-2.01) Management Support System 21 28.6 0.50 (0.18-1.40) 9.52 0.46 (0.1-2.16) 28.6 0.89 (0.32-2.53) Sector Wide Approach 8 75.0 7.31 (1.4-38.2) 50.0 5.88 (1.3-25.9) 62.5 4.25 (0.96-18.85) Other Approaches 15 33.3 0.67 (0.21-2.10) 6.67 0.32 (0.04-2.55) 33.3 1.16 (0.37-3.68) Project Inputs Salaries 25 52.0 1.72 (0.71-4.17) 24.0 1.78 (0.61-5.24) 48.0 2.65 (1.07-6.60) Training 89 47.2 2.81 (1.09-7.24) 15.7 0.72 (0.25-2.07) 34.8 2.57 (0.89-7.39) Technical Assistance 104 43.3 1.91 (0.56-6.48) 16.4 0.72 (0.18-2.84) 31.7 1.70 (0.45-6.52) Pharmaceuticals 71 47.9 1.96 (0.91-4.23) 18.3 1.28 (0.47-3.49) 35.2 1.78 (0.77-4.09) Equipment 78 46.2 1.78 (0.80-3.95) 15.4 0.72 (0.27-1.96) 38.5 3.54 (1.33-9.14) Buildings 69 47.8 1.89 (0.88-4.05) 15.9 0.84 (0.32-2.22) 40.6 3.50 (1.43-8.58) Software 58 53.5 2.68 (1.26-5.70) 22.4 2.19 (0.80-5.95) 39.7 2.38 (1.06-5.33) Project Activities Human Resources Systems 95 45.3 2.34 (0.85-6.46) 11.6 0.20 (0.07-0.58) 34.7 3.55 (0.98-12.82) Health Services Management Systems 85 48.2 2.91 (1.18-7.18) 17.7 1.20 (0.40-3.62) 38.8 6.35 (1.79-22.47) Logistics Systems 80 42.5 1.13 (0.52-2.49) 13.8 0.51 (0.19-1.37) 37.5 3.2 (1.20-8.55) Contracting with Organizations 32 59.4 2.73 (1.19-6.28) 21.9 1.57 (0.56-4.38) 37.5 1.55 (0.66-3.65) Regulation of health providers 46 50.0 1.76 (0.83-3.75) 23.9 2.20 (0.83-5.82) 37.0 1.64 (0.74-3.62) Public involvement in planning 19 42.1 1.03 (0.38-2.78) 10.5 0.53 (0.11-2.50) 36.8 1.41 (0.50-3.94) Country Context Country Policy & Inst. Assessment Low: <3 18 33.3 Reference 0 N/A 33.3 Reference Moderate: 3-3.5 37 24.3 0.64 (0.19-2.21) 18.9 Reference 24.3 0.64 (0.19-2.21) High: >3.5 63 54 2.34 (0.78-7.03) 20.6 1.11 (0.40-3.10) 33.3 1.0 (0.33-3.04) Gross National Income Low income 64 50 Reference 21.9 Reference 37.5 Reference Lower middle income 40 22.5 0.29 (0.12-0.71) 15 0.63 (0.22-1.80) 22.5 0.48 (0.20-1.19) Upper middle income 14 57.1 1.33 (0.42-4.28) 0 N/A 21.4 0.45 (0.12-1.79) 13 C. ORGANIZATIONAL APPROACHES General budget support projects were statistically no different from projects that took a sectoral approach in terms of improvements in health services, health financing, or health status. However, when examining specific types of health sector approaches, there was a clear association between SWAps and all three study outcomes. Projects that were primarily organized as SWAps were significantly more likely to show improvements in health services (UOR: 7.3, 95% C.I. 1.4- 38.2), health financing (UOR: 5.9, 95% CI 1.3- 25.9), though associated improvements in health status did not quite reach statistical significance at p<0.05 in the bivariate analysis (UOR: 4.25, 95% C.I. 0.96-18.85). Disease program, geographic priority, and management support systems approaches were not significantly associated with improvements in health services, health financing, or health status (see Figure 4). Figure 4: Percent of HNP Project ICRs Showing Improvements in Health Services, Health Financing, or Health Status According to Primary Organizational Approach 100% ent 90% m Health Services ve 80% Health Financing Health Status Impro 70% ingw 60% Sho 50% ts 40% Projec oft 30% en 20% Perc 10% 0% Budget Support Sector Wide Disease Others (51) Total (118) (32) Approach (8) Program (27) Note: Parenthesis indicates number of projects 14 D. PROJECT INPUTS & ACTIVITIES In the bivariate analysis (Table 6), project inputs involving training and software appeared to be statistically related to improvements in health services, whereas equipment, buildings and software appear associated with improvements in health status. None of the inputs examined were statistically associated with improvements in health financing. Project activities involving strengthening health management systems and contracting with organizations were also associated with improvements in health services. Activities involving human resource systems, health services management, and logistics systems appeared statistically associated with health status in the bivariate analysis. E. COUNTRY CONTEXT The level of per capita gross national income did not appear related to improvements in health services, financing, or outcomes in the bivariate analysis (Table 6). Similarly, the average levels of the CPIA did not appear to be statistically associated with improvements in any of the project outcomes, either when treated as a continuous variable or when treated as a categorical variable. There was also no obvious trend in the association with health services improvements (the middle level of CPIA had the higher UOR), though it was noted that none of the lowest level of CPIA had measures of improving health financing (Table 6). F. MULTIVARIATE ANALYSIS Table 7 reports the results of the multivariate logistic regression models for improvements in health services and health status using the full sets of variables. These analyses confirm the statistically significant association between SWAps and improvements in health services (AOR 14.3, 95% CI 1.6-130.0) and health status (AOR 8.89, 95% CI 1.21-65.42). In the full model, only one other variable was statistically associated with improved health services: contracting (AOR 4.1, 95% CI 1.1-15.9). When examining health status improvements, no particular type of project input was associated with health status improvements. However, project activities involving support for logistics systems was associated with improved health status (AOR 3.7, 95% ACI 1.02-13.41), and those involving health services management systems nearly reached statistical significance (AOR 4.6, 95% CI 0.93-22.7). 15 Table 6: Logistic Regression Results for Improvement in Health Services and Health Status Variable Health Services Improvement Health Status Improvement Odds Ratio P-value 95% Odds Ratio P-value 95% Confidence Confidence Interval Interval Organizational Approach (Other) 1.0 - - 1.0 - - General Budget Support 1.49 0.69 0.21-10.50 1.45 0.77 0.12-16.93 Disease Program 2.49 0.19 0.65-9.61 2.38 0.22 0.60-9.38 Sector Wide Approach 14.30 0.02 1.57-130.01 8.89 0.03 1.21-65.42 Project Inputs Salaries (none) 0.88 0.84 0.24-3.15 2.01 0.25 0.61-6.60 Training (none) 1.60 0.63 0.23-11.21 0.54 0.60 0.05-5.39 Technical Assistance (none) 2.45 0.39 0.32-18.92 0.69 0.76 0.07-6.93 Pharmaceuticals (none) 1.39 0.63 0.37-5.18 0.54 0.39 0.14-2.17 Equipment (none) 0.50 0.42 0.09-2.68 1.20 0.86 0.16-9.06 Buildings (none) 1.97 0.34 0.50-7.82 3.69 0.10 0.79-17.29 Software (none) 2.63 0.14 0.73-9.52 1.27 0.71 0.36-4.45 Project Activities Human Resource Systems (none) 0.22 0.17 0.02-1.96 0.62 0.69 0.06-6.50 Health Services Management (none) 1.25 0.76 0.29-5.40 4.59 0.06 0.93-22.69 Logistics Systems (none) 0.62 0.42 0.19-2.00 3.69 0.05 1.02-13.41 Contracting (none) 4.12 0.04 1.07-15.94 1.03 0.97 0.31-3.42 Regulations (none) 1.23 0.72 0.39-3.84 0.57 0.34 0.18-1.79 Public Involvement in Planning (none) 1.59 0.50 0.42-6.08 1.37 0.66 0.34-5.49 Country Policy & Inst. Assessment (Low: <3) 1.0 - - 1.0 - - Moderate: 3-3.5 0.28 0.17 0.04-1.75 0.92 0.92 0.17-4.99 High: >3.5 2.00 0.39 0.41-9.62 1.76 0.47 0.37-8.25 Region (AFR) 1.0 - - 1.0 - - EAP 1.56 0.66 0.22-11.17 1.21 0.84 0.19-7.65 ECA 4.01 0.20 0.47-34.19 0.29 0.32 0.03-3.28 LAC 1.98 0.55 0.21-18.46 0.46 0.49 0.05-4.18 MENA 2.37 0.53 0.16-34.31 0.65 0.75 0.05-8.64 SAR 0.89 0.90 0.15-5.22 3.21 0.20 0.54-19.02 Economy (Low income) 1.0 - - 1.0 - - Lower middle income 0.14 0.07 0.02-1.19 1.83 0.56 0.24-14.05 Upper middle income 1.17 0.90 0.12-11.58 1.27 0.84 0.12-13.54 N 118 118 Pseudo R2 0.274 0.226 Note: Parenthesis indicates reference group 16 DISCUSSION AND CONCLUSIONS These findings show that there has been very little measurement of health services, health financing, and health status in the Implementation Completion Reports of most Bank- assisted projects. Given the central importance of achieving the MDGs and related results in health, nutrition, and population, these findings are worrisome. It is a wake-up call that more needs to be done in measuring results in health services, financing, and health status. The good news from this study is that it seems that those projects that pay attention to measurement are associated with better health services and health status. One advantage of the approach taken in this study, in contrast to the evaluation formally used in ICRs and IEG project reviews, is that the evaluation criteria are objective and not dependent on subjective ratings. Only reported measurements in health services, financing, and health outcomes were used in this analysis, with the indicators defined by the projects. The requirements needed to demonstrate change were minimal: only two measurements were needed to demonstrate any kind of quantitative improvement of any relevant indicator defined by the project. One disadvantage of this approach is that other important effects of a project that are not captured by these types of measurements are not considered. A large range of project outputs related to development of physical, human, and institutional capital were not considered in this study. For example, establishing new institutions or improving processes were not assessed by this study unless they could be assessed through the type of outcomes measured. Construction of health facilities, provision of drugs, equipment, and training of health workers were treated as independent variables, but were not treated as evaluation outcomes. Furthermore, many other useful activities may not be captured by quantitative measurements of health services, health financing, or health outcomes. For example, about 48% of the projects involved pilot testing of new schemes in a country. However, in most instances, it was not clear what became of the pilot study, and it was not possible in this review of ICRs to fairly assess their status. Finally, it is also likely that important improvements in health services, financing, or health outcomes actually did occur through Bank assistance, but that they were not incorporated into the evaluation framework or were never measured or reported in the ICR. Although it is standard practice in medicine and public health to base an assessment on what is documented, the analysis used in this report should not be considered a definitive evaluation of project performance because it was not designed to assess individual project performance, and did not consider other types of outcomes or activities or documentation outside the ICRs. It can be considered an objective evaluation of the degree to which quantitative measurements in health services, health financing, and health status have been reported in the self evaluation of Bank-assisted projects. There are many reasons why measurement of health services, financing and health status in HNP is poorly done. This could be attributed to logistical difficulties in obtaining the necessary data, reporting bias (i.e. not wanting to report negative results), a lack of resources to conduct monitoring and evaluation, and particularly a lack of incentives to use project resources for proper monitoring and evaluation. Regardless of the reason, it is imperative for policy makers to encourage and support project managers to focus on 17 measuring and reporting results in health services, health financing, and health status. When such activities are consistently promoted and taken up, policy makers will be better able to identify more effective ways to improve health services, health financing, and the health of their populations. What can we learn about what approaches are more likely to be associated with better outcomes? Crude analysis from this study showed that measured improvements in health services were significantly associated with positive changes in health status. This association may simply imply that projects measuring health services were generally better organized, making them more likely to achieve health gains. It is hoped that when people are better able to access higher quality health services, premature disability and death should decline. When specific health targets were developed, project managers could have allocated resources more effectively towards improving a health condition. That is, more attention to results could have contributed to better strategies to achieve the result. Yet more attention to measurement may have biased the results upwards through a Hawthorne effect (i.e. if workers knew they were being observed, they may have worked harder to achieve their objectives). This study also suggests that certain types of organizational approaches may be associated with better health services and health status in low and middle income countries. The significantly higher odds of improving health services, financing and health status for programs using SWAps compared to others is potentially important, though the sample size was quite small. It is possible that the emphasis on regular review of performance that is part of most SWAps means that they are more likely to measure and report on health services, health financing and health status. Whether SWAps are actually more effective at achieving improvements is not clear, largely because of poor reporting of performance by most projects, and weak evaluation designs. However, the multivariate analysis reinforced the association of SWAps with measurement of improved outcomes, along with use of contracting and attention to management of health services and logistics. The main limitation of this study is that it is not clear if the relationships described above are causal or merely associated with better outcomes. Since measuring improvement is endogenous to measuring change, these associations could be demonstrating the likelihood of measuring outcomes rather than causing the improvement. The primary project approaches and other characteristics were not randomly assigned, and there are a multitude of factors outside the control of a project that may have influenced the outcomes. We tried to control for the influence of country context through the CPIA ratings, income levels, and geographic regions, but this is unlikely to be a full control of confounding variables. We note that the analysis was not sensitive to changes in CPIA and income levels, and only the levels were used in the analysis. There is also a possible reporting bias: improvements in health services and status may be biased upwards because the health indicators that show improvement were more likely to be measured and reported. Because of the lack of control groups, non-random allocation of the project sites, and unavailability of time series data, most of the project ICRs would not qualify as scientifically robust ways to demonstrate attribution to changes in health services, 18 financing, or status to the projects. For this reason, we restrict our interpretation to detecting associations between the project factors and these measuring these outcomes. Another important limitation of the study is that the sample size was relatively small, restricting the ability to detect statistically significant differences that may actually exist. On the other hand, the study may have overestimated the contribution of projects because positive changes in the services, financing and status were not assessed as to whether those changes were statistically significant. The study definitions may have categorized some indicators as having a positive change when in fact the changes were not statistically significant. Despite its limitations, there is clearly room for improving the measurement of results in health projects and ICRs. It will be difficult to focus on any kind of "Results Agenda" without more attention to measuring results. Providing better incentives for staff to undertake robust evaluations and embed the use of these results into management decision-making would help. One such option is to establish a dedicated unit to enhance monitoring and evaluation, similar to what is being done with the Global AIDS Monitoring and Evaluation Team (GAMET). If such expertise were provided with independent funding and be available to Bank team and client country requests, better monitoring and evaluation is likely to occur. Other helpful steps would be to find ways to ensure that sufficient project funds are committed for monitoring and evaluation, perhaps through benchmarking the proportion of funding used for these activities. Providing better guidelines, tools, and training to staff to define quantifiable outcome measures and to at least include baseline and follow-up data would also help. Much more could be done to improve the body of evidence around what Bank assistance in health sector is achieving at the country level. Incorporating evaluation designs that have comparison groups, particularly if they can be randomly allocated, would provide a much stronger basis for understanding results. Indicators should be selected that are relevant and can be measured more than once during the life of a project. Monitoring and evaluation designs should employ sample sizes that will have the statistical power to detect important changes, and use techniques able to minimize biases for potentially confounding variables. If randomization of interventions cannot occur, at least some consideration and measurement of factors that determine program placement should be incorporated into the analysis framework. In summary, in order to understand the factors that influence health services, health financing, health status, it is important for future World Bank projects be more consistent and rigorous in monitoring and evaluation. When projects are able to consistently do this, policy makers will be better informed in deciding what strategies are most effective and efficient in achieving the MDGs and other important outcomes. 19 ANNEX1 List of Project Implementation Completion Reports Reviewed REGION PROJECT ID COUNTRY PROJECT TITLE AFR P000118 Benin Population and Health AFR P001214 Cote d'Ivoire Integrated Health Services Development AFR P001331 Kenya Arid Lands Resource Management Project AFR P001564 Madagascar Rural Water Supply and Sanitation Pilot AFR P001792 Mozambique Health Sector Recovery AFR P001999 Niger Health Sector Development Program AFR P002369 Senegal Integrated Health Sector Development AFR P002422 Sierra Leone Integrated Health Sector Investment Project AFR P002957 Uganda Small towns water and sanitation project AFR P002963 Uganda A sexually transmitted infections project AFR P002971 Uganda District Health Project AFR P034180 Kenya Early Childhood Development AFR P035689 Mauritania Health Sector Investment Project AFR P036038 Malawi Population and Family Planning Project AFR P041567 Senegal Endemic Disease Control Project AFR P041568 Guinea Population & Reproductive Health Project AFR P043124 Eritrea Health Project AFR P050619 Ghana Third Economic Reform Support Operation Credit AFR P052887 Comoros Health Project AFR P053200 Lesotho Health Sector Reform Project AFR P055003 Mauritania Nutrition, Food Security and Social Mobilization Project AFR P058627 Tanzania Health Sector Development Program AFR P064556 Burundi Emergency Economic Recovery Credit AFR P065725 Guinea-Bissau Economic Rehabilitation and Recovery Credit AFR P066385 Rwanda Institutional Reform Credit AFR P066571 Nigeria Second Primary Education Project AFR P069570 Niger Second Public Expenditure Adjustment Credit AFR P070999 Madagascar First Poverty Reduction Support Operation AFR P071375 Cote d'Ivoire Economic Recovery Credit AFR P073832 Malawi TA-Adjustment (FRDP III) or the Third Fiscal Restructuring and Deregulation Program Technical Assistance Project AFR P074081 Uganda Poverty Reduction Support Credits (PRSCs 1-3) AFR P076908 Burkina Faso Poverty Reduction Support Operation (3) AFR P077781 Chad Chad Fifth Structural Adjustment Credit AFR P078994 Burkina Faso Poverty Reduction Support Credit 4 AFR P080345 Madagascar Emergency Economic Recovery Credit AFR P080368 Malawi Emergency Drought Recovery Project AFR P080612 Zambia Emergency Drought Recovery Project AFR P083246 Ghana Second Poverty Reduction Support Credit EAP P004399 Papua New Population Project Guinea EAP P003589 China Disease Prevention Project EAP P003967 Indonesia ID-Fifth Health Project EAP P004034 Cambodia P004034 EAP P004200 Lao Health Systems Reform and Malaria Control Project EAP P004841 Vietnam Population and Family Health Project EAP P036956 Indonesia Safe Motherhood Project EAP P042540 Indonesia Intensified Iodine Deficiency Control Project EAP P070294 Timor-Leste Health Sector Rehabilitation and Development Project EAP P073911 Timor-Leste Second Agriculture Rehabilitation Project ECA P008414 Georgia Health Project 20 REGION PROJECT ID COUNTRY PROJECT TITLE ECA P008797 Romania Health Sector Reform ECA P008814 Russian Health Reform Pilot Project Federation ECA P008867 Turkmenistan Water supply and sanitation project ECA P009076 Turkey Second Health Project ECA P009125 Uzbekistan First Health ECA P034807 Latvia Welfare Reform ECA P035761 Russian Community Social Infrastructure Project Federation ECA P044523 Bosnia and Basic Health Project Herzegovina ECA P045312 Albania Health System Recovery and Development Project ECA P049894 Tajikistan Primary Health Care Project ECA P050140 Armenia Health Financing and Primary Health Care Development Project ECA P051026 Armenia Second Structural Adjustment Technical Assistance Credit ECA P051418 Slovenia Health Sector Management Project ECA P058520 Latvia Health Reform Project ECA P069516 Kosovo Education and Health Project ECA P074586 Yugoslavia Structural Adjustment Credit ECA P074893 Macedonia Public Sector Management Adjustment Loan 2 ECA P075758 Armenia Fifth Structural Adjustment Credit ECA P078390 Serbia and SOSAC (SERBIA) Montenegro LAC P006196 Bolivia Integrated Child Development Project LAC P006522 Brazil Esprito Santo Water and Coastal Pollution Management Project LAC P006554 Brazil BR- Health Sector Reform- Reforsus LAC P006854 Colombia Municipal Health Services Project LAC P006954 Costa Rica Health Sector Reform Project LAC P007015 Dominican Provincial Health Services Project LAC P007720 Mexico Health System Reform LAC P007832 Panama Basic Education Project LAC P007837 Panama Social Investment Project LAC P007846 Panama Rural Health Project LAC P007927 Paraguay Maternal Health and Child Development LAC P035753 Nicaragua NI Health Sect II LAC P040179 Panama PA Health Pilot LAC P043418 Argentina AR-AIDS and STD Control LAC P043874 Brazil BR-Disease Surveillance- Vigisus LAC P054120 Brazil Second AIDS and STD control project LAC P055061 Mexico Health Systems Reform Technical Assistance Loans LAC P055480 Chile Municipal Development Project II LAC P060392 Bolivia BO-Health Reform APL I LAC P069861 Colombia, Social Sector Adjustment Loan Republic of LAC P073572 Colombia Structural Fiscal Adjustment Loan LAC P073817 Peru PE-Programmatic Social Reform Loan II LAC P074760 Nicaragua Programmatic Structural Adjustment Credit LAC P079060 Colombia Programmatic labor Reform and Social Structural Adjustment Loan LAC P082700 Bolivia Social Safety Net SAC LAC P083074 Argentina Argentina Economic and Social Transition Structural Adjustment Loan LAC P087841 Bolivia BO-Social Sectors Programmatic Structural Adjustment Credit MENA P005152 Egypt National Schistosomiasis Control Project MENA P005163 Egypt Population project MENA P005521 Morocco Water resources management project 21 REGION PROJECT ID COUNTRY PROJECT TITLE MENA P005746 Tunisia TN-Health Sector Loan MENA P005910 Yemen Family Health Project MENA P034004 Lebanese Health Project MENA P039749 Jordan Health Sector Reform MENA P040556 Morocco Rural Water Supply & Sanitation Project MENA P042415 Morocco Social Priorities Program MENA P053892 West Bank and Health System Development Project Gaza MENA P075984 West Bank and Emergency Services Support Project Gaza MENA P078136 West Bank and Emergency Services Support Project II Gaza SAR P009977 India (Bihar) Second Integrated Child Development Project SAR P009977 India (MP) Second Integrated Child Development Project SAR P010531 India Reproductive and Child Health SAR P035825 India (Karnataka) State Health Systems II SAR P035825 India (Punjab) State Health Systems II SAR P035825 India (West State Health Systems II Bengal) SAR P037857 Bangladesh Health and Population Program Project SAR P067543 India Second National Leprosy Elimination Project SAR P077834 Pakistan NWFP Structural Adjustment Credit SAR P078806 Pakistan Pakistan Poverty Reduction Support Credit (PRSC) I SAR P079635 Pakistan NWFP SAC II 22 ANNEX 2 Types of Health Services, Health Financing and Health Status Indicators Used in Implementation Completion Reports of Health Projects (FY 2003- 2005) Percent of Percent of Projects with Type of Indicator Frequency Indicators Indicator Health Services Indicators Child immunization coverage 66 17% 32% Maternal immunization coverage 2 1% 2% Antenatal care coverage 25 6% 20% Delivery care coverage 32 8% 21% Post-natal care coverage 5 1% 3% Outpatient services use 30 8% 25% Hospital services use or efficiency 39 10% 19% Technical quality of health care 53 14% 19% Patient/client satisfaction 8 2% 7% Equity in health services 0 0% 0% Family planning use 50 13% 17% Sexual health behavior 18 5% 4% Other health behavior (e.g. breastfeeding) 6 2% 5% Water quality or use 21 5% 9% Sanitation use 3 1% 2% Other public health services (e.g. surveillance) 29 7% 18% Total Number of Service Indicators 387 100% 75% Health Financing Indicators Public sector health allocations 28 60% 13% Public financing procedures 10 21% 6% Household health financing 4 9% 3% Health insurance program 5 11% 5% Total Number of Financing Indicators 47 100% 21% Health Status Indicators Infant or neonatal mortality rates 30 16% 20% Child mortality rates 14 8% 10% Maternal mortality ratios 17 9% 16% Life expectancy 2 1% 2% Cause-specific mortality rates 12 7% 6% Disease prevalence or incidence 68 37% 25% Malnutrition prevalence 29 16% 9% Fertility rates 10 5% 7% Total Number of Health Status Indicators 182 100% 46% Total Number of Projects 118 Note: Projects may use more than one indicator of each type in its evaluation framework 23 ANNEX 3 Graphical Representation of results 1. HNP Projects With At Least One Measurement of Services, Financing or Status 100 90 80 70 s 60 Projectfo 50 40 Percent 30 20 10 0 AFR EAP ECA LAC MENA SAR Total Any Health Services Indicator Any Health Financing Indicator Any Health Status Indicator 24 2. HNP Projects Measuring an Improvement in Health Services, Financing, or Status 100 90 80 70 s ect 60 oj Pr of 50 cent 40 Per 30 20 10 0 AFR EAP ECA LAC MENA SAR Total Health Services Health Financing Health Status 25 3. Odds Ratio For Improving Health Services According to Organizational Design & Project Activity 15 14 13 12 11 10 otiaR 9 8 Note: Red indicates p<0.05 7 Odds6 5 4 3 2 1 0 General Disease SectorWide Human Health Logistics Contracting Regulations Public Budget Program Approach Resource Services Systems Involvement in Support Systems Management Planning 26 4. Odds Ratio For Improving Health Status According to Organizational Design & Project Activity 10 9 8 7 6 Ratio Note: Red indicates p<0.05 s 5 Odd4 3 2 1 0 General Disease SectorWide Human Health Logistics Contracting Regulations Public Budget Program Approach Resource Services Systems Involvement in Support Systems Management Planning 27 About this series... This series is produced by the Health, Nutrition, and Population Family (HNP) of the World Bank's Human Development Network. The papers in this series aim to provide a vehicle for publishing preliminary and unpolished results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For free copies of papers in this series please contact the individual authors whose name appears on the paper. Enquiries about the series and submissions should be made directly to the Managing Editor Janet Nassim (Jnassim@worldbank.org) or HNP Advisory Service (healthpop@worldbank.org, tel 202 473-2256, fax 202 522-3234). For more information, see also www.worldbank.org/ hnppublications. THE WORLD BANK 1818 H Street, NW Washington, DC USA 20433 Telephone: 202 473 1000 Facsimile: 202 477 6391 Internet: www.worldbank.org E-mail: feedback@worldbank.org