PURCHASING IN A PANDEMIC: LESSONS FROM THE HEALTH SYSTEMS RESPONSE IN ARMENIA AND ROMANIA DISCUSSION PAPER FEBRUARY 2023 Adanna Chukwuma Juan Carlos Rivillas Estelle Gong Huihui Wang Dorothee Chen Hratchia Lylozian Radu Comsa Diana-Luliana Pirjol Tania Dmytraczenko Katie Sonnefeldt PURCHASING IN A PANDEMIC: LESSONS FROM THE HEALTH SYSTEMS RESPONSE IN ARMENIA AND ROMANIA Adanna Chukwuma, Juan Carlos Rivillas, Estelle Gong, Huihui Wang, Dorothee Chen, Hratchia Lylozian, Radu Comsa, Diana- Luliana Pirjol, Tania Dmytraczenko, Katie Sonnefeldt February 2023 Health, Nutrition and Population (HNP) Discussion Paper This series is produced by the Health, Nutrition, and Population Global Practice of the World Bank. The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations, or to mem- bers of its Board of Executive Directors, or to the countries they represent. Citation and the use of the material presented in this series should take into account this provisional character. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. For information regarding the HNP Discussion Paper Series, please contact the Editor, Jung-Hwan Choi at jchoi@worldbank.org or Erika Yanick at eyanick@worldbank.org. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because the World Bank encourages the dissemination of its knowledge, this work may be reproduced, in whole or in part, for non- commercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. © 2023 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved. ii Health, Nutrition and Population (HNP) Discussion Paper Purchasing in a pandemic: Lessons from the Health Systems Response in Armenia and Romania Adanna Chukwumaa, Juan Carlos Rivillasa, Estelle Gonga, Huihui Wanga, Dorothee Chenf, Hratchia Lyloziana, Radu Comsaa, Diana-Luliana Pirjola, Tania Dmytraczenkoa, Katie Sonnefeldta a Health Nutrition and Population, World Bank, Washington DC, USA Paper prepared as a part of the Collectivity Project that examines adjustments made in purchasing arrangements that occurred during the Armenian and Romanian response to the COVID-19 pandemic. This project was funded by The World Bank Group, Abstract: Purchasing for health, which includes what, how, and from whom services are purchased, was one of the policy levers available to countries as part of their health sys- tems’ response to the COVID-19 pandemic. Ideally, the purchasing function should align with the broader health financing functions. Empirical evidence indicates that purchasing arrangements transformed during the pandemic across the world. A systematic examina- tion of these changes can inform ongoing efforts to leverage purchasing to strengthen health system performance. The Collectivity Project is a global community of practitioners, decision makers, and re- searchers contributing to collaborative health system projects. The thematic group con- vened experts from eight countries to systematically assess the changes in purchasing arrangements as part of the COVID-19 health response and their implications for health system objectives. This report examines adjustments made in purchasing arrangements that occurred during the Armenian and Romanian response to the COVID-19 pandemic. The research adopted mixed methods and a deductive approach. Data were obtained from a scoping literature review, key informant interviews, and an exploratory analysis of quantitative health system indicators. The study was informed by a framework for understanding purchasing changes that adapted existing frameworks to explore the implications of purchasing adjustments on critical outcomes. The research describes critical changes in purchasing, provider and user responses to these changes, and health system outcomes that accompanied the COVID-19 response in Armenia and Romania. For example, it was essential to have a governance environment that defined shared objectives and facilitated coordination across stakeholders. During this time benefits expanded, and contracts changed, including payment mechanisms to offset the decline in essential service use and incentivize care delivery for COVID-19. Fur- thermore, the pandemic saw the accelerated adoption of innovation, particularly telemed- icine, within service delivery. The lessons from purchasing during the pandemic have im- plications for improving coverage, quality, and adaptability to a crisis, including beyond the contexts studied. iii This report concludes that intergovernmental cooperation, agile purchasing tools, and in- novation in purchasing and service delivery are important to effectively coordinate essen- tial services to key populations in rapid changing times. Moving forward, it is essential to assess the sustainability of these adjustments over time and the implications for health system performance. Keywords: purchasing, health financing, health system, performance, innovations. Disclaimer: The findings, interpretations and conclusions expressed in the paper are entirely those of the authors, and do not represent the views of the World Bank, its Exec- utive Directors, or the countries they represent. Correspondence Details: Adanna Chukwuma, 1818 H Street, NW Washington, DC 20433, USA, telephone: 202-212-9000, e-mail: achukwuma@wordbank.org, website: www.worldbank.org iv Table of Contents FOREWORD......................................................................................................... 3 ACKNOWLEDGMENTS ....................................................................................... 5 ACRONYMUS ...................................................................................................... 6 LIST OF TABLES AND FIGURES ....................................................................... 7 PART I: INTRODUCTION .................................................................................... 8 1.1. THE COLLECTIVITY PROJECT ................................................................................... 8 1.2. PURCHASING AS PART OF THE POLICY RESPONSE TO THE COVID-19 PANDEMIC ....... 8 1.3. PREPANDEMIC HEALTH SYSTEMS IN ARMENIA AND ROMANIA................................... 10 1.4. STUDY OBJECTIVE ................................................................................................ 14 PART II: METHODS ........................................................................................... 15 2.1. CONCEPTUAL FRAMEWORK ............................................................................... 15 2.1.1.Purchasing arrangements ................................................................... 16 2.1.2.Provider and user engagement and responses ................................... 17 2.1.3.Intermediate and final health systems outcomes................................. 17 2.2. DATA COLLECTION AND ANALYSIS...................................................................... 17 PART III: RESULTS ........................................................................................... 19 2.3. ARMENIA .......................................................................................................... 19 2.3.1.Governance of purchasing .................................................................. 19 2.3.2.What to Buy ........................................................................................ 21 2.3.3.From Whom to Buy ............................................................................. 22 2.3.4.How to Buy ......................................................................................... 23 2.3.5.Stakeholder engagement and responses ............................................ 24 2.3.6.Health system outcomes ..................................................................... 24 2.4. ROMANIA .......................................................................................................... 27 2.4.1.Governance of purchasing .................................................................. 27 2.4.2.What to Buy ........................................................................................ 28 2.4.3.From Whom to Buy ............................................................................. 29 2.4.4.How to Buy ......................................................................................... 30 2.4.5.Stakeholder engagement and responses ............................................ 31 2.4.6.Health system outcomes ..................................................................... 31 PART IV: DISCUSSION ..................................................................................... 33 PART V: APPENDIX .......................................................................................... 36 2.5. APPENDIX A. DETAILED METHODS FOR DATA COLLECTION AND ANALYSIS............ 36 2.5.1.Scoping literature review ..................................................................... 36 2.5.2.Key informant interviews and focus group discussions ....................... 37 2.5.3.The Recruitment of the key participants .............................................. 38 2.5.4.Interview guide for purchasers (Ministries of Health) ........................... 39 2.5.5.Health systems outcome indicators..................................................... 40 2.5.6.Data analysis ...................................................................................... 41 2.5.7.Scoping literature review ..................................................................... 41 2.5.8.Key informant interviews and focus group discussions ....................... 42 2.5.9.Quantitative indicator analysis ............................................................ 42 REFERENCES ................................................................................................... 43 2 FOREWORD Purchasing for health, which includes what, how, and from whom services are purchased, was one of the policy levers available to countries as part of their health systems’ response to the COVID-19 pandemic. Ideally, the purchasing function should align with the broader health financing functions (revenue raising, pooling, and public financial management) and policy levers (service delivery, health workforce, health technologies, and governance). Empirical evidence indicates that purchasing arrangements transformed during the pandemic across the world. As COVID-19 infection rates grew, countries expanded cov- ered health services, removed copayments, designated specific facilities for COVID-19 care, increased private sector participation, and implemented new payment mechanisms. In most cases, these changes aimed to ensure access to quality care for COVID-19 while maintaining essential care use. A systematic examination of these changes can inform ongoing efforts to leverage purchasing to strengthen health system performance. The Collectivity Project is a global community of practitioners, decision makers, and researchers contributing to collaborative health system projects. A thematic group was set up within the Collectivity Project to explore the impacts of adjustments to purchasing arrangements during the pandemic. This thematic group is part of the larger collaborative learning initiative on strategic purchasing for universal health coverage (UHC), led by the World Health Organization (WHO), the Institute of Tropical Medicine, and the Financial Access to Health Services and Performance Based Financing Communities of Practice. The thematic group convened experts from eight countries to systematically assess the changes in purchasing arrangements as part of the COVID-19 health response and their implications for health system objectives. As part of the thematic group, this report examines adjustments made in purchasing arrangements that occurred during the Armenian and Romanian response to the COVID-19 pandemic. The research adopted mixed methods and a deductive approach. Data were obtained from a scoping literature review, key informant interviews, and an exploratory analysis of quantitative health system indicators. The study was informed by a framework for understanding purchasing changes that adapted existing frameworks to explore the implications of purchasing adjustments on critical outcomes. The research describes critical changes in purchasing, provider and user responses to these changes, and health system outcomes that accompanied the COVID-19 response in Armenia and Romania. For example, it was essential to have a governance environment that defined shared objectives and facilitated coordination across stakeholders. During this time benefits expanded, and contracts changed, including payment mechanisms to offset the decline in essential service use and incentivize care delivery for COVID-19. Furthermore, the pandemic saw the accelerated adoption of innovation, particularly tele- medicine, within service delivery. The lessons from purchasing during the pandemic have implications for improving coverage, quality, and adaptability to a crisis, including beyond the contexts studied. 3 This report concludes with three ideas. First, intergovernmental cooperation is important to effectively coordinate and disseminate essential services to key populations. Second, while purchasing tools were adapted for agile pandemic policy response, governance and delivery modifications were needed to accompany these changes. Finally, it highlights the reality of changing needs and how a crisis can spur innovation in purchasing and service delivery. Moving forward, it is essential to assess the sustainability of these adjustments over time and the implications for health system performance. 4 ACKNOWLEDGMENTS The authors are grateful to the World Bank for publishing this report as an HNP Discussion Paper. Valuable comments from Denizhan Duran, Health Economist (HMNHN); Isidore Sieleunou, Senior Health Specialist (HHNGF); Olena Doroshenko, Senior Health Econo- mist (HECHN); and Irina Tevosyan, Operations Analyst (ECCAR) are gratefully acknowl- edged. Special thanks to the policy makers, purchasers, health care providers, and service users whose insights contributed to the study. We also thank colleagues in the Collectivity Project for helpful feedback during the different stages of the project, including Inke Mathauer, Uju Onyes, and Danielle Bloom. 5 ACRONYMUS ANMCS National Authority for Quality Management in Health BBP Basic Benefits Package CHE Current Health Expenditure DHIH District Health Insurance House DPHA District Public Health Authority ICU Intensive care unit NCDs Noncommunicable diseases NHIF National Health Insurance Fund NHIH National Health Insurance House OOP Out-of-pocket SHA State Health Agency SHAEI State Hygiene and Anti-Epidemic Inspectorate THE Total Health Expenditure UHC Universal Health Coverage WHO World Health Organization 6 LIST OF TABLES AND FIGURES FIGURES Figure 1. Regional Cumulative COVID-19 Cases per Million Population (October 2021) Figure 2. The Projected Need for Hospital and ICU Beds in Europe and Central Asia Figure 3. Core Areas of Strategic Purchasing Figure 4. A Conceptual Framework for Understanding Purchasing Changes Figure 5. Methods for Data Collection and Data Triangulation Figure 6. Health System Response to the COVID-19 Pandemic in Armenia Figure 7. Health System Response to the COVID-19 Pandemic in Romania Figure A.1. PRISMA Flow Chart Detailing Numbers of Studies Screened, Assessed, and Included in the Review TABLES Table 1. Armenia: Key Characteristics of Health System Governance, Health Financing, and Health Service Delivery Table 2. Romania: Key Characteristics of Health System Governance, Health Financing, and Health Service Delivery Table A.1. Literature Search Criteria Table A.2. Number of Participants by Country and Gender 7 PART I: INTRODUCTION 1.1. THE COLLECTIVITY PROJECT The Collectivity Project is a global community of practitioners, decision makers, and re- searchers contributing to collaborative health system projects. A thematic group was set up within the Collectivity Project to explore the impacts of adjustments to purchasing ar- rangements during the pandemic. This thematic group is part of the larger collaborative learning initiative on strategic purchasing for universal health coverage (UHC), led by the World Health Organization (WHO), the Institute of Tropical Medicine, and the Financial Access to Health Services and Performance Based Financing Communities of Practice. The COVID-19 pandemic raised critical questions regarding the reconfiguration of pur- chasing arrangements to ensure the provision of COVID-19-related health services. Care included supplying essential non-COVID-19 health services, cushioning the health system from shocks, and enabling functioning despite fiscal austerity. Therefore, the thematic group convened experts from eight countries to systematically assess the changes in pur- chasing arrangements as part of the COVID-19 health response and their implications for health system objectives (The Collectivity 2021). This report contributes to the group’s work by focusing on the experiences of Armenia and Romania. 1.2. PURCHASING AS PART OF THE POLICY RESPONSE TO THE COVID-19 PANDEMIC Since the first case of COVID-19 was reported in December 2019, there have been over 560 million confirmed cases and 6 million deaths across the globe (Ritchie and Mathieu 2022). Europe and Central Asia account for over 7 percent of cases and 5 percent of deaths globally. Figure 1 shows the cumulative reported cases per million people in coun- tries in the region as of October 2021, ranging from 1,793 cases per million in Tajikistan to 228,938 cases per million in Montenegro. Figure 1. Regional Cumulative COVID-19 Cases per Million Population (October 2021) Source: Our World in Data. 8 With the introduction of COVID-19 vaccines in December 2020, over 12 million vaccines have been administered globally (Mathieu et al. 2021). However, vaccination rates have varied significantly within Europe and Central Asia. For example, as of October 2021, 7 percent of the population in Armenia was fully vaccinated compared to 57 percent in Tur- key (Hannah Ritchie 2022). The pandemic challenged health systems worldwide, requiring them to reduce COVID-19 transmission, treat cases, and ensure the availability of essential non-COVID-19 services. To this end, several public health measures were adopted, including restrictions on travel, stay-at-home orders, contact tracing, social distancing, and masking measures (Blavatnik School of Government 2022). In addition, countries invested in increased testing capacity and case management at home, in first-level facilities, or via hospitalization and intensive care. The health system response to growing COVID-19 infection rates required a surge in hos- pital capacity for case management. In November 2021, the Institute of Health Metrics and Evaluation (IHME) estimated the number of hospital beds needed in Europe and Central Asia to be about 332,600, of which 37.6 percent were intensive care unit (ICU) beds. Fig- ure 2 shows the projected hospital beds needed during the region's fourth wave of COVID- 19 infections. The total beds needed in November 2021 were projected to approach the December 2020 peak of about 347,000 beds (IHME 2021). Figure 2. The Projected Need for Hospital and ICU Beds in Europe and Central Asia Source: IHME (Institute for Health Metrics and Evaluation). 2021. "COVID-19 Estimate Downloads." Seattle. https://www.healthdata.org/covid/estimate-downloads-archive The pandemic also presented challenges in ensuring the use of essential care. Particularly with movement restrictions, the delivery and utilization rates of essential services were interrupted. For example, in a World Health Organization (WHO) survey of 105 countries in May and June 2020, 76 percent reported reduced ambulatory care utilization, and 66 percent reported cancellation of elective procedures (WHO 2020b). These disruptions 9 persisted into the first half of 2021, with the most significant impacts on primary, rehabili- tative, palliative, and long-term care. In response to these demands, health systems increased public health funding. Increases in fiscal space for health were enabled by borrowing, reallocations within the state budget to the health sector, support from development agencies, and leveraging declarations of states of emergency. More than 40 countries established predominantly extrabudgetary funds to support their broader response to the COVID-19 crisis (IMF 2022). Countries also exempted funding increases from existing budget deficit laws (WHO Europe 2020). In addition to expanding fiscal space for health, many countries adjusted purchasing ar- rangements to align fund flows with emerging needs (Mathauer, Dkhimi, and Townsend 2020). The benefits package was expanded to include COVID-19 related to testing, tele- medicine, and home and hospital care. Some countries removed copayments for essential non-COVID-19 services to reduce barriers to use. Furthermore, new payment mecha- nisms were implemented to protect providers from decreased revenue from output-based payments. On the service delivery side, governments designated specific facilities to provide care for moderate-to-severe COVID-19 and increased their engagement with the private sector to expand available hospital beds. A rapid assessment reviewed the changes in purchasing globally and noted the need for more in-depth examinations of the experiences in specific countries (Merkur et al. 2020). Hence, the thematic group under the Collectivity Project examined the changes in purchasing in selected countries in depth and their implications for universal health coverage and health system resilience. 1.3. PREPANDEMIC HEALTH SYSTEMS IN ARMENIA AND ROMANIA This report focuses on Armenia and Romania. In both countries, life expectancy at birth has increased over the past decades. Life expectancy rose from 68 years in Armenia and 70 years in Romania in 1990, to 75 years in Armenia and 74 years in Romania in 2020 (WHO Europe 2022). A change in the epidemiologic picture has accompanied the demo- graphic transition. For example, between 2009 and 2019, maternal mortality rates fell in both countries as the prevalence of chronic diseases rose (World Bank Group 2021). Ar- menia and Romania's top two causes of death are ischemic heart disease and stroke (IHME 2021). The research aimed to provide a baseline against which the changes in Armenia and Ro- mania during the pandemic can be assessed. Arrangements for purchasing before the pandemic are described below, followed by definitions for how, what, and from whom ser- vices are purchased. Armenia and Romania present interesting and contrasting cases, as these countries both emerged from the centralized Semashko-style health system model in the 1990s with divergent health financing paths. Armenia, an upper-middle-income country, has a small public health budget and frag- mented funding flows, with multiple private insurers administering one of three benefits packages. These arrangements limit the monopsony power of the public purchaser and, 10 thus, its ability to shape incentives. Additionally, payment mechanisms are primarily out- put-based. Romania, a high-income country, has a relatively high per capita public health budget. In addition, a significant proportion of funding is prepaid and pooled in a single public purchaser. This financing arrangement confers significant monopsony power on the purchaser. At the service delivery level, output-based payment methods link fund flows to essential service use. Tables 1 and 2 describe the key characteristics of the health system in Armenia and Romania before the pandemic. 11 Table 1. Armenia: Key Characteristics of Health System Governance, Health Financing, and Health Service Delivery Health system governance Health system financing Health service delivery Health information sys- tems Political economy Revenue collection Delivery capacity • The ArMed national digital health • An upper-middle-income country with a per capita • The current health expenditure (CHE) per capita in US • The unequal distribution of skilled workers system has been used in Armenia income of $4,605 in 2019. dollars was $422.3 (2018). limits the readiness of the service delivery since 2017. • The current government has implemented • Fifteen percent of CHE is from the general government system to support high-quality health care. • The platform enables the collection prudent macroeconomic policies supported by an budget; 85 percent comprises out-of-pocket (OOP) • Armenia has a comparable density of of clinical, administrative, and active inflation-targeting regime, an effective payments. physicians to the average in Europe and financial data from the provision of fiscal rule, sound financial sector oversight, and • Public spending on health is financed via the general Central Asia (43 per 10,000), at 44 per standard health care services and continued pro-competition reforms. government budget and funded via tax revenue (22.2 10,000 in 2017. aims to facilitate patient • About 44.5 percent of the employed population is percent of GDP in 2018). • However, the density of physicians varies engagement. in the informal sector, and 49 percent of women from 88 per 10,000 in Yerevan, the capital, • The privately operated ArMed are unemployed. Purchasing to 14 per 10,000 in the village of allows real-time access to data on • High informal employment and an aging • Multiple packages for different segments of the population, Gegharkunik. service use. Patients can access population limit opportunities to raise revenue, including a social package for formal sector workers, the • In 2015, Armenia had a slightly lower ArMed via a mobile phone or including health revenue, via direct taxes on state order for social and vulnerable groups, and the general hospital bed density (4.2 per 1,000 people) computer. The ArMed dashboard wages. package for the rest of the population (including primary than the regional average. is also accessible to government care, maternal health services, emergency care, and care • There are 501 primary health care facilities officials via a desktop platform. Institutional arrangements for selected infectious diseases). and 124 hospitals within the country. • Before 2020, the fully implemented • The Ministry of Health (MoH) holds a regulatory • Most of the population does not have coverage for more • Outside Yerevan, the national capital of registries in ArMed supported the role, responsible for drafting health policy, setting comprehensive inpatient and diagnostic services. Armenia, there are spatial monopolies in coding of medical services, claim service standards, and overseeing state- • Six private insurers are administering the social package, a service delivery, such that excluding a preauthorization and administered public health programs. separate benefits package for public employees with a provider for poor quality via selective reimbursement, and registration of • The MoH is responsible for revising the Basic health insurance component distinct from other packages. contracting may result in loss of access to contracted health care providers. Benefits Package (BBP) that outlines the service • The SHA administers the state order and the package for care for a community. • Data collected through ArMed are scope for different groups of beneficiaries. the general population. not routinely analyzed to • The State Health Agency (SHA) is the designated understand service delivery • Providers sign contracts annually with the MoH. purchaser of state-provided health services. Provider autonomy performance or emerging health • Payment to facilities is output-based, with a combination of care needs. However, the MoH now contracts with providers capitation, fee-for-service, performance-based payments, • With broader decentralization, the directly. management of health facilities has been • The modules that supported and case-based payments (inpatient settings). • Funds for the social package are allocated via the extended to the subnational level. electronic referrals and • Health workers are paid fixed wages, bonuses, or both. SHA to multiple private insurers. • However, the MoH also owns and operates coordination across service • To receive reimbursement for services outlined in the BBP, delivery levels were not utilized • MoH, SHA, and private insurers have a broad providers have formal requirements related to licensure, some tertiary hospitals. until 2020. For instance, the e- stakeholder representation. Still, this tripartite equipment, personnel, financial reporting, and a minimum • Marz (provincial) governments are prescription registry remains participation of purchasers does not fully cover income from paid services, as specified in Decree 49-N. significantly involved in facility management, partially implemented. However, other important stakeholders’ views and interests, • In practice, facilities are rarely excluded, regardless of and there is autonomy in decisions regarding this registry could support including patient groups, citizens, doctors, and whether they meet requirements. service delivery organization in response to monitoring prescription patterns, nurses. • Contracting is also not conditioned on other measures of purchasing incentives. potentially preventing harmful drug • Overlaps in the roles of the MoH, SHA, and quality of care (e.g., adherence to clinical guidelines, interactions and polypharmacy. private insurers lead to inefficiencies. procurement capacity). Source: Authors. 12 Table 2. Romania: Key Characteristics of Health System Governance, Health Financing, and Health Service Delivery Health systems Health systems financing Health service delivery Health information systems governance Political economy Revenue collection Health care provision • The National Centre of Statistics and • A high-income country with a • Public spending on health is financed via four primary sources: Informatics in Public Health at the NIPH is the per capita income of $12,610 in national health insurance funds, the state budget, local budgets, and • Family medicine physicians provide primary primary and oldest health information system 2019. out-of-pocket (OOP) payments. care. managed by the MoH through the National • Romania's economy contracted • The publicly funded proportion is relatively high (over 80 percent, • Specialized ambulatory care is provided Centre of Statistics and Informatics in Public by 3.9 percent in 2020, one of just above the EU average of 76 percent). through a network of hospital outpatient Health. It collects a large volume of data on the lowest contractions in the • Most public funding comes from the health insurance contributions departments and polyclinics, specialized health services and utilization (number of European Union (EU). to the National Health Insurance Fund (NHIF) (67 percent in 2013); medical centers, centers for diagnosis and medical consultations, inpatient days, • The fiscal deficit surged to 9.2 the share of OOP payments is the second-largest source (19 treatment, and individual specialist average length of stay, or bed occupancy) percent of GDP at the end of percent in 2014), and voluntary health insurance (VHI) is marginal physician offices under contract with the and data on morbidity. 2020. (0.1 percent). DHIHs. • Data are published annually in statistical • Given Romania's limited fiscal • Financing capital investments in health care facilities and equipment • Specialized physicians in ambulatory care reports and specific bulletins, restricted to space, maximal absorption of is mainly from the state budget, through the MoH, but can also come generally divide their time between the health care units, and not publicly available. EU funds will be crucial to a from local budgets (according to investment plans preapproved by public and private sectors. Family However, they may be made available upon sustainable recovery. the MoH) and external funds (World Bank, EU structural funds). practitioners also make referrals to request. • In 2021, the employment rate specialist care. • Data collected are often incomplete or not was 69.1 percent, with a higher • Regarding the number of publicly compiled. For example, data on rate for men (72.2 percent) than Purchasing contracted beds, the share of public vs. determinants, disease risk factors, and women (65.6 percent). private suppliers is 95 percent vs. 5 percent. information on utilization services do not • There is no competition, and the District Health Insurance Houses • Other outpatient services, such as imagistic reflect the activity of the entire system. Also, (DHIHs) usually sign contracts with all providers in the district. services, are mainly provided by private data on private physicians are not always • In addition to the roles of the MoH and NHIH, the Ministry of Public Institutional arrangements Finances influences purchasing by defining expenditure ceilings for suppliers under contracts with NHIH. gathered. • The Ministry of Health and critical cost categories, such as human resources and • Provision is characterized by an oversupply • Data are aggregated at the district level, District Public Health Authorities of highly specialized inpatient care and which includes multiple levels of care and pharmaceuticals. (DPHA) has regulatory underutilization of primary and community population groups, making it impossible to • All purchasing contracts between the DHIHs and providers must oversight over health care. care. identify and analyze the activities of individual comply with the Framework Contract norms. • The Ministry of Health (MoH) providers. • The NHIH and DHIH monitor contract compliance. While and the National Health • The Unique Integrated Health Informatics compliance checks verify the volume of services, provisions for Insurance House (NHIH) jointly System in Romania (SIUI) was designed in monitoring quality are not implemented in practice due to a lack of sponsor the regulation for the Provider autonomy 2002 and not fully implemented until 2008. It capacity. Noncompliant DHIHs and service providers, both public framework contract that • Most public hospitals (80 percent) are under is the health information system managed by and private, may be subject to sanctions. regulates purchasing. the administration of local councils. Others the NHIH. • Private family physicians are contracted to provide the basic and • The framework contract, are administered by the MoH (13 percent) • The SIUI, as a national system, is dedicated minimum service packages at the primary care level. They also can introduced in 1999, is the or other ministries or governmental to managing the funds for over 21 million provide national health services such as immunizations and primary legislative tool institutions (7 percent). citizens and over 26,000 health service screenings. To provide services under the benefits package, regulating health service providers. providers must have between 800 and 2,200 registered patients per • Public hospitals are established or closed purchasing. by governmental decisions initiated by the • The SIUI collects information on contracted physician working 35 hours a week. Public or private providers contracted by NHIH are reimbursed based on the same calculation relevant administrative authority. providers, reporting, billing, settlement, and medical information for insured patients. formulas, valid for all providers. • Capital investments in public health care • Framework contracts are consulted with service providers, patients, institutions are approved by ministerial • The system focuses primarily on and civil society representatives. order. administrative patient data review and less on aspects of the individual's clinical history. • For pharmaceuticals, a positive list is elaborated with input from the • There is significant autonomy over Each module corresponds to software Health Technology Assessment (HTA) department at the MoH spending allocation at the provider level, available to health care providers. established in 2012. The framework contract banned charging and profits are retained. copayments for the health care services provided to insured patients. • VHI plays a marginal role in the financing of health care. These plans mainly offer access to superior hospital accommodation and choice of provider and private care (supplementary cover). It is impossible to opt out of the statutory scheme and purchase substitutive VHI cover. • Payment for primary care is via capitation and fee-for-service, with hourly tariffs for permanence centers (after-hour care), while hospitals are primarily reimbursed via diagnosis-related groups. Source: Authors. World Bank Group Europe and Central Asia Office 13 Health, Nutrition and Population Global Practice 1.4. STUDY OBJECTIVE This study examines the adjustments made in purchasing arrangements as part of the Armenian and Romanian response to the COVID-19 pandemic, their implications for provider and user be- havior, and the associated changes in health system performance. The rest of the report is struc- tured as follows. Chapter 2 reviews the study methodology, while Chapter 3 discusses the results along the dimensions of the study framework. Finally, Chapter 4 provides concluding remarks and reflects on the strengths and limitations of the study. 14 PART II: METHODS This section describes (i) the study framework for analyzing purchasing changes during the pan- demic, (ii) data collection approaches, and (iii) analysis. The analysis focuses on 2021 due to data availability. 2.1. CONCEPTUAL FRAMEWORK Purchasing refers to the allocation of funds toward health care goods and services. Households purchase when they pay providers out-of-pocket (OOP) for services. However, this is an inequitable and inefficient arrangement. Third-party purchasers, however, allocated pooled funds toward health services on behalf of households (Cashin et al. 2018). Passive allocations often follow historical patterns, with little regard for changing population needs or service quality. However, if strategically deployed, purchasing arrangements can be policy levers that facilitate health system goals. Stra- tegic health purchasing aims to align health care provider and service user behavior with the pur- chaser’s objectives for the health system. During the pandemic, these goals may have included ensuring access to high-quality health care for essential services and COVID-19. Strategic purchasing involves several interrelated areas, which need to be aligned and addressed jointly within a relevant overarching governance architecture. The strategic health purchasing pro- gress framework described by Cashin and others (2018) provides an overview of these areas, in- cluding specification of services and interventions (“what to buy”); choice of providers (“from whom to buy”); design of financial and nonfinancial incentives (“how to buy”); and the governance of these decisions (Cashin et al. 2018). In the broader health system context, they also identify factors, including revenue raising, pooling, and governance, that can strengthen or weaken the power of purchasers to influence provider behavior. These dimensions are summarized in Figure 3. Figure 3. Core Areas of Strategic Purchasing What to buy From whom to buy? How to buy? Which services and From which providers Which provider payment ` medicines to purchase? to buy? methods? What cost-sharing and What contractual obliga- referral mechanisms are How to select them? tions and performance utilized? incentives? Purchasing governance How is information generated and managed for strategic purchasing decisions? ` What oversight mechanisms exists for accountability and coordination? Source: Cashin, Cheryl, Sharon Nakhimovsky, Kelley Laird, Altea Cico, Sharmini Radakrishnan, Tihomir Strizrep, Ali Lauer, et al. 2018. "Strategic Health Purchasing Progress: A Framework for Policymakers and Practitioners." This study aims to link the core purchasing areas to provider and user behavior and health systems performance. Changes in purchasing arrangements are expected to incentivize behavior 15 modification among providers and users. The actions of these stakeholders, in turn, have implica- tions for health systems' performance. The latter are identified by drawing on health systems frame- works. Hence, the revised conceptual framework merges core purchasing areas with stakeholder behavior and health systems performance, proxied by intermediate and final outcomes. Figure 4 outlines the revised conceptual framework described above. Figure 4. A Conceptual Framework for Understanding Purchasing Changes Intermediate out- comes Purchasing ar- E.g., Utilization/Access rangement Quality Stakeholder engage- Efficiency ment and ` Health systems Governance UHC responses needs What to buy Resilience From whom to buy How to buy Provider Final User outcomes E.g., Financial risk pro- tection Health outcomes Source: Authors. Note: UHC = Universal health coverage Feedback relationships are likely between purchasing arrangements, stakeholder responses, and health system outcomes. For simplicity, the analysis only considers correlations between these aspects and cannot clarify the direction of changes outside cases when one event occurred before another. Below, the dimensions of the framework are described in more detail. 2.1.1. Purchasing arrangements The conceptual framework outlined in Figures 3 and 4 describe the following four functions of pur- chasing: “Governance” concerns the institutional arrangements for purchasing decisions. Where purchas- ing is strategic, these interactions are guided by clearly articulated health system goals, backed by an appropriate regulatory framework, and may involve consultations with relevant stakeholders. In addition, purchasing governance should be supported by data systems that can collect rich data on service use, quality, and population health needs for monitoring and accountability purposes. “What to buy” includes the specific goods and services purchased with available funds. Ideally, the included or excluded goods and services and the depth of coverage are explicitly defined. In 16 addition, stipulations on how services and medicines are accessed may be used as gatekeeping. For example, stipulations may encourage patients to access services at the primary care level in- stead of from the more costly inpatient level. “From whom to buy” refers to the providers contracted to provide services. Strategic purchasing involves selective contracting, which delineates which qualifications providers must hold and the performance standards they must meet to enter contracts for services provided. Regular review of providers’ qualifications and performance can hold providers accountable for delivering high-quality services. “How to purchase” involves the payment and monitoring mechanisms providers agree to when entering contracts with purchasers. Payment may be conditioned on whether a procedure is nec- essary for that condition. Additionally, payment mechanisms can be combined to achieve health system goals. Monitoring contract provisions may also introduce nonfinancial incentives to prioritize some services and report them diligently. 2.1.2. Provider and user engagement and responses The research captures potential responses by providers to purchasing changes by modifying service delivery inputs, organization, and financial management. These responses are constrained by the degree of provider autonomy and mediate the effects of purchasing on health system out- comes. Provider responses depend on their knowledge about the changes in purchasing, which are included in stakeholder engagement by the purchaser. Similarly, service user behavior may change in response to changes in purchasing arrangements, including benefits package coverage and selective contracting, which in turn affects health system outcomes, such as utilization rates. This report also explores service user engagement, and thus information on purchasing changes. 2.1.3. Intermediate and final health systems outcomes Among intermediate health systems goals are promoting equitable service use and financing, improving transparency and accountability, promoting quality, and improving efficiency in delivery and financing. In contrast, ultimate health systems goals promote financial risk protection, im- prove health outcomes, and enhance patient satisfaction. The study framework aims to link these goals or outcomes to purchasing arrangements via provider and service user responses. However, causal relationships are not specified as the pandemic may have affected these outcomes through different pathways. There is also the potential for reverse causality. 2.2. DATA COLLECTION AND ANALYSIS Data collection The researchers employed a mixed-methods approach and triangulated quantitative and qualitative data. Specifically, secondary data were collected from administrative sources and routine surveys. Researchers also undertook a scoping literature review and conducted key informant interviews. Figure 5 summarizes these sources, described in detail in Appendix A. 17 Figure 5. Methods for Data Collection and Data Triangulation Scoping review: Qualitative data collection: Quantitative data collec- Peer-reviewed papers Focus group discussions tion: and grey literature and interviews Administrative databases and surveys All framework All framework Health systems dimensions dimensions outcomes Source: Authors. Scoping review: Two researchers independently identified a total of 240 studies from Embase and Google Scholar using search terms based on dimensions of the conceptual framework. After screening the studies for relevance, 50 studies were selected for further review. Qualitative data collection: Twenty-four key informants were selected from Armenia and Roma- nia's purchasers, providers, and service users. Separate focus groups were conducted for each participant type. However, individual interviews were held for senior policy makers and in cases more data collection was needed following the group discussions. Interview guides were con- structed along dimensions of the conceptual framework. Appendix A includes a detailed description of qualitative data collection. Quantitative data collection: Data on health system outcomes for 2019 and 2020 were collected and disaggregated by gender and age where possible. Data were collected from administrative databases, including those hosted by the National Center for Disease Control (NCDC) and the National Institute of Health (NIH) in Armenia, the National Institute of Statistics (INS), the National Health Insurance House (NHIH), and the National Institute of Public Health (INSP) in Romania. The analysis included prepandemic and postpandemic comparisons, where possible, to assess disrup- tion resulting from COVID-19 in health system performance. Unfortunately, the emergency pre- sented limitations for collecting comprehensive data. Data analysis Findings from the literature review and qualitative and quantitative data collection were triangulated using thematic analysis. Initial coding was undertaken on data collected from the scoping review and the summary notes from the interviews and focus group discussions, organized in line with the framework. The second round of coding was undertaken to highlight the frequency of mentioned topics, tensions between the interviews and the review, and the extent to which study participants agreed on specific issues. The exercise was completed separately for both countries. 18 PART III: RESULTS This section reflects the study's findings. The first section includes country highlights, changes in purchasing arrangements, and provider and user responses to these changes. Then the associated differences in health system outcomes are described by comparing indicators before (2019) and during (2020) the pandemic. These changes were considered significant if the p-value of the test of difference in means was below 0.05. 2.3. ARMENIA 2.3.1. Governance of purchasing • Decisions were initially made at the federal level, with expanded fiscal space for health and purchasing power. • Technical gaps within health authorities, including the purchaser, limited the capacity to support the pandemic response. • There were significant electronic health information systems improvements, with some duplication and paper reporting. COVID-19 policy was made at the highest levels of government. The Commandant’s Office was formed and granted authority in March 2020 to implement response measures across sectors. It was led by the deputy prime minister and included policy makers from the Ministries of Finance, Health, and other sectors (European Observatory on Health Systems and Policies 2021). However, the Commandant’s Office was disbanded after the State of Emergency was lifted in September 2020. At this point, the Ministry of Health (MoH) was tasked to lead the COVID-19 response, in- cluding purchasing (Nalbandian 2020). Significant domestic funding was mobilized for the response. Due to the State of Emergency, the government allocated dram 150 billion ($300 million) to mitigate the health and social conse- quences of COVID-19 without Parliament’s consent. Previously, parliamentary approval was nec- essary for any reallocation that exceeded 3 percent of the budget. This adaptive measure expanded budgetary space for the pandemic response in the face of significant social needs. The funds mo- bilized included direct spending, state-sponsored loans, and investments (European Observatory on Health Systems and Policies 2021). Additional external funding expanded fiscal space further. Armenia obtained donor grants to complement government funds (Thomson and Habicht 2020). Donor contributions were also in- kind, including ventilators, echocardiography, pulse oximeters, and other medical equipment. The combined increase in domestic and external funding for social services, including health, translated to higher purchasing power in the public sector. Furthermore, the provision of supplies and equip- ment helped offset the revenue losses from reduced utilization. 19 “Comparing the budget today and a year ago (before the pandemic), it is the same. Before the pandemic, the hospital could offer services for fees. These tariffs constituted a large portion of the profits. However, during the pandemic, [the medical center] was deprived of such services. Hence, it was compensated by the MoH in the form of renovations and new equipment.” Public health care provider, Armenia While there was higher purchasing power in the health sector, human resource constraints limited the purchaser's ability to respond effectively. There were shortages of technical and administrative staff in the MoH and the State Health Agency (SHA). These staffing shortages led to overwhelming workloads and staff burnout. In some cases, new positions were created and filled. However, staff in the SHA reported that despite the emerging needs, no additional hiring occurred. “The Economic and Financing Department [of the MoH] … is functioning as usual. However, the Health Care Policy Department was short of staff before the pandemic. That deficit worsened during the pandemic. However, some positions have been filled.” MoH staff, Armenia “No changes in expertise involved in purchasing occurred at the SHA during the pandemic. However, the staff worked under more stress and for longer hours.” SHA staff, Armenia Adaptations in the health information system facilitated purchasing and service delivery. For example, new modules were added to ArMed to track COVID-19 tests, hospitalizations, isola- tions, and vaccinations. These figures were published daily in statistical dashboards on the MoH’s and NCDC’s websites and informed changes in the pandemic response. Clinicians also used data in ArMed to identify patients for close monitoring in their catchment areas. Furthermore, an app was launched for patients to document their vaccination status (European Observatory on Health Systems and Policies 2021). As a result, there were reported improvements in the quality of data and its use to support decision making. “During the pandemic, a unified database was introduced into ArMed, showing COVID test results in real time. Now, vaccination data are shown in the same manner. Also, the hospitals could electronically receive reports on their performance within minutes.” Public health care provider, Armenia “ArMed underwent software developments and was upgraded with new servers. It informs the whole process, from case identification to reimbursement. COVID-19 test results are shown in ArMed, and the primary care doctor sees if a person in their catchment area has COVID-19. They can begin the treatment accordingly.” MoH staff, Armenia 20 Duplications in reporting contributed to inefficiencies in purchasing. For example, ArMed monitored COVID-19 cases, care, and reimbursement through the SHA. However, the actual pro- cessing of some expenses, such as maintenance, required a separate electronic or paper-based reporting mechanism via the MoH. “ArMed is being used to track the epidemiology of COVID-19 in the country. However, for reimbursing the facilities financed through maintenance costs, their reporting and reimbursement were not conducted using ArMed. Instead, those facilities fill in and separately submit a report of their actual expenses directly to the MoH, based on which they are reimbursed.” MoH staff, Armenia 2.3.2. What to Buy • The benefits package was expanded to include care for COVID-19. • For non-COVID-19 care, some services were deprioritized, and mechanisms were in- troduced to maintain essential care access. The benefits package was expanded to include care for COVID-19, including vaccines, accessible and free of charge to the entire population. Noncitizens could also access these services. In addition, the reimbursed services followed new national guidelines. Primary health care providers were responsible for sampling, referral testing, and counseling newly diagnosed patients on treatment. Hospitals initially treated all cases. Later, hospitalization was restricted to specific situations involving the elderly, pregnant women, people living with comorbidities, and complica- tions (such as severe pneumonia or multi-organ failure). “Since the government covers COVID-19 care, the BBP [Basic Benefits Package] and the medicines list were changed. As a result, COVID-19 patients received free care regardless of nationality and medicines necessary for treatment.” MoH staff Some non-COVID-19 care was deprioritized, and mechanisms were introduced to maintain essential care access. For example, elective surgeries were canceled. Also, remote consultations were held by phone or video, such as for directly observed treatment for tuberculosis. However, telemedicine was not formally reimbursed by the SHA, and these consultations were not routinely monitored in health information systems. When the demand for COVID-19 care fell, facilities in- creased access to elective services. For example, after August 1, 2022, medical centers in the municipalities of Vedi, Vanadzor, and Dilijan resumed regular care. 21 2.3.3. From Whom to Buy • The MoH designated specific facilities for COVID-19 services and equipped them to support intensive care. • Recruitment, incentives, and training were used to fill worker shortages in specialties necessary for COVID-19 care. • Mechanisms to facilitate the patient's choice of provider were also introduced. COVID-19 care was reimbursed via the state in specific facilities identified in ministerial or- ders. The number of facilities varied over time, up to 21 public and six private hospitals. By March 2021, bed capacity had doubled, but cases still needed hospitalization (Figure 6). For the public secondary and tertiary care hospitals designated for COVID-19 care, operational plans were devel- oped to repurpose them accordingly. This process involved ensuring adequate supplies and equip- ment for case management, including intensive care. In addition, medical centers that aimed to be reimbursed for COVID-19 treatment could submit applications to the MoH if they met the specified standards. Armenia filled health worker shortages for high-demand specialties, particularly in intensive care and anesthesiology. Approximately 4,500 health workers were identified to provide COVID- 19 care as of November 2020. These included physicians, nurses, medical residents, medical stu- dents, and other junior staff who volunteered to serve, including from other countries (France, Ger- many, Italy, Lithuania, Russian Federation, and United Kingdom) (WHO 2020a). An additional 1,500 Armenian Red Cross volunteers helped launch a national public awareness campaign (Eu- ropean Observatory on Health Systems and Policies 2021). The National Institute of Health trained the recruited staff before their involvement in care for COVID-19. In addition, bonuses were made (see below), labor conditions were improved, and new positions were opened to help close labor supply gaps. “Currently, the staff is returning. During [the pandemic], recruits such as volunteers, other specialists, and residents were contracted to help due to shortages.” Private health care provider, Armenia Other reforms introduced during the pandemic strengthened the link between patient pref- erences and provider selection. For example, regulation was introduced to allow patients to “vote with their feet” for some non-COVID-19 care, such as cardiac or obstetric care. This initiative shifted payment systems at the primary care level from capitation based on living in a facility catchment area to capitation based on the needs of the individual (Chukwuma, Meessen et al. 2020). Further- more, in September 2021, a free app was introduced to enable users to select registered doctors, consider reviews by other users, and rate the physician's strengths and weaknesses. “The MoH and SHA also introduced a new purchasing mechanism in 2020 called the ‘Oferta’ contracts. The new mechanism is conducted through posting a service delivery contract over the Internet for some services, mainly for military personnel, heart conditions, and obstetric care. Providers express their interest and start offering them following a ministerial order. The Oferta contracts are open contracts, and any provider can apply.” SHA staff, Armenia 22 2.3.4. How to Buy • Facility reimbursement mechanisms temporarily shifted from case-based reimburse- ment to prospective payments. • Payments to health workers increased through bonuses and salary adjustments. Payments were made to the 20 public hospitals providing COVID-19 care by the MoH to cover fixed and variable costs. These prospective payments temporarily replaced output-based reimbursements and ensured a flow of funds despite service disruptions. These transfers aimed to avoid “long difficult price calculations.” The funds were calculated based on the hospital’s capacity to cover utilities, medicine, food, equipment, and facility upkeep. In addition, due to their specialized care capacity, hospitals in Yerevan were reimbursed at higher rates. This payment mechanism was discontinued in the fall of 2020. Since then, reimbursement has transitioned back to case-based payments for the 10 private hospitals contracted by the SHA for COVID-19 care. There is no limit to the number of COVID-19 cases that can be reimbursed, and some other services, such as hu- man immunodeficiency virus (HIV) care, also had annual limits removed. Furthermore, there were special rates for services for children and pregnant women aimed at preventing their neglect by busy providers. “When COVID-19 cases fell, the maintenance costs reimbursement for hospitals were suspended, with a transition to case-based financing. In my opinion, comparing the two rates, the latter was very low. The maintenance cost covered included salaries, utilities, medication, and administrative costs. There was no limitation in reimbursement in the maintenance cost method.” Public health care provider, Armenia The MoH also centralized purchasing COVID-19 supplies for inpatient and primary care fa- cilities, including the private sector. For example, one private provider reported receiving new oxygen generators, X-ray machines, and CT scanners. In addition, some supplies were donated, including COVID-19 tests from the WHO and the Armenian Relief Fund (European Observatory on Health Systems and Policies 2021). Health workers received bonus payments and increased salaries during the pandemic. In March and April 2020, bonuses were provided to doctors, nurses, and ambulance drivers. In addi- tion, some volunteers recruited for COVID-19 care were compensated through a civil service con- tract that paid them under the same conditions, including via salary, as full-time providers. In June 2020, medical staff salaries increased two to threefold (European Observatory on Health Systems and Policies 2021). These salary adjustments created disparities in payments to providers who treated COVID-19 and non-COVID-19 care. Hence, in some facilities, salaries were raised for non- COVID-19 providers as well. “Since the salaries of physicians who treat COVID-19 were substantially higher than their normal salaries, it was an incentive to recruit and retain them. However, it was an issue for medical facilities since all the other doctors started complaining about their low salaries and demanded raises. Hence the doctors' salaries in Nairi Medical Center were raised by 20 percent.” Private health care provider, Armenia 23 2.3.5. Stakeholder engagement and responses • There was no systematic reported engagement of stakeholders on purchasing changes. • Provider autonomy and decision space were reduced as the MoH assumed oversight of facilities. The COVID-19 response was not associated with better stakeholder engagement in the health sector. Decision making in the coordinating body for the COVID-19 response involved sig- nificant discussion across policy-level stakeholders. As a result, these policy changes and resource allocations were intentionally publicized. Service users reported being informed of policy changes, such as covered services for COVID-19. However, in many cases, this information was received from informal sources, such as family and friends. There was also a lack of clarity on the require- ment for copayments. For example, some participants believed that outpatient care required fees, while inpatient care for COVID-19 was free. Respondents did not report formal forums or commu- nication with policy makers regarding stakeholder consultations for purchasing changes impacting providers and patients. The emergency response was associated with decreased provider autonomy and decision space. The Legal Regime of the State of Emergency authorized the government to transfer the management of all medical facilities in Armenia to the MoH, regardless of their affiliation or owner- ship (European Observatory on Health Systems and Policies 2021). As a result, the MoH assumed oversight of public hospitals previously owned and operated by Marz (provincial) governments. Private hospitals designated for COVID-19 care were contracted through the SHA. According to a purchaser in the MoH, this allowed for the central coordination of COVID-19 care. Furthermore, this centralized coordination allowed the MoH to maintain non-COVID-19 services by designating facil- ities to provide these services, even if they had not in the past. In addition, selected facilities were also directed to provide medical care for those affected by military activity in October 2020. 2.3.6. Health system outcomes • Financial risk protection improved, but the changes in health outcomes were mixed. • Health care use decreased, with mixed findings on access and quality of care. Changes to mortality were mixed, but financial risk protection improved. Between 2019 and 2020, monthly deaths due to ischemic heart disease significantly increased from 6,386 to 8,743, but maternal mortality rates did not change. However, by December 2020, Armenia recorded over 1,000 deaths per million due to COVID-19. This figure was higher than the global and upper-middle- income averages of 240 and 277, respectively (Our World in Data 2023). However, there was an apparent improvement in measures of financial risk protection. Between 2019 and 2020, health spending as a proportion of total household health expenditure fell from 7.2 to 3.2 percent. The population eligible for state funding for health care rose by nearly 9 percentage points, and the percentage of extremely impoverished households who reported the cost of services as the main reason for not using needed care fell from 49 to 33 percent (ARMSTAT, The World Bank, 2020) (ARMSTAT, The World Bank 2021). In addition, respondents reported lower informal 24 OOP payments for health care in the study interviews. One public health care provider noted that any OOP payments made for non-COVID-19 services were declined, especially inpatient care. This finding was attributed to facilities receiving extra funds, so the prices were revised to market value. A few users reported being required to make OOP payments for diagnostics, testing, and medicine linked to COVID-19 care. Sometimes, patients sought paid services for subjectively higher-quality COVID-19 care. "For COVID-19 care, it was free of charge. So, no OOP payments were asked of patients. However, for non-COVID-19 cases, OOP payments have decreased. This experience could be explained by the additional financial resources included in the health care system. Also, there was better coordination of funds and a revision of services prices based on market actual prices.” Public health care provider, Armenia Health care utilization rates declined between 2019 and 2020, and subjective health care access assessments were mixed. For example, the average monthly measles vaccinations fell from 73,317 to 69,308. Hospital bed occupancy declined, but the change was not significant. How- ever, there was a reported increase in consultations via telephone and video for services that did not require in-person interaction. In addition, participants reported improved primary care access due to less crowding at facilities, home visits, and telephone access, including after working hours. During the pandemic, one patient noted that care was not interrupted for a non-COVID-19 condition and commended the coordination between physicians. Nevertheless, in other cases, continuity of care was reportedly interrupted for non-COVID-19 ser- vices, such as in cases where health workers were reassigned to COVID-19 care. There was no information on utilization or other access measures disaggregated by income status or different social groupings, which may have served as an objective proxy for equitable access to care. While there were interventions to improve quality, anecdotal evidence is mixed. Quality im- provement interventions included training providers in home-based care and pneumonia manage- ment for COVID-19 patients. This was led by the WHO and a partnership between New York Uni- versity and Armenian radiologists for image interpretation. Also, to provide care for COVID-19, fa- cilities were required to meet hygiene standards and have adequate human resources and beds. Furthermore, in a 2020 survey, over 80 percent of respondents indicated some degree of trust in the health care system, a subjective quality proxy (The World Bank, CRRC - Armenia 2020). However, respondents noted that the reallocation of health workers and supplies to COVID-19 care may have reduced the quality-of-service delivery for non-COVID-19 care. Also, there were reports that some volunteer medical staff provided substandard COVID-19 care. Unfortunately, no other data were available on the quality-of-service delivery for COVID-19 and non-COVID-19 care. “Unfortunately, many volunteer doctors are not qualified to treat COVID-19. Also, PHC (Primary Health Care) doctors have needlessly prescribed many antibiotics to COVID-19 patients.” Public health care providers, Armenia 25 Time line: Figure 6 illustrates selected milestones in the Armenian health system’s response to the COVID-19 pandemic Figure 6. Health System Response to the COVID-19 Pandemic in Armenia Medical staff salaries in- Bonuses provided to creased by 2-3x doctors, nurses, and Opening of applications to ambulance drivers the MoH for medical cen- ters to provide COVID-19 care Call for clinical and nonclinical volunteers Source: Elaborated by authors. Note: ICU = Intensive care unit; MoH = Ministry of Health. 26 2.4. ROMANIA 2.4.1. Governance of purchasing • The MoH centralized coordination, including regulatory and budgetary changes, to en- sure continuity of care for patients and to support the needs of providers and hospitals to combat COVID-19 • Budgetary allocations were increased for health, funded via state and loans, despite persistent budgetary pressure at the service delivery level • Pandemic needs spurred innovation in health management information systems and telemedicine MoH Operational Coordination Center led COVID-19 decision making (Petcu et al. 2020). The center also provided information on the real-time stocks of medicines, medical equipment, beds, personnel, and disinfectants nationwide in hospital units. MoH extended provider payment mechanisms and benefits, ensuring continuity of care. The MoH extended the Framework Contract beyond its initial expiration date of March 31, 2020, for all providers, adding additional amendments to adjust to the pandemic and avoiding disruptions (Preda 2020). Amendments included the removal of caps on the number of consultations delivered by PHC providers, including telemedicine consultations, and changed hospital payments to no longer be conditional on achieving target volumes. However, in 2020, payments for non-COVID-19 hospitals were based on invoices for delivered services and up to the estimated budget for hospitals operating at capacity, whereas COVID-19 hospitals had no restrictions. This action allowed the continuation of reimbursement mechanisms for care in the health system at all levels. Extensions occurred until July 2021, when a new Frame- work Contract was established (Hintea, Ticlau, and Neamtu 2020). Budgetary allocations were increased for health and funded via state loans to provide hos- pital services and reduce the transmission of COVID-19. On April 17, 2020, an Emergency Ordinance increased the public health budget by about 808 million euros and allocated 594 million euros to the social health insurance fund and the remainder to health worker bonuses and equip- ment purchases. Additionally, the government negotiated donor grants to complement further funds from public sources (WHO 2020a). New functionality for health reporting was introduced, including coding systems for mortal- ity and COVID-19-related health outcomes. Data collection platforms such as Coronaforms and AlerteMS were introduced. However, reporting methods differed between platforms and the sys- tems used by the INS (Salceanu, 2021). The Association of Family Physicians Bucuresti-Ilfov main- tained resources such as www.atelieremedicale.ro to facilitate COVID-19 information exchange through online symposiums and webinars. Also, the National Electronic Vaccine Registry (RENV) was updated to include COVID-19 vaccines for adults (Dascalu 2020). Pandemic needs spurred innovation in health management information systems, telemedi- cine, and health communication, but improvement is needed. Establishing a monitoring plat- form and conceptual program for telemedicine used data storage, text messages, and a freely available video application (Doica et al. 2021). However, despite demonstrating promise, the health 27 system was slow to adapt to new resources and communication channels, especially during an ongoing health crisis (Cernicova-Buca and Palea 2021). "There were some changes—a little digitalization, especially for general practitioners. If you want to receive a referral online, in the past, that was not possible. But now this has changed." A patient diagnosed with COVID-19, Romania 2.4.2. What to Buy • Benefits included free access to COVID-19 care and certain telemedicine services, regardless of insurance status • Gatekeeping mechanisms were relaxed to reduce the need for in-person contact, especially for chronic disease management All COVID-19 services are free to all persons in Romania, even those not paying contributions to the national health insurance scheme (Scintee and Farcasanu 2020). Essential service packages included services related to the pandemic and were made available to the uninsured population. However, certain services, such as COVID-19 testing, require a physician's recommendation or were otherwise paid for out-of-pocket. The MoH permitted outpatient medical consultation through telemedicine for everyone, re- gardless of insurance status (Florea et al. 2021). In April 2020, Romania established a legal framework for telemedicine that allowed certain medical specialists to serve remotely. These enti- ties included the Romanian Association of Psychiatry and Psychotherapy, the Romanian College of Physicians, the MoH, the National Insurance House, and other eligible institutions and authorities (Băcilă and Anghel 2020). Before the COVID-19 pandemic, regulations stipulated that every patient had to be evaluated face-to-face by health professionals. “Regarding the Ministry's duties on purchasing arrangements, there is the same legislation in this area that governed both in the prepandemic period and which still governs today (Law No. 98/2016 on public procurement and Government Decision no. 395/2016 on the methodological rules governing the application of this law), with the specification that during the period of the emergency state, the emphasis was placed on those provisions of the law that facilitate public procurement of goods and services dedicated to combating COVID-19.” Participant, MoH, Romania Gatekeeping mechanisms were relaxed to reduce the need for in-person contact, especially for chronic disease management. These regulations expanded the role of the family physician, allowing these practitioners to prescribe drugs for chronic illnesses with only a letter from a special- ist, and did not require regular reevaluation. Extensions for referrals and medical recommendations, and evaluations were granted. Additionally, all contracting documents with health service providers and the District Health Insurance House (DHIH) were extended to continue purchasing arrange- ments (Scintee and Farcasanu 2020). However, one provider expressed confusion about working with the NHIH on new contracting practices. 28 "We have a new contract process in place, but we have unclear issues about how to move forward with the payments in specific circumstances. We have been proactive and initiated the conversations with the NHIH. Providers are still waiting to work jointly with NHIH in this matter. We also have complaints about the procurement process. However, we have not been involved in the decision- making process for the selection of suppliers. We have not been consulted either on this!" Health care provider, Romania 2.4.3. From Whom to Buy • Medical personnel shortages were addressed by hiring, retraining, and repurposing personnel • Testing capacity increased through partnerships with privately owned laboratories The MoH addressed medical personnel shortages by approving additional positions, retraining personnel, and repurposing family physicians. In April 2020, 2,000 additional positions were ap- proved within the District Public Health Authorities (DPHAs) and ambulance services. This amend- ment was extended twice, through October 2020, but data are missing on the number of positions filled (Hintea, Ticlau, and Neamtu 2020). In addition, some specialized medical personnel, such as primary care physicians, specialists, and residents, were retrained to treat COVID-19 and assigned to other public health units with personnel shortages (Ohanyan et al. 2015). The MoH also provided the necessary medical personnel at bor- der crossings, with specialists from the DPHA (Petcu et al. 2020). Although family physicians were not initially involved in COVID-19 care, many were tasked with monitoring mild cases or assisting with vaccination efforts as case numbers grew (Scintee and Farcasanu 2020). Private medical laboratories offered COVID-19 testing. An order from the MoH added several private facilities to the list of those contracted to perform COVID-19 polymerase chain reaction PCR testing. Since this decree, testing capacity has increased from 7 testing centers in March 2020 to 176 in January 2021 (Scintee and Farcasanu 2020). Phase I and Phase II hospitals split COVID-19 care. Phase I facilities are a network of infectious disease hospitals. The DPHA coordinates these hospitals and may initiate a transfer of patients if one facility is overwhelmed/overcapacity or if a patient requires specialty care. Phase II facilities are equipped with intensive care units and appropriate COVID-19 equipment and can take on ad- ditional COVID-19 patients once Phase I facilities have reached capacity. In April 2020, 16 hospitals were designated as Phase I or II, specifically to treat COVID-19 patients, and 80 were listed as "support hospitals" to treat both COVID-19 and non-COVID-19 cases. By the beginning of 2021, this increased to 63 public hospitals designated as Phase I or II and 180 desig- nated for support. These facilities were identified by Ministerial Order no. 555/2020, and its amend- ment no. 623/2020 specifies the hospitals and facilities providing COVID-19 care in Phases I and II (Government of Romania 2021). 29 2.4.4. How to Buy • State of Emergency declarations and subsequent ordinances allowed for centralized procurement of emergency medical supplies • Hospital-based COVID-19 care was fully reimbursed • Telemedicine for primary care use and reimbursement is expanding • Bonus payments were provided to COVID-19 medical providers The Ministry of Internal Affairs (MoIA) centralized procurement for personal protective equipment (PPE) and sanitation supplies. The MoIA coordinates all emergency responders in the country, including hospitals, emergency rooms, ambulances, and paramedic services (Government of Romania 2020). Following the state of emergency declaration on March 16, 2020, and the first measure, Ordinance 11, in February 2020, the Romanian National Office carried out centralized purchasing of emergency medical equipment and supplies. This process also included a series of statutes issued as government emergency ordinances meant to provide additional supplies to the health system to support medical personnel (Mirica et al. 2020). Amendments to procurement regulations permitted district health authorities to procure medical equipment and expedite acquisition (Government of Romania 2020). Local procure- ment units were allowed to procure COVID-19-related equipment and medicines above maximum prices using MoH funds. To access MoH funds, those involved in procurement were obliged to justify the purchases upon approval, these funds removed price barriers to acquiring necessary supplies (Petcu et al. 2020). Hospital financing changed to full reimbursement. Before the pandemic, hospitals were fi- nanced through output-based reimbursement. During the pandemic, the DHIH disbursed funds to cover all hospital expenditures up to the amount calculated for operating at maximum capacity. Hospitals that treated COVID-19 patients could exceed this limit if they were over capacity. Telemedicine was recognized as a reimbursable service by the National Health Insurance House when used to provide primary care. Almost 92 percent of general practitioners surveyed supported continued telemedicine reimbursement beyond the pandemic. Since the government has decided to scale up telemedicine in Romania, it is partnering with the World Health Organization to launch a pediatric phone triage service (Florea et al. 2021). Bonus payments to COVID-19 medical providers compensated for increased risk and addi- tional services outside the providers' usual scope of practice (WHO Europe 2020). Health workers involved in COVID-19 care were eligible to receive ~500 euros. The government paid this sum with support from the European Commission through unspent prefinancing for the European Structural and Investment Funds from 2019 to 2020 (Scintee and Farcasanu 2020). Over 60,000 health workers also received a bonus of 30 percent of their base salary. State-affiliated health workers were eligible for a risk allowance ranging between 75 percent to 85 percent of their base salary, depending on the health care setting. In addition, community nurses received a base risk allowance of lei 2,500 and free PPE. Family physicians were reimbursed for remote monitoring of COVID-19 cases at le 105 per patient and entered separate monitoring contracts with DHIHs in October 2020 (Government of Romania 2021). However, some bonus payments had the unin- tended consequence of demotivating health workers who were not involved in providing COVID-19 care. 30 "Before the pandemic, after receiving the financial incentives authorized for the staff during the emergency, many staff became unmotivated due to financial differences in the amount. For instance, staff not involved in hospitalization and ICU care were assigned the same amount as staff in other departments without any direct COVID-19-related care. Another category of staff involved in the diagnosis, treatment, and transport of patients was not included in this scheme of financial incentives. This was unfair." Health care provider 2, Romania 2.4.5. Stakeholder engagement and responses • Prior to the COVID-19 pandemic, some physician’s associations advised on purchasing decisions • Neither patients nor providers were consulted on COVID-19 purchasing decisions Stakeholder involvement in policy making was varied. Providers acknowledged that some physician's groups and associations were involved in the purchasing decisions during non- COVID-19 times. However, the research did not indicate that their expertise was consulted during the pandemic. In addition, purchasing decisions did not consider patients' perspectives. "The GP was not consulted for any changes in purchasing arrangements made by the MoH, the NHIH or DHIH in 2020–21. Nonetheless, in previous years (up to 2019), [he] was involved in most discussions with the government." Health care provider in a rural area, Romania "The ideal structure would allow us (patient associations) to be involved since the beginning when legislations are being discussed. To be included in the specialized commissions. We have expertise in many domains, including law, economics, medicine, and IT [information technology], that would be very valuable to these discussions. However, our inputs need to be considered prior to the formulation or drafting of official documents." Patient 4, Romania 2.4.6. Health system outcomes • Inconsistent data collection may mask the true impact of COVID-19 on health status of the population • Financial risk protection improved for all needing COVID-19 care There were variable outcomes to health status between 2019 and 2020. Following the onset of the COVID-19 pandemic in Romania, there was no significant change in maternal mortality. How- ever, ischemic heart deaths increased from 6,386 to 8,743 deaths per month (p < 0.05). By the end 31 of December 2020, Romania recorded nearly 800 deaths per million due to COVID-19 (Our World in Data 2023). This figure was higher than the global and upper-middle-income averages of 240 and 277, respectively. There was also a substantial decrease in the percentage of children receiv- ing the first dose of their vaccine schedule, from 89.5 percent to 87.3 percent. There was also a decrease in the percentage of children with two doses of MMR (Measles, Mumps, and Rubella) vaccines, from 75.8 percent to 75.1 percent. It is important to note that data availability at the time of the analysis limited the scope of this anal- ysis. Financial risk protection improved for all needing COVID-19 care. All COVID-19 services were free for all persons in Romania, even those not enrolled or contributing to the National Health In- surance Fund. No copayments were required for testing (according to protocols) or vaccines, and no copayments were charged for services falling within the competence of the MoH. However, financial access to some non-COVID-19 care services still needs to be improved for the uninsured population. Participants reported the out-of-pocket payments with specialists and online family medicine. Interventions to improve utilization rates and the quality of health care services produced unclear results. The use and subsequent telemedicine coverage under the national health insur- ance scheme permitted continuity of care for COVID-19 and non-COVID-19 patients. Additionally, there were parallel incentives to increase health care worker supply by hiring 2,000 additional med- ical personnel and offering bonuses meant to ensure access to care; however, no data are available to confirm the number of personnel hired. Time line: Figure 7 illustrates selected milestones in the Romanian health system’s response to the COVID-19 pandemic. Figure 7. Health System Response to the COVID-19 Pandemic in Romania Family physicians paid Lei 105 per patient for online monitoring of asymptomatic or mild cases. State of Emergency extended tele- medicine regulations. Monthly COVID-19risk payment of Lei 2,500 was provided to health workers Source: Elaborated by authors. 32 PART IV: DISCUSSION This report examines adjustments made in purchasing arrangements as part of the Armenian and Romanian response to the pandemic. The learnings from these two case studies may inform on- going efforts to leverage strategic purchasing to strengthen health outcomes and health system performance in the region and beyond. Although the two countries have different health systems and forms of governance, during the COVID-19 pandemic, they shared the same ultimate objective of maintaining high-quality COVID- 19 and non-COVID-19 care in both inpatient and outpatient settings. The achievement of this goal is mediated by the country’s governance and purchasing functions, which are reflected by their intermediate objectives: (1) maintaining access to essential health services despite restrictions in seeking in-person care, and (2) providing COVID-19 services at no cost to the entire population. The above evidence indicates that the following three factors should be considered when deter- mining potential purchasing changes in response to health system shocks. First, an enabling governing environment, requires purchasing decisions to be made at the appropriate level. In both countries the research found that the high level and centralized coordi- nation and additional fund flows increased purchasing power. For instance, in Armenia, the gov- ernment involved the main stakeholders in the decision making and implementation of purchasing changes, made consultations, and fostered collaboration with health care providers. Furthermore, both countries, but mainly Romania, implemented emergency regulatory changes. Given significant purchaser influence in Romania, there was no need to commandeer facilities. Although changes varied significantly across countries, leadership was combined with coalition- building, regulation, attention to system design, and accountability among stakeholders. Health systems in Armenia and Romania centralized coordination for COVID-19 response. Romania lev- eraged the existing MoH authority to manage hospitals, and Armenia issued mandates to coordi- nate all hospitals providing COVID-19 care under the MoH. This enabled governance was particu- larly relevant for strategic purchasing, and it facilitated the operationalization of benefits, including telemedicine services, COVID-19 medicines and supplies, and management strategies for chronic diseases. Next, targeted changes in benefits, payment incentives, and contracts to respond to emerg- ing needs represent the potential of strategic (versus passive) purchasing. In both countries an expanded health benefits package, including COVID-19 care and hospitalization, was provided to the entire population. In Armenia telemedicine for COVID-19 within essential care was not reim- bursed, whereas in Romania it was. However, users in Romania reported paying to cover COVID- 19 testing and medicines. Regarding design of financial incentives, there were payment incentives for COVID-19 cases in primary health care, such as bonus payment, bonuses for additional work- load, increased salaries, and reimbursement to the private providers through case-based, contract, and accreditation of the COVID-19 health service providers (coordination of public and private hos- pitals). Additionally, both countries paid hospitals through full reimbursement mechanisms during the height of the pandemic, temporarily supplanting output-based payment mechanisms. Payment methods were based on the needs and circumstances of the providers and population. Regarding 33 contracting processes, fund flows to facilities (full reimbursement in Armenia, lifting limits on cases in Romania, and covering costs of key inputs in both countries) increased the supply of health care services, extended contracts, and improved the accreditation process. Finally, the introduction of agile innovation methods supported the delivery of COVID-19 care. Purchasing functions were an enabling innovation factor during the crisis. In Armenia and Romania, social distancing and restricted movement limited in-person consultations for primary care and other outpatient services. Consequently, this contributed to the purchase of more robust information systems, new frameworks for telemedicine and testing capacity, and new payment mechanisms. The rapid introduction of this innovation demonstrates the importance of an enabling governance environment, which is imperative for supporting a new legislative framework and man- aging data collection and digital solutions. This also includes alignments in payment methods to keep incentives aligned with purchasing decisions within the country’s COVID-19 responses. The WHO Regional Office for Europe and the European Commission contributed timely evidence on health systems innovations in the European region. The European Observatory addressed gov- ernance, paying for services, effective provision of health services, ensuring sufficient workforce capacity, and preventing transmission. This study is complementary in providing more evidence from Armenia and Romania and suggests that during the pandemic, health systems have witnessed significant transformation in PHC services across Europe. Data availability was a significant limitation that may have affected the internal validity of this study. Due to the fragmented nature of data sources across various health institutions in both countries, health data may be out-of-date or unavailable. Data collection and availability may have also been interrupted by the COVID-19 pandemic and emergency response. Additionally, only a small sample of individuals was chosen for qualitative interviews, which may not be fully representative of the entire population. To overcome these limitations, this study used a mixed-methods approach, which incorporated qualitative findings from participant interviews with quantitative data from peer-re- viewed research to ensure that the findings presented are reliable. Additionally, this study can make no causal argument about the purchasing changes made during the COVID-19 pandemic and health system outcomes in Armenia and Romania. This is due to the lack of high-quality quantitative data available on relevant health system indicators and a lack of data on possible confounding variables, such as the reactions of other government sectors to COVID-19. Instead, this study relied on the conceptual framework to understand the provider and user responses to purchasing changes. This study may serve as the basis for other countries to investigate the effects of health purchasing changes on consumer and provider behavior. Finally, these findings may not be externally valid. The limitations mentioned above, along with the specific nature of the Armenian and Romanian responses to the COVID-19 pandemic, prevent the results of this study from being generalizable and applicable to other countries. However, a com- parison may be possible after considering other important contextual factors aside from the struc- ture of the health systems. Political, ethical, and social factors, priorities, and technical ability may influence purchasing decisions and consumer behavior during a health crisis such as COVID-19. Future research should focus on assessing the impact of certain purchasing changes and gathering evidence for sustainability. Purchasing adjustments described in this report demonstrate that a range of services can be available to the entire population at no cost and that health systems have the operational capacity to provide high-quality services. However, further research is needed to understand the financial sustainability of these practices, which of the changes can be sustained, 34 and how. Also, an impact evaluation should be performed to determine the causal effect of pur- chasing changes, like full reimbursement, on health system outcomes. Further research is needed to fully understand and assess how purchasing changes can be synchronized with specific health services and sustained in the long term. Finally in an emergency context, it is imperative to act quickly and consider the local context. To promptly adjust purchasing mechanisms in response to a public health emergency, identify mech- anisms to increase financial resources for increased service purchasing, and designate the pur- chasing body or another administrative group to communicate centrally and monitor service deliv- ery. Additionally, other factors outside of the health system may impact the availability and quality of services, and implications may differ based on their context. For example, armed conflict in Ar- menia and an unstable political environment in Romania are externalities to consider within these purchasing decisions. The research described in this report is the result of a systematic analysis of the purchasing changes made in Armenia and Romania in response to the COVID-19 pandemic in 2020. The findings indicate three factors that influenced health care purchasing decisions in the countries under study. First, intergovernmental cooperation is of paramount importance to effectively coordi- nate and disseminate essential services to key populations. Second, while purchasing tools were adapted for an agile pandemic policy response, guidelines need to support these changes. Finally, the report highlights the potential for changing needs and a crisis to spur innovation in purchasing and service delivery. Moving forward, it is essential to assess the sustainability of these payment adjustments over time and the implications for health system performance. The ability to apply these learnings to purchasing adjustments during future emergencies makes all systems more re- silient, effective, and inclusive. 35 PART V: APPENDIX 2.5. APPENDIX A. DETAILED METHODS FOR DATA COLLECTION AND ANALYSIS 2.5.1. Scoping literature review This study employed a scoping review methodology to provide an overview of the available re- search. The objectives of this review were the following: 1. Identify relevant and high-quality published studies, reports, and data examining the changes in purchasing arrangements. Identify the associations between purchasing changes and impacts on health systems outcomes (COVID-19 and non-COVID-19 health care). 2. Provide context for changes, driving forces, and key mechanisms in purchasing arrange- ments during the pandemic. 3. Demonstrate knowledge of current literature in the field. 4. Highlight any gaps in the existing literature and identify how this research fills those gaps in knowledge. The search strategy was limited to English-language literature using two databases: Embase and Google Scholar. This study employed two searches. An initial search in May 2021 and a second literature review in June 2021 to address previously uncovered topics. This search strategy was adapted appropriately to the wildcards, the syntax of exact keywords, and adjusted to the individual databases' requirements. Table A.1 presents the search terms in May 2021, including MESH terms and other relevant words and expressions using Boolean logic ("AND" or "OR"). Table A.1. Literature Search Criteria Outcome/ Ovid databases (Embase) Google Scholar Database Changes The search strategy covered all the keywords Advanced searching allowed researchers in pur- and subject headings or combined with “AND” to limit the search to specific fields (title, chasing 1. All the keywords and subject headings for author, journal, and date), but not to title, purchasing, providers, and patients/users abstract, and keywords fields only. combined with “AND.” 2. All the keywords and subject headings for 1. Purchasing changes; purchasing ad- coordination, public involvement, infor- justments; health care provision mation systems, roles, stakeholder, per- changes; “patient engagement" or formance system, staff communication "the changes in purchasing arrange- combined with “AND.” ments during the pandemic in Arme- 3. All the keywords and subject headings for nia." AND. benefit package, services covered, medi- 2. "Purchasing| COVID-19 |Armenia" All cines, vaccines, medical device reim- in the title: Limit your search to terms bursement, cost-sharing, copayment, clini- appearing in the title only, e.g., cal guidelines combined with “AND.” "SARS-CoV-2." 4. All the keywords and subject headings for 3. Filetype: Limit your search to specific payment methods, mix-payments, rates, file types by using filetype: or ext: negotiations, centralized purchasing com- e.g., "SARS-CoV-2" filetype: pdf. bined with “AND.” 4. Site: Limited search to certain 36 Outcome/ Ovid databases (Embase) Google Scholar Database 5. All the keywords and subject headings for websites or domains, e.g., "Non- supplier, retailer, distributors, private pro- COVID-19." Site: journals.plos.org. viders, accreditation, licensing, provider By searching within certain domain monitoring combined with “AND.” extensions, limited search by country 6. COVID-19, pandemic, COVID-19 re- or type of institution, e.g., "Non- sponse, SARS-CoV-2, coronavirus. COVID" site: .edu (academic institu- 7. Proximity searching using purchasing adj tions in Armenia). COVID-19. 8. Proximity searching using provision adj COVID-19. 9. Proximity searching using patients adj COVID-19. 10. Combination of 1–5 AND 6 AND 7–9 Health A comprehensive search for health outcomes system is listed below. outcomes The search strategy covered: 1. All the keywords and subject headings for utilization combined with OR quality com- bined with OR financial protection com- bined with OR equity in utilization com- bined with OR accountability combined with OR 2. Proximity searching using utilization adj3 pandemic* 3. Proximity searching using quality* adj3 4. pandemic* 5. Proximity searching using financial protec- tion*adj3 pandemic* 6. Proximity searching using equity in utilisa- tion*adj3 pandemic* 7. AND COVID-19 care, non-COVID-19 care, health care, primary health care, tel- emedicine, hospitalization, surgery, and hospital management. 8. Combination 1–5 and 6–11 and 12 Source: Elaborated by authors. A data extraction sheet was used to extract data from articles individually (Appendix A). An Excel spreadsheet was then developed to organize the extracted data into a matrix to help analyze the data within and across papers. The extracted data were completed using COVIDENCE to facilitate this process deductively. Subsequently the data were inductively analyzed into themes and sub- themes drawing on the conceptual framework. 2.5.2. Key informant interviews and focus group discussions The research team designed and tested interview questionnaires for each participant type along the dimensions of the conceptual framework. They piloted the questionnaires with one participant in each country and adjusted the purpose, scope, and clarity of the questions. Participants were recruited via the network of the research team, who work with or within country health agencies. 37 Based on the conceptual framework, we recruited participants responsible for purchasing deci- sions, providers involved in service delivery, and service users. Formal e-mail invitations were sent to participants inviting them to interview via video call. Each participant signed an informed consent form for each interview. Study team members of both countries, who have previous experience conducting qualitative data collection, led the interviews with the assistance of one other team member as a notetaker. All interviews were conducted in local languages with simultaneous translation for notetakers, and all participants agreed to the in- terview being video recorded. Immediately after each interview, field notes were compiled. Re- sponses were organized and coded into emerging topics of interest according to the conceptual framework. Ethical approval for this study was granted by the Center of Medical Genetics and Primary Health Care in Armenia on May 19, 2021, and by the Scientific Council of the Babes-Bolyai University in Romania on May 26, 2021. 2.5.3. The Recruitment of the key participants The recruitment strategy was based on a list of potential participants provided by the World Bank Group and team members in the countries. In addition, the number of participants increased as the people initially selected to interview invited others to participate, mainly users of health care ser- vices. A nonprobabilistic sample was used based on saturation point, which means that people were interviewed until no new themes emerged from the interviews. The saturation points in this study were reached at a total of 24 interviews. The fieldwork and logistics of the data collection plan were held successfully between May and June 2021 in both countries. Participants were invited by e-mail invitation. No financial reward was offered to participants in this study. The groups of key participants. The eligibility criteria focused on covering the themes and subthemes of the conceptual framework: including participants from the government, purchasing agencies and providers, the Ministry of Fi- nance, those responsible for COVID-19's policy implementation, and users of the health care ser- vices. Table A.2 shows the gender and type of respondents by country for this study. Table A.2. Number of Participants by Country and Gender Armenia Romania Total Participants/Country Female Male Female Male Purchasers 2 1 4 0 7 Providers 1 3 2 3 9 Patients 3 0 2 3 8 Total 6 4 8 6 24 Source: Elaborated by authors. Type of interview A semistructured interview guide developed along the dimensions of purchasing was used to struc- ture interview questions. All interviews were conducted in local languages with simultaneous trans- lation for English-speaking team members, and all participants agreed to the interview being video recorded. Detailed notes were recorded throughout the interview and with reference back to video recordings, if needed. Responses were organized and coded into emerging topics of interest 38 according to the conceptual framework using deductive reasoning. This study has a low risk for research participants. The interview guides are listed below. 2.5.4. Interview guide for purchasers (Ministries of Health) Section 1. Purchasing governance • Can you tell me about any changes by the main actors involved in purchasing, and mechanisms for coordination in decision making and governance of purchasing? a. To what extent did this include providers and patients? • Can you tell me how any changes to purchasing were documented? a. Were these in official or unofficial documents? b. What, if any, specific goals and objectives were acknowledged regarding purchas- ing? • Can you tell me about any changes to revenue flows and pooling to support purchasing ar- rangements during the COVID-19 pandemic? • Can you tell me about any modifications made to health information systems, including hard- ware, software, monitoring framework, data analysis, and use, to support purchasing arrange- ments? • Can you tell me about any changes to the roles and responsibilities of key actors involved in purchasing? a. How were these changes documented? b. How were these changes communicated? • Can you tell me about any changes to the staffing and expertise involved in purchasing ar- rangements during the pandemic? • Can you tell me about any changes to the performance systems and actors involved in moni- toring the implementation of purchasing decisions? • What channels and means were introduced to communicate purchasing decisions to provid- ers and patients? • What were the unintended (or intended) consequences resulting from changes to the govern- ance of purchasing? Section 2. What to buy • Can you tell me about any changes to the services covered in the benefits package and medi- cines in the essential medicines list? • Can you tell me about any changes to the state-reimbursed costs and a change in cost-shar- ing (including copayments) for services covered in the benefits package and medicines in the essential medicines list? • Can you tell me about any changes to which groups of individuals are covered by the state? • Can you tell me about any changes to clinical guidelines, including gatekeeping and referral requirements, for reimbursement of services covered by the government? • What unintended (or intended) consequences resulted from changes to purchasing in the cat- egory of “What to Buy”? Section 3. How to buy • Can you tell me about any changes to payment methods for providers of medicines, medical devices, and supplies? • Can you tell me about any changes to the process or system for calculating and negotiating payment rates? • Can you tell me about any changes to the systems for monitoring provider performance by the purchaser? 39 • Can you tell me about any disagreement between central (government) purchasing policies and district performance? • What unintended (or intended) consequences were there resulting from changes to purchas- ing in the category of “How to Buy”? Section 4. From whom to buy • Can you tell me about any changes to the suppliers of medicines, medical devices, and sup- plies and rules for payment to suppliers, distributors, and retailers? • Can you tell me about any changes to the conditions and rules for contracting health care pro- viders, including licensing, accreditation, and quality requirements? • What was the role of private providers in health care provision during the pandemic? o Can you tell me about any changes to guidelines for contracting private providers? • What were the unintended (or intended) consequences resulting from changes to purchasing in the category of “From whom to Buy”? Section 5. Centralized procurement of emergency equipment and supplies (Romania-specific, questions for the Ministry of Interior) • What are the main equipment and supplies procured by facilities through the mechanism? • What were the benefits of this mechanism relative to the status quo of decentralized procure- ment? • What was the driving force for this mechanism to be developed? • Do you have suggestions for improving this mechanism or sustaining it? Closing questions Note: For the closing questions, you may emphasize the unintended (or intended) consequences resulting from changes to the governance of purchasing, what to buy, how to buy, and from whom to buy. Finally, ask the group whether there is anything else they would like to add. Thank you for your participation. 2.5.5. Health systems outcome indicators Three different analyses were proposed to capture relevant health indicators during the pre- pandemic period and during the pandemic: 1) Descriptive analysis by category (utilization, equity of utilization, quality, and financial protection). 2) Prepandemic and postpandemic analysis to analyze disruption in non-COVID-19 health care services and provision of COVID-19 care. 3) A time series analysis to analyze changes throughout the pandemic in the short term (next three months) to test assumptions of normality, stationarity, and volatility of data. Data sources: The data on the health system outcome indicators were obtained from multiple offi- cial data sources. The indicators of COVID-19 disease, maternal and child health, immunization, cardiovascular disease, hospital capacity, and financial protection were collected in Armenia and Romania. In Armenia, data were provided by the National Center for Disease Control (NCDC) and National Institute of Health (NIH). In Romania, data were provided by the National Institute of Sta- tistics (INS) and the National Institute of Public Health (INSP). Both countries provided data for 2019 and 2020. A limited number of indicators allowed gender and age group disaggregation. 40 Minimal requirements of data collection are described as follows: • Data collection is estimated in monthly information intervals for each of the years of analysis o Prepandemic: 2018–2019 o Pandemic: 2020–2021. • After this, the prepandemic and postpandemic prevalence will be calculated, if required. • (Always) try to obtain disaggregated data by gender and any other disaggregation that re- quires some specific indicator. • A time series analysis is proposed for each country if data with long periods of time are available. This is to predict the behavior of the indicators in the short term (e.g., the next three months). Statistical analysis: • Unit of analysis: counts, percentages. • The analysis of the information begins with a descriptive analysis that is supported by graphs that help to interpret the data, such as bar graphs, boxplots, and scatter plots, among others. Among the analyses, some hypotheses will be tested to verify the following: o Distribution of data o Difference in groups (women and men or other disaggregation's levels (elderly vs. young), according to the requirement of the indicator) o Analysis of trends using monthly data o In the time series, the series stationarity and volatility will be verified • Subsequently, a statistical inference analysis was done to characterize the country's popu- lation against the selected indicators. A multivariate analysis technique was used to describe similarities and differences between population groups in time in the selected variables. • The logit regression will determine the probability of occurrence of an event, or a behavior given some characteristics of the independent variables. With the logit regression, it is pos- sible to estimate the probability that a country has made a change associated with the var- iables of interest. 2.5.6. Data analysis The researchers used thematic analysis, applying codes using Microsoft Excel according to the conceptual framework. Throughout the exploration, emerging categories were noted, which func- tioned as subthemes that allowed for a more detailed analysis of the contexts. Based on initial field notes and subsequent thematic analysis, thematic saturation was reached. Data management and analysis used STATA to perform three different analyses: 1. Descriptive analysis by dimensions (utilization, equity in the utilization, quality, and financial protection). 2. Prepandemic and postpandemic analysis to analyze disruption in non-COVID-19 health care services and provision of COVID-19 care. 3. A time series analysis to analyze changes throughout the pandemic in the short term (next three months) to test assumptions of normality, stationarity, and volatility of data. 2.5.7. Scoping literature review The literature search yielded 240 publications, out of which 130 were peer-reviewed academic jour- nal articles. After removing duplications and reviewing titles and abstracts, 120 articles were 41 identified for full-text screening. Ninety full-text articles were selected based on the inclusion criteria, and 50 studies were included for data extraction. Figure A.1 summarizes the literature search and selection based on PRISMA. Figure A.1. PRISMA Flow Chart Detailing Numbers of Studies Screened, Assessed, and In- cluded in the Review Source: Elaborated by authors. 2.5.8. Key informant interviews and focus group discussions We interviewed a total of 24 participants, with 10 from Armenia and 14 from Romania, from May to June 2021 (Table A.2). Of the participants, seven were involved in purchasing decisions, nine were providers, and eight were service users. Interviews averaged 60–90 minutes. More than half of the participants were female (58 percent). 2.5.9. Quantitative indicator analysis The data on the health system outcome indicators were obtained from multiple official data sources. The indicators of COVID-19 disease, maternal and child health, immunization, cardiovascular dis- ease, hospital capacity, and financial protection were collected in Armenia and Romania. In spite of the efforts to collect data in both countries, the availability of information at the time of the analysis (number of relevant indicators, periods of time, and disaggregation levels) limited the scope of the research. 42 REFERENCES Andreasyan, D., Al. Bazarchyan, S. Manukyan, G. Muradyan, A. Torosyan, A. Chamanyan, L. Bi- dzyan, and P. Zelveyan. 2016. "A 720 Health System Performance Assessment, Armenia, 2016." Yerevan: National Institute of Health. ARMSTAT, the World Bank. 2020. Social Snapshot and Poverty in Armenia, Part III Non-Material Poverty. The Statistical Committee of the Republic of Armenia (ARMSTAT). https://arm- stat.am/file/article/poverty_2020_e_4.pdf. ARMSTAT, the World Bank. 2021. Social Snapshot of Poverty in Armenia, Part III Non-Material Poverty. The Statistical Committee of the Republic of Armenia (ARMSTAT). https://arm- stat.am/file/article/poverty_2021_e_4..pdf. Băcilă, Ciprian, and Claudia Anghel. 2020. "Telepsychiatry in the Romanian Health System during COVID-19 Pandemic." Acta Medica Transilvanica 25 (3): 75–77. https://doi.org/10.2478/amtsb-2020-0056. Bedrule-Grigoruta, M. V. 2017. "Public-Private Partnership in Romania." European Procurement and Public Private Partnership Law Review 4 (3): 5. https://doi.org/10.21552/EPPPL/2009/3/81. Blavatnik School of Government. 2022. "Variation in Government Responses to COVID-19. Uni- versity of Oxford. https://www.bsg.ox.ac.uk/research/publications/variation-government-re- sponses-covid-19. Cashin, Cheryl, Sharon Nakhimovsky, Kelley Laird, Altea Cico, Sharmini Radakrishnan, Tihomir Strizrep, Ali Lauer, et al. 2018. "Strategic Health Purchasing Progress: A Framework for Poli- cymakers and Practitioners." Photo: Donald Batson, Courtesy of Photoshare Photo: Gonralo Guajardo for Communication for Development. Bethesda, Marylan, USA. www.abtassoci- ates.com. Cernicova-Buca, Mariana, and Adina Palea. 2021. "An Appraisal of Communication Practices Demonstrated by Romanian District Public Health Authorities at the Outbreak of the COVID- 19 Pandemic." Sustainability 13 (5): 2500. https://doi.org/10.3390/su13052500. Chukwuma, Adanna, Radu Comsa, Dorothee Chen, and Estelle Gong. 2021. "Provider Payment Reforms for Improved Primary Health Care in Romania." The World Bank, Washington, D.C., USA. https://openknowledge.worldbank.org/handle/10986/36189. Chukwuma, Adanna, Srinivas Gurazada, Manoj Jain, Saro Tsaturyan, and Makich Khcheyan. 2020. "FinHealth Armenia." FinHealth Armenia, November. https://doi.org/10.1596/34747. Chukwuma, Adanna, Hratchia Lylozian, and Estelle Gong. 2021. "Challenges and Opportunities for Purchasing High-Quality Health Care: Lessons from Armenia." Health Systems and Re- form 7 (1). https://doi.org/10.1080/23288604.2021.1898186. Chukwuma, Adanna, Bruno Meessen, Hratchia Lylozian, Estelle Gong, and Emma Ghazaryan. 2020. "Strategic Purchasing for Better Health in Armenia." Strategic Purchasing for Better Health in Armenia, September. https://doi.org/10.1596/34491. Dadulescu, Ana-Maria. 2015. "Hospital Management between the Modern Image and Aging." ACTA Universitatis Cibiniensis 67 (1): 201–4. https://doi.org/10.1515/aucts-2015-0090. Dascalu, Stefan. 2020. "The Successes and Failures of the Initial COVID-19 Pandemic Response in Romania." Frontiers in Public Health. Frontiers Media S.A. https://doi.org/10.3389/fpubh.2020.00344. Doica, Irina Paula, Dan Nicolae Florescu, Ion Rogoveanu, Dan Ionut Gheonea, Bogdan Silviu Un- gureanu, Carmen Nicoleta Oancea, Adina Turcu-Stiolica, Mihaela-Simona Subtirelu, and Gindrovel Dumitra. 2021. "Telemedicine Chronic Viral Hepatitis C Treatment during the Lock- down Period in Romania: A Pilot Study. "International Journal of Environmental Research and Public Health 18 (7): 3694. https://doi.org/http://dx.doi.org/10.3390/ijerph18073694. 43 Duran, Antonio, Tata Chanturidze, Adrian Gheorghe, and Antonio Moreno. 2019. "Assessment of Public Hospital Governance in Romania: Lessons From 10 Case Studies." International Journal of Health Policy and Management 8 (4): 199–210. https://doi.org/10.15171/ijhpm.2018.120. European Observatory on Health Systems and Policies. 2021. "COVID-19 Health System Re- sponse Monitor (HSRM) Armenia." Brussels, Belgium. https://eurohealthobserva- tory.who.int/monitors/hsrm/hsrm-countries/hsrm/armenia. Florea, Mira, Cecilia Lazea, Remus Gaga, Genel Sur, Lucia Lotrean, Aida Puia, Ana Maria Ale- xandra Stanescu, Monica Lupsor-Platon, Horatiu Florea, and Maria Lucia Sur. 2021. "Lights and Shadows of the Perception of the Use of Telemedicine by Romanian Family Doctors during the COVID-19 Pandemic." International Journal of General Medicine 14 (April): 1575– 87. https://doi.org/10.2147/IJGM.S309519. Fraser, Nicole, Adanna Chukwuma, Marianna Koshkakaryan, Lusine Yengibaryan, Xiaohui Hou, and Tommy Wilkinson. 2021. "Reforming the Basic Benefits Package in Armenia: Modeling Insights from the Health Interventions Prioritization Tool." The World Bank, Washington, D.C., USA. Government of Romania. 2020. "DECRET 195 16/03/2020 - Portal Legislativ." https://legis- latie.just.ro/Public/DetaliiDocument/223831. ———. 2021. "ORDIN 623 14/04/2020—Portal Legislativ." https://legislatie.just.ro/Public/Detali- iDocument/224887. Hannah Ritchie, and Edouard Mathieu. 2022. "Coronavirus Pandemic (COVID-19)—Our World in Data." University of Oxford, England. https://ourworldindata.org/coronavirus. Hintea, Calin, Tudor Ticlau, and Bogdana Neamtu. 2020. "The Romanian National Government Experiences in Dealing with the COVID-19 Crisis." In Good Public Governance in a Global Pandemic, 1st ed. Edited by Paul Joyce, Fabienne Maron, and Purshottama Sivanarain Reddy, 359–75. Brussels: The International Institute of Administrative Sciences. https://doi.org/10.46996/pgs.v1e1. IMF (International Monetary Fund). 2022. "Government Finance Statistics—At a Glance." Wash- ington, DC, https://data.imf.org/?sk=a0867067-d23c-4ebc-ad23-d3b015045405. Inke Mathauer, Elina Dale, and Joe Kutzin, and Joe Jowett. 2019. "Purchasing Health Services for Universal Health Coverage: How to Make It More Strategic?" World Health Organization WHO, Geneva. IHME (Institute for Health Metrics and Evaluation). 2021. "COVID-19 Estimate Downloads." Seat- tle. https://www.healthdata.org/covid/estimate-downloads-archive. Kutzin, Joseph. 2013. "Health Financing for Universal Coverage and Health System Perfor- mance: Concepts and Implications for Policy." Bull World Health Organ. 2013 Aug 1;91(8):602-11. doi: 10.2471/BLT.12.113985. Epub 2013 Jun 17. Lavado, Rouselle F., George Schieber, Ammar Aftab, Saro Tsaturyan, and Hiddo A. Huitzing. 2020. "An Actuarial Model for Costing Universal Health Coverage in Armenia." Edited by Rouselle Lavado, George Schieber, Ammar Aftab, Saro Tsaturyan, and Hiddo Huitzing. Busi- ness & Economics. Manila, Philippines. https://doi.org/10.22617/TCS200395-2. Lylozian, Hracthia. 2016. "The Health Component of the Social Package: A Cross-Sectional Sur- vey of Beneficiaries." American University of Armenia, Yerevan. Mathauer, Inke, Elina Dale, and Bruno Meessen. 2017. "Strategic Purchasing for Universal Health Coverage: Key Policy Issues and Questions: A Summary from Expert and Practitioners’ Dis- cussions." World Health Organization WHO, Geneva. https://apps.who.int/iris/han- dle/10665/259423. Mathauer, Inke, Fahdi Dkhimi, and Matthew Townsend. 2020. "Adjustments in Health Purchasing as Part of the Covid-19 Health Response: Results of a Short Survey and Lessons for the Fu- ture," 1–15. World Health Organization WHO, Geneva. 44 Mathieu, Edouard, Hannah Ritchie, Esteban Ortiz-Ospina, Max Roser, Joe Hasell, Cameron Ap- pel, Charlie Giattino, and Lucas Rodés-Guirao. 2021. "A Global Database of COVID-19 Vac- cinations." Nature Human Behaviour 5 (7): 947–53. https://doi.org/10.1038/s41562-021- 01122-8. Merkur, Sherry, Jonathan Cylus, Ewout van Ginneken, Suszy Lessof, and Anna Maresso. 2020. " Lessons from the first wave: the COVID-19 Health System Response Monitor (HSPM) an evidence resource and a source of analysis." Eurohealth, 26 (2), 5 - 9. World Health Organi- zation. Regional Office for Europe. Mihalas, George I., Dan D. Farcas, Diana Lungeanu, and Mircea Focsa. 2009. "Building EHealth National Strategies—The Romanian Experience." Studies in Health Technology and Infor- matics 150 (February): 33–37. https://doi.org/10.3233/978-1-60750-044-5-33. Mihalas, George I., and Lacramioara Stoicu-Tivadar. 2018. "History of Romanian Medical Infor- matics: Learning from the Past to Reshape the Future." Yearbook of Medical Informatics 27 (1): 243–51. https://doi.org/10.1055/s-0038-1641203. Mirica, Cristian, Monica Laura Zlati, Angela Eliza Micu, Silvius Stanciu, Violeta Sapira, and Andrei Mirel Florea. 2020. "Managing the Financial Stress Generated by the COVID-19 Pandemic in the Public System. Solutions for Economic Restart in Romania." Dunarea de Jos Univer- sity of Galati, Romania. Nalbandian, Naira. 2020. "Armenia Lifts Coronavirus State Of Emergency." https://www.azatutyun.am/a/30833889.html. RFE/RL, Prague, Czech Republic. Ohanyan, Marina, Armine Danielyan, Kevork Hopayian, and Bob Mash. 2015. "International Pri- mary Care Snapshots: Armenia and South Africa." The British Journal of General Practice: The Journal of the Royal College of General Practitioners 65 (635): 308–9. https://doi.org/10.3399/bjgp15X685381. Our World in Data. 2023. Cumulative Confirmed COVID-19 Deaths per Million People. COVID-19 Data Explorer: https://ourworldindata.org/explorers/coronavirus-data-ex- plorer?facet=none&Interval=Cumulative&Relative+to+Popula- tion=true&Color+by+test+positivity=false&country=OWID_WRL~Upper+middle+in- come~ROU~ARM&Metric=Confirmed+death.University of Oxford, England. Petcu, Laura Bostangiu, Alexandru I. Iordache, Nicolae D. Garofil, and Raluca S. Costache. 2020. "CoViD-19 Disease, Romanian Health System Response to Outbreak and Economic Im- pact," 284–94. Vol. CXXIII, No. 4/2020, November, Romanian Journal of Military Medicine, Bucharest, Romania. Petrosyan V, and H. Martirosyan. 2016. "Armenia—Voluntary Health Insurance in Europe—NCBI Bookshelf." Observatory Studies Series, No. 42. Sagan A, Thomson S, editors. Copenhagen, Denmark. European Observatory on Health Systems and Policies; 2016. https://www.ncbi.nlm.nih.gov/books/NBK447709/?report=reader. Preda, Ionel. 2020. "Challenges in the Management of Emergency Public Procurement Per- formed in Romania to Combat the COVID-19 Pandemic." Business Excellence and Manage- ment 1: 67–87. Purcărea, V. L., B. I. Coculescu, and E. C. Coculescu. 2015. "Improving the Quality of Health Care in the Romanian Public Health System—A Priority in the Reform Process." Journal of Medicine and Life 8 (2): 166–70. https://pubmed.ncbi.nlm.nih.gov/25866573. Richardson, Erica. 2013. "Health Systems in Transition." Armenia Health System Review 15 (4). Sagan, Anna, and Sarah Thomson. 2016. "Voluntary Health Insurance in Europe: Role and Regulation." Copenhagen, Denmark. European Observatory on Health Systems and Poli- cies; 2016. European Observatory Health Policy Series. PMID: 28825784 Salceanu, Diana. 2021. "Report of Covid Deaths Released: Up to 13 Percent More Deaths on Alerte.ms Than the Data of the National Institute of Statistics." The Romania Journal web- site. Bucharest, Romania. https://www.romaniajournal.ro/society-people/health/report-of- 45 covid-deaths-released-up-to-13-more-deaths-on-alerte-ms-than-the-data-of-the-national-in- stitute-of-statistics/. Scintee, Silvia Gabriela, and Dana Farcasanu. 2020. "Romania." https://www.covid19healthsys- tem.org/countries/romania/countrypage.aspx. European Observatory on Health Systems and Policies, Brussels, Belgium. SIUI. (n.d.). "Unique Integrated Health Informatics System in Romania (SIUI)." Accessed May 13, 2022. https://www.simavi.ro/en/siui. SIMAVI. Bucharest, Romania. The Collectivity. 2021. "Strategic Purchasing and Covid-19." https://www.thecollectivity.org/en/pro- jects/strategic-purchasing-and-covid-19. Institute of Tropical Medicine Department of Public Health. Antwerp, Belgium. The World Bank, CRRC—Armenia. 2020. Armenia: Citizens' Perceptions on COVID-19 Pan- demic. Washington, DC. https://thedocs.worldbank.org/en/doc/720051597910561126- 0080022020/original/ArmeniaCitizensPerceptionsonCOVID19pandemic.pdf Thomson, Sarah, and Triin Habicht. 2020. "How Are Countries Mobilizing Additional Public Funds for Health? – Cross-Country Analysis." https://analysis.covid19healthsystem.org/in- dex.php/2020/04/27/how-are-countries-mobilizing-additional-public-funds-for-health/.WHO Europe Regional Office, Copenhagen, Denmark. Vlaˇdescu, Cristian, Vlaˇdescu Silvia, Gabriela Scîntee, Victor Olsavszky, Cristina Hernández- Quevedo, and Anna Sagan. 2016. "Health Systems in Transition: Romania." Romania Health System Review 18 (4). Vlădescu, Cristian. 2008. "Romania: Health System Review—LSE Research Online." .http://eprints.lse.ac.uk/25708/. WHO (World Health Organization=). 2020a. "Archived: WHO Timeline—COVID-19." https://www.who.int/news/item/27-04-2020-who-timeline---covid-19. ———. 2020b. "Pulse Survey on Continuity of Essential Health Services during the COVID-19 Pandemic." https://apps.who.int/iris/bitstream/handle/10665/334048/WHO-2019-nCoV- EHS_continuity-survey-2020.1-eng.pdf. World Health Organization WHO, Geneva. WHO Europe. (n.d.). "ArMed National Digital Health System." Accessed May 13, 2022. https://www.euro.who.int/__data/assets/pdf_file/0006/509595/vignette-digital-health-system- Armenia.pdf. ———. 2020. "Strengthening the Health Financing Response to COVID-19 in Europe, April 22, 2020 (Produced by WHO/Europe)," October. https://www.euro.who.int/en/health-top- ics/Health-systems/health-systems-financing/publications/2020/strengthening-the-health-fi- nancing-response-to-covid-19-in-europe,-22-april-2020-produced-by-whoeurope. ———. 2021. "European Health for All Database (HFA-DB) Data Source—European Health Infor- mation Gateway.’ https://gateway.euro.who.int/en/datasets/european-health-for-all-data- base/. ———. 2022. "Armenia Country Profile—European Health Information Gateway." https://gate- way.euro.who.int/en/country-profiles/armenia/. WHO (World Health Organization), OECD (Organisation for Economic Co-operation and Develop- ment), and World Bank Group. 2018. Delivering Quality Health Services. http://apps.who.int/bookorders. World Bank Group. 2021. "World Development Indicators | DataBank." https://data- bank.worldbank.org/source/world-development-indicators. 46 This report examines adjustments made in purchasing arrangements that occurred during the Armenian and Romanian response to the COVID-19 pandemic. The research adopted mixed methods and a deductive approach. Data were obtained from a scoping literature review, key informant interviews, and an exploratory analysis of quantitative health system indicators. The study was informed by a framework for understanding purchasing changes that adapted existing frameworks to explore the implications of purchasing adjustments on critical outcomes. The research describes critical changes in purchasing, provider and user responses to these changes, and health system outcomes that accompanied the COVID-19 response in Armenia and Romania. For example, it was essential to have a governance environment that defined shared objectives and facilitated coordination across stakeholders. During this time benefits expanded, and contracts changed, including payment mechanisms to offset the decline in essential service use and incentivize care delivery for COVID-19. Furthermore, the pandemic saw the accelerated adoption of innovation, particularly telemedicine, within service delivery. The lessons from purchasing during the pandemic have implications for improving coverage, quality, and adaptability to a crisis, including beyond the contexts studied. This report concludes that intergovernmental cooperation, agile purchasing tools, and innovation in purchasing and service delivery are important to effectively coordinate essential services to key populations in rapid changing times. Moving forward, it is essential to assess the sustainability of these adjustments over time and the implications for health system performance. ABOUT THIS SERIES: This series is produced by the Health, Nutrition, and Population Global Practice of the World Bank. The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For free copies of papers in this series please contact the individual author/s whose name appears on the paper. Enquiries about the series and submissions should be made directly to the Editor Jung-Hwan Choi (jchoi@worldbank.org) or HNP Advisory Service (askhnp@worldbank.org). For more information, see also www.worldbank.org/hnppublications. 1818 H Street, NW Washington, DC USA 20433 Telephone: 202 473 1000 Facsimile: 202 477 6391 Internet: www.worldbank.org E-mail: feedback@worldbank.org