IMPLEMENTATION KNOW-HOW BRIEF Regulatory Sandboxes for Digital Health D igital technology, applications, data, and information systems, as part of the ongoing transformation of health and health care can help ensure universal and equitable access to affordable, people-centered, and integrated quality care, contributing to the goal of reaching Universal Health Coverage (UHC). Intelligent use of data and digital technologies can elevate patient experience, improve DIGITAL-IN-HEALTH FLAGSHIP PROGRAM clinician and staff satisfaction, drive operational efficiency, improve patient outcomes and create new business models, with benefits for both the public and private sectors. This Implementation Know-How Brief provides World Bank Group staff, country teams, and other organizations involved in the implementation of Digital-in-Health activities with practical discussions, key terms and considerations, and broad guidance on how to engage with clients on the topic of regulatory sandboxes for digital health. This Brief Will Help Stakeholders to: Learn what regulatory sandboxes are and why, and how, they are used in digital health Learn about the benefits of regulatory sandboxes for digital health Understand the steps involved in designing, implementing and monitoring sandboxes Be aware of success factors, challenges and pitfalls in setting up regulatory sandboxes Why are Regulatory Sandboxes Important for Digital Health? Regulatory sandboxes—tools “for developing evidence about how a new product, technology, or business model (innovation) works and the outcomes it produces” (Jeník and Duff 2020)—seek to promote innovation in highly regulated sectors, giving innovators access to regulatory experts. Regulatory sandboxes are controlled environments in which innovators can:1 test new products and services; find out if business models are viable and how new technologies work in the market; introduce an innovation to market potentially more quickly and at lower cost; and REGULATORY SANDBOXES FOR DIGITAL HEALTH find out what consumer protection safeguards may be needed in the context of their innovations. Regulatory sandboxes can have benefits for multiple stakeholders, including benefits for:2 ▪ Innovators and entrepreneurs. Benefits include access to funding and investment, to markets, and to regulatory guidance. Innovators also have an opportunity to learn how they can integrate their products and services into the existing infrastructure and systems, finding solutions early for problems that would have only been discovered later without a regulatory sandbox. They can reduce their exposure to risks and increase confidence in the ability of their products and services to conform to regulatory requirements (something that patients will also value thereby contributing to the business model for those products and services). Besides receiving guidance and learning about regulatory best practices, innovators and entrepreneurs may also contribute to the development of new regulations and to the revision of existing DIGITAL-IN-HEALTH FLAGSHIP PROGRAM regulations. Even innovators that do not participate in regulatory sandboxes may benefit from new guidance and regulations associated with regulatory sandboxes. ▪ Patients and populations. Benefits include early access to innovative products that are found to be in conformity with existing or new regulations (including safety and risk mitigation) at potentially lower costs, as well as increased transparency and trust in both regulators and innovations. Because products and services have been tested already and their consequences studied, once the innovations are licensed patients and populations can access more and better evidence to make informed decisions. ▪ Governments and regulators. Benefits include the promotion of innovative products and services that conform to regulations and patient rights (such as rights to privacy and security), greater understanding and awareness of the benefits and risks of new products and services (like artificial intelligence, or AI, and genomics) that might fall under their regulatory purview and learning about the effectiveness of existing and new regulations. Claims made by innovators can be tested within the sandbox environment, pre-empting problems. Regulators can also learn about how regulations are perceived by innovators, patients, and populations, and how to better communicate the need for and value of existing or new regulations. Given these potential benefits, it is unsurprising that there is a growing interest in regulatory sandboxes, and an increasing number of countries using them in digital health. Data and digital technologies, products and services can contribute to the transformation of health and health care, but transformations can have both positive and negative consequences, and some consequences may not be predictable at the time that innovations are being developed and adopted. Regulatory sandboxes allow all interested parties to come together to test new digital health products and services, including innovative uses of data, in a controlled and safe environment. In doing so, regulatory sandboxes can highlight unforeseen and unintended costs and consequences of data and digital health innovations, allowing regulators, innovators, and populations to work together to minimize the risks of these innovations and maximize the potentially very high rewards. IMPLEMENTATION KNOW-HOW BRIEF | 2 REGULATORY SANDBOXES FOR DIGITAL HEALTH What Is a Regulatory Sandbox and What Are its Uses? The World Bank defines regulatory sandbox as "a controlled, time-bound, live testing environment, which may feature regulatory waivers at regulators’ discretion” (M. S. Appaya, Gradstein, and Haji Kanz 2020). The Consultative Group to Assist the Poor (CGAP) defines regulatory sandbox “a tool for developing evidence about how a new product, technology, or business model (innovation) works and the outcomes it produces”, as “a controlled, time-bound, live testing environment, which may feature regulatory waivers at regulators’ discretion”.3 For the Organisation for Economic Co-operation and Development (OECD), a “regulatory sandbox refers to a limited form of regulatory waiver or flexibility for firms, enabling them to test new business models with reduced regulatory requirements” (Attrey, Lesher, and Lomax 2020). Other initiatives similar to, or even synonymous with, regulatory sandboxes include “living labs”, “innovation spaces”, “regulatory testbeds”, or “real-life experiments” (BMWi 2019). Put simply, regulatory sandboxes seek to DIGITAL-IN-HEALTH FLAGSHIP PROGRAM promote innovation in highly regulated sectors. Regulatory sandboxes share certain characteristics (BMWi 2019). First, they are established for limited periods with a specific focus area (for example, the Singapore Ministry of Health’s regulatory sandbox for telemedicine and mobile medicine, which ran from 2018 to 20214). Second, they provide regulatory waivers and flexibility, although these are typically within the limits afforded by legislation. Third, their goal is not only to develop evidence on the effects of innovations in real life but also, crucially, to provide insights into how innovations might be regulated going forward. Importantly, while undeniably similar in spirit, regulatory sandboxes are not to be confused with the term sandbox as used in software development, as a testing environment that mirrors “live” systems and that allows experimentation while preserving the integrity of the “live” environments (such as websites).5 The British foundation Nesta—an innovation agency for social good—has proposed the AAA model of regulation as one way to characterize different uses of and objectives for regulatory sandboxes (Armstrong and Rae 2017): ▪ Advisory approaches give innovators access to regulatory guidance and expertise so that new products and services can be adapted to existing regulatory regimes. Innovators benefit from a better understanding of how their products and services affect and are affected by existing regulations, while regulators benefit from learning about possible new disruptions to existing regulatory regimes ▪ Adaptive approaches allow for existing regulations to be adapted if the value of new products and services justifies such changes. Because adapting regulations can affect the industry or sector more broadly, this approach requires consultation and engagement with relevant parties. It is important to note that legislation may limit flexibility in changing regulations ▪ Anticipatory approaches focus on the impacts on the economy and society of emerging technologies with a view to understanding the regulatory implications. The IMPLEMENTATION KNOW-HOW BRIEF | 3 REGULATORY SANDBOXES FOR DIGITAL HEALTH objectives are not just to support innovation but to collect as much information and evidence as possible to minimize uncertainty regarding the potential risks and benefits Figure 1 NESTA’s AAA model of regulation and regulatory sandboxes DIGITAL-IN-HEALTH FLAGSHIP PROGRAM Source: Armstrong and Rae 2017. A regulatory sandbox might not be the most appropriate solution for a specific regulatory challenge in a specific context. It is crucial for their success that sandboxes be fit-for- purpose: that they have specific objectives. Sandboxes are time and resource intensive, and typically not available to all market participants, so regulators determine whether a regulatory sandbox is the best approach (see Figure 2). IMPLEMENTATION KNOW-HOW BRIEF | 4 REGULATORY SANDBOXES FOR DIGITAL HEALTH Figure 2 Key steps when deciding whether a regulatory sandbox is needed DIGITAL-IN-HEALTH FLAGSHIP PROGRAM Source: Jeník and Duff 2020. IMPLEMENTATION KNOW-HOW BRIEF | 5 REGULATORY SANDBOXES FOR DIGITAL HEALTH Countries with Regulatory Sandboxes for Digital Health The concept of a regulatory sandbox was first introduced in 2015 in a report by the United Kingdom (UK) Financial Conduct Authority (FCA) “on the feasibility and practicalities of developing a regulatory sandbox that is a ‘safe space’ in which businesses can test innovative products, services, business models and delivery mechanisms without immediately incurring all the normal regulatory consequences of engaging in the activity in question” (FCA 2015). Since then, the UK has been distinctly prolific in the use of regulatory sandboxes across multiple economic sectors and levels of government. Besides the still active regulatory sandbox of the FCA, and of specific interest to the health care sector, the UK Care Quality Commission (CQC) has ran three regulatory sandboxes to date,6 the UK Information Commissioner’s Office (ICO) has an ongoing regulatory sandbox "to support organizations who are creating products and services which use personal data in innovative and safe ways” (including many in health),7 and the UK Medicines and Healthcare products Regulatory Agency DIGITAL-IN-HEALTH FLAGSHIP PROGRAM (MHRA) has announced its own regulatory sandbox, called AI-Airlock, that “will provide a regulator-monitored virtual area for developers to generate robust evidence for” AI technologies.8 In Cornwall, a general practice sandbox is being set up, and the Island of Jersey has a digital health sandbox.9 While a pioneer in the field, the UK is far from alone, and many more countries are using regulatory sandboxes for digital health. As previously mentioned, Singapore ran a regulatory sandbox for telemedicine and mobile medicine between 2018 and 2021, and is now running a new regulatory sandbox for small and medium enterprises on generative AI. In the United States, the Massachusetts eHealth Institute at the MassTech Collaborative runs, since 2019, a Sandbox Challenge Program to identify and support digital health companies with solutions that address key health care challenges in Massachusetts.10 In Malaysia, the National Technology and Innovation Sandbox, launched in 2020, allows researchers, innovators and entrepreneurs to test their products and services in a live environment and qualify for grants to bring those products and services to market. As of Jul 2021, 26% of 139 approved applications were in the medical and health care sectors, and 44% of companies were assisted by the Ministry of Health.11 In India, the National Digital Health Mission Sandbox helps organizations seeking to integrate their innovations into the national digital health ecosystem (National Digital Health Mission 2020). In Norway, Datatilsynet, the Norwegian Data Protection Authority, has recently focused its regulatory sandbox on AI technologies, with some of the participating innovators focusing on health.12 In Colombia, the Superintendence of Industry and Commerce has proposed creating a regulatory sandbox for AI applications focusing on e-commerce and/or advertising and marketing.13 In France, the Commission Nationale de l’Informatique et des Libertés (CNIL) recently published the results of its regulatory sandboxes on digital health and EdTech.14 In Brazil, the Data Protection agency has prepared a benchmarking report on sandboxes, including the health sector, as a first step towards creating a regulatory sandbox (ANPD 2023). In Indonesia, the SatuSehat health services platform includes a regulatory IMPLEMENTATION KNOW-HOW BRIEF | 6 REGULATORY SANDBOXES FOR DIGITAL HEALTH sandbox as a testing mechanism to innovate and test digital applications, services, and regulations before market launch.15 At the international level, interest in cross-country or even global regulatory sandboxes is also growing. In preparation for the European Union (EU) AI act (article 53 of which specifically discusses regulatory sandboxes), the Spanish government is funding a pilot for an AI sandbox that can test “future obligations and requirements in a controlled environment and provide practical learning experience to support the development of standards, guidance and tools at national and European level”.16 The initiative will give EU Member States the possibility to participate, and may eventually become a pan-European sandbox for AI. While not specifically focused on health, it is likely—based on past country experiences with AI sandboxes—that some of the innovations will focus on the health sector. At the global level, in the financial sector, the UK FCA proposed in early 2018 a global sandbox for firms to test innovative products and services, or business models, across jurisdictions (M. S. Appaya, DIGITAL-IN-HEALTH FLAGSHIP PROGRAM Gradstein, and Haji Kanz 2020). The resulting Global Financial Innovation Network, launched in 2019, and which includes the World Bank Group among its 60 members and observers, ran a cross-border testing pilot in 2020 for which there was strong demand. The network is working to build on the pilot project and increase collaboration within the international regulatory community (ibid). How to Create a Regulatory Sandbox for Digital Health As previously stated, regulatory sandboxes are time and resource intensive, and may not be the most cost-effective way to promote data and digital health innovations. In the context of the financial sector (fintech) but with applicability to the health sector, the World Bank advises that “before embarking on creating a regulatory sandbox, authorities should step back and objectively review the environment in which they operate, specifically, (i) the existing legal and regulatory framework within that jurisdiction; (ii) the capacity and resources available to the regulator; (iii) the maturity and pervasiveness of the fintech market; and (iv) broader market conditions, including competition criteria” (S. Appaya and Haji 2020). If a decision is made to proceed with the creation of a regulatory sandbox for digital health, the Colombian Consejería Presidencial para Asuntos Económicos y Transformación Digital has proposed a structure for designing cross-sectoral regulatory sandboxes (Español 2020) which is helpful. A modified version of this proposal – adapted to focus on digital health, taking inspiration from an Inter-American Development Bank report (Herrera and Vadillo 2018) – includes the following 12 steps. 1 Definition of the leadership model. The leadership model provides clarity and coherence in public policy objectives by defining the entity responsible for leading the design, implementation, and evaluation of the regulatory sandbox. Given the focus of the sandbox is digital health, typically the Ministry of Health will play a leading role, supported by the Ministers responsible for economic affairs and the digital transformation. The offices of the Presidency or Prime Minister may also be involved. IMPLEMENTATION KNOW-HOW BRIEF | 7 REGULATORY SANDBOXES FOR DIGITAL HEALTH 2 Identification of the problems to be addressed and the relevant emerging technologies. As is best practice, the development of data and digital solutions in health should be guided by priorities in health and health care policy, typically set out in national health strategies or plans. Priority should be given to emerging technologies, products and services that seek to address strategic health challenges, technologies that are still developing, and whose regulatory, economic, and social implications are unclear but expected to be important. 3 Regulatory mapping. Having selected the broad health challenges to be addressed and the emerging technologies to be considered, it is now possible to identify the laws, regulations, norms, and standards that are applicable. This regulatory mapping should benefit from wide participation from sectors and parts of government outside of health that may be affected by the new products and services (such as data protection agencies). Moreover, a mandate or administrative act (such as a law, decree, or similar) DIGITAL-IN-HEALTH FLAGSHIP PROGRAM is essential to allow the creation of the sandbox, and to give powers and authority to the supervisor of the sandbox. 4 Identification of the entities who could be involved as well as those responsible for monitoring and evaluating. The entity responsible for supervising the regulatory sandbox, as well as the agencies or departments responsible for monitoring and evaluating its use should be identified. All entities that could benefit from being involved in the design, implementation, and monitoring should also be identified and invited to participate, setting out clearly the mechanisms for participation. Stakeholders with an interest in participating could include data protection agencies, cybersecurity agencies, health regulators, health care quality agencies, medical professional associations, industry associations, and civil society representatives, to name just a few (ANPD 2023). It is important to define the roles of stakeholders, shore up support, and establish coordination and support mechanisms. Common types of stakeholders involved in regulatory sandboxes Core Active Occasional Surrounding stakeholders participants participants environment Decide on the set-up, Supply services, key Support via sharing of Observes and may design, and components of an contacts and positive participate, potentially implementation of the innovation, access to backing. passively. regulatory sandbox. and influence on decision-makers. Source: BMWi 2019. A World Bank and CGAP survey of innovation facilitators in fintech found that the two most common governance models for regulatory sandboxes in fintech were dedicated units and hub-and-spoke models (S. Appaya and Haji 2020). The first model involves setting up potentially large teams dedicated exclusively to running the sandbox, while the second model relies on a smaller team that collaborates with government agencies and other stakeholders to extend capacity. A third model involves IMPLEMENTATION KNOW-HOW BRIEF | 8 REGULATORY SANDBOXES FOR DIGITAL HEALTH completely outsourcing the implementation and operation of the regulatory sandbox to a firm. Whatever model is chosen, having the requisite skills and capacity is essential for success. 5 Capacity building and training. All entities involved in the regulatory sandbox should receive training from technical experts and, if possible, individuals or organizations previously involved in the creation of regulatory sandboxes. The training and capacity building should include consideration of the potential risks and benefits of the new products and services. Disruptive innovations can be complex, and regulators must be capable of assessing applications to the regulatory sandbox, devise plans for testing new products and services, and supervising participants once the sandbox is up and running, all of which require skills and time (S. Appaya and Haji 2020). Because of the broad set of skills DIGITAL-IN-HEALTH FLAGSHIP PROGRAM needed (ranging from technological to legal to financial skills) to run a regulatory sandbox, it is essential to have subject matter experts either join the supervisory team, or be made available through external stakeholders involved in the regulatory sandbox. Lack of resources, capacity, and skills may result in serious risks to consumers and expose regulators to reputational, legal, and financial risks. 6 Establishment of working group(s). These will be the vehicles through which participating entities coordinate and carry out the design, implementation, and monitoring of the regulatory sandbox, and where risks, challenges, and interests are presented and discussed. 7 Definition of a risk model for implementation. Given the nature of regulatory sandboxes, there is the potential for unintended and unexpected consequences. Despite the uncertainty that is inherently associated with sandboxes, it is essential that the supervisor and the multiple parties involved identify possible effects of operating the regulatory sandbox and establish possible contingencies and mechanisms for protecting all stakeholders, but especially patients and the public. Even with a risk model in place, it is important that patients and the public understand and are conscious of the risks of participating in regulatory sandboxes. 8 Establishment of the criteria for eligibility and selection of innovators. Among possible eligibility and selection criteria are: novelty (for example, the product or service is not currently being offered in the market), development stage (the product or service, and the business plan, are sufficiently developed to be tested in a live environment), benefit (the innovator is able to make a reasonable case for the benefits of the product or service to the health care system, health care workers, and/or patients and communities), regulatory interest (it is unclear how the product or service affects and is affected by existing or planned regulations), and financial capacity (innovators have the funding necessary to participate). Leadership should be ready to not move forward with a regulatory sandbox if there are no eligible innovators (in other words, there needs to be demand for the regulatory sandbox, this is not a supply-driven model). Regulatory IMPLEMENTATION KNOW-HOW BRIEF | 9 REGULATORY SANDBOXES FOR DIGITAL HEALTH sandboxes need to meet their stated objectives to justify the significant resources allocated to them. 9 Fit-for-purpose design of the sandbox. Design choices include the duration of the regulatory sandbox and admission windows (for example, cohort and rolling basis), restrictions on the functioning of the sandbox (such as the number of innovators that will be accepted, or geographical boundaries), information disclosure policies, transmission of data to the supervisor and potentially participating stakeholders, and exit options. According to CGAP, regulatory sandboxes generally have five core design elements that may be combined in various ways to accommodate specific objectives and contexts (some of which have already been mentioned in previous steps): eligibility, governance, timing, test restrictions, and exit options (see table below). DIGITAL-IN-HEALTH FLAGSHIP PROGRAM Design elements of regulatory sandboxes with descriptions and examples DESCRIPTION EXAMPLES Design element: Eligibility Defines who can participate in the sandbox ▪ Open to incumbents only Eligibility should be articulated clearly to ensure a ▪ Open to newcomers only level playing field across all market participants Design element: Governance Defines the internal operating structure of the ▪ Specialized sandbox unit sandbox, roles and responsibilities, and key ▪ Hub-and-spoke: a central point of contact operational processes coordinating sandbox inquiries with other units of the regulator Design element: Timing Includes: duration of the admission window, and ▪ Periodic admission (cohort-based) duration of the test ▪ Permanent admission window (on-tap) ▪ Testing periods range from 3 to 36 months Design element: Test restrictions Limits to the scope, scale, and/or conduct of the ▪ Number of clients sandbox test to minimize potential ▪ Number of transactions ▪ Volume of transaction ▪ Geographical limits ▪ Consumer protection safeguards Design element: Exit options Includes: individual test outcomes (graduation, ▪ KPIs in terms of the absolute output terminated test, etc.); program-level key (number of graduated firms) performance indicators (KPIs); and incorporation ▪ KPIs in terms of a regulatory change of insights and lessons learned into the broader regulatory age promoted Source: (Jeník and Duff 2020) The CGAP suggests that testing plans be customized to a particular objective and regulatory context (Jeník and Duff 2020). Before any tests are conducted, the owner of IMPLEMENTATION KNOW-HOW BRIEF | 10 REGULATORY SANDBOXES FOR DIGITAL HEALTH the sandbox should be clear on what next steps will be once testing is finalized and should map out every possible outcome under its own regulatory framework. Setting up a regulatory sandbox without legal clarity on each potential exit option is not advisable. Possible exit options are shown in the figure below. As shown, licensing (possibly with certification) is only one option after a successful test. Figure 3 Exit options for regulatory sandboxes DIGITAL-IN-HEALTH FLAGSHIP PROGRAM Source: Jeník and Duff 2020. 10 Publication of regulatory sandbox plans for public consultation. Documentation on the proposed regulatory sandbox (such as white papers or discussion papers) should be disseminated and a period for public consultation should be provided. Once this period is over, leadership should publish documentation describing the comments received and how these will be addressed (for example, the UK ICO conducted a public consultation on its regulatory sandbox in 2018 before moving forward with its implementation; the public consultation showed an overwhelmingly positive response to the plans17). 11 Launch and implementation. The regulatory sandbox is ready for launch and implementation. Innovators are now able to submit applications, and the supervisor can decide which products and services to include. All parties involved, especially government agencies and departments, should coordinate their efforts during this time to ensure wide dissemination and promote interest and demand for participation. IMPLEMENTATION KNOW-HOW BRIEF | 11 REGULATORY SANDBOXES FOR DIGITAL HEALTH One option to keep in mind, especially for first-time implementers of a regulatory sandbox who may be unsure of the viability of a sandbox and who want to make sure their plans are realistic before and after launch, is a World Bank Sandbox Simulation exercise. “The World Bank Group has helped many policy makers assess (i) the feasibility of implementing a sandbox as compared to other tools, (ii) the effectiveness of sandbox frameworks, and (iii) the alignment of the process to the goals and objectives of the sandbox” S. Appaya and Haji 2020. The World Bank Sandbox Simulation exercise gives participants the benefit of the World Bank’s global expertise, and brings together sandbox staff, members of the sandbox working groups and committees, representatives from all government agencies and DIGITAL-IN-HEALTH FLAGSHIP PROGRAM departments involved, other regulators, and sometimes market innovators. Outputs include, for example (S. Appaya and Haji 2020): ▪ At the design and pre-launch stages: identification of sandbox objectives, scope, and remit; identification of applicable best practices; definition of eligibility and evaluation criteria, as well as testing and exit procedures; creation of sandbox prototype; understanding of relevant governance mechanisms; support for the creation of a sandbox manual, and communication strategies ▪ After the sandbox is operating: simulation of each stage of sandbox using relevant case studies, stages such as review of applications, selection, and testing, reporting and risk mitigation, oversight of supervision, and options for sandbox exit; identification of areas for improvement in governance; possibility of memorandum of understanding with other regulators 12 Monitoring, evaluation, and learning. Once the regulatory sandbox is closed and all innovators have exited, a report should be made publicly available by the leadership and supervisor, in close collaboration with the parties involved in the design, implementation, and monitoring and evaluation of the regulatory sandbox. Moreover, whenever an innovator exits the sandbox, an exit report should be written up and, if possible, made publicly available.18 Both individual exit reports and the general monitoring, evaluation and learning report should explain how the regulatory sandbox contributed to the development of new information and evidence on the innovations included in the sandbox and highlight what changes, if any, will be made to the regulatory regime because of operating the regulatory sandbox. The results of this monitoring, evaluation, and learning exercise will be instrumental in streamlining the process of setting up future regulatory sandboxes. As mentioned, it is challenging to predict the effects of innovative products and services. This makes it difficult to design measurement frameworks to determine that regulatory sandboxes are reaching their objectives. Having a measurement framework is IMPLEMENTATION KNOW-HOW BRIEF | 12 REGULATORY SANDBOXES FOR DIGITAL HEALTH essential, however challenging it may be to design one. The World Bank has proposed the framework in the figure below “to measure the impact of a sandbox on country, regulatory and market level outcomes” (S. Appaya and Haji 2020). Figure 4 Measurement framework for regulatory sandboxes DIGITAL-IN-HEALTH FLAGSHIP PROGRAM Source: Adapted from S. Appaya and Haji 2020. During initial measurement, the focus is on establishing the indicators that innovators applying for the regulatory sandbox should consider given the sandbox’s objectives. Possible indicators, at this stage, could include business metrics, regulatory and market outcomes, and test assumptions. As successful applicants go through the sandbox, authorities should continue to monitor these indicators. During the ongoing monitoring phase, indicators could include confirmed suitability of innovator’s products and services against the objectives of the sandbox, changes that can be attributed to the operation of the sandbox, and operational efficiency of the sandbox. Finally, evaluations should be conducted at defined points in time, and definitively at the closing of the sandbox, as already mentioned. For all stages, levels at which indicators should be collected include country, regulatory, firm, and operational/institutional. Both stages and levels of measurement can overlap. Examples of quantitative indicators include (S. Appaya and Haji 2020): ▪ Number of applicants and number of applicants accepted into the sandbox ▪ Average length of time to accept applicants into sandbox, to test offerings, and to exit ▪ The time to come to market with and without a sandbox ▪ Number of sandbox tests (successful and unsuccessful) ▪ Number of companies graduated from the sandbox or successfully completing testing IMPLEMENTATION KNOW-HOW BRIEF | 13 REGULATORY SANDBOXES FOR DIGITAL HEALTH ▪ Number of firms formally registered or with formal authorization to operate in the market following successful testing ▪ Number of firms successfully graduated from the sandbox that are currently still in operation ▪ Number of non-sandbox firms operating in the market under an adjusted legal or regulatory framework based on sandbox test ▪ Number of firms receiving investment as a result of the sandbox program ▪ Number of new consumers receiving services or products under the sandbox tests Besides exit and final reports, other documents and dissemination tools that can be derived from regulatory sandboxes include guidelines (for example on AI and privacy enhancing technologies), podcasts, blogposts, and workshops (ANPD 2023). DIGITAL-IN-HEALTH FLAGSHIP PROGRAM Key Success Factors, Challenges and Pitfalls A World Bank and CGAP survey of innovation facilitators in fintech found that regulatory sandboxes require significant resources (from US$25,000 to US$1 million in the survey) and that policy makers tended to underestimate these during planning and design (S. Appaya and Haji 2020). The resource intensity of regulatory sandboxes was considered a major weakness of the approach by respondents. Crucially, a regulatory sandbox contributes to an existing legal and regulatory framework, they do not decrease – if anything they increase – the need for supervision. While it is possible that regulatory sandboxes in the health sector could require fewer resources than in the financial sector, this is unlikely to be the case given the organizational, economic, regulatory, and technical complexity of the health sector. Lessons from the World Bank and CGAP survey for countries seeking to use regulatory sandboxes for digital health include (S. Appaya and Haji 2020): ▪ Aligning the regulatory sandbox with the country needs ▪ Having focused, well-developed scope and objectives ▪ Making sufficient financial and technical resources available ▪ The digital health market is mature, ready and has demand for the sandbox. ▪ Having well-thought-out exit plans ▪ Engaging with interagency and international partners ▪ The operation of the sandbox is agile and flexible A crucial lesson from World Bank work is that a well-functioning regulatory sandbox needs to meet existing market demand (S. Appaya and Haji 2020). There must already be an environment for innovation, or one must be possible to develop through foreign investment or entry. A key lesson from the UK CQC is that all relevant regulators and government agencies need to be involved in regulatory sandboxes for digital health, given the IMPLEMENTATION KNOW-HOW BRIEF | 14 REGULATORY SANDBOXES FOR DIGITAL HEALTH multidisciplinary nature of data and digital health innovations.19 For example, in the UK, the MHRA is responsible for confirming manufacturer’s claims regarding how their devices perform, but the agency makes no assurances as to whether they contribute to high-quality care when implemented in practice in a specific context (an area that the UK CQC is very much focused on). This is a lesson that the OECD also supports, in the context of AI innovations: “AI products and services are complex and often affect several areas, so several regulatory authorities must be involved in their testing”.20 The OECD points out other lessons (again in the context of AI): ▪ Promote expertise, skills, and capacity-building in AI within regulatory bodies ▪ Explore international regulatory interoperability and the role of trade agreements. ▪ Cooperate internationally on sandbox eligibility and testing criteria ▪ Consider impact of regulatory sandboxes on impact and competition as early as possible DIGITAL-IN-HEALTH FLAGSHIP PROGRAM ▪ Understand how regulatory sandboxes interact with other pro-innovation mechanisms The Brazilian Data Protection Agency’s benchmarking report on sandboxes highlights some common risks and possible mitigating actions (ANPD 2023): ▪ Unauthorized access or disclosure of sensitive data and/or information protected by intellectual property rights, in the context of outsourcing activities in the conduct of the program. Mitigating options include to stipulate liability in agreements/contracts and to limit, or simply bar, outsourcing activities ▪ Competitive imbalance. Mitigating options include to define clear, objective, and transparent criteria for selecting participants and disseminate these widely, to share knowledge with non-participants through reports and other publications, to not grant any regulatory privileges to participants, and to potentially involve antitrust authorities in the sandbox ▪ Harm to patients and populations participating in the sandbox. Mitigating options include to inform patients and populations of the risks involved and who is liable for damages, and to formalize compensation policies for patients and populations who may be harmed ▪ Breach of property rights, including intellectual property and business secrets. Mitigating options include setting scope and remit of sandbox to limit risks, as well as to consider participation mechanisms for stakeholders who may stand to benefit from access to proprietary information ▪ Security incidents. Mitigating options include setting obligations for participating innovators and stakeholders to adopt certain security measures to be involved ▪ Incidents related to centralization of commercial information. Mitigating options include to use decentralized approaches such as federated data models and privacy enhancing technologies ▪ High financial and human resources costs. Mitigating options include sharing costs among all participants (innovators and stakeholders) IMPLEMENTATION KNOW-HOW BRIEF | 15 REGULATORY SANDBOXES FOR DIGITAL HEALTH Among the challenges of running a regulatory sandbox, the main challenge with regulatory sandboxes is the inherent difficulty of predicting the potential risks associated with a truly novel innovation, and thus introducing unknown risks, no matter how small, to the market (Attrey, Lesher, and Lomax 2020). Second, regulatory sandboxes are time and resource intensive programs, providing personalized advice to sandbox participants on very specific challenges that they may face. This naturally limits the ability of regulatory sandboxes to scale, while also putting significant pressure on the budgets of regulators. Third, regulators are typically specialized in one domain, but innovations can be cross-sectoral, creating challenges for proper design and operation of regulatory sandboxes in such situations. Fourth and finally, regulatory sandboxes are recent tools, and evidence of their effectiveness is limited. This, coupled with the significant investments needed to set up sandboxes, means that there are legitimate questions regarding their value for money, and whether alternatives could be better. DIGITAL-IN-HEALTH FLAGSHIP PROGRAM Other Resources Links to exit reports of regulatory sandboxes for digital health: France’s CNIL: https://www.cnil.fr/en/digital-health-and-edtech-cnil-publishes-results-its-first- sandboxes Norway’s Datatilsynet: https://www.datatilsynet.no/en/regulations-and-tools/sandbox-for- artificial-intelligence/reports/ UK ICO: https://ico.org.uk/for-organisations/advice-and-services/regulatory- sandbox/previous-participants/ Relevant External Case Studies Mass Digital Health Success Stories: https://massdigitalhealth.org/sandbox-success-stories Malaysia National Technology and Innovation Sandbox (NTIS) snapshot: https://drive.google.com/file/d/16e5FtSJPRq19rzpN3c-a7JdhDYG2rrEH/view IMPLEMENTATION KNOW-HOW BRIEF | 16 REGULATORY SANDBOXES FOR DIGITAL HEALTH Regulatory Sandboxes for Digital Health Checklist This checklist can be printed as a stand-alone document. Define the leadership model  Define the entity responsible for leading the design, implementation, and evaluation Identify problems to be addressed and emerging technologies  Determine whether a sandbox is the most appropriate solution for your context. Establish the key goals of the regulatory sandbox and what it seeks to discover Map the relevant regulations  Collaboratively identify the laws, regulations, norms, and standards that are applicable, and map out possible outcomes. Introduce mandate or administrative act to allow creation DIGITAL-IN-HEALTH FLAGSHIP PROGRAM of the sandbox Identify entities that could be involved  Identify/invite all relevant entities to participate under clear participation mechanisms. Build capacity  Provide training and capacity-building for all entities involved. Establish working group(s)  Establish models for participating entities to coordinate and work together. Define risk model for implementation  Identify possible effects of the regulatory sandbox and establish contingency and mitigation plans. Decide on the design of the regulatory sandbox  Set the design elements of the regulatory sandbox (eligibility, governance, timing, test restrictions, and exit options) to achieve its stated objectives. Publish plans for public consultation  Have a period of public consultation and respond to comments received in a public forum. Launch and implement  Launch the regulatory sandbox allowing innovators to apply and implementing planned design. Monitor, evaluate, and learn  Have a measurement framework in place, and publish exit and final reports. IMPLEMENTATION KNOW-HOW BRIEF | 17 REGULATORY SANDBOXES FOR DIGITAL HEALTH Project-level Considerations The Technical Guide “How to Build a Regulatory Sandbox” published by the Consultative Group to Assist the Poor (or CGAP, a global partnership of more than 30 leading development organizations), provides four templates to help sandbox teams (Jeník and Duff 2020). As detailed in the Technical Guide: The templates are not intended to be universal blueprints and should always be adapted to specific circumstances. The templates are for the following: Regulatory Sandbox Feasibility Assessment. Helps create a structured, high-level project plan for assessing, designing, and implementing a regulatory sandbox. Preparing and announcing a sandbox framework document is relatively straightforward. However, experience in other jurisdictions suggests that the success of sandbox initiatives requires careful initial evaluation of the legal, regulatory, market, and political economy conditions and the regulatory DIGITAL-IN-HEALTH FLAGSHIP PROGRAM capacity (the threshold restrictions) to help tailor the sandbox and any related initiatives. Sandbox Project Plan. Helps create a Gantt chart with a detailed step-by-step process, assigned responsibilities, and timeline. Internal Operating Guidelines. Outlines the sandbox process and summarizes common activities performed at each stage of the process. It can be used to inform the process flow or facilitate drafting of an internal regulatory sandbox manual where needed. Sandbox Testing Plan. Offers an example of a testing plan. The actual testing plan always must be adapted to the regulatory sandbox framework and the innovation being tested. Regulators should not attempt to create a one-size-fits-all testing plan.” Refer to the Technical Guide for more details. The German Federal Ministry for Economic Affairs and Energy (BMWi) has published a handbook for regulatory sandboxes, which includes the following set of questions that policy makers considering the use of regulatory sandboxes for digital health should consider (BMWi 2019): ▪ What are the key goals of the regulatory sandbox? ▪ What does the regulatory sandbox seek to discover? ▪ How can attainment of the goals be measured? ▪ Are there networks in place already which can participate in the regulatory sandbox? ▪ How else can potential participants be grouped? ▪ How will cooperation within the network be governed? ▪ What are the timings for sandbox preparation, planning and implementation? ▪ What resources will need to be allocated to each phase of the development process? ▪ What public and private funding sources are there to meet needed resources? ▪ What legal areas and legal provisions are important for the regulatory sandbox? IMPLEMENTATION KNOW-HOW BRIEF | 18 REGULATORY SANDBOXES FOR DIGITAL HEALTH ▪ What rules or regulations may act as barriers to the regulatory sandbox? ▪ What exemptions or experimentation clauses are available to enable the regulatory sandbox? ▪ What conditions are needed to issue an exemption? ▪ Which authorities are responsible to issuing an exemption? ▪ Have similar exemptions been issued in the past, and in what circumstances? ▪ What are the potential risks to users, observers and third parties of this sandbox? ▪ Who would be liable for any damages associated with operation of the sandbox? ▪ How can risks be insured? ▪ How much time will be needed to attain the goals of the regulatory sandbox? ▪ Who are the key stakeholders that need to be involved in the regulatory sandbox? DIGITAL-IN-HEALTH FLAGSHIP PROGRAM ▪ What roles and expectations are associated with each stakeholder? ▪ What are the interests surrounding the regulatory sandbox? ▪ Who will evaluate the regulatory sandbox? ▪ How will evaluation results lead to actions? ▪ What indicators are suitable to evaluate the impact of the regulatory sandbox? ▪ What data are already available? What data need to be collected? ▪ What are the reporting requirements to stakeholders regarding the sandbox? ▪ How will results from regulatory sandbox use inform legislative/regulatory actions? IMPLEMENTATION KNOW-HOW BRIEF | 19 REGULATORY SANDBOXES FOR DIGITAL HEALTH Acknowledgements This implementation know-how brief was written by Tiago Cravo Oliveira Hashiguchi, Malarvizhi Veerappan, and Marelize Görgens. It benefited greatly from comments and feedback from Ivo Jeník and from experts in the United Kingdom’s Care Quality Commission. The brief was graphically designed by Theo Hawkins. The development of the implementation know-how brief series was prepared under the supervision of Malarvizhi Veerappan and Marelize Görgens. Background on Implementation Know- How Briefs DIGITAL-IN-HEALTH FLAGSHIP PROGRAM What is an Implementation Know-how Brief and What is it for? Implementation Know-How Briefs serve as practical, implementable extensions to the Digital Health Flagship Program report. The Implementation Know-How Briefs take a practical approach to discussing a topic with the aim of describing the topic, the key terms and technical considerations, guidance on how to start an operational engagement with clients on the topic, relevant checklists (if applicable), links and places to go for help. The aim of Implementation Know-How Briefs is to give Task Teams enough information to figure out how a given topic fits into Health, Nutrition and Population (HNP) investments, and what are the right questions to ask. The aim is not to make Task Teams topic experts. The Implementation Know-How Briefs also tackle the dependencies between different topics. Who is this Implementation Know-How Brief For? The Implementation Know-How Briefs are focused on World Bank Task Teams and extend the discussion on the topics covered in the Digital Health Flagship Program report. Who is Responsible for Implementation Know-How Briefs? Digital Health Flagship Research Program: digitalinhealth@worldbank.org. IMPLEMENTATION KNOW-HOW BRIEF | 20 REGULATORY SANDBOXES FOR DIGITAL HEALTH | ANNEXES Annex 1 Theory of Change Regulatory Sandboxes for Digital Health Theory of Change Limited uptake of Regulatory Expected longer data and digital sandboxes for Outcomes term impact health innovation digital health DIGITAL-IN-HEALTH FLAGSHIP PROGRAM ▪ Innovators are ▪ Understand when Shorter- and medium- ▪ Universal and equitable discouraged from regulatory sandboxes term access to affordable, entering the complex are useful and when ▪ Innovators have better people-centered, and highly regulated health other approaches are access to funding, to integrated quality care market due to better markets and to ▪ Good governance of uncertainty ▪ Establish leadership regulatory expertise health systems for ▪ Regulatory uncertainty and governance ▪ Experimentation in sustainable financing increases risks and models for the controlled environment and accountability for costs for innovators, sandbox leads to more health outcomes and consequently for ▪ Determine set of evidence-based ▪ Augmented service health care providers challenges addressed decisions and better delivery value chain and patients and technologies of products/services ▪ Reinvigorated essential ▪ Patients and interest ▪ Safer, more effective, public health functions populations have ▪ Map the relevant and potentially cheaper limited or no access to regulations and products and services innovative data and establish legal clarity available to patients digital health products for sandbox and communities and services ▪ Invite all interested ▪ Regulations that ▪ Adopters of parties, build capacity, provide a better innovations are unsure form working groups, balance between whether claims made decide on risk model, protecting the health by innovators are and define the design care system and trustworthy of the sandbox patients, and ▪ Regulators have ▪ Publish plans for public promoting innovation limited insight into the consultation before Longer-term innovation pipeline and launch and ▪ Better health system how new technologies implementation performance due to affect and are affected ▪ Monitor, evaluate, and rapid access to well- by existing and new learn, documenting regulated data and regulations and publishing lessons digital health and consequences of innovations sandbox REGULATORY SANDBOXES FOR DIGITAL HEALTH | ANNEXES Notes 1 See (Leckenby et al. 2021) and the United Kingdom’s Financial Conduct Authority’s webpage on regulatory sandboxes, available from https://www.fca.org.uk/firms/innovation/regulatory- sandbox 2 See (Leckenby et al. 2021; ANPD 2023; National Digital Health Mission 2020), as well as the Singapore Ministry of Health’s press release on its first regulatory sandbox to support the development of telemedicine available from https://www.moh.gov.sg/news- highlights/details/moh-launches-first-regulatory-sandbox-to-support-development-of- telemedicine, and the Bank Rakyat Indonesia sandbox webpage available from https://developers.bri.co.id/en/news/get-know-sandbox-payment-business-system. 3 See (Jeník and Duff 2020). Established in 1995 and housed at the World Bank, CGAP is DIGITAL-IN-HEALTH FLAGSHIP PROGRAM supported by over 30 leading development organizations committed to making financial services meet the needs of poor people. 4 The Licensing Experimentation and Adaptation Programme (LEAP) was launched in 2018, and included a regulatory sandbox for telemedicine and mobile medicine to inform the licensing of these new service delivery models. In February 2021, the Ministry closed the sandbox as it had achieved its objectives. For more information see https://www.moh.gov.sg/home/our- healthcare-system/licensing-experimentation-and-adaptation-programme-(leap)-a-moh- regulatory-sandbox 5 See the Wikipedia page for sandbox in the context of software development: https://en.wikipedia.org/wiki/Sandbox_(software_development). 6 See the CQC webpage available from https://www.cqc.org.uk/what-we-do/how-we-work- people/regulatory-sandbox 7 See the ICO webpage available from https://ico.org.uk/for-organisations/advice-and- services/regulatory-sandbox/the-guide-to-the-sandbox/. 8 See the MHRA announcement available from https://www.gov.uk/government/news/mhra-to- launch-the-ai-airlock-a-new-regulatory-sandbox-for-ai-developers. 9 See Kernow Health CIC webpage available from https://www.cornwallti.com/2019/04/02/kernow-health-cic-supercharges-cornish-healthtech- innovation-with-new-sandbox-offering/, and Digital Jersey webpage available from https://www.digital.je/choose-jersey/sandbox-jersey/digital-health-sandbox/. 10 See Mass Digital Health webpage available from https://massdigitalhealth.org/mass-digital- health-programs/digital-health-sandbox-program. 11 See the NTIS progress report 2020-21 available from https://drive.google.com/file/d/1LbICiI7rEPzuH_ShatD-s-CCsUSJ6lYl/view. Some of the health care innovations included in the sandbox can be found in the report titled “2 years of accelerating innovations" available from https://drive.google.com/file/d/16e5FtSJPRq19rzpN3c- a7JdhDYG2rrEH/view. 12 See Datatilsynet webpage available from https://www.datatilsynet.no/en/news/aktuelle- nyheter-2022/sandbox-forever/. REGULATORY SANDBOXES FOR DIGITAL HEALTH | ANNEXES 13 See the report available from https://www.sic.gov.co/sites/default/files/files/2021/150421%20Sandbox%20on%20privacy% 20by%20design%20and%20by%20default%20in%20AI%20projects.pdf 14 See the CNIL webpage available from https://www.cnil.fr/en/digital-health-and-edtech-cnil- publishes-results-its-first-sandboxes 15 See, for example, the following press release available from https://en.antaranews.com/news/278697/health-ministry-launches-regulatory-sandbox-to- secure-telemedicine. 16 See dissemination of launch event available from https://digital- strategy.ec.europa.eu/en/events/launch-event-spanish-regulatory-sandbox-artificial- intelligence. DIGITAL-IN-HEALTH FLAGSHIP PROGRAM REFERENCES ANPD. 2023. “Sandbox Regulatório - Estudo Técnico ANPD.” Brasília: Autoridade Nacional de Proteção de Dados. https://www.gov.br/anpd/pt-br/documentos-e-publicacoes/documentos-de- publicacoes/sandbox_regulatorio___estudo_tecnico__versao_publica_.pdf/view. Appaya, Mandepanda Sharmista, Helen Luskin Gradstein, and Mahjabeen Haji Kanz. 2020. “Global Experiences from Regulatory Sandboxes.” Text/HTML. Washington, D.C.: World Bank. https://documents.worldbank.org/en/publication/documents- reports/documentdetail/912001605241080935/Global-Experiences-from-Regulatory-Sandboxes. Appaya, Sharmista, and Mahjabeen Haji. 2020. “Four Years and Counting: What We’ve Learned from Regulatory Sandboxes.” Private Sector Development Blog (blog). November 18, 2020. https://blogs.worldbank.org/psd/four-years-and-counting-what-weve-learned-regulatory- sandboxes. Armstrong, Harry, and Jen Rae. 2017. “A Working Model for Anticipatory Regulation.” London: Nesta. https://media.nesta.org.uk/documents/working_model_for_anticipatory_regulation_0.pdf. Attrey, Angela, Molly Lesher, and Christopher Lomax. 2020. “The Role of Sandboxes in Promoting Flexibility and Innovation in the Digital Age.” Going Digital Toolkit Notes 2. Vol. 2. Going Digital Toolkit Notes. https://doi.org/10.1787/cdf5ed45-en. BMWi. 2019. “Making Space for Innovation - The Handbook for Regulatory Sandboxes.” Berlin: Federal Ministry for Economic Affairs and Energy (BMWi). https://www.bmwk.de/Redaktion/EN/Publikationen/Digitale-Welt/handbook-regulatory- sandboxes.pdf?__blob=publicationFile&v=2. Español, Armando Guío. 2020. “Modelo Conceptual Para El Diseño de Regulatory Sandboxes & Beaches En Intelegencia Artificial.” Bogotá: Consejería Presidencial para Asuntos Económicos y Transformación Digital. REGULATORY SANDBOXES FOR DIGITAL HEALTH | ANNEXES https://dapre.presidencia.gov.co/AtencionCiudadana/DocumentosConsulta/consulta-200820- MODELO-CONCEPTUAL-DISENO-REGULATORY-SANDBOXES-BEACHES-IA.pdf. FCA. 2015. “Regulatory Sandbox.” London: Financial Conduct Authority. https://www.fca.org.uk/publication/research/regulatory-sandbox.pdf. Herrera, Diego, and Sonia Vadillo. 2018. “Regulatory Sandboxes in Latin America and the Caribbean for the FinTech Ecosystem and the Financial System.” Washington, D.C.: Inter-American Development Bank. https://publications.iadb.org/en/regulatory-sandboxes-latin-america-and-caribbean-fintech- ecosystem-and-financial-system. Jeník, Ivo, and Schan Duff. 2020. “How to Build a Regulatory Sandbox: A Practical Guide for Policy DIGITAL-IN-HEALTH FLAGSHIP PROGRAM Makers.” Technical Guide. Washington, D.C.: CGAP. https://documents1.worldbank.org/curated/en/126281625136122935/pdf/How-to-Build-a- Regulatory-Sandbox-A-Practical-Guide-for-Policy-Makers.pdf. Leckenby, Emily, Dalia Dawoud, Jacoline Bouvy, and Páll Jónsson. 2021. “The Sandbox Approach and Its Potential for Use in Health Technology Assessment: A Literature Review.” Applied Health Economics and Health Policy 19 (6): 857–69. https://doi.org/10.1007/s40258-021-00665-1. National Digital Health Mission. 2020. “NDHM Sandbox.” https://abdm.gov.in:8081/uploads/sandbox_guidelines_b39bcce23e.pdf.