SOCIAL PROTECTION AND JOBS DISCUSSION PAPER No 2306 | SEPTEMBER 2023 NOVISSI TOGO Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection Cina Lawson Morlé Koudeka Ana Lucía Cárdenas Martínez Luis Iñaki Alberro Encinas Tina George Karippacheril © 2023 International Bank for Reconstruction and Development/The World Bank. 1818 H Street NW, Washington, DC 20433, USA. Telephone: 202–473–1000; Internet: www.worldbank.org. Some rights reserved This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other in- formation shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) http://creativecommons.org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, trans- mit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: Lawson, Cina, Morlé Koudeka, Ana Lucía Cárdenas Martínez, Luis Iñaki Alberro Encinas, and Tina George Karippacheril. “Novissi Togo: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection.” ©World Bank Group, Washington, DC. License: CC BY 3.0 IGO. Third‑party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third-party-owned individual component or part contained in the work will not in- fringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202–522–2625; e-mail: pubrights@ worldbank.org. Cover design: Andres de la Roche Cover photo: Novissi Togo Typesetting and layout: Michael Alwan ii NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection Contents Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v I. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 II. Enabling environment and program overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 III. A deep dive into NOVISSI’s high-tech delivery model (model 2). . . . . . . . . . . . . . . . . . . . . . 11 1. Outreach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2. Intake and registration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3. Assessment of needs and conditions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4. Eligibility determination, enrollment, and notification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5. Payments administration, provision, and reconciliation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 6. Beneficiary operations management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 IV. Preliminary assessment: achievements and challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Potential efficiency gains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Financial inclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Risks of exclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Model limitations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Performance of NOVISSI’s model 2 approach against other methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 V. Going forward. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Annex. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Contents iii Boxes Box 1. Supervised machine learning algorithms in a nutshell......................................................................................................................................15 Figures Figure 1. Existing social protection programs needed to be shock-responsive to expand horizontally................................................. 2 Figure 2. How NOVISSI Works: Journey mapping the perspectives of beneficiaries and administrators............................................. 6 Figure 3. NOVISSI’s implementation timeline..................................................................................................................................................................... 11 Figure 4. NOVISSI’s Model 2 delivery chain process mapping...................................................................................................................................12 Figure 5. Prioritizing the poorest cantons in Togo (Model 2).....................................................................................................................................16 Figure 6. Prioritizing the poorest individuals within the poorest cantons (Model 2)......................................................................................17 Figure 7. Performance of feasible targeting approaches..............................................................................................................................................25 Figure 8. Performance of different targeted approaches in the context of a hypothetical nationwide program..........................26 Tables Table 1. NOVISSI (model 1 and model 2) compared to targeted cash transfer programs............................................................................... 5 Table 2. Data sources used for assessing needs and conditions...............................................................................................................................14 Table A.1. Overview of NOVISSI’s stakeholders...............................................................................................................................................................29 iv NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection Acknowledgments This case study was jointly prepared by the Government of Practice Manager, Social Protection and Jobs for West and Togo and the World Bank as a Social Protection and Jobs Central Africa. The authors gratefully acknowledge helpful Paper. It is also to be included as part of the “Innovations insights from Joshua Blumenstock and Emily Aiken (Univer- in social assistance programs in data scarce environments. sity of Berkeley), Han Sheng Chia (GiveDirectly), Phillippe Use of non-traditional approaches and novel data sources Leite, Sveta Milusheva, Emil D. Tesliuc, Joachim Boko, Julian during COVID and beyond” by Yuko Okamura, Tim Ohlen- Koschorke, Aissatou Ouedraogo, Felicien Donat Edgar burg and Emil D. Tesliuc with financing from the G2Px trust Towenan Accrombessy (World Bank), Attia Byll, and Kafui fund. Overall guidance was provided by Christian Bodewig, Ekouhoho (MENTD). Acknowledgments v Photo: © Stephan Gladieu / World Bank I. Introduction The COVID-19 crisis drove reversals in global poverty Fund and a grant from the French Development Agency reduction in 2020, with an estimated 40 million new poor (Agence Française de Développement, AFD in French). The in Sub-Saharan Africa alone1 (World Bank, 2020). With expansion of the transfers to rural areas was financed by people at the heart of the COVID-19 recovery, more than the American not-for-profit organization GiveDirectly. 200 countries around the world launched over 3,000 social NOVISSI’s delivery systems were funded with government assistance measures in record time to tackle the economic spending and partly through International Development and social impacts of the crisis (Gentilini et al., 2021). Togo Association (IDA) from the World Bank. With a spend of was no exception. Located in West Africa, Togo is home US$34 million, NOVISSI benefited over 920,000 of the poor to 8.3 million people with a GDP per capita of US$888 and and vulnerable (around 25 percent of the adult population poverty incidence standing at 45 percent using the national of Togo), 63 percent of whom were women. poverty line2 (World Bank, 2021e). On April 8, 2020, just one month after the country’s first reported case of COVID-19,3 Prior to the pandemic, steady economic growth contrib‑ the Government of Togo launched NOVISSI, which means uted to moderate declines in poverty, yet some of those solidarity in the local Éwé language. NOVISSI is a large-scale gains slowed due to COVID-19. Between 2008 and 2019, unconditional emergency cash transfer program that initially Togo grew at a 5.7 percent per annum pace and benefited supported informal workers whose incomes were disrupted from a stable macroeconomic framework. Togo’s growth or negatively affected by containment measures and was momentum was driven by the agriculture and service later extended to rural areas with higher poverty incidence. sectors, contributing to a decrease in headcount poverty This initiative was a milestone for the social protection rates, from 61.7 percent in 2006 to 45.5 percent in 2018–20194 sector in Togo and, indeed, internationally, having been (INSEED, 2020). While health impacts have been limited, the designed from scratch in ten days and delivered digitally effects of the COVID-19 pandemic are estimated to have from end to end. The program and the digital platform severely impacted welfare, leading to an increase in food underpinning it were developed and implemented by the insecurity and extreme poverty,5 particularly in rural areas Government of Togo under the Ministry of Digital Economy (World Bank, 2021e). Poverty continues to be widespread, and Transformation (Ministère de l’Économie Numérique et especially in rural areas, where 58.8 percent of people live in de la Transformation Digitale, MENTD in French) with over- poverty compared to 26.5 percent in urban areas.6 Limited sight from an interministerial steering committee. NOVIS- economic opportunities in productive activities have SI’s transfers in urban and peri-urban areas were financed perpetuated low levels of income, and informality levels through Togo’s National Solidarity and Economic Recovery are striking, with more than 85 percent of the labor force 1. Measured using the international poverty line of US$1.90/day. 2. Corresponding to 4.3 million people under the national poverty line, which is estimated at CFA 273,619 per capita per year (about US$475). 3. The first COVID-19 case in the country was reported on March 6, 2020. 4. Based on the national survey, Questionnaires Unifiés des Indicateurs de Base du Bien-être (QUIB) which is not directly comparable to the Enquête Har- monisée des Conditions de Vie des Ménages (EHCVM) 2018–2019. 5. Food poverty is defined as the share of the population whose food consumption is below the food poverty line; extreme poverty is defined as the proportion of the population whose total consumption (including food, rent, clothing, energy, health expenditures, and education) is below the food pov- erty line (World Bank 2020e). 6. INSEED, 2020. I. Introduction 1 working in the informal sector. With the unfolding COVID-19 tion insurance for civil servants and formal workers, and crisis, poverty conditions are expected to worsen, partic- price subsidies on fuel, electricity, transport, and agriculture. ularly among persons working in economic sectors such as tourism, transport and logistics, manufacturing, agriculture, Indeed, the existing social protection paradigm left infor‑ and agribusiness, which were severely affected by internal mal economy workers out of the picture, emerging as the and external shocks (World Bank, 2020a). “new poor” in the aftermath of the COVID-19 pandemic. The informal sector forms the backbone of most devel- While the Government of Togo has made significant oping economies and employs a wide range of individuals, efforts to lay down the foundations of a robust social including smallholder farmers, street vendors, small traders, protection system over the past decade, its weaknesses porters, casual laborers, artisans, hairdressers, fabric resellers, were exposed by the crisis. Government spending on social tailors, among others. In Sub-Saharan Africa, 89 percent of protection and jobs programs has moderately increased employed women and girls are in the informal economy over the last decade from 1.85 percent of GDP in 2009 to workforce, accounting for 80 percent of total employment. 2.06 percent in 2018 but remains below the levels of regional They fall through the cracks of existing social protection peers such as Mali and Senegal. Several programs were in programs as they are often not eligible for social safety net place before the COVID-19 shock and continue to operate. benefits, and being outside of the formal economy, they However, these programs are limited in geographic scope are ineligible for social insurance programs mandated for and coverage. Recent evidence indicates low coverage of the formal sector. They are difficult to reach and tend to be social protection, with only 38 percent of the poor receiv- mobile. Furthermore, the irregular and low earnings of infor- ing some kind of social safety nets transfer or subsidy in mal workers leave them particularly vulnerable to economic 2019 (World Bank Group, 2021d). Existing social protection shocks. In Togo, 84 percent of women and 74 percent of programs include a national school feeding program, social men work in the informal sector, with a higher incidence in safety nets transfers for the rural poor, public works, and rural areas. The containment measures to curb the spread of entrepreneurship programs for rural youth, social protec- the virus impacted domestic economic activity, dispropor- FIGURE 1 Existing social protection programs needed to be shock-responsive to expand horizontally Source: Adapted from Bowen et al. (2020). 2 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection tionately affecting workers in the informal economy, partic- NOVISSI was launched on April 8, 2020, it initially priori- ularly women who work in the hardest hit sectors, implying tized informal workers in areas under a declared state of severe consequences for the well-being of children’s health, emergency—mainly located in urban areas—that involved nutrition, and education (World Bank 2020e). Existing social strict social distancing measures (henceforth referred to protection programs and delivery systems in Togo were as model 1). Subsequently, the program was extended on not designed to reach the informal sector, limited to those November 2, 2020, to rural areas with a high prevalence of already beneficiaries of programs (Figure 1). poverty (henceforth referred to as model 2). By August 2021, over 920,000 individuals had benefited from the program, The response to shock caused by the COVID-19 pandemic of which 63 percent were women. stress-tested the capabilities of social protection deliv‑ ery systems worldwide. Countries with universal coverage This case study, jointly authored by the Government of Togo of unique identification systems and social registries cover- and the World Bank, documents the innovative features of ing a large part of the population with frequently updated the NOVISSI program and posits some directions for the data on households and individuals were better prepared way forward. The study examines how Togo leveraged artifi- to deliver social assistance to those in most need.7 In the cial intelligence and machine learning methods to prioritize absence of both a robust shock-responsive social protec- the rural poor in the absence of a shock-responsive social tion policy framework and recent comprehensive data on protection delivery system and a dynamic social registry.8 household income and wealth, Togo’s ability to implement It also discusses the main challenges of the model and the shock-responsive social assistance using existing methods risks and implications of implementing such a program. The was put to the test. Moreover, traditional in-person data study builds on consultations with program staff, partners, intake, registration, and cash payment delivery methods and academic researchers who provided technical support proved impractical for crisis response, requiring minimal to the program’s design. The study is organized as follows. physical contact to avoid the spread of the virus and agility Section II provides an overview of the NOVISSI program and to identify and register vulnerable people. the context in which it was deployed. Section III describes the delivery processes of NOVISSI’s model 2 (high-tech The Government of Togo took swift decisions to innovate, approach) and dives deeper into the implementation of breaking with traditional paradigms and offering a mobile advanced spatial analysis techniques and mobile call detail phone-enabled delivery of a social protection program to records to assess needs and conditions to determine eligi- affected people. In record time, NOVISSI introduced a 100 bility. Section IV presents a preliminary assessment of the percent “mobile phone-enabled” approach for on-demand results of the program and discusses the main challenges self-registration, determination of eligibility, and payment and limitations of the model. Finally, Section V concludes of benefits. The program leveraged available sources of with a discussion of the lessons learned and what NOVISSI’s government administrative data sources, combined with innovations can implicate for the future of social protec- satellite imagery, data science, and machine learning meth- tion delivery. ods, to prioritize those most affected by the crisis. When 7. For instance, Egypt, Turkey, Peru, and Argentina proved agile with interoperable systems, pulling administrative data from different databases that tend to be updated more frequently (Gentilini et al., 2021) 8. Social registries are information systems that provide a gateway for potential inclusion into social protection programs. They support the “Assess” stage of the delivery process—from outreach to intake and registration to the assessment of needs and conditions—to help social systems determine potential eligibility for one or more programs. When Social Registries are dynamic, registration to social protection programs is open and continuous for all house- holds, families, and individuals, allowing them to register when needed and update their information as it changes (Playbook on Dynamic Social Registries and Interoperable Social Protection Information Systems, forthcoming). I. Introduction 3 II. Enabling environment and program overview At the onset of the COVID-19 crisis, the Government program to scale up faster than traditional cash transfer of Togo swiftly enacted strict policies9 to manage the programs, which rely heavily on face-to-face interactions spread of the disease, accompanied by interventions to and prevalent statistical methods to infer well-being (Table limit socioeconomic impacts. Containment measures to 1). NOVISSI was facilitated by high mobile phone pene- curb the spread of the virus disproportionately affected tration rates and near-universal mobile phone coverage. the informal economy. The majority of the labor force Recent estimates from the Enquête Harmonisée des Condi- comprises workers who earn their livelihoods in the infor- tions de Vie des Ménages (EHCVM) 2018–2019 indicate that mal economy, with little to no access to social protec- 65 percent of individuals in Togo own a mobile phone (54 tion (World Bank 2021d). On the external side, global trade percent in rural areas), and 85 percent belong to a house- disruptions combined with the country’s response to the hold with one or more phones. At the end of 2020, SIM card pandemic affected export value chains. Critical economic penetration was 83.6 percent in Togo (ARCEP, 2021). While sectors were affected by internal and external shocks, which some people still do not possess a mobile phone, owning a translated into price increases in goods and services, includ- SIM card and using it on a family member or friend’s mobile ing food staples and imports. The resulting income losses device is not uncommon. Recent estimates using NOVISSI’s and consumption shocks likely forced an additional 0.3 administrative data show that 22 percent of SIM cards and 7 million people into extreme poverty (World Bank Group, percent of SIM slots are shared in Togo (Aiken, Takhur, and 2021e). With limited government spending on safety nets, Blumenstock, 2021). low-income families are typically forced to rely on detri- mental coping strategies that disrupt long-term human capi- NOVISSI’s assistance unit were individual informal sector tal formation. To counter these effects, the Government workers. This was a policy choice made by the Govern‑ laid down an emergency response plan to protect the lives, ment of Togo, given the context of the pandemic and livelihoods, and future growth prospects to mitigate the data constraints. In the absence of a dynamic social regis- negative impacts of the pandemic. Among others, the plan try with up-to-date data on households and individuals, and sought to prevent an increase in poverty by introducing an without unique identifiers with universal coverage, some unconditional emergency cash transfer and electricity and households may have received multiple transfers, provided water utility waivers. that the assistance unit was determined to be individuals. It follows therefore that NOVISSI did not set out to ensure Togo implemented a fully digital process supported by just one beneficiary within a household. mobile devices without requiring in-person contact to register, enroll, and transfer financial aid to thousands From the potential beneficiary’s perspective, NOVISSI of vulnerable people. This approach allowed the NOVISSI operated in three simple steps (Figure 2): 9. Togo’s containment measures included, among others, cordoning off villages under a declared state of emergency, curfews, and border closures. 4 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection TABLE 1 NOVISSI (model 1 and model 2) compared to targeted cash transfer programs Characteristic NOVISSI model 1 NOVISSI model 2 Social safety nets 1 Low-tech approach High-tech approach Geographic coverage Urban and peri-urban Rural Rural 200 poorest cantons based on Poverty maps based on Areas under declared state satellite imagery, population Geographic assessment 2011 household surveys of emergency census data and machine conducted by INSEED learning Categorical filter based on Welfare assessment based Proxy means testing and Individual assessment occupation and location as on machine learning and community validation recorded in voter list anonymized call detail records Benefit amount Women: 12,250 monthly Women: 8,170 per installment 5,000 monthly (CFA) Men: 10,500 monthly Men: 7,000 per installment Monthly (100 poorest cantons Bi-weekly (half of the benefit in phase 1) and then bi-weekly Frequency of payments amount was disbursed Quarterly (following 100 poorest cantons bi-weekly) phase 2) 5 months for the 1st phase (100 0.5 to 2 months, depending poorest cantons) and then 2.5 Duration of benefits on the length of the state of 24 months months (following 100 poorest emergency by area cantons) for the 2nd phase 1 to 4, depending on the Number of installments length of the state of 5 8 emergency by area On-demand self-registration On-demand self-registration Registration In-person survey via USSD via USSD Initially cash based. At the Payment Mobile money Mobile money onset of COVID-19, mobile money February 2018 – December Period April 2020 – March 2021 November 2020 – August 2021 2021 Number of beneficiaries 819,972 138,531 40,309 Share of female beneficiaries 63% 52% 60% Number of registrants 1,632,942 519,972 N.A.  2 Government and AFD grant Source of Funds as refinancing of the initial GiveDirectly IDA grant transfer Spending (USD) 23.9 million 10 million N.A. Source: Authors’ elaboration using administrative data from NOVISSI and World Bank Group (2021c). Note: 1. The Togo Safety Nets and Basic Services Project (Filets Sociaux de Base, FSB in French) seeks to provide poor communities and households with greater access to basic socioeconomic infrastructure and social safety nets, including cash transfers and school feeding. Data are reported for the cash transfer component under the existing arrange- ments when NOVISSI was deployed. Note that additional financing has been recently approved, entailing some improvements in the project design. 2. N.A. Not available II. Enabling environment and program overview 5 FIGURE 2 How NOVISSI Works: Journey mapping the perspectives of beneficiaries and administrators Learns about Novissi NOVISSI HOW IT WORKS through peers (Word of mouth), radio, community leaders *855# and targeted SMS ID NUMBER NAME CONSENT BENEFICIARY'S JOURNEY SELF REGISTRATION NOTIFICATION PAYMENT Receives Dials *855# and Receives notification on provides payment through registration (via minimal data via mobile money USSD) and on USSD. account. eligibility status to the program (via SMS) ASSESSMENT OF ELIGIBILITY & INTAKE & NEEDS AND ENROLLMENT PAYMENT OUTREACH REGISTRATION CONDITIONS DECISIONS NOTIFICATION ADMINISTRATION MANAGEMENT ADMINISTRATIVE PROCESSES Multi-layered Data pulled Geographic assessment Cross-validation Immediate Instant Beneficiary outreach from various (areas under curfew of registrants’ notification mobile operations strategy sources, (model 1) or poorest data to of money management supported by including districts (model 2)) and determine enrollment payments through a massive radio administrative individual assessment eligibility and status toll-free number campaigns, data, big data, (based on occupation benefit amount, to address community and (model 1) or poverty level based on complaints, leaders, and self-reported assessment using predefined external audits, targeted SMS data machine learning and criteria and and dashboard algorithms (model 2)). gender, for real-time respectively. analytics At every step, a specialized call center is available ASSESS ENROLL PROVIDE MANAGE to handle problems and complaints. THE DELIVERY CHAIN Source: Original figure for this publication. 1. Self-registration. Interested individuals would to utilize their personally identifiable information (PII) self-register by dialing *855# from a mobile phone to be further assessed by the NOVISSI program. and inputting basic information in an Unstructured Supplementary Service Data (USSD) form for intake 2. Notification. Applicants would be notified by USSD and registration. This information included identifica- about their registration in the program and will then tion number (from the voter list),10 name, and consent get an SMS to inform them about their eligibility status and enrollment decision.11 The eligibility requirements 10. An administrative dataset produced by the national independent electoral commission. The voter list was used as a means to identify beneficiaries as it was the most widespread, up-to-date, and structured database available to the program. About 86 percent of the adult population owns a voter’s ID cre- dential, compared to about 30 percent of people who own a national ID card. 11. Eligibility decisions consist of determining which applicants qualify for a program based on pre-defined eligibility criteria. Enrollment decisions, that is to say, whether an eligible applicant is enrolled in the program (becomes beneficiary) or not (waitlisted) are typically made based on budget and operational capacity criteria (Lindert et al., 2020). In NOVISSI, all eligible applicants were enrolled in the program. 6 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection included being a Togolese citizen and resident, aged 18 applicants received an SMS acknowledging registration years or older, possessing a valid voter ID credential, and informing them that they would be considered for living in one of the eligible areas12,13 and meeting the financial aid if they became eligible for the program. individual vulnerability assessment criteria based on occupation14 (model 1) or welfare status (model 2). 2. Enroll. Registrants’ profiles were cross-referenced against eligibility criteria. Subsequently, eligible appli- 3. Payment. Immediately after eligibility notification, cants received a second notification informing them beneficiaries would receive their first payment on the about the enrollment decision. Non-eligible individuals mobile money account linked to the phone number were still considered for automatic future enrollment they registered with. For beneficiaries who did not into the program if the eligibility criteria changed (for have a mobile money account before enrollment into example, if a new geographic location became eligible). the program, one was automatically created by their All eligible applicants were enrolled in NOVISSI. corresponding mobile network operator (MNO). The know-your-customer (KYC) processes were conducted 3. Provide. Enrolled beneficiaries received the emer- by MNOs using their own procedures at the time gency cash transfer in their mobile money account. of account opening. The amount disbursed ranged This involved, among other key partnerships, strong from CFA 10,500 for men to CFA 12,250 for women, collaboration with MNOs to ensure interoperabil- per month, in model 1 (with half of the amount being ity between the NOVISSI platform and their mobile disbursed every 2 weeks), and CFA 7,000 for men to CFA payment systems. 8,170 for women, per installment, in model 2 (Table 1). 4. Manage. Accompanying measures such as a solid griev- A toll-free number was set up to accompany the process ance redress mechanism (GRM), external audits, and so that registrants and beneficiaries could reach out to a real-time analytics enabled the program to operate in specialized call center to handle problems and complaints. a transparent and traceable way and adapt to the bene- ficiary’s needs. From an administrative perspective, the beneficiary’s journey was facilitated at the backend by four key steps In the context of limited administrative16 and survey consistent with the Social Protection Delivery Systems data17 combined with restrictions on in-person program Framework15 (Figure 2): delivery, NOVISSI had to innovate to reach individuals who needed assistance the most. Administrative data 1. Assess. Individuals’ characteristics, needs, and condi- collected by governments and service providers during their tions were assessed through several sources, including day-to-day business is an increasingly important source the voter list, big data, and self-reported data via USSD, for evidence-informed policy-making. Survey data can be in order to prioritize people for social assistance. All combined with administrative data to increase the preci- 12. Model 1’s eligible geographic areas were those placed under a declared state of emergency where containment measures were stricter. These areas are predominantly urban or peri-urban and include: Agoè-Nyivé, Golfe, Tchaoudjo, Soudou, Cinkassé, Kpendjal, Kpendjal-Ouest, Oti, Oti-Sud, Tandjouaré and Tône. 13. The 200 poorest cantons were eligible for NOVISSI’s Model 2. 14. 2,328 unique occupations were considered informal and eligible in the scope of NOVISSI model 1. The most common eligible professions included retail- ers, seamstresses, hairdressers, housekeepers, and drivers. 15. Developed by Lindert et al. (2020). 16. Government agencies do not incur additional costs for administrative data collection, so they also do not impose an additional burden on respondents. They typically contain the full population of participants in the program, so the sizes of the datasets are often much larger than those of a statistical survey. Administrative data are often longitudinal, which enables tracking individuals or businesses over time. 17. Household surveys are an important source of socio-economic data. II. Enabling environment and program overview 7 sion of survey estimates with little additional cost. In other residence data. After intense data scrubbing work to cate- cases, different administrative datasets can be combined to gorize occupations, NOVISSI model 1 used the voter list not replace existing surveys or to reduce reliance on survey data only to guarantee the unicity of the beneficiaries but also to (Harris-Koetjin et al., 2017). However, Togo lacks a unique verify whether registrants met geographical and occupation identifier covering all persons in the territory and a dynamic criteria. Individuals were deemed eligible if their occupation social registry, allowing to prioritize those most affected and home residence, as reported in the voters’ list, matched by the shock. In terms of survey data, the first round of the predefined eligible geographic areas and occupations. the EHCVM was carried out in Togo in 2018–2019 and data processing was completed in 2020. The most recent popu- Expanding NOVISSI to rural areas required a more inno‑ lation census was completed in 2011 without collecting vative approach to prioritize the poorest. Hence, delivery welfare or income information. Collecting new survey data model 2, a high-tech approach, was developed to leverage through traditional methods was not possible due to the administrative, survey, and big data. NOVISSI continued health crisis: the country required quick and decisive action relying on the voter list to verify a person’s uniqueness and while being subject to limited fiscal resources and social home residence for rural expansion. However, given insuf- distancing constraints. ficient variation in rural occupations and limited budget to cover entire districts (or cantons), NOVISSI utilized new NOVISSI’s deployment consisted of two complementary technologies and leveraged big data from satellites and delivery models (Table 1). In delivery model 1, a low-tech mobile phone call detail records (CDR)20 to prioritize the approach, the program leveraged the voter list to deter‑ delivery of benefits to the most vulnerable individuals. The mine applicants’ uniqueness, location, and occupation. Government of Togo established two guidelines for this This was a pragmatic choice based on the readily avail- endeavor, first to focus on the poorest districts and second able administrative data on registered voters to allow for to prioritize the poorest individuals within those districts. the verification of unique individuals. The voter list was Togo’s Government partnered with a group of academics compiled based on a mass registration campaign for Togo- from Innovations for Poverty Action (IPA), the Center for lese citizens aged 18 years or older, conducted over a three- Effective Global Actions (CEGA), and the University of Cali- day period at the end of 2019.18 Registered voters received fornia, Berkeley, to build a poverty map of Togo using satel- a biometric voter identity credential for the February 2020 lite imagery. Based on this exercise, Togo’s 397 cantons were presidential elections. A total of 3,633,898 individuals were ranked from poorest to richest.21 The 100 poorest cantons registered, representing an estimated coverage of 86.6 were selected by the Government and GiveDirectly (the percent of the adult population (Aiken et al., 2021).19 The funding partner for NOVISSI model 2) to provide them with voter list includes a unique number for each voter, which, in financial aid. This number was subsequently extended to the the absence of a foundational unique identification system following 100 poorest cantons. Within each selected canton, with universal coverage of the population, was helpful to the poorest individuals were prioritized by leveraging big reliably verify registrants to the NOVISSI program, thereby data and machine learning to estimate individual consump- limiting the duplication of benefits delivery. Additionally, tion using CDR, as described in further detail in Section III. the voter’s registration exercise collected occupation and 18. Voter’s registration took place nationwide and in selected countries between November 29 and December 1, 2020. 19. Alternative functional identification systems had lower coverage and did not include information on key eligibility criteria. According to Government administrative data, over 1.3 million individuals possess a national identity card and almost 0.5 million possess a passport. 20. A call detail record (CDR) is a data record produced by a telephone exchange or other telecommunications equipment that documents the details of a telephone call or other telecommunications transactions (e.g., text message) that passes through that facility or device. The record contains various attri- butes of the call, such as time, duration, completion status, source number, and destination number (Wikipedia). 21. Togo has a total of 397 cantons. Cantons are Togo’s smallest administrative divisions (Admin-3). 8 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection Geospatial data22 sourced from private data producers23 Deploying NOVISSI required the engagement of multi‑ was employed to create fine-grained poverty maps to ple partners. The Government of Togo collaborated with locate the least wealthy cantons. Using machine learning various stakeholders, each of whom played a specific role algorithms, geospatial satellite, connectivity, demographic, in the delivery chain and across delivery models (Annex and geographic data were analyzed to find patterns that 1). Implementation was overseen by the Government of predict wealth and poverty. Images such as roofing materi- Togo. An inter-ministerial steering committee provided als, road surfaces, size plots of land, proximity to bodies of oversight of the program’s implementation and supported water, among others, were interpreted to produce high-res- strategic decision-making. MENTD operated the program, olution estimates of consumption.24 These estimates were including setting up the information system to assess (regis- then combined with population density estimates to ter with available data), enroll (determine eligibility, enroll, generate a weighted average of wealth per canton. The and notify), and provide (deliver the payments). This also calculations were calibrated using the EHCVM 2018–2019 involved outreach to the population, putting in place a toll- household survey containing the exact geo-coordinates free number managed by a call center to address griev- of each surveyed household. This survey data provided a ances, and program monitoring and evaluation. GiveDirectly “ground truth” against which the model was trained. Finally, provided funding and implementation support for NOVIS- cantons were ranked to identify those with a higher concen- SI’s expansion to rural areas (model 2). Academics developed tration of people living under US$1.25 per day. the prioritization26 methodology for the deployment of NOVISSI in rural areas (model 2), which resulted in a potential Afterward, individual-level mobile phone usage data was eligibility list containing minimal data (phone numbers) that analyzed to prioritize the poorest individuals in selected was transferred to MENTD for eligibility check purposes. cantons. Researchers analyzed mobile phone metadata The World Bank, through IDA, provided financing for data containing information on calls, SMS, mobile data usage collection for research and feasibility testing of machine frequencies, and mobile money transactions to predict learning and data-driven methods to prioritize the poor- consumption at the individual level. Specifically, a phone- est, as well as for equipment and audit. Togo’s two lead- based survey was administered among a large and repre- ing mobile phone operators provided anonymized mobile sentative sample of mobile phone subscribers from the phone metadata (also referred to as CDR) to the team of poorest cantons. These data were then matched to each academics for selected periods to support the deployment subscriber’s history of phone usage in order to train a super- of an emergency cash transfer in the midst of a historic crisis. vised machine learning algorithm to predict average daily consumption at the individual level.25 Based on this infor- Transparent institutional arrangements were put in place mation, roughly 750,000 individuals across the country were to mitigate data privacy concerns. To protect the confi- shortlisted as potentially eligible, given their likelihood of dentiality of CDR data, researchers hash-encoded each living under US$1.25 per day. phone number into a unique identifier prior to analysis and stored all data on secure servers. Call-detail records were never transmitted to anyone else, not even the Government 22. As with administrative records, data on weather conditions, water quality, traffic, geospatial data, among others, are not created with the primary inten- tion of being used for statistical or social protection purposes, yet they may provide valuable information for official statistics (Harris et al. 2017). 23. NASA Harvest, University of Berkeley School of Informatics. 24. Using this methodology, a total of 10,119 wealth estimates for 2.4 km2 tiles were produced for Togo. 25. Participants’ consent was requested to match mobile phone metadata to survey responses. 26. Throughout this report, we use the terminology of prioritizing people for social assistance in place of the term “targeting.” See Lindert et al. (2020) for a detailed discussion. II. Enabling environment and program overview 9 of Togo. Only specified researchers accessed them under a those most affected by the COVID-19 pandemic, combined non-disclosure agreement and a data use agreement signed with restrictions on in-person social protection delivery, by the University of Berkeley with each of the two MNOs. NOVISSI was the only feasible alternative at the time for a lightning response. Digital approaches, such as the one Relatively speaking, NOVISSI operated in a policy envi‑ introduced by NOVISSI, have the potential to introduce ronment with limited choices for shock-responsiveness, significant gains in terms of efficiencies for governments and prompting the Government to make strategic design financial inclusion. However, as such, they can also exclude decisions to proactively respond to the emergency. With some groups and raise important data privacy concerns, as the lack of a shock-responsive social protection policy explained in greater detail in the final sections of the report. framework and a dynamic social registry in place to prioritize 10 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection III. A deep dive into NOVISSI’s high- tech delivery model (model 2) In expanding NOVISSI to rural areas, the delivery 1. Outreach model covered the entire gamut of end-to-end delivery processes. Using the Social Protection Delivery Systems Immediately after launch, the program quickly attained Framework from Lindert et al. (2020) as a guiding frame- high take-up rates.27 Part of the success was probably work, this section illustrates NOVISSI’s implementation due to a multi-layered outreach strategy. Outreach for phases and key actors, focusing on the program’s expan- NOVISSI was supported by proactive and extensive radio sion to rural areas. The section also highlights the role of campaigns in approximately 30 stations in seven languages technology as an enabler in automating processes and (French, Ewé, Mina, Kotokoli, Losso, Kabyè, and Moba). In manage information. addition to announcing the program objective and the steps 27. Although the systems were designed to take over 700 registration requests per second, the program’s high demand in the first three days of implemen- tation resulted in systems saturation. FIGURE 3 NOVISSI’s implementation timeline Survey on the impact of NOVISSI Phone-based survey to train model 1 on the living machine learning algorithms conditions of the and model calibration population Phone-based survey to train (200 poorest cantons) MAY 30–JUNE 14, 2020 machine learning algorithms MARCH 22–APRIL 16, 2021 and model calibration (100 Declaration of state poorest cantons) of emergency by SEPTEMBER 18–OCTOBER 12, 2020 NOVISSI Model the President 2 end date APRIL 1, 2020 AUGUST 31, 2021 APRIL 8, 2020 JULY 1, 2020 NOVEMBER 2, 2020 Launch of the Meeting between the Launch of the NOVISSI program Government, IPA, and APRIL 23–MAY 18, 2021 NOVISSI program model 1 and the Give Directly. Beginning of Phone-based survey model 2 (100 NOVISSI platform work on categorization of evaluate the impact of poorest cantons) cantons based on poverty MARCH 6, 2020 APRIL 12, 2021 NOVISSI model 2 (100 maps and planning poorest cantons). First COVID-19 Extension of the case in Togo JUNE 6, 2020 NOVISSI model 2 Impact evaluation State of emergency is lifted. program to 100 analysis (undergoing) NOVISSI model 1 is additional cantons implemented in localities with (200 poorest cantons) increasing cases of COVID-19 Source: Original figure for this publication. III. A deep dive into NOVISSI’s high-tech delivery model (model 2) 11 FIGURE 4 NOVISSI’s Model 2 delivery chain process mapping GeospaƟal data 3A. Assesses needs (Big Data) and condiƟons using Poorest Academics machine learning + cantons and Phone Surveys (2020) big data + survey individuals (Survey Data) data + admin data EHCVM (2018–2019) INSEED Pop Census (2011) (Survey Data) Mobile Network 11. Provisions 8. Creates mobile Operators Call Detail payments to money account if mobile money Records (Big Data) it doesn’t exist accounts 2A. Registers by 6A. Does not receive a 6B. Typically those 2B. NoƟficaƟon dialing *855# noƟficaƟon because they who don’t hear back 6C. Receives 8A. May receive that individual Individual and providing is registered could be considered call the hotline noƟficaƟon of noƟficaƟon of new 12. Receives requested data successfully eligible later based on operator who explains enrollment decision mobile money transfer via USSD geographic locaƟon or that they could sƟll be via USSD account created other criteria considered later from MNO End No NOVISSI BOMS System 3B. Cross-validates unique 4. Determines 5. 7. System sends a number from voter list + benefits Makes money transfer checks validaƟon rule that Yes 10. Sends package as part decision on enrollment request to the takes into account poorest payment as a beneficiary, for cantons/ individuals + of validaƟon MNO to pay the instrucƟon payment of phone number to check if rule on individual to Telcos benefits potenƟally eligible eligibility 1. Outreach focused on selected 9. Give instrucƟons for Government communiƟes and Voters list payment for model 2 individuals (Gov Admin data) directly to telco account. In model 1, MoF gives instrucƟons Start for payments to telco Source: Original figure for this publication. necessary to register and receive the benefit, frequently them to apply for NOVISSI. The ads asked people to dial asked questions were answered through radio ads and state- *855#, a special number that would prompt them to respond ments from Government officials. They were also shared to a brief questionnaire over USSD to register for NOVISSI. with the call center so teleoperators could understand the 2. Intake and registration program and have the relevant information to respond to people’s questions. A user-friendly website containing program information was created. Social media handles NOVISSI’s data intake to allow for the assessment of also provided daily registration and beneficiary metrics to needs and conditions and eligibility determination used the population. At the local level, NOVISSI was promoted four primary sources: actively through local radio networks. Community leaders were onboarded to inform communities about the program. • Government administrative sources. Data were pulled While these features of the outreach strategy were standard from the voters’ list to verify the uniqueness of individ- across both models, model 2 introduced an innovation by uals and obtain demographic information for eligibil- sending targeted outreach messages. Individuals identified ity verification and determination of benefits package. as potential beneficiaries received text messages inviting These variables included the voter’s unique identification 12 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection number, a second validation number (Numéro de Suivi ter for NOVISSI’s benefits by dialing *855#. This would du Formulaire d’Inscription, NSF in French) available only prompt a USSD menu where applicants were asked to in printed credentials, name, sex, home location, and provide minimal data to verify their eligibility. Registrants occupation. Data-sharing protocols to access the voters’ would need to grant their consent to share information list were facilitated by the declaration of emergency for the purposes of the NOVISSI program, including their measures to combat the crisis. voter’s unique identification number, a second validation number (NSF) only available in printed voter credentials, • Non-traditional data sources. Big data were gathered and their last name. The NSF helped the program ensure and analyzed offline to prioritize areas and individu- that the beneficiary was actually in possession of the als. Independently, the team of academics leveraged voter’s card.29 These data were automatically processed geospatial data and mobile phone metadata to assess and triggered the eligibility determination process. needs and conditions at the canton and individual levels, 3. Assessment of needs and respectively. In many cases, datasets were publicly avail- conditions able, while others required specific licenses from interna- tional entities. Due to the nature and sensitivity of data, access to CDR data required non-disclosure and data The assessment of needs and conditions for expand‑ use agreements between academics and MNOs. All of ing NOVISSI to rural areas utilized machine learning to these datasets were accessed and processed exclusively analyze non-traditional data in two steps. When NOVISSI by academics. Ultimately, the ranking of the poorest was rolled out in urban areas (model 1), the Government used cantons, along with the list of phone numbers of poten- a categorical approach based on individuals’ geographic tially eligible individuals meeting the criteria of predicted location and occupation to identify the program’s poten- wealth below a predetermined threshold (US$1.25/per tially eligible population. Prioritizing informal workers in day), were transmitted to the Government for eligibility places with a declared state of emergency, implying harsher determination. mobility restrictions, helped target benefits to vulnera- ble individuals affected by the COVID-19 pandemic. This • Survey data. Survey data collected by the National Insti- initial approach focused deliberately on the hardest-hit tute of Statistics in Togo (Institut National de la Statis- urban areas. However, that approach would not have been tique et des Études Économiques et Démographiques, conducive to reaching the poorest households in the INSEED in French) and researchers were used to cali- rest of the country, primarily in rural areas. For NOVISSI’s brate the machine learning algorithms. These included expansion to rural areas, the task of locating the poorest the EHCVM 2018–2019 survey and phone-based surveys. geographic regions in the country and the poorest mobile subscribers in those regions was performed by a team of • Self-reported data. NOVISSI introduced an on-demand academics30 with a two-step approach that used geospa- registration system to apply for the program using a tial and mobile phone data as inputs. The outputs were (i) simple USSD28 form. Interested individuals could regis- a poverty map of Togo ranking the cantons (districts) from 28. USSD is communication protocol used to send text messages by opening a real-time session between a mobile phone and a network or server that enables a two-way exchange of information. Users interact directly from their mobile phones and select different options among predefined menus. 29. Debenedetti (2021) documents that “[…] voter rolls (used to verify that voters are correctly registered ahead of casting their votes) were still posted at some of the country’s polling stations from the February elections, which left open the possibility of attempting to register for the program through other people’s voter ID numbers on the posted registers. The government was able to detect and prevent fraudulent registration attempts by requiring applicants to furnish their NSF number as part of the registration process. The NSF is a security code included on each person’s voter identification card which, criti- cally, is not published on voter rolls.” 30. See Annex 1. III. A deep dive into NOVISSI’s high-tech delivery model (model 2) 13 TABLE 2 Data sources used for assessing needs and conditions Data Source 1. Road density Open Street Map 2. Urban (or built up) NASA (MODIS) 3. Elevation USGS 4. Slope USGS 5. Precipitation NASA/Japan Aerospace Exploration Agency 6. Population Humanitarian Data Exchange 7. Number of cell towers Facebook 8. Number of WiFi access points Facebook 9. Number of mobile devices Facebook 10. Number of Android devices Facebook 11. Number of iOS devices Facebook National Center for Environmental Information Earth 12. Nightlights / Radiance (VIIRS) Observation Group 13. Satellite imagery Digital Globe 14. EHCVM 2018–2019 INSEED 15. CDR Togocel and Moov 16. Phone survey Academics Sources: Aiken et al. (2021) and Chi et al. (2021). Note: Data sources 1–14 were used for the prioritization of cantons’ algorithm (step 1), while data sources 14–16 were used for the prioritization of individuals’ algorithm (step 2). the poorest to the richest and (ii) a potential eligibility list Model 2—Step 1. Selecting the with minimal information on individuals living under the poorest areas in Togo poverty line (US$1.25/day) located in the poorest cantons prioritized by the program (200 cantons in total). The The first step of model 2 was to find the poorest areas in potential eligibility list was transferred to the Govern- Togo. Before partnering up with academics, Togo did not ment of Togo to determine eligibility and enroll benefi- possess recent poverty estimates at the levels of prefecture ciaries. A summary of the data sources used for this process (Admin-2)31 or canton (Admin-3).32 The only available, most is provided in Table 2. updated, representative estimates were aggregated at the 31. The most recent estimates of poverty representative at the prefecture level were from 2015. 32. Latest available data from the Demographic and Health Survey (DHS) 2013-2014 surveyed 185 (47 percent) of cantons, while the EHCVM 2018–2019 sur- veyed 260 (65 percent) cantons. This implies that one or more households per canton provided socioeconomic information; however, poverty estimates are not representative of the canton. 14 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection Supervised machine learning algorithms model a function to predict an outcome BOX 1 variable that cannot be easily obtained (e.g., wealth, consumption, population, Supervised machine etc.) based on a set of predictors readily available (e.g., non-traditional data). In learning algorithms in order to train the model to predict the outcome, a statistical learning method is applied to extract features and find patterns in ground truth data (also referred a nutshell to as input data). The model is then extrapolated to obtain outcome variable predictions beyond existing data points. Essentially, the algorithm learns to iden- tify patterns in input data indicative of an outcome variable. national and regional levels.33 However, prioritizing support The ground truth data used to train the predictive model to the poorest cantons in Togo required more granularity. of consumption was derived from the EHCVM 2018–2019 NOVISSI’s geographic prioritization strategy followed Chi survey. The EHCVM 2018–2019 is a nationally and regionally et al.’s (2021) methodology to produce high-resolution esti- representative household survey administered by INSEED. mates of wealth and poverty, with some tweaks specific to A total of 6,172 households were surveyed, covering 265 the country. Chi et al. (2021) estimations for over 135 low- cantons (65 percent) and 922 unique tiles (9.1 percent) (Chi and middle-income countries (LMICs) are available for free et al., 2021). Data collected include questions on educa- public use.34 NOVISSI’s expansion to rural areas was initially tion, health, labor, household characteristics, assets, and planned for the 100 poorest cantons and extended in April expenditures, among other topics, along with the exact 2021 to an additional 100 cantons. geo-coordinates of household location. Using the EHCVM 2018–2019, the researchers constructed a measure of daily NOVISSI’s selection of the poorest cantons was based per capita consumption. This indicator was used as a ground on a model producing micro-estimates of consump‑ truth measure of consumption, while the tile-level average tion (Figure 5). In Togo, the approach predicted average of consumption was the target variable for the machine consumption (outcome variable) for each 2.4 km2 grid cell learning model. or tile.35 The size of the tile was determined based on the resolution of input data, but also considering that smaller Survey data were linked to a vast range of non-tradi‑ grid cells could compromise the privacy of individual house- tional big data records to predict average consump‑ holds (Chi et al., 2021). While wealth estimates were also tion. Geospatial satellite, connectivity, demographic, and produced for the various cantons in Togo, the NOVISSI geographic data were integrated into the model as predic- program employed consumption estimates to prioritize tors of consumption (Chi et al., 2021). Specifically, satellite benefits, given the direct connection of this indicator to data included hi-resolution imagery and nightlights. Connec- poverty lines measurement and a more intuitive interpre- tivity data contained characteristics of telecommunica- tation of the index. tions infrastructure, like cell towers and WiFi access points, 33. Togo has only five Admin-1 level regions. 34. https://data.humdata.org/dataset/relative-wealth-index 35. The statistical learning method applied by Chi et al. (2021) was a gradient boosted regression tree. The standard reference for gradient boosting is Fried- man (2001), while the standard reference for random forests is Breiman (2001). Natekin and Knoll’s (2013) provide an introduction. III. A deep dive into NOVISSI’s high-tech delivery model (model 2) 15 FIGURE 5 Prioritizing the poorest cantons in Togo (Model 2) NOVISSI PRIORITIZING GRID CELL LEVEL: 2.4KM2 THE POOREST CANTONS TRAINING DATA SURVEY DATA EHCVM 2018-2019 survey with consumption data and geocoordinates used as ground truth GEOSPATIAL DATA SATELLITE DATA MACHINE Hi-res imagery, night lights LEARNING CONNECTIVITY DATA Cell towers, devices TRAINING GEOSPATIAL DEMOGRAPHIC DATA DATA DATA POVERTY MAP Population, urban/rural These data sources were matched to train a The result was a high-resolution map supervised machine learning algorithm to find with the estimated average daily patterns of poverty and identify a model to consumption per capita at the grid cell GEOGRAPHICAL DATA predict consumption level (2.4km2) across Togo Road density, elevation Source: Original figure for this publication based on Chi et al. (2021) with inputs from Josh Blumenstock. and mobile device features, such as quantity and operating Micro-estimates of consumption were combined with systems. Demographic features included high-resolution population density estimates to identify the poorest population density maps and urban-rural domain markers. cantons. Using high-resolution population density esti- Geographic properties such as road density, elevation, and mates from the Humanitarian Data Exchange,37 the team precipitation were also considered. All geospatial features of academics calculated a weighted average of consump- were aggregated for each grid cell to mitigate privacy tion for all 2.4 km2 grid cells within each canton. This concerns. Previous preparation to compress satellite imag- allowed cantons to be ranked by their predicted degree ery was required since these data are usually unstructured of consumption as well as their population density, giving and multi-dimensional.36 112 features of non-traditional big more prominence to cantons where a larger share of the data were then matched to the ground truth data to find total population was expected to live under US$1.25 per patterns of poverty in geospatial data and identify a model day. The result of this exercise located and ranked the 397 to predict consumption in tiles without ground truth data. districts in Togo from the poorest to the richest. The 200 The result was a high-resolution map with the expected poorest cantons were selected to receive benefits during average daily consumption per capita at the grid cell level. the program’s extension to rural areas. The first phase of the program’s extension to rural areas focused on the 100 poor- 36. Compressing satellite data required using unsupervised machine learning techniques, specifically, convolutional neural network methods, to produce a vector of 2,048 features from 256x256 pixel images. Then, principal component analysis (PCA) helped reduce the number of features to only 100 (Chi et al., 2021). 37. Available at https://data.humdata.org/dataset/highresolutionpopulationdensitymaps 16 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection est cantons, then on the following 100 poorest cantons in explored groundbreaking opportunities to prioritize the the second phase. Although the canton selection process poor. In this context, the team of academics proposed a responded to technical criteria, the team of academics vali- methodology that had only previously been theoretically dated the model calibrations through various exchanges tested38 in which poverty was proxied using mobile phone with government officials to take into account their deep metadata and machine learning algorithms. The approach knowledge of practical realities. was replicated for the first time in a real cash transfer deliv- ery context in Togo. Model 2—Step 2. Prioritizing the poorest individuals The approach builds a predictive model to estimate daily consumption for each of the country’s 5.83 million The second step of model 2 consisted of prioritizing the mobile phone subscribers (Figure 6). The intuition behind poorest individuals living in the poorest cantons. With- the method is similar to the one employed to estimate out an up-to-date and dynamic social registry to assess poverty maps in the previous step. A supervised machine needs and conditions of households and/or individu- learning algorithm was trained on survey-based measures als, combined with constrained resources to collect new of individual-level consumption to recognize patterns of survey data amidst the pandemic, the Government of Togo poverty in CDR input data.39 Mobile phone usage is likely FIGURE 6 Prioritizing the poorest individuals within the poorest cantons (Model 2) TRAINING DATA NOVISSI PRIORITIZING THE POOREST INDIVIDUALS Phone survey data collected in September 2020 was used as ground truth. A total of 8,915 individuals in the 100 poorest cantons responded to the survey and provided their consent to match their responses to call detail records. CALL Cell phone records DETAIL transformed into RECORDS metrics describing behaviors. MACHINE Volume, intensity, timing, social network characteristics, LEARNING CALLS AND patterns of mobility and SMS locations, international INDIVIDUAL ASSESSMENT transaction features. TRAINING CALL DETAIL Mobile data transactions, DATA RECORDS The result was a model allowing to MOBILE DATA USAGE days on which data is estimate average daily consumption consumed. These data sources were matched to train a supervised for each of Togo’s 5.83 million machine learning algorithm to find patterns of poverty in mobile phone subscribers. MOBILE MONEY Amount, duration, CDR data and identify a model to predict consumption. TRANSACTIONS direction Source: Original figure for this publication based on Aiken et al. (2021) and inputs from Josh Blumenstock. 38. See Blumenstock, Cadamuro and On (2015) and Blumenstock (2018) and Aiken et al. (2020). 39. The machine learning method used to train models that predict poverty from CDR features was a gradient boosted regressor tree (Aiken et al., 2021). III. A deep dive into NOVISSI’s high-tech delivery model (model 2) 17 to be different between poor and non-poor people. For predicting consumption given its more straightforward instance, individuals with a larger volume of international interpretability, academics developed a consumption proxy phone calls, a higher mobile money balance, or more exten- to be used as ground truth for training the machine learning sive networks are likely better off than their counterparts. model. Using the EHCVM 2018–2019 survey, a Ridge regres- sion (similar to a proxy means test) was used to identify a Data collected through a phone survey of active mobile small subset of the most predictive features of consump- phone subscribers were used as ground truth to train tion and the weight associated with each of them. These the model. In September 2020, before the expansion of features can be roughly grouped into household assets, NOVISSI to rural areas, the team of academics adminis- location, education, and the number of children. The fitted tered a phone-based survey financed by the World Bank to model was then used to produce consumption estimates provide ground truth information representative of mobile for each phone survey observation, interpreted as per capita network subscribers inferred to be living in the 100 poor- daily US dollars consumption per day. est cantons eligible for NOVISSI. When the Government decided to extend the program to the following 100 poorest Mobile phone metadata obtained from mobile network cantons (200 in total), further data were collected to retrain operators were used as inputs to do out-of-sample the models. The sampling frame included all mobile phone consumption predictions. CDR captures rich information subscribers active on one of the two mobile networks in beyond phone usage. They also embed information on an Togo between March 1 and September 30, 2020, equivalent individual’s social network, travel patterns and location, and to 5.83 million subscribers.40 histories of consumption and expenditure (Blumenstock, 2015). Raw input data from mobile phones used for NOVISSI The phone surveys solicited consent from respondents contained records of calls, SMS messages, mobile data usage, to match their information to mobile phone transaction and mobile money transactions (Aiken et al., 2021). Specifi- data. The phone surveys were conducted over two weeks cally, data logs like the call or SMS recipient’s number, date, with an average duration of 40 minutes. They gathered data time, and duration of each operation, identification number on demographics, asset ownership, and well-being. A survey of the associated cell tower or antenna, among others, were was considered complete if participants consented to obtained for each mobile phone subscriber. These transac- merge their data with mobile phone records and responded tion logs were transformed or “featurized” into a large set to the entire set of questions. 28 percent of the phone of metrics or variables describing behaviors like the volume, numbers sampled met these conditions, amounting to 8,915 intensity, timing, and directionality of communication; social individual observations available for training data in the first network characteristics; patterns of mobility and locations; survey wave (100 poorest cantons). international transaction features; and characteristics of mobile money transactions like amount, duration, direction, Since phone surveys did not collect consumption data, and other descriptive statistics. An anonymized version of academics developed a consumption proxy. Phone surveys these CDR features was matched to all 8,915 observations did not collect data on consumption because these are in the training data to do out-of-sample predictions of indi- usually more complicated and time-consuming to enumer- vidual-level consumption for all mobile-phone subscribers ate. Considering that the Government was interested in in Togo. 40. The sample was stratified based on four criteria: (i) inferred probability of living in one of the 100 poorest cantons; (ii) predicted probability of having registered before to the NOVISSI program; (iii) expected wealth; and (iv) total mobile phone expenditure. A more detailed description of the sampling methodology and weighting is provided in Aiken et al. (2021), including a description on the methods used to predict each metric employing machine learn- ing algorithms. 18 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection This process resulted in a potential eligibility list trans‑ 2. Location. The individual’s home location, as reported mitted to the Government for eligibility and enroll‑ in the voter list, was verified against the list of selected ment decisions. Roughly 750,000 individuals from all over poorest cantons. the country were shortlisted to be potentially eligible for NOVISSI, given their predicted daily consumption of US$1.25 3. Individual eligibility. The individual’s phone number, or less per day. The threshold corresponds to the 29th retrieved automatically from the USSD system, was percentile of the predicted poverty distribution obtained cross-checked against the individual-level potential from the training data survey. The total number of short- eligibility list. listed individuals was constrained by budget and assumed an 85 percent registration rate among eligible individuals. An additional validation step conducted concomitantly The resulting potential eligibility list containing only the was the verification of sex which determined the amount phone number of shortlisted individuals was conveyed to of benefit each person would receive. NOVISSI’s poten- the Government under strict data protection protocols. tial eligibility list factored in information on sex and the The Government did not access CDR, nor did it directly corresponding amount of benefit that each beneficiary observe individuals’ consumption; instead, they received the was entitled to. Since its conception, NOVISSI has sought processed list of eligible individuals. This list was uploaded to actively include and empower women and reduce the into NOVISSI’s system to support eligibility and enrollment existing gender gaps by offering them more incentives to decisions. Importantly, access to CDR data was exceptionally register and benefit from the program. Women tend to be granted by Togo’s MNOs to the team of academics under poorer, more informally employed, and more excluded from non-disclosure and data use agreements. digital and financial services than men.41 At the same time, an extensive body of research has documented how women 4. Eligibility determination, are more likely than men to take care of basic household enrollment, and notification needs (for example, Rubalcava, Teruel and Duncan, 2009; Attanasio and Lechene, 2010; De Brauw et al., 2014). In Once registration was completed, it automatically trig‑ NOVISSI’s delivery model 2, women received US$15 per gered a cross-validation process within NOVISSI’s bene‑ installment, almost 17 percent more than men, receiving ficiary operations management system to determine US$12 per installment. Once registrants were deemed eligi- eligibility. Through an eligibility and enrollment decisions ble, NOVISSI’s systems verified the sex of the individual module within NOVISSI’s system, self-reported data through using the voter’s list to determine the benefit amount to the USSD self-registration platform were validated against be disbursed. the voter’s list and the potential eligibility list produced by the team of academics (in the case of NOVISSI model 2). All eligible registrants were enrolled in the program and Three specific validations were simultaneously performed: notified about the enrollment decision. After complet- authentication, location, and individual eligibility: ing the USSD-based application form, all registrants were systematically notified of their successful registration by 1. Authentication. The first check verified the voter’s SMS, letting them know that they would receive a subse- unique identification number and the NSF security quent notification in the event of being eligible. If the number reported by the registrant against the voter registrant’s self-reported data met the eligibility crite- list to verify uniqueness. ria, they were enrolled in NOVISSI and received a second 41. For instance, according to the Global Findex 2017 survey, in Togo, mobile money account ownership is 27 percent among men and 16 percent among women. III. A deep dive into NOVISSI’s high-tech delivery model (model 2) 19 SMS with information on the decision. Beneficiaries were beneficiary’s residence. The contracts also laid down also informed that their benefits would soon be reflected the terms so that each company’s payment system in their mobile money account. Non-eligible applicants was interoperable with NOVISSI’s system to ensure the remained under consideration for future enrollment into speed of payments. the program in case of changes in their eligibility status if, for instance, their canton was selected for assistance. The • Establishing the payroll and issuing payment instruc‑ list of beneficiaries with information on their identity and tions. Once a person was deemed eligible, NOVISSI’s benefits level was systematically produced and fed into the system automatically produced an individual payroll payroll for the administration of benefits. schedule based on enrollment data. This was verified and certified through automatic cross-checks within 5. Payments administration, NOVISSI’s systems and facilitated through an applica- provision, and reconciliation tion programming interface (API). Payroll included infor- mation on beneficiaries such as name, voter’s unique Instant mobile cash payments were made to NOVISSI’s identification number, phone number, benefit amount, beneficiaries. Following enrollment confirmation, the and payment date. The payroll schedule was transferred payment of benefits was issued automatically facilitated by to the corresponding telecom operator, triggering an the interoperability of NOVISSI’s system with mobile money instruction to issue the first payment. platforms which enabled the direct transfer of payments from GiveDirectly to beneficiaries. Funds were preemptively • Mobile-money account opening for those without an deposited into the MNOs operating account in order to active one and disbursement. Once MNOs received ensure timely payments. The consecutive disbursements the payroll, they automatically created a mobile money were automatically set up following a payroll schedule account for beneficiaries without one using the asso- defined by the program. Note that in NOVISSI’s model ciated mobile phone number and issued the payment. 1, payments ran similarly, with differences in the amount, Know-Your-Costumer (KYC) processes for mobile frequency, and duration of benefits and with direct payment money account opening occurred at the MNO level from the Ministry of Finance to beneficiaries. Payments were and followed their own internal procedures. Once the deposited into the beneficiaries’ mobile money account, deposit was issued, MNOs notified beneficiaries via managed by the corresponding MNO. SMS, per its pre-established procedures. Payments administration involved the following subpro‑ • Payment reconciliation. The Government of Togo cesses: retained an external auditor to conduct daily reconcilia- tion of payments and determine whether the transfers • Payment provision contract. A contract between were successfully credited on time. The external audi- the Government of Togo and each of the two MNOs tor would receive the payment logs from NOVISSI and (Togocom and Moov Africa Togo) was signed for the MNOs to ensure that all payment instructions matched provision of payments to program beneficiaries. The the eligibility criteria and were indeed processed. contracts established the terms and conditions of the Payment logs contained information such as payment service, including creating a mobile money account for reference number, MNO, date of payment issuance, their subscribers (if they did not already have one), the payment date, region and prefecture of the benefi- disbursement of benefits, and ensuring the availability ciary, beneficiary’s unique identifier, beneficiary’s phone of mobile money agents within a 5 km radius of the number, transfer amount, sex, and occupation. 20 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection • Sensitization and nudges. Although NOVISSI is an to the call center to support the operations of the GRM. A unconditional cash transfer program, beneficiaries were report of complaints received, processed, and resolved was encouraged to use their benefits to cover basic expen- prepared on a daily basis. The call center was instrumental in ditures, including food and water, water and electric- gathering and resolving problems. For instance, with NOVIS- ity bills, or airtime. They were also spurred to transact SI’s delivery model 1, the NSF security code available only electronically instead of cashing out the benefits to on voters’ physical credentials was requested at the time of avoid overcrowding at payment points and to maintain intake and registration after multiple fraudulent registration social distancing. attempts were reported to the call center (Debenedetti, 2021). Feedback could also be submitted through the USSD 6. Beneficiary operations form. Moreover, an external auditor was hired to ensure management proper program accountability through daily reconciliation of accounts. Real-time monitoring provided helpful infor- Accompanying measures such as a solid grievance redress mation to improve the program’s operations. For example, mechanism (GRM), external audits, and real-time analyt‑ the USSD-based registration system integrates audit logs ics enabled the program to operate in a transparent and to inform program operators about difficulties encoun- traceable way and adapt to needs. From the beginning, a tered during registration. Moreover, when NOVISSI was toll-free number dedicated to the program was set up to file extended in rural areas, GiveDirectly, which financed the complaints at any stage of the delivery process, from intake cash transfers, continually conducted phone check-ins with and registration to enrollment decisions to payment provi- program beneficiaries and community members to monitor sion. A frequently asked questions guide was made available program integrity. III. A deep dive into NOVISSI’s high-tech delivery model (model 2) 21 IV. Preliminary assessment: achievements and challenges Since its launch in April 2020, NOVISSI has supported the the beneficiaries’ perspective, streamlined processes could livelihoods of more than 920,000 individuals. Shock-re- have improved their experience and increased the quality sponsive programs, which face constraints on available time of the services by decreasing overcrowding, waiting times, and fiscal space, are much more binding than during regu- transportation costs, and the number of visits needed to lar times and call for greater cost-efficiency and speed of receive benefits. implementation. NOVISSI’s approach—model 1 low-tech and model 2 high-tech—allowed the Government to Although not conclusively evaluated yet, it is also likely rapidly disburse emergency cash transfers at scale amidst that NOVISSI’s digital payments approach helped reduce the COVID-19 shock while sidestepping risks of exclusion leakages and errors, while the intake of unique identifiers errors, out-of-date data, limited survey, and administrative via USSD forms may have minimized corruption and fraud data, and absence of a dynamic social registry, notwith- at the registration stage. Nonetheless, further analysis will standing less feasible alternative scenarios at that time. be required to quantify the efficiency gains introduced by Togo’s digital infrastructure readiness was key to NOVISSI’s NOVISSI and benchmark them to other approaches, as well apparent success. Good pre-existing conditions such as as to identify whether these gains were concentrated in a mobile network coverage, mobile phone penetration, and specific population segment. digital literacy were essential to deploying NOVISSI swiftly. Financial inclusion Nevertheless, despite its perks, some caveats are worth considering when assessing NOVISSI’s results. Mobile money payments introduced by the program may Potential efficiency gains have accelerated financial inclusion in Togo. Ever since the introduction of mobile money in Togo in 2013, the number NOVISSI’s innovative technology and data-driven of users has shown an upward trend. The NOVISSI program approach can potentially reduce the costs of social has encouraged individuals without an account to open protection delivery, increase efficiency for adminis‑ one and has expedited the process. Between April 2020 trators, and introduce convenience for beneficiaries. and January 2021, the program created a total of 170,278 NOVISSI has likely had important fiscal efficiencies for Togo. new mobile money accounts, accounting for a 7 percentage Compared to traditional cash transfer payments, efficiency point increase in mobile money penetration in the coun- gains due to the digital delivery of payments have reduced try (World Bank Group, 2021b). Mobile money payments the Government’s transaction costs and increased the speed have enabled users to make deposits, withdraw money, of disbursements to only a few minutes after registration. purchase airtime, pay for goods and services, and receive The use of big data and on-demand USSD intake and regis- social protection payments. A frequent constraint to mobile tration has resulted in a faster and more cost-efficient way payments is the limited liquidity of mobile money agents, to collect data than prevailing in-person collection. From especially in rural areas. However, in Togo, the availability 22 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection of agents within a close range of program beneficiaries may • Digital literacy. Similarly, some may have faced chal- have helped lessen this barrier, thus mitigating the risk of lenges completing the USSD-based registration due to mobile money being less attractive to rural populations. a lack of reading skills and digital competencies. Literacy Despite mobile money gaining ground through NOVISSI, rates in Togo have improved but remain low: in 2019, merchants in Togo have been slow in accepting it,42 leading only 67 percent of the population aged 15 years old or to most beneficiaries transacting in cash within the first 48 older could read or write.45 For NOVISSI, 72 percent of hours of the transfer. Whether access to digital accounts, mobile phone subscribers who initiated the registration incentivized by NOVISSI, served as a gateway to access process were able to complete it, while the average other financial services remains to be studied in order to registration required four attempts (Aiken et al., 2021). inform future operations. Moreover, according to govern- While some beneficiaries were unfamiliar with mobile ment officials, NOVISSI contributed to maximizing the technologies, preliminary qualitative evidence indicates compatibility of mobile money platforms with government that in many cases, applicants relied on social, family, payments administration systems and improving the overall and peer learning networks to complete the registra- quality of MNOs payments systems due to increased equip- tion procedures. ment investments. • Limitations of the voters’ registry. Due to a lack of Risks of exclusion universal coverage of unique identifiers and a dynamic social registry, the only available source at the time With an end-to-end digital approach for cash transfer to verify the uniqueness of potential beneficiaries, delivery, specific design features could have led to the in addition to location and occupation data, was the exclusion of some interest populations. voters’ registry. An estimated 86.6 percent of the Togo- lese adult population possesses a voter’s identification • Mobile phone adoption and use. The latest available credential (Aiken et al., 2021). This card, however, is only estimates suggest that 85 percent of individuals in Togo available to Togolese citizens aged 18 years old or more, are in a household with one or more phones,43 and meaning that foreign residents, individuals who could around 83.6 percent of individuals own a SIM card.44 not prove citizenship, or underaged individuals were While this share is likely to have increased by the time excluded by design. Moreover, those having recently NOVISSI was scaled to rural areas, part of the popula- moved from ineligible cantons to eligible cantons may tion still did not meet this requirement. Research shows have also been excluded from the program due to the that socioeconomic and demographic traits are driv- static nature of data contained in the registry. ers of digital technologies’ adoption and use (Rodri- guez-Castelán et al., 2021). Unequal access to mobile • Program awareness. Although the Government of Togo phones for some groups could have excluded those in implemented a multi-layered approach to outreach, most need; however, the extent to which the existing a lack of awareness of the existence of the program digital divide excluded some people from accessing the could have negatively affected registration rates. Around program is yet to be quantified and addressed in the 40 percent of mobile phone subscribers were aware of future through an all-doors-open approach. the NOVISSI program as they attempted to register for NOVISSI in delivery model 2 (Aiken et al., 2021). 42. World Bank Group (2020b). 43. EHCVM 2018–2019. 44. ARCEP (2021). 45. UNESCO Institute for Statistics (uis.unesco.org). Data as of September 2021. IV. Preliminary assessment: achievements and challenges 23 While the Government recognized these potential risks household asset-based wealth.46 While asset-based of exclusion, they were outweighed by the risks of not and consumption-based wealth are highly correlated, being responsive to vulnerable people in a historic crisis. the latter is more challenging to measure. For instance, Due to the rapid transmissibility and the virulence of the some features more directly related to consumption COVID-19 virus, in-person registration could have exposed are not observable on geospatial data, unlike in the and put at risk the lives of many; accordingly, registration asset-based wealth index where roofing materials and was to be done on-demand and via mobile devices. The plot characteristics, for example, are observed. Since no Government of Togo needed to ensure that payments were alternative consumption data are available for Togo, no made only to verified unique individuals. In the absence of further validations of the model’s accuracy have been a foundational unique identification system, Togo’s most made to date. Therefore, the accuracy of asset-based pragmatic option was to utilize unique identifiers generated poverty maps is likely to present an upper bound of the by the voter’s list to allow people to register. At the time accuracy of consumption-based poverty maps. the program was launched, the voter’s registry was the most up-to-date and widespread means of identification. Further- • The algorithm to prioritize amongst rural individuals more, it provided information on the place of residence was trained on a proxy of consumption calibrated and occupation of the card owner, which were the main with survey data collected before the shock hit. The eligibility criteria for NOVISSI model 1. Without a dynamic consumption index developed to train the machine social registry and universal coverage of unique identifiers, learning algorithm for individual-level prioritization was Togo leveraged available administrative data, survey data, computed using data collected before the crisis. More- and big data to swiftly expand social protection benefits over, variables such as household assets, location, educa- horizontally to as many vulnerable individuals as possible tion, and the number of children—used in NOVISSI’s in a short period of time. model 2—while correlated, are less likely to fluctuate as much as consumption during crises. Although more Model limitations data and analyses are needed, the ground truth data against which the model was trained may have intro- Similar to other statistical methods traditionally used duced some bias by potentially excluding individu- to prioritize benefits based on an estimation of welfare, als who were directly hit by the shock but reduced machine learning algorithms are also prone to prediction consumption instead of, for example, household assets errors. In the case of NOVISSI, some of the model limita- as a coping strategy. The Ridge regression, used to esti- tions include the following: mate consumption, as other estimation methods tradi- tionally used, is also subject to prediction errors. • Poverty maps produced with machine learning algo‑ rithms have proven to be highly accurate. However, Importantly, while machine learning algorithms can validations have mainly focused on estimates of encode bias, several tests confirm the fairness of the asset-based rather than consumption-based wealth. phone-based algorithm for different groups in Togo. Using census data for a sample of 15 countries on three Algorithmic decision-making can unwittingly discriminate continents and independently collected micro-data against some groups. Aiken et al. (2021) audited the fairness from three countries in Africa, Chi et al. (2021) found of the phone-based prioritization model for a large number that their model explains more than 70 percent of of groups. No evidence of systematic exclusion was found 46. Using data from Togo (not used to train the machine learning model), Chi et al. (2021) find that the model’s predictions explain 76 percent of the variation in wealth at the 2.4 km2 grid cell level, and 84 percent of the variation in wealth of cantons, 24 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection by gender, ethnicity, religion, age, or household character- a comprehensive social registry (not available at the istics. Compared to the alternative scenarios mentioned moment) outperformed the model. In the context a hypo- below, the authors find that none achieve perfect parity, thetical cash transfer program, Aiken et al. (2021) compared but the most prominent differences occur in the purely the performance of the machine-learning phone-based geographic approach. approach against four alternative prioritization methods: (1) universal coverage of poorest prefectures, (2) univer- Performance of NOVISSI’s model 2 sal coverage of poorest cantons, (3) prioritization based approach against other methods on an asset-based index, and (4) prioritization based on proxy means testing. The authors found that the phone- Compared to available and feasible alternatives to based machine learning algorithm performed best than the prioritize transfers, the phone-based machine learn‑ geographical targeting methods (1 and 2) to prioritize the ing approach reduced targeting errors. Aiken et al. (2021) poorest individuals (Figure 8). In contrast, the well-being tested the phone-based machine learning model’s perfor- assessment methods (3 and 4) outperformed the phone- mance against alternative strategies to assess needs and based machine learning algorithm in terms as seen in Figure conditions available to the Government of Togo at the time. 9. Nevertheless, well-being assessment methods, which rely The first scenario consisted of universal coverage in the on a comprehensive social registry, were not feasible at the poorest locations (geographic approach), while the second time of NOVISSI’s deployment. replicated NOVISSI’s Phase 1 occupation-based prioritization strategy (occupation approach). Aiken et al. (2021) compute An impact evaluation of NOVISSI’s welfare effects is performance using indicators of accuracy (proportion of ongoing. Data for the impact evaluation were collected in observations correctly identified as poor and non-poor) and a phone survey between April and May 2021, with around precision and recall (interpreted as one minus the exclusion 12,500 respondents. The results will make it possible to error). Using the phone survey data, their results show that assess the program results on a variety of welfare outcomes the phone-based model significantly reduces exclusion and and food security, detect unintended biases against vulnera- inclusion errors relative to the feasible alternatives (Figure 7). ble groups, and evaluate exclusion and inclusion errors more broadly. It can also set the ground for a cost-effectiveness However, when comparing the phone-based approach analysis to inform future operations. to traditional targeting methods, models requiring FIGURE 7 Performance of feasible targeting approaches 70% 59% 45% 47% 33% 19% Occupation Geographic Phone-based ML Recall and precision Accuracy Source: Aiken et al. (2021). IV. Preliminary assessment: achievements and challenges 25 FIGURE 8 Performance of different targeted approaches in the context of a hypothetical nationwide program Feasible Not feasible 85% 73% 75% 66% 68% 63% 54% 48% 50% 45% Geographic Geographic Phone-based ML Asset index Proxy means test (poorest (poorest cantons) prefectures) Recall and precision Accuracy Source: Aiken et al. (2021). 26 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection V. Going forward NOVISSI is a prime example of the potential of techno‑ Leveraging machine learning and big data, combined logical innovation and leapfrogging in building adaptive with administrative and survey data for social program and shock-responsive social protection systems. Togo’s targeting, shows promise; however, the accuracy, impli‑ experience has shifted the paradigm of how we think about cations, and trade-offs of this novel approach are yet social protection delivery and offers valuable lessons for the to be exhaustively studied. The literature on the use of region and beyond. Under specific circumstances, NOVISSI machine learning for social protection prioritization is was able to reach scale quickly by using novel approaches limited and inconclusive. When compared to a traditional and building a new delivery model for social protection proxy means testing, the performance of machine learning systems. In the absence of a reliable foundational unique algorithms depends heavily on (i) the data sources used, identification system and a dynamic social registry, these (ii) the performance metric, and (iii) the program policy innovative methods provided a way to support Togolese objectives (whether it is a welfare threshold or a quota). residents in the context of urgency. The NOVISSI approach For example, exclusion errors were larger in the phone- constitutes a new tool within the social protection deliv- based machine learning algorithm than in alternative ery toolkit, expanding the choices available to policymakers but unfeasible scenarios depending on a comprehen- but also introducing new challenges, such as greater depen- sive social registry (Figure 9) (Aiken et al., 2021). Moreover, dency on digital readiness. all algorithms can encode bias. As training data becomes outdated or is not representative of the population, the NOVISSI’s innovations inspire optimism by introduc‑ model performance declines and can increase both types ing significant efficiency gains in the delivery of social of targeting errors. Careful design, calibration, and audit protection programs, yet these gains need to be balanced of decision-making algorithms are crucial to examine if against new concerns. By heavily depending on digital tech- specific vulnerable groups are more likely to be excluded. nologies, this novel delivery model can potentially exacer- Whether individuals or households change their behavior bate exclusion risks that are yet to be quantified. Previous as they learn that transactional or satellite data are used infrastructure investments in mobile network infrastructure, to determine eligibility for benefits remains unanswered; payment ecosystems, and digital literacy enabled NOVISSI’s however, ongoing efforts seek to make these approaches delivery model, yet these enabling factors are not necessar- more robust to certain strategic behaviors.47 ily distributed uniformly or progressively within Togo’s socio- economic landscape. Data privacy is another concern that The risks of using private consumer data are profound is potentially heightened by NOVISSI’s delivery approach and require robust institutional arrangements. Building since mobile subscribers do not necessarily intend for their institutional safeguards is imperative. This should include, phone usage to be a proxy of their socioeconomic status. at least, minimizing access to sensitive data, anonymiza- New opt-in and opt-out mechanisms should be considered tion, encryption, and strict storage protocols. A registrant’s as these innovations mature and inform other initiatives. consent to use their data for program administration 47. See for example, Hardt et al. (2015) and Björkegren, Blumenstock and Knight (2020). V. Going forward 27 purposes is not only necessary but can also be an oppor- Having robust foundational unique identification systems tunity to inform them about their data privacy rights. This and dynamic social registries is instrumental to a truly requires capacity and a regulatory framework to manage and adaptive social protection delivery system. Leveraging protect data appropriately. In Togo, MNOs provided access unique identification platforms can help countries quickly to CDR data to researchers for selected periods and areas scale social registries and benefits payments platforms to on an exceptional basis due to the pandemic. It needs to deploy shock-responsive programs with potentially univer- be further determined how best to put in place a sustain- sal coverage. Under a 72 million IDA financing through the able and feasible longer-term arrangement. However, data WURI program, the Government of Togo is building a foun- trusts48 can be a viable option to look after and make deci- dational unique identification platform to register all indi- sions about data on behalf of communities. viduals in the territory, which will be the basis of a dynamic social registry of individuals and households for on-de- Technology can be both an enabler and a source of exclu‑ mand, multi-channel intake and registration. Both proj- sion for human capital services absent complementary ects are listed as priorities on the Government’s 2020-2025 investments in digital transformation. Countries have roadmap and will contribute to creating an innovative and made significant strides in mobile phone coverage and pene- shock-responsive social protection system that can serve tration over the past decades. Nevertheless, more action is as a model for other countries. still needed as technology can support the post-COVID-19 economic recovery and alleviate poverty. Democratizing Going forward, countries can leverage the innovations access to mobile phones through sustainable models such introduced by NOVISSI and combine them with more as pay-as-you-go schemes is an essential step toward digi- conventional approaches to increase the reach of social tal transformation. Funding digital infrastructure, increasing protection systems. Local conditions such as digital read- access to electricity, and building digital skills will pave the iness or urgency call for differentiated, complementary, way for inclusive digitalization. Accompanying measures to and flexible policies to optimize program delivery. Hybrid develop a digital economy ecosystem will also be critical. models that combine traditional with new approaches can Increasing the density of mobile money agents, proliferating be a viable alternative. For example, in-person census sweeps the acceptance of mobile payments among local traders, can be used to build the foundations of a social registry and multiplying acceptance technologies would further that could rely on remotely collected and passively gener- incentivize users to transact electronically and become a ated data for dynamic updating. Combining approaches conduit for financial inclusion while driving costs down both requires forward-thinking design, including using modular for customers and governments. data structures to integrate future data sources and ensure the interoperability of administrative, survey, and big data sources through unique identification systems. 48. Ruhaak (2021) 28 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection Annex TABLE A.1 Overview of NOVISSI’s stakeholders Stakeholder Role The President’s political leadership was key to the success of NOVISSI in Togo. He The President of the Republic galvanized the different stakeholders and ensured access to government resources needed to successfully implement NOVISSI. Led by the President of Togo, the committee defined eligibility criteria for state-funded Inter-ministerial Steering Committee NOVISSI-powered social cash transfer programs. Implemented and shaped state policies around the NOVISSI digital cash transfer Ministry of Digital Economy and system. Supervised relationships among government agencies, funding partners, and Digital Transformation (MENTD) NOVISSI technical delivery unit to ensure successful execution. Managed the state accounts and made payments to the NOVISSI account with the Ministry of Economy and Finance MNOs to ensure the availability of funds for direct transfer to beneficiaries. Developed the core NOVISSI technical platform and integrations with APIs while NOVISSI technical delivery unit ensuring that NOVISSI responds to high performance and security standards. Facilitated acquisition and uniformization of USSD codes to ensure cross-network Telecommunications regulator functionality. MNOs provided free access to their APIs, created mobile money accounts for Mobile network operators (MNOs) beneficiaries, provided anonymized CDR needed to identify vulnerable persons (model 2), and ensured that USSD platforms could support the long sessions NOVISSI needed. Developed and coordinated training of machine learning algorithms using satellite Academics/research team (Center for imagery and CDR to build a poverty map of Togo and identify the poorest individuals. Effective Global Action at University They also conducted an impact evaluation of the program through a mix of high- of California Berkeley, Northwestern frequency phone surveys and analysis of state-provided data and mobile phone University, University of Mannheim) metadata. Ensured customer relations management and grievance redress through a toll-free Call Center hotline. Independent daily auditing and data reconciliation. This involved confirming that External auditor NOVISSI payment orders were in line with the program requirements and that these orders, as triggered by the platform, matched the actual payments made by the MNO. INSEED carried out high-frequency mobile phone surveys to collect data needed for National Statistics Office (INSEED) NOVISSI’s impact evaluation. Provided funding for direct transfer to beneficiaries in a bespoke program designed by Donors (GiveDirectly) UC Berkeley and powered by the NOVISSI platform cf. GIVEDIRECT-NOVISSI. Development partners (World Bank- Provided funding for various aspects of the implementation NOVISSI program, IDA, AFD) including refinancing of cash transfers and funding for studies and IT equipment. Source: Original for this publication. Annex 29 References Aiken, E.L., Bedoya, G., Coville, A., & Blumenstock, J.E. (2020). Tar- De Brauw, A., Gilligan, D.O., Hoddinott, J., & Roy, S. (2014). The geting Development Aid with Machine Learning and Mobile Impact of Bolsa Familia on Women’s Decision-Making Power. Phone Data. COMPASS ‘20: Proceedings of the 3rd ACM SIG- World Development, 59, 487-504. doi: 10.1016/j.world- CAS Conference on Computing and Sustainable Societies, dev.2013.02.003 310–311. doi:10.1145/3378393.3402274 Friedman, J.H. (2001). Greedy Function Approximation: A Gradient Aiken, E.L., Bellue, S., Karlan, D., Udry, C.R., & Blumenstock, J.E. Boosting Machine. The Annals of Statistics, 29(5), 1189-1232. (2021). Machine learning and mobile phone data can improve http://www.jstor.org/stable/2699986 the targeting of humanitarian assistance (NBER Working Paper Gentilini et al. Social Protection and Jobs Responses to COVID-19: No. 29070). National Bureau of Economic Research. A Real-Time Review of Country Measures. Living paper version doi:10.3386/w29070 15. May 14, 2021. World Bank Aiken, E.L., Thakur, U.V., & Blumenstock, J.E. (2021). Phone sharing Hardt, M., Megiddo, N., Papadimitriou, C., & Wootters, M. (2015). and cash transfers in Togo: Quantitative evidence from mo- Strategic classification. https://arxiv.org/abs/1506.06980 bile phone data. https://arxiv.org/abs/2111.00175 Institut national de la statistique et des études économiques et Attanasio, O. & Lechene, V. (2010). Conditional cash transfers, démographiques (INSEED). (2020, December 20). Présentation women and the demand for food (IFS Working Paper No. des chiffres de pauvreté issus de l’enquête EHCVM 2018–2019. W10/17). Institute for Fiscal Studies. https://ideas.repec.org/p/ https://inseed.tg/presentation-des-chiffres-de-pauvrete-is- ifs/ifsewp/10-17.html sus-de-lenquete-ehcvm-2018–2019/ Autorité de Régulation des Communications Électroniques et INSEED. (n.d.). https://inseed.tg/ des Postes (ARCEP). (2021). Observatoire du marché des com- Leite, Phillippe; George, Tina; Sun, Changqing; Jones, Theresa; munications électroniques : Tableau de bord au 4eme trimes- Lindert, Kathy. (2017). Social Registries for Social Assistance tre 2020. https://arcep.tg/wp-content/uploads/2021/09/ and Beyond : A Guidance Note and Assessment Tool. Social Tableau-de-bord_2020-T4.pdf Protection & Labor Discussion Paper;No. 1704. World Bank, Björkegren, D., Blumenstock, J.E. & Knight, S. (2020). Manipula- Washington, DC. © World Bank. https://openknowledge. tion-proof machine learning . https://arxiv.org / worldbank.org/handle/10986/28284 License: CC BY 3.0 IGO. abs/2004.03865 Lindert, K., Karippacheril, T.G., Rodriguez Caillava, I., & Nishikawa Blumenstock, J.E. (2018). Estimating Economic Characteristics with Chavez, K. (2020). Sourcebook on the foundations of social Phone Data. AEA Papers and Proceedings, 108, 72-76. protection delivery systems (1st ed.). https://openknowledge. doi:10.1257/pandp.20181033 worldbank.org/handle/10986/34044 Blumenstock, J.E., Cadamuro, G., & On, R. (2015). Predicting pover- Natekin, A. and Knoll, A. (2013). Gradient boosting machines, a ty and wealth from mobile phone metadata. Science, tutorial. Front Neurorobot, 7(21). doi:10.3389/fnbot.2013.00021 350(6264), 1073-1076. doi:10.1126/science.aac4420 National Academies of Sciences, Engineering, and Medicine. Breiman, L. (2001). Random Forests. Machine Learning, 45, 5-32. (2017). Innovations in Federal Statistics: Combining Data doi:10.1023/A:1010933404324 Sources While Protecting Privacy. Washington, DC: National Chi, G., Fang, H., Chatterjee, S., & Blumenstock, J.E. (2021). Mi- Academies Press. doi: 10.17226/24652. cro-estimates of wealth for all low- and middle-income coun- Rodriguez-Castelan, C., Granguillhome Ochoa, R., Lach, S., and tries. https://arxiv.org/abs/2104.07761 Masaki, T. (2021). Mobile Internet Adoption in West Africa. Debenedetti, L. (2021, July). Togo’s Novissi cash transfer: Designing Policy Research Working Paper;No. 9560. World Bank, Wash- and implementing a fully digital social assistance program ington, DC. © World Bank. https://openknowledge.world- during COVID-19 (RECOVR Case Study). Innovations for Pov- bank.org/handle/10986/35209 License: CC BY 3.0 IGO. erty Action. 30 NOVISSI TOGO: Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection Ruhaak, A. (2021, February 24). How data trusts can protect privacy. World Bank Group. (2021a). The Republic of Togo: Macro-Poverty MIT Technology Review. https://www.technologyreview. Outlook. Washington, DC: World Bank. https://pubdocs. com/2021/02/24/1017801/data-trust-cybersecurity-big-tech- worldbank.org/en/471581492188174413/mpo-tgo.pdf privacy/ World Bank Group. (2021b, April 13). Prioritizing the poorest and Rubalcava, L., Teruel, G. & Duncan, T. (2009). Investments, time most vulnerable in West Africa: Togo’s Novissi platform for preferences, and public transfers paid to women. Economic social protection uses machine learning, geospatial analytics, Development and Cultural Change, 57(3), 507-538. and mobile phone metadata for the pandemic response. doi:10.1086/596617 https://www.worldbank.org /en/results/2021/04/13/ World Bank. 2020. Poverty and Shared Prosperity 2020: Reversals prioritizing-the-poorest-and-most-vulnerable-in-west-africa- of Fortune. Washington, DC: World Bank. doi: 10.1596/978-1- togo-s-novissi-platform-for-social-protection-uses-machine-l 4648-1602-4. License: Creative Commons Attribution CC BY World Bank Group. (2021c). Project paper of a proposed addition- 3.0 IGO https://openknowledge.worldbank.org/bitstream/ al credit and a proposed additional grant to the Republic of handle/10986/34496/9781464816024.pdf Togo for the Safety nets and basic services project. Washing- World Bank Group. (2020a). Rapport sur la situation économique ton, DC: World Bank. du Togo : Dynamiser l’investissement privé pour plus de World Bank Group. (2021d). The Republic of Togo: Public Expendi- croissance et d’emploi (French). http://documents.worldbank. ture Review. Washington, DC: World Bank. org /curated/en/329991599079506564/Rapport-sur-la- World Bank Group. (2021e). Forthcoming - The Republic of Togo: Situation-Economique-du-Togo-Dynamiser-l-Investissement- Poverty and Gender Assessment. Washington, DC: World Prive-Pour-Plus-de-Croissance-et-d-Emploi Bank. World Bank Group. (2020b). Togo Digital Economy Diagnostic R e p o r t . h t t ps : / /o p e n k n ow le d ge .wo r ld b a n k . o rg / handle/10986/34996 References 31 NOVISSI TOGO Harnessing Artificial Intelligence to Deliver Shock-Responsive Social Protection