Mobilizing Technology for Development An Assessment of World Bank Group Preparedness © 2021 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: World Bank. 2021. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness. Independent Evaluation Group. Washington, DC: World Bank. COVER PHOTO Adapted from shutterstock/ Vladimir Vihrev EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Luísa Ulhoa This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: Mobilizing Technology for Development An Assessment of World Bank Group Preparedness April 7, 2021 Contents Abbreviations v Acknowledgments vii Overview viii World Bank Group Management Response xx Report to the Board from the Committee on Development Effectiveness xxxiv 1. Background and Context���������������������������������������������������������������������������������������� 1 About This Evaluation 10 Evaluation Challenges 12 Report Organization 13 Harnessing DTT for Development: Key Bank Group 2.  Enablers and Links with the Twin Goals��������������������������������������������������������������� 16 Key Enablers of the Bank Group’s DTT Support 18 Links between the Bank Group’s DTT Support Relating to Two Corporate Priorities and the Twin Goals 22 Staff Skills and Mindsets for DTT Support����������������������������������������������������������� 33 3.  DTT-Relevant Staff Skills and Mindsets 34 Bank Group Recognition of Key Issues 34 IEG Findings 35 Internal Processes and Procedures for DTT Support������������������������������������������ 45 4.  Collaboration 47 Procurement (World Bank Only) 58 Institutional Culture and Incentives for Risk Taking and Innovation�������������������� 65 5.  DTT-Relevant Risks and Innovation 66 Bank Group Recognition of Key Issues 67 IEG Findings 67 Moving Forward: Directions of Travel������������������������������������������������������������������� 74 6.  Building on the Bank Group’s Existing Strengths 75 ii Enhancing Bank Group Capabilities That Are Not Yet Its Forte 76 Developing New Strengths 77 Conclusion and Recommendations��������������������������������������������������������������������� 82 7.  Conclusion 82 Recommendations 85 Bibliography��������������������������������������������������������������������������������������������������������������� 87 Boxes Disruptive and Transformative Technologies Have Box 1.1.  Created New Development Opportunities 3 Box 1.2. Risks Posed by DTT 5 World Bank Group and Independent Evaluation Box 1.3.  Group Definition of Disruptive and Transformative Technologies  7 Disruptive and Transformative Technologies Programs and Box 1.4.  Units in the World Bank Group 8 Box 2.1. Lessons from Identification for Development 20 The Unpredictability of the Future Job Market and the Box 2.2.  Importance of a Growth Mindset 26 Leveraging World Bank Group Experience with Off-Line Box 2.3.  Distance Learning and Online Learning in Responding to COVID-19  29 IEG and DECDG’s Two-Part Approach to the Staff Skills Assessment Box 3.1.  36 Recent Initiatives to Build Skills Relevant to Box 3.2.  Disruptive and Transformative Technologies 40 Institutional Arrangements for Digital Technologies and Box 4.1.  DTT in the World Bank Group  51 Cross-Sectoral Collaboration in the ID4D and Digital CASA Programs  Box 4.2.  53 Division of Roles and Responsibilities for Digital Technologies and DTT 57 Box 4.3.  Box 4.4. Legal Challenges in Procurement  62 Hit Refresh: Concrete Actions Taken by Microsoft Box 6.1.  to Acquire a Growth Mindset 78 iii Appendixes Appendix A. Methodological Approach 108 Appendix B. The World Bank Group’s DTT Mainstreaming Approach 149 Appendix C. Gender and Disruptive and Transformative Technology 160 Appendix D. World Bank Data Initiatives 168 Appendix E. Foresight and Anticipation 172 Appendix F. Creating a Culture of Risk Taking and Innovation 174 Appendix G. External Partnerships 176 Appendix H. Evolution of the World Bank Group’s Approach to DTT 179 Appendix I. ID4D: Three Pillars of the World Bank Group’s ID4D Support 181 Select Examples of the World Bank Group’s Appendix J.  COVID-19 Activities Across Sectors 184 Appendix K. Scaling Access to Solar Energy: Lighting Global and Scaling Solar 185 iv Abbreviations AI artificial intelligence ASA advisory services and analytics CASA Central Asia–South Asia CHIP Connect, Harness, Innovate, Protect CoP Community of Practice COVID-19 coronavirus CPF Country Partnership Framework CV curriculum vitae DE4A Digital Economy for Africa DEC Development Economics DECDG Development Economics, Development Data Group DTT disruptive and transformative technologies FY fiscal year World Bank Group Independent Evaluation Group    v GEMS Geo-Enabling for Monitoring and Supervision ICT information and communication technology IDA International Development Association ID4D Identification for Development IEG Independent Evaluation Group IFC International Finance Corporation IT information technology ITS Information and Technology Solutions MDDT Mainstreaming Digital and Disruptive Technologies MNA Middle East and North Africa SCD Systematic Country Diagnostic TMT Telecoms, Media, and Technology All dollar amounts are US dollars unless otherwise indicated. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Abbreviations vi Acknowledgments This evaluation was prepared under the leadership of Soniya Carvalho, Stephan Wegner, and Mitko Grigorov. The team consisted of Amshika Amar, Amitava Banerjee, Andrew Bent, Yue Chen, Anis Dani, Arunjana Das, Shahrokh Fardoust, Breda Griffith, John Heath, Morgan Holmes, Anjali Ku- mar, Andres Liebenthal, Kristin Little, Johan Lopez, Daniel Palazov, Xiaoxiao Peng, Mari Noelle Roquiz, Paul Schapper, Shiva Sharma, Anthony Tyrrell, Shahid Yusuf, and Disha Zaidi. The peer reviewers for the evaluation were Monika Huppi (principal adviser in the Evaluation Office of the Inter-American Development Bank), Charles Kenny (director of Technology and Development at the Center for Global Development), and Romain Murenzi (former science minister for Rwanda and current executive director of the World Academy of Sciences for the Ad- vancement of Science in Developing Countries, Italy). Collaboration on human resources analysis with the Development Econom- ics, Development Data Group, in particular Haishan Fu and Craig Hammer, is gratefully acknowledged. Advice from Nagy Hanna, Rasmus Heltberg, Manuel Penalver, Luis Alvaro Sanchez, and Jozef Leonardus Vaessen is much appreciated. The team is indebted to World Bank Group staff who shared World Bank Group Independent Evaluation Group    vii their insights, perspectives, and time. Data analytics support was provided by the Chennai team, led by Bhagyam Chandrasekharan. Gaby Loibl provided team support. Maximillian Ashwill provided content coaching, and William Hurlbut edited the report. Overview Highlights The World Bank Group adopted a new approach to disruptive and transformative technologies (DTT) in 2018, later merging it with the 2019 DTT Mainstreaming approach, and further developing it through the 2020 Mainstreaming Digital and Disruptive Technolo- gies Initiative. Given that the Mainstreaming approach is relatively new, this evaluation focuses on the Bank Group’s preparedness for providing DTT support rather than on the outcomes of that support. It establishes a baseline and highlights specific areas for the Bank Group’s attention, action, and monitoring that can help it enhance the effectiveness of its future DTT support. The main findings of this evaluation are twofold: (i) the Bank Group’s traditional areas of strength, such as its support for global public goods, honest broker role, production of quality advisory services and analytics, and ability to mobilize International Development Association resources and trust funds, have enabled its support for DTT for development; (ii) given the accelerating pace and complex- ity of technological change, the Bank Group is not yet sufficiently well prepared to help clients harness the opportunities and mitigate the risks posed by DTT, despite some areas of strength. The Bank Group will need to seize opportunities where DTT offer the potential to achieve the twin goals more effectively or efficiently and play a particular role in addressing DTT risks. Support for DTT by the Bank Group may not be relevant in all contexts and at all times. viii   Context for World Bank Group Engagement Disruptive and transformative technologies (DTT) have far-reaching implica- tions for development. Traditional development models are being disrupted by the accelerating pace of technological change and the convergence of multiple technologies, among other things. What distinguishes the current DTT revolution from past technological revolutions (such as the industrial revolution) is the explosion of data. In 2015 and 2016 alone, more data were created than in all previous years combined. Recognizing the implications of DTT for development, the World Bank Group adopted a new approach to DTT in 2018, later merging it with its 2019 DTT Mainstreaming approach and further developing it through the 2020 Main- streaming Digital and Disruptive Technologies Initiative. The Bank Group’s approach encompasses five DTT corporate priorities (country diagnostics, agile regulations, digital connectivity, digital government, and skills and capabilities for the new economy and the role of education), the Bali Fintech Agenda (financial technology and digital entrepreneurship), and sectoral and regional programs (for example, Digital Economy for Africa Moonshot/ Accelerate and MNA [Middle East and North Africa] Tech). The Bank Group aims to help clients harness the opportunities and mitigate the risks of DTT to accelerate progress toward achieving the twin goals of ending extreme poverty and boosting shared prosperity. The transition to a mainstreaming approach from earlier discrete areas of support marked a significant increase in the Bank Group’s ambition regarding DTT. Accordingly, the Bank Group’s World Bank Group Independent Evaluation Group    ix definition of DTT broadened beyond digital technologies to include other technologies (such as robotics, artificial intelligence, the internet of things, biotechnology, 3D printing, solar power, and batteries) and the analog com- plements (such as policies, institutions, and skills). It also extended beyond the “application” of technology to development challenges (for example, using computers in the classroom) to the “response” to technology (for example, imparting socioemotional skills to equip people for future jobs that machines will not be able to perform). The Bank Group’s definition of DTT, however, lacks operational clarity and does not sufficiently distinguish among digital, disruptive, and transformative. To date, much of the Bank Group’s focus has been on digital technologies. The Bank Group is implementing an array of DTT initiatives across all Re- gions and is engaged in innovative DTT initiatives, such as Development Data Partnership (formerly Data Collaboratives), Development Marketplace to Address Gender-Based Violence (in which the International Finance Corporation [IFC] also participated), Disruptive Technologies for Develop- ment Trust Fund, Geo-Enabling for Monitoring and Supervision, Geospatial Operations Support Team, the Information and Technology Solutions (ITS) Technology and Innovation Lab, Lighting Global, Social Development Data Lab, and IFC’s TechEmerge and ScaleX. Adding significance and urgency to the Bank Group’s DTT support are the DTT-related commitments for the 19th Replenishment of the International Development Association (IDA), the capital increase package, and opportunities offered by DTT to respond to Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Overview the disruptions caused by the coronavirus (COVID-19). Framework for Analysis This evaluation sought to answer the following question: How well prepared is the Bank Group to help clients harness the opportunities and mitigate the risks posed by DTT? The evaluation covered the World Bank and IFC (henceforth collectively referred to as the Bank Group) and drew on multiple sources. A literature review encompassing the organizational effectiveness literature and relevant Independent Evaluation Group evaluations identified the main preparedness dimensions for DTT support, which included dimensions of organizational ability and organizational willingness. Organizational ability was examined in terms of staff skills and mindsets, as well as internal processes and proce- dures. Organizational willingness was examined in terms of institutional cul- ture and incentives. The relevance, timeliness, and effectiveness of the Bank Group’s support for DTT will also be influenced by how well that support is tailored to and affected by country context factors. Thematic reviews were conducted of the five DTT corporate priorities, the Bali Fintech Agenda, and selected sectoral and regional programs. Semi- structured interviews of 105 Bank Group staff probed their experiences with DTT and were analyzed using NVivo, a qualitative data analysis software. Interviews of 57 external stakeholders were also conducted. x Bank Group Preparedness Given the accelerating pace and complexity of technological change, the an- swer to the evaluation question is that the Bank Group is not yet sufficiently well prepared to help clients harness the opportunities and mitigate the risks posed by DTT, despite some areas of strength. Where the Bank Group Is Best Prepared The Bank Group’s traditional areas of strength are also where it is best pre- pared to support DTT. Supporting Global Public Goods The Bank Group has a solid reputation for supporting global public goods, which increasingly use DTT. For example, the Global Facility for Disaster Re- duction and Recovery supports the use of DTT through open-source technol- ogy and geospatial data-sharing platforms to help people worldwide access risk information. Furthermore, the Bank Group is recognized for its expertise in development data and is helping Bank Group operations take advantage of the DTT-enabled opportunities for data production, analysis, and use through initiatives such as the Development Data Partnership (formerly Data Collaboratives), Geo-Enabling for Monitoring and Supervision, and the Geospatial Operations Support Team. World Bank Group Independent Evaluation Group    xi Serving As an Honest Broker The Bank Group’s reputation as an honest broker stems from its record of neutrality in dealing with public and private sector technology initiatives, its access to worldwide academic and commercial technological knowledge, and its prominent role in advising and negotiating with governments. By leverag- ing its honest broker role and adopting a problem-driven approach, the Bank Group—in its support for the Identification for Development program—has aimed to address the long-standing challenges of targeting the right benefi- ciaries and delivering multiple services through interoperable systems. Undertaking Quality Analytical Work The Bank Group’s ability to offer quality advisory services and analytics (ASA) has also enabled it to support DTT. ASA has equipped the Bank Group with actionable recommendations to support clients in putting their digital economies on a robust footing. Providing Trust Funds and IDA Resources The Bank Group’s ability to mobilize trust funds and IDA grants and credits has bolstered its DTT support. For example, in Benin, IDA financing (for the West Africa Regional Communications Infrastructure Program—Benin proj- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Overview ect) enabled the development of a competitive telecoms sector with open access to all providers of communications infrastructure. Furthermore, trust funds have supported World Bank task teams in piloting projects and gaining a better understanding of constraints. For example, in the Kenya Industry and Entrepreneurship Project, the use of trust funds to launch a competi- tion among local start-up companies enabled the World Bank team to better understand issues such as local intellectual property rights. Where the Bank Group Is Less Prepared Links between the Bank Group’s DTT Support and the Twin Goals Links between the Bank Group’s DTT support and the twin goals were as- sessed for two of the five Bank Group DTT corporate priorities, noting that all DTT corporate priorities are relevant for achieving the twin goals. The two selected DTT corporate priorities were (i) country diagnostics that help chart the new drivers of growth and (ii) skills and capabilities for the new economy and the role of education. Country diagnostics. The Bank Group has given increased attention to the digital economy in a few recent country diagnostics and strategies. Of those referred to in the 2019 World Bank Mainstreaming Paper (World Bank 2019h), this evaluation found that discussion of the digital economy in the Morocco and Senegal Country Partnership Frameworks was informed by xii findings in their respective Systematic Country Diagnostics. However, this was not always the case. There has also been insufficient effort in the Sys- tematic Country Diagnostics in linking the support for digital technologies to the twin goals (for example, when identifying key constraints). Further- more, the newly introduced digital economy diagnostics have not adequately considered the broader development agenda and the twin goals. Contribut- ing reasons include (i) limited participation by relevant Global Practices in the preparation of digital economy diagnostics and (ii) insufficient disaggre- gated data, such as internet usage by income group. Skills and capabilities for the new economy and the role of education. In light of the changing nature of work, the World Development Report 2019 notes that lack of education is likely to be one of the strongest mechanisms for transmitting inequalities from one generation to the next (World Bank 2019m). It emphasized the need to invest early in twenty-first century skills to be prepared for the changing nature of work. DTT have a critical role to play in education, to equip workers and children for the job market of the future. For instance, the application of DTT can improve the delivery of education. In ad- dition, the education system can facilitate a response to DTT by fostering the twenty-first century skills necessary for the job market of the future. The Bank Group’s top corporate initiatives in education—the Human Capital Project and the Learning Poverty initiative—spotlight “necessary” skills such as literacy and numeracy. The Bank Group has an opportunity to move the narrative World Bank Group Independent Evaluation Group    xiii forward from necessary skills to include twenty-first century “sufficient” skills, such as advanced cognitive skills, digital literacy, and socioemotional skills, in particular a growth mindset or the ability to “learn to learn” and adapt. Necessary and sufficient skills are critical for ensuring that a workforce is well prepared for the future labor market. Staff Skills and Mindsets for DTT Support The Bank Group’s fiscal year (FY)20–22 Human Resources Strategy presents key insights about staffing issues that confront the Bank Group, acknowledg- ing the need to build the expertise required to cope with rapid technological change. This evaluation found that the Bank Group has yet to (i) identify the DTT-relevant skills that it needs; (ii) ensure that its information systems and databases contain the necessary information on its current DTT-relevant skills; and (iii) take action to fill any gaps. In interviews conducted for this evaluation, Bank Group staff noted that the number of existing staff with DTT-relevant skills was insufficient to meet client demand. In the educa- tion sector, staff referred to the EdTech Fellows program as an initiative that could provide lessons, positive and negative, on how to address skills shortages in DTT for development. Interviewees also noted the need to raise awareness and better leverage existing staff expertise and resources across different sectors. Furthermore, they reported that a mindset for continuous learning and adaptation would help enhance the effectiveness of the Bank Group’s DTT support. IFC interviewees perceived that DTT expertise was fragmented after the realignment of Telecoms, Media, and Technology teams, which initially diluted IFC’s ability to maintain thought leadership Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Overview in DTT. The more recent absorption of the Telecoms, Media, and Tech- nology group into the Infrastructure group and the Fintech team into the Financial Institutions Group, and the realignment and hiring of staff in the Disruptive Technologies and Funds group, in addition to the respec- tive industries’ Upstream units (including specialists in artificial intelli- gence and machine learning), have sought to address gaps in DTT-relevant thought leadership and expertise. However, it is too early to assess the effectiveness of these measures. Internal Processes and Procedures for DTT Support Collaboration. The need for collaboration is underlined by the wide-ranging nature of DTT projects that cut across sectors and often involve both public and private institutions. The Independent Evaluation Group found examples of effective collaboration in providing DTT support in the Bank Group, for example the Development Data Partnership (formerly Data Collaboratives), the Digital Central Asia–South Asia program, Geo-Enabling for Monitoring and Supervision, and Identification for Development. However, interview- ees from across Global Practices perceived that collaboration is insufficient among the different Global Practices, with instances of competition for task team leadership. The lack of collaboration between the World Bank and IFC is also an issue, because successful DTT-related outcomes require knowledge and inputs from both the public and private sectors. xiv Procurement (World Bank only). Procurement was a major constraint to the smooth implementation of DTT-related projects in the eyes of most World Bank interviewees. There is currently only fragmented guidance for complex technology projects, specifically on their associated risks (such as vendor lock-in, lack of interoperability of systems, potential loss of data ownership and lack of data privacy, limitations in technology-specific legislation and regulation, and mismatch between business processes and the requirements of the particular technology). The World Bank’s procurement systems have provided some flexibility for complex technology projects, for example the use of framework agreements for software maintenance or servicing. However, interviewees noted that there was insufficient guidance on how to apply the available flexibility in different situations, suggesting the need for such guidance. Reluctance to use the available flexibility may also result from misaligned incentives that encourage risk aversion. Institutional Culture and Incentives for Risk Taking and Innovation Harnessing DTT for development often demands innovation, which by definition is without precedent and inevitably risky. However, interviewees reported that DTT were treated in the same way as other sectors or themes, despite their fluid and fast-moving nature, and that relevant staff were not encouraged to keep at the cutting edge of technological advances and DTT trends. Furthermore, the levers at the disposal of the Bank Group (such as leadership signaling, questions asked in operational review meetings, and World Bank Group Independent Evaluation Group    xv the performance management system) were not yet being effectively used to facilitate informed risk taking, learning from successes and failures, or innovation for DTT for development. Interviewees saw the operational review process as constraining innovation and creative solutions, given the general support for continuing on a familiar path rather than for breaking new ground. Interviewees reported that legal, procurement, and external relations departments often asked for precedents. Finally, risk taking— however informed and calculated—means failing at least some of the time, and interviewees said they would benefit from knowing how failure would be treated by Bank Group management. Moving Forward: Directions of Travel The Independent Evaluation Group presents—for the Bank Group’s consid- eration—possible directions of travel, including examples for each, that can help the Bank Group move forward in strengthening its preparedness for DTT for development. 1. Building on the Bank Group’s Existing Strengths Mining DTT-generated development data to create even greater social value. DTT enable development data to be collected and analyzed cost- effectively, with high frequency, and at fine levels of granularity. This gran- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Overview ularity can allow for personalizing services, for example tailoring learning to individual students in poor areas whose performance may be lagging and thus potentially helping reduce high dropout rates. Both the Bank Group’s World Development Report 2021 on data (World Bank 2021), as well as the recently established Data Governance body and its supporting Data Gover- nance Steering Committee, could boost the World Bank’s ability to support development data. Optimizing the use of public and private data—both those made available by DTT and those that can be analyzed more effectively by using DTT—can help the Bank Group become a data-driven organization. Leveraging the honest broker role by advising clients, especially on DTT risks. The Bank Group’s perceived neutrality positions it well to advise on DTT risks such as income inequality, data privacy, data security, and cyber- surveillance. Attention to DTT risk is particularly important in Identification for Development systems, which are especially vulnerable to inadvertent data spills or willful misuse. Addressing DTT risks and adhering to the do no harm principle may require the Bank Group to navigate sensitive ethical and political issues and advise clients on them. Where it lacks the necessary technical expertise to help clients mitigate DTT risks, the Bank Group could use its convening power to bring together relevant experts. Using explicit corporate metrics to track the implementation of the Bank Group’s DTT Mainstreaming approach. Explicit metrics—defined at the corporate level—can help track progress and enable timely course correc- xvi tions, thereby facilitating learning and the successful implementation of the Mainstreaming approach. Enhancing Bank Group Capabilities 2.  That Are Not Yet Its Forte Addressing gender-differential impacts of DTT, beginning with ASA work. Although the Bank Group’s strategic documents recognize the gender- differential impacts of DTT, a review of gender and DTT for this evalua- tion found that just 7 percent of World Bank ASA in the information and communication technology (ICT) sector (ICT sector percentage weight of 50 percent or more) approved during FY15–18 were gender-relevant ASAs, a particular concern since it is in ASA that new opportunities to address the gender-differential impacts of DTT can be explored. (Gender-relevant ASAs were identified using a keyword search and desk review. Appendix C provides methodological details.) Furthermore, although 76 percent of the 51 World Bank ICT projects (ICT sector percentage weight of 50 percent or more) approved during FY15–19 had one or more gender-relevant flags, 37 percent had all three gender-relevant flags. 3. Developing New Strengths Fostering a growth mindset in the Bank Group. Given the uncertainties created by technological change, the Bank Group could enhance its pre- World Bank Group Independent Evaluation Group    xvii paredness for DTT by ensuring a growth mindset that promotes continuous learning and adaptation among its staff. This would entail encouraging staff to recognize—and learn from—mistakes without fear of repercussion. Systematically applying foresight and anticipation to inform and guide the Bank Group’s support of DTT for development. Given the accelerating pace of technological change and the associated uncertainty, foresight and anticipation have a particular role in DTT for development. Foresight and an- ticipation can enable the Bank Group to regroup and take on challenges in real time and thrive even in markedly changed circumstances. The Bank Group’s Country Partnership Framework is already applying foresight and anticipation to outline the Bank Group’s country strategy four to six years out. More specif- ically for DTT, the recent exploration of foresight and anticipation by the Bank Group’s Technology and Innovation Lab is a promising step. Recommendations This evaluation makes three recommendations: Recommendation 1: Where DTT offer opportunities to make progress on the twin goals more effectively or efficiently, ensure that the Bank Group avails itself of those opportunities and addresses, in particular, the risks posed by DTT. This will entail, for example: » Systematically identifying in both country diagnostics and digital economy diag- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Overview nostics the opportunities and risks posed by DTT for achieving the twin goals; » Consistently asking in operational review meetings (from Concept Note to approval and implementation stages) whether and how the use of DTT in operations can bring (or is bringing) effectiveness and efficiency gains in addressing the twin goals; and » Strengthening cross-sectoral linkages and synergies between DTT and sec- toral issues, including through enhanced collaboration across Global Prac- tices and between the World Bank and IFC, particularly where different units support technological solutions with the same client. Where appropriate, a complement to country and digital economy diagnos- tics could be a review of ongoing and pipeline projects for DTT opportuni- ties, such as the one undertaken recently in Vietnam. Recommendation 2: Build a Bank Group workforce with the skills re- quired to harness DTT opportunities and mitigate DTT risks by identi- fying DTT-relevant skills, determining gaps in these skills, and filling these gaps. This will entail: » New recruitment of staff or consultants to fill specific skills gaps, retraining of existing staff, outsourcing, secondments, external partnerships, or a com- bination of these; xviii » Ensuring a more efficient use of the skills of available specialized staff and a better bridging of technology expertise with that of sector specialists and task team leaders; and » Supporting a growth mindset for continuous learning as noted in the Bank Group’s FY20–22 Human Resources Strategy. Recommendation 3 (World Bank only): Improve the effectiveness and efficiency of World Bank procurement for complex technology projects. This will entail, for example: » Strengthening procurement guidance for staff and borrowers, for early-stage project scoping as well as subsequent capacity development, on the identi- fication, prioritization, and mitigation of the risks associated with complex technology projects. The guidance could cover: availing of existing flexibili- ties for complex technology projects, such as two-stage bidding; monitoring the extent to which World Bank staff teams adopt these flexibilities; and institutionalizing arrangements for technology project management. » Ensuring that teams benefit from effective and efficient innovation while protecting the World Bank and the borrower from procurement-related reputational risks. » Preparing a roster of the world’s leading experts on the procurement of com- plex technology projects and encouraging market consultations on technical World Bank Group Independent Evaluation Group    xix requirements in the preparation of bidding documents. World Bank Group Management Response Management welcomes the Independent Evaluation Group (IEG) report, Mo- bilizing Technology for Development: An Assessment of World Bank Group Pre- paredness. This topic is key for the Bank Group, given its relevance to both the 19th Replenishment of the International Development Association (IDA) Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Management Response policy commitments and the International Bank for Reconstruction and Development–International Finance Corporation (IFC) capital package com- mitments. It has become even more timely in the context of the coronavirus pandemic (COVID-19). Management would like to thank IEG for undertaking this evaluation and consulting with management throughout the process. World Bank Management Response Management is pleased with the report’s appreciation of the World Bank’s solid reputation in supporting global public goods using disruptive and transformative technologies (DTT). The report recognizes that the Bank Group plays an important role as (i) an honest broker in dealing with public and private sector technology initiatives, (ii) a provider of robust advisory services and analytics on DTT, and (iii) a catalyzer for concessional funding in support of DTT initiatives. Management will build on these strengths and develop new ones by incorporating lessons learned from the IEG evaluation. Given the relative novelty of the mainstreaming approach (articulated in 2019), many improvement opportunities identified by the report (such as the/a proposed shift of the human capital and learning narrative[s?] from “necessary” skills of literacy and numeracy to include 21st century “suf- ficient” skills of digital literacy to meet the demands of the future labor market) are subject to ongoing efforts by management across a variety of fronts. Most of these efforts are still at early stages. For example, the FY21 Education Global Practice (GP) pipeline includes 42 projects that support the development of digital skills (both in basic and higher education) for teach- ers, principals, and learners. xx Management believes that the report’s conclusion that the World Bank’s focus is largely limited to digital technologies (as opposed to DTT), is prompted by an incomplete account of World Bank initiatives. The report appears to fo- cus mostly on activities undertaken by the Digital Development Group, the Data Development Group, and the Education GP. Many more groups across the World Bank spearhead initiatives that are relevant to this agenda. There is significant analytical and operational work across various Practice Groups (particularly in the Sustainable Development Practice Group) on data plat- forms, disruptive technologies, and technology innovations. Many of the more recently developed action plans, studies, and initiatives on DTTs provide a better sense of World Bank preparedness for supporting clients on DTTs. Management recognizes that even more systematic internal collaboration is needed, given the wide-ranging nature of DTT projects that cut across sectors and involve both public and private institutions. The report describes several initiatives, notably Identification for Development (ID4D), that have generated collaboration across GPs. Notwithstanding these initiatives, man- agement recognizes that the multisectoral dimensions of the agenda and the rapid evolution of the sector require further efforts. Therefore, it recently established the Digital and Disruptive Technologies Initiative, housed within the Digital Development GP, as an avenue to link different GP initiatives on DTT. This initiative will continue to mainstream digital and DTT solutions across sectors and work across the GPs to scale up digital and DTT applica- World Bank Group Independent Evaluation Group    xxi tions in World Bank projects. Management does not share the negative assessment of the Digital Econo- my for Africa Initiative (DE4A) yet recognizes room for improvement. DE4A fosters significant levels of collaboration across GPs and IFC. Each diagnostic is conducted with other GPs and IFC to avoid fragmentation of the analysis. Any digital economy assessment is done by a team of task team leaders from Finance, Competitiveness, and Innovation; Governance; Digital Develop- ment; Education; and IFC. The DE4A initiative and analytics by the offices of the Middle East and North Africa and Africa Region chief economists demonstrated the benefits of increased digital technology adoption and the links to both the twin goals and the Jobs and Economic Transformation agenda. The preparation of DE4A assessments (over 20 countries in Afri- ca in FY19 and FY20) enhanced Bank Group readiness to respond to client demand. The initiative evolved after an adaptative process and will continue to do so by carefully examining the assessment of this report. The methodol- ogy was updated and tighter links between diagnostics and implementation enabled based on the initial experiences of preparation of DE4A diagnostics. Management agrees with the report’s first recommendation, namely, to seize the opportunities brought by DTT to make progress on the twin goals and will continue doing so wherever relevant. This is fully in line with the 19th Replenishment of IDA policy commitments and the International Bank for Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Management Response Reconstruction and Development-IFC capital package commitments, which conceive the adoption of inclusive transformative technologies as promising means to achieve the twin goals. In this regard, management welcomes the report’s recognition that “more recent country diagnostics and strategies have increased attention to the digital economy.” Management emphasizes that the World Bank’s country engagement model is based on a principle of sovereign selectivity; hence, there should be no expectation that all World Bank client countries will systematically seek to assess or exploit opportu- nities for DTT. In this context, when it is justified by the congruence of client demand, policy dialogue and analysis, and the World Bank’s comparative advantage, management will continue making efforts to ensure appropri- ate diagnostics are conducted and a line of sight to the twin goals explicitly established. It is precisely for this purpose that the Bank Group recently endeavored to improve the digital economy country diagnostic methodolo- gy (version 2). Although the diagnostic is focused on the foundations of the digital economy, it recognizes that additional deep dives may be needed to further study specific areas of development, especially for key socioeconomic sectors. The diagnostic intends to provide actionable recommendations to governments and stakeholders on priority areas of development, with a mix of possible policy reforms and financing needs, and guidance on potential areas to achieve the twin goals. Management will continue developing a robust pipeline of analytical activi- ties and cross-sectoral initiatives with the intention of implementing the re- port’s recommendations. The knowledge and analytical pipeline is designed to provide practical tools for World Bank staff to harness opportunities and mitigate risks of DTT through the World Bank portfolio, enhance the skills xxii of World Bank staff, and provide guidance on procurement for DTT. These activities also support knowledge creation and technical assistance to World Bank clients. Many of these activities are managed jointly across different Practice Groups and between the World Bank and IFC.1 The World Bank will leverage this broad spectrum of efforts by continuing to work closely with public and private partners and by enhancing its capacity through externally funded activities (for example, Digital Development Partnership). Management also agrees with the report’s second recommendation, to build a workforce with the skills required to harness DTT opportunities and ad- dress DTT risks. Management welcomes the report’s acknowledgment of World Bank measures to strengthen the skills of staff working on digital development. As the report indicates, the World Bank has facilitated several staff to get online university degrees on artificial intelligence (AI), cyberse- curity, 5G, and regulation. There is a large amount of learning on DTT in the Open Learning Campus. Disruptive Technologies is the main “Cross–Cut- ting” learning theme for staff learning and includes a package of learning on blockchain, AI, and internet of things. The Digital Development GP is sponsoring a wide range of training on mobile technologies, cybersecuri- ty, and connectivity, among other topics. Information technology services has a Technology and Innovation Lab whose goal is to “serve as a catalyst, enabler and accelerator for WBG [World Bank Group] staff to learn about and build expertise around emerging technologies’ potential to support the WBG development agenda.” It also has an active staff learning program World Bank Group Independent Evaluation Group    xxiii (Learn2Innovate) that partners with GPs for learning sessions on topics such as blockchain, AI, and other technologies with potential applications to the World Bank’s work. The Sustainable Development Practice Group has a ro- bust learning program on the use of technologies for remote preparation and supervision for projects under its Remote Supervision and Preparation of Projects initiative. Additionally, the World Bank benefits from secondments of DTT experts from donor countries and the private sector. These initiatives demonstrate a growing commitment of time and resources, which will be sustained by management. Finally, management agrees with the report’s third recommendation, to improve the procurement of complex technology projects, and believes that continuing to invest in client capacity and staff knowledge and skills is the most promising pathway to the outcome desired. Management appreciates the report’s illustrative example that upgrading the current guidance on complex procurement could be one solution to that end, but it must be com- plemented with other efforts to be sufficiently consequential. A fundamental issue with complex information and communication technology and DTT projects is that client countries’ institutional strength has a material impact on the course of procurement. Countries with strong control institutions typically discourage the use of innovative procurement methods or award processes that are more flexible. This offers strong disincentives for govern- ment decision makers to innovate, reinforcing behaviors to stick to a more Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Management Response objective, price-based award decision to avoid criticism by controllers. This needs to be more systematically factored in during project preparation and negotiations with clients, including through better reflection in procurement documents for specific contracts. As mentioned in the report, these issues are discussed in analytical reports being prepared by management under the GovTech Global Partnership (“GovTech Procurement Strategy”) and by the Procurement Global Unit (“Disruptive Technologies in Public Procurement”). Management will ensure that as these reports are finalized, appropriate lessons are compiled and guidance revised, as warranted, to inform staff and client capacity-building efforts.2 International Finance Corporation Management Response DTTs play an increasingly critical role in a country’s economic growth and are fundamentally transforming how we and our clients do business. COVID-19 has only accelerated this trend of DTT and digitalization, impact- ing a wide spectrum—from productivity, scale, and the very nature of work, to financial inclusion, access to health and education, global value chains, and food security. Given the potential of DTT to help our client countries achieve their goals and the novel risk considerations digital development presents, this evaluation on the Bank Group preparedness is timely and important. IFC management welcomes this evaluation and would like to acknowledge the IEG team’s effort in evaluating thisimportant subject. IFC has been strategically planning and implementing various organizational xxiv changes to position itself to more effectively integrate DTT across all aspects of its business, from strategy formulation, investment, advisory, upstream, and diagnostics, to thought leadership. IFC management is critically aware that all industries need to understand the new reality that rapid innovation is changing business economics and therefore business models. It will change the constituency of IFC’s clients and how they operate their business. With this awareness, IFC management implemented two major organizational changes—IFC sector team realignment and upstream business establishment. In retrospect, these changes coincided with the time when the data collection and Bank Group interviews were being conducted for this evaluation. In FY19–20, IFC management restructured its organization around DTT to better reflect the critical role of technology in development and strategically position IFC toward tapping into these investment themes. First, digital in- frastructure, including telecommunications technology and services (TMT), moved to the Infrastructure Department. Second, financial technology (fin- tech), which has been investing in early-stage digitally-enabled payments and financial services since 2007, moved to the Financial Institutions Group Department (FIG). This was done in recognition of the impact fintech was having on financial services and the need to more closely integrate learnings from the technology portfolio into our work across the spectrum of financial services. Third, what we now call the Disruptive Technologies and Funds Department (CDF) was established as an Industry Department focused on the intersection of technology, manufacturing, agriculture, climate, and in- World Bank Group Independent Evaluation Group    xxv frastructure that combines both funds and direct investments in early-stage businesses. CDF now integrates closely with our mainstream teams in FIG; Manufacturing, Agribusiness, and Services; and the Infrastructure Depart- ment to inform and share knowledge on disruptive technology trends while also cooperating through investment joint ventures on all aspects of invest- ment decisions, knowledge transfer, and other cross-sectoral upstream and advisory initiatives in the space. In addition, IFC upstream business, which was launched in FY19, was further formalized in FY20 with the creation of the Global and Industry Upstream units. This aimed to better position IFC to support IFC 3.0’s vision toward market creation with a strong focus on har- nessing the opportunities enabled by DTT and the digital agenda. Although the report notes the reorganization, IFC management appreci- ates that the timing ofthe IEG evaluation may not have fully captured the extent of the above realignment and enhancement efforts in DTT across its investment, upstream, advisory and thought leadership businesses, some of which were in their infancies of establishment. It is notable to highlight that these activities have translated into increased capacity and expertise in DTT across IFC and enhanced collaboration within the Bank Group and private sector partners. Noting that it was too early to assess the effectiveness of its realignment and enhancement measures, IFC management would welcome IEG’s further assessment at a later time on the outcome of the efforts to strengthen its readiness to help clients harness the opportunities offered by Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Management Response DTT and mitigate the risks posed by them. In the interim, we highlight key enhancements that have taken place primarily since the above realignment, complementing the report. Investments in Disruptive Technologies, Fintech, and Digital Infrastructure. As a leading emerging markets Venture Capital (VC) investor in disruptive technologies and fintech, IFC has invested over $1.5 billion in the VC and fintech asset class, building a portfolio that has approximately 120 direct investments in tech-enabled startups alongside over 50 limited partnership investments in early-stage VC funds across the emerging markets. IFC has developed unique expertise to support innovative business models leverag- ing technology in this space and built a portfolio with a diverse set of DTT solutions—from financial inclusion, access to high quality health care and education, to climate innovation—demonstrating a capacity for continuous learning and adaptation. In addition, we have provided financing or equity of $5 billion in digital infrastructure and the broader TMT space over the past 10 years, including mobilization.3 IFC is championing the pioneering of a transactional approach in several areas, such as electric vehicles, cold chain logistics, virtualization of infrastructure, digital twins of utilities operations, Cloud Platform as a Service, and green data centers. As of January 2020, IFC has 60 active portfolio projects and 66 pipeline projects with a digital com- ponent, totaling $3.1 billion in investments. The development impact and financial returns of these investments have been quite attractive. Below are some examples: » Keeping supply chains afloat: E-logistics platform TradeDepot in Nigeria con- nected small businesses directly with suppliers and supplied food to informal xxvi retailers during lockdown, helping over 40,000 microretailers stay in business. The business is now expanding to other countries, including Ghana.4 » Expanding access to education: Ed-tech companies connect students to teachers without ever having to set foot in a classroom. The example men- tioned in the report, Byju’s in India, has enabled over 900,000 students in India to expand their education.5 » Extending reach of health care: Health-tech companies like 1mg in India are providing critical medicines through e-pharmacy services to remote villages and connecting patients to doctors through e-consultation, for which de- mand has spiked over 440 percent during the pandemic. The data advantage has led this business to become the leader in AI powered diagnostics through e-medicine in remote parts of the country.6 » Financial inclusion: Fintech companies like Fawry, which started the first and largest electronic payment platform in Egypt and now serves over 30 million customers, bring convenient, lower cost payment systems to a country where as recently as 2014, 94 percent of all transactions were cash.7 Advisory services. In addition to IFC’s investments in DTT, IFC has devel- oped extensiveadvisory capacity, including in-house expertise, to support the digital agenda. With over 220 advisory projects with digital or technol- ogy components that were launched since FY13 and currently in pipeline, World Bank Group Independent Evaluation Group    xxvii IFC’s advisory assistance cuts across the themes of digital financial services, digital infrastructure, digital entrepreneurship, and digital skills. One such example, the Digital Gender-Ethiopia Program with Gebeya, provides critical software development skills to female students and helps build robust digital economies and competitive markets. IFC has particularly taken a program- matic approach with initiatives, such as IFC Startup Catalyst,8 ScaleX,9 and TechEmerge,10 to support the development of digital entrepreneurship. Similarly, we have taken a programmatic approach to underpin technology adoption in innovative climate programs, such as in Scaling Solar, Light- ing Global, and EDGE.11 In the financial sector, IFC has supported financial institutions in adopting digital channels to drive efficiencies and expand the reach of financial services, and in leveraging data and technology for credit underwriting, risk management and transformation of back-end processes. The continuous evolution and sophistication of IFC’s product offering in this space demonstrates a capacity to adapt and evolve. Some important collab- orations with donors have supported this work, such as the Partnership for Financial Inclusion supported by the Mastercard Foundation and the United Kingdom-funded Harnessing Innovation for Financial Inclusion program, a partnership across World Bank Finance, Competitiveness, and Innovation; the Consultive Group to Assist the Poor; and IFC. Upstream. IFC management highlights that its investment and advisory support in DTT are well complemented by the upstream business, Diagnostics Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Management Response and Analytics, Thought Leadership, and other activities to maximize our de- velopment contribution. Since its establishment, IFC’s Upstream efforts have been strengthening the institution’s ability to bring together different teams from IFC and the World Bank to catalyze highly effective financial solutions for development and to impact policy and regulatory interventions. Eight TMT related Upstream Platforms, including Energy Service Companies for Telecom Anchor Clients; Satellite, Rural Communications, Cloud and Data Services, and internet of things, have been launched, targeting investment opportuni- ties in cutting-edge market segments. To facilitate private investments, each of these platforms is scheduled to publish a study for public use and undertake some analytics that can be used by IFC and the World Bank teams. Country diagnostics and analytics. Complementing the work on Systematic Country Diagnostics (SCDs), IFC, together with the World Bank, produc- es Country Private Sector Diagnostics (CPSDs) since 2017 as a core tool to deliver on IFC 3.0. The diagnostics then feed into two country-level strat- egy documents, the IFC Country Strategy and/or the Bank Group overall Country Partnership Framework. In this regard, many CPSDs have included the assessment of the DTT and sectors at their nexus and have led to World Bank-IFC joint interventions in the digital sector, for instance, in Kyrgyz and Madagascar. Furthermore, a separate Disruptive Technology and Dig- ital Economy guidance note is being created to help each CPSD consider DTT opportunities and risks in the country. In terms of Analytics, IFC is enhancing its internal capabilities by undertaking a series of DTT analyti- cal research pieces, such as ones stated above related to the eight Upstream Platforms and a Sector Deep Dive on digital infrastructure. Recent analytics also studied a model to assess the economic impact of reforms in the tele- xxviii com sector; a benchmarking tool to assess areas of regulatory interventions to enable digital entrepreneurship; and the impact of digital connectivity on entrepreneurship and innovation in Africa. Thought leadership. We further emphasize the role of IFC in exercising Thought Leadership in the DTT space. IFC has published a rich collection of related Thought Leadership pieces. For instance, recent pieces discuss the potential long-term impact of the COVID-19 pandemic on digitalization and adoption of disruptive technologies and a framework for anticipating the potential impact of COVID-19 on digital connectivity. Other pieces have contributed research and ideas to the application of AI in emerging markets (including its ethical uses), the Africa internet economy, sector trends and investment opportunitiesoffered by DTT, and digital connectivity through infrastructure sharing. Bank Group collaboration. IFC management appreciates an extensive sec- tion dedicated to thistopic in the report and agrees that the World Bank and IFC could play separate but complementaryand highly coordinated roles to enhance DTT projects. As identified in the assessment, there has been in- creasing effective collaboration examples in the DTT and related sectors, in- cluding the “Digital Economy for Africa (DE4A)” initiative to digitally enable Africa by 2030 and the “Identification for Development (ID4D)” initiative highlighted in box 4.2 of the report; CDF Senior Director is a core member of the latter’s Senior Directors’ Group. IFC has contributed to 30 DE4A diag- nostics by sharing its private sector perspectives and knowledge centrally World Bank Group Independent Evaluation Group    xxix collectedand coordinated from across IFC by CDF. IFC continues to work with the World Bank as the Digital Economy Initiatives roll out in Central America (DE4CA) and South Asia (DE4SA) in alignment with the Cascade ap- proach. For successful DTT-related outcomes, IFC teams are on thesame page with the report that contributions from both public and private sectors are needed. Wewill further seek opportunities to improve the collaboration with the World Bank teams. Gender. Although the report’s gender analysis focused on the World Bank, IFC management welcomes an increased focus on the cross-section of gender and technology. IFC’s Gender and Economic Inclusion Group hosts a team focused exclusively on inclusive technology. Since FY19, IFC has invested over $140 million in projects with a gender focus across funds and disruptive technologies and launched multiple advisory initiatives and market leading research reports. Examples include the first-ever research on women and ride-hailing, globally and in Sri Lanka, and the first investigation into emerging women-only solutions, which informed the development and rollout of increasingly prevalent solutions in the industry. The organization has also taken a lead convening the sector to develop and mainstream best practice in inclusive tech; for instance, the Digital2Equal initiative, the pro- gram cited in the report, has brought together 17 companies in the platform economy to close gender gaps. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Management Response The pandemic has exacerbated the need for more investments in this space and a burgeoning of new and innovative technologies that require our sup- port. Digital transformation is key to ensuringthat countries are well poised for the future. To that end in particular, the digital gender divide must be bridged by focusing closely on the role women play in this space–either as en- trepreneurs,clients, or end users and decision makers. We also cannot afford to lose generations of youth dueto lack of access to technologies for education or growth. IFC management will prioritize its effortsin this area, keeping in mind the report’s findings and two recommendations given to IFC. Response to Recommendations Recommendation 1. Given the above progress, IFC management is aligned with the thrust of the first recommendation. As noted, IFC has implemented two major organizational changes that we believe would yield a more effec- tive and efficient response to opportunities to make progress on the devel- opment challenges, such as the twin goals, and to address risks posed by the adoption of DTT. The organizational changes have enabled us to establish CDF as a focal point to promote increased collaboration within the Bank Group and with external partners. In parallel, IFC management would like to further highlight the extensive work that we are doing in the Sector Deep Dives, CPSDs, Country Strategies (including the business plans) as described above. This host of institutionalized instruments offers opportunities for IFC to design, implement and invest in DTT private sector solutions in our coun- tries of operations. Senior Managers within our organization are taking the lead in aligning the DTT agenda, ensuring that it is sufficiently incorporated xxx into the direction of our investment, advisory services, and upstream work programs. We will continue our efforts to leverage the changes implemented. Recommendation 2. With respect to the second recommendation, IFC management acknowledges that DTT is a rapidly changing area where we need more skilled staff. Steps were being taken to address those gaps as this evaluation was being carried out. The aforementioned restructuring has prioritized challenges related to skills in recognition of areas mentioned in the IEG report, and it has paved the way to enhance skillsets. Given the fast-changing pace and evolution of the DTT space, it would be very chal- lenging and virtually impossible to keep expertise embedded in the institu- tion. However, IFC has the core foundations of in-house DTT generalists who understand technology and its impact on our business embedded within our industry groups. IFC management commits to continue assessing its DTT skills and expertise needs as markets and technologies evolve, and leverage external consultants to ensure that our work in the space remains relevant to our clients and impactful in our markets and operations. IFC manage- ment will also continue to strengthen the expertise required to run IT for the Corporation. As also briefly introduced in the report, rapid response training has been offered in key areas, such as privacy, cybersecurity, AI, regulatory reform, digital finance, and technology investing. We are actively collaborat- ing with the information technology services Technology and Innovation Lab to enhance staff understanding of new technology opportunities and are also World Bank Group Independent Evaluation Group    xxxi proactively seeding new activities, such as cybersecurity appraisal, with staff who have previously acquired expertise in these areas. Several units have worked programmatically to upskill their own staff to better support IFC’s technology activities. We agree that harnessing DTT for development demands innovation and involves controlled risk taking. Spaces for such innovations exist in IFC with early-stage fintech and VC investment activities, including recently de- veloped venture ecosystem building programs to further embrace the DTT agenda, such as Startup Catalyst, Scale X, and TechEmerge, as mentioned above. IFC will continue its effort to enhance risk taking and innovative, growth mindsets. 1  The most prominent example is the forthcoming World Development Report 2021: Data for Better Lives, which will highlight ways in which the Bank Group can both harness opportunities and mitigate risks in the area of data. Other forthcoming knowledge products cover topics such as disruptive technologies and sustainable development, the converging technology revolution and human capital: potential and implications for South Asia, clean energy, artificial intel- ligence, cloud and data storage, cybersecurity, digital gender gap, resilience, blue economy, remote monitoring, digital health, ed-tech, citizen engagement, digital government, financial technology, digital solutions to COVID-19 vaccine distribution, and digital ID. The Bank Group will also continue to support cross-sectoral initiatives such as Identification for Development Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Management Response (ID4D), Digital Economy for Africa (DE4A), and the Digital Central Asia-South Asia program. Disruptive and transformative technology (DTT) is also an important element of the World Bank’s response strategy to the COVID-19 crisis, as DTT offers the opportunity for governments, individuals, and businesses to cope with the pandemic, ensure business continuity, prevent ser- vice interruptions, and ensure social distancing. The Bank Group is committed to building back better and working with clients and partners to exploit innovative approaches that can speed progress in fighting the pandemic and transform crisis into opportunity. 2  ID4D was also a leader in ICT procurement. ID4D developed, in collaboration with World Bank procurement specialists, a Procurement Guide and Checklist for Digital Identification Systems. This helped countries (i) understand the risks of vendor and technology lock-in, and possible mitigation measures; (ii) develop a robust procurement strategy for ID systems based on key design decisions; and (iii) develop effective requests for proposals (RFPs) aligned with short and long term goals and sufficiently elaborate business and technical requirements for the ID system. 3  The TMT investment is expected to further grow based on its quadrupled pipeline over the past 12 months and increased sector presence. 4  A co-investment. $4.5 million committed in FY20. 5  IFC/R2016-0247. $7.8 million committed in FY16. 6  IFC/R2019-0174. $12.5 million committed in FY19 7  IFC/R2012-0346. $6 million committed in FY13. 8  A facility through which IFC makes equity and quasi-equity investments in a number of com- mercially-oriented Incubators, Accelerators, Seed Funds, and similar vehicles and structures, xxxii across emerging markets. 9  An incentive program to increase equity funding access for women entrepreneurs. 10  A program seeking to pilot technology projects uniquely tailored to local needs, supported by a global network of industry advisors. 11  https://edgebuildings.com/ Scaling Solar and Lighting Global are described in appendix K of the report. World Bank Group Independent Evaluation Group    xxxiii Report to the Board from the Committee on Development Effectiveness The Committee on Development Effectiveness met to consider the Inde- pendent Evaluation Group evaluation entitled Mobilizing Technology for WMobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Report to the Board Development: An Assessment of World Bank Group Preparedness and the draft management response. The committee welcomed the timely report and commended the Indepen- dent Evaluation Group for the quality and focus of the evaluation. Members were appreciative of management’s broad agreement with the evaluation recommendations and its recognition that while it had begun implementing the recommendations, there was scope for enhancing an already substantial effort undertaken by the World Bank Group to improve staff skills and capac- ities, expand the use of procurement flexibilities, and establish incentive for risk taking and innovation with regard to disruptive and transformational technologies (DTT). They stressed the crucial role that digital technology has played in helping governments, businesses, and people cope with the coronavirus pandemic (COVID-19) as well as the role it can have in driving growth in the economic recovery from the pandemic. In this regard, mem- bers acknowledged the far-reaching and important implications DTT can have in (i) improving lives and fostering growth and development and (ii) accelerating progress toward reducing poverty, promoting shared prosperi- ty, and achieving the Sustainable Development Goals. Noting that the Bank Group was a trusted partner and was well positioned to be effective in help- ing clients integrate innovative solutions to development challenges using transformative technologies, they underscored the importance of assisting client countries in assessing and mitigating risks associated with these tech- nologies. Management reiterated its agreement with all three recommenda- tions with the caveat that (i) the investment in requisite staff skills needed to be viewed while taking into consideration tradeoffs among competing xxxiv staffing needs in the context of a flat real budget and (ii) the fact that, while procurement flexibility existed in the Bank Group, clients were often reluc- tant to use it given their internal control environments. Members agreed with the Independent Evaluation Group’s assessment that the Bank Group needed to demonstrate organizational nimbleness to con- stantly anticipate, innovate, and adapt as part of its mission to achieve the twin goals and thus urged management to continue to build partnerships and develop knowledge and skills for staff to equip them to better support clients in harnessing DTT. They noted the link between discussions on knowledge management, learning, and outcome orientation and the need for stronger collaboration within the Bank Group, a strengthened incentive system, and the creation of a workplace culture of willingness to (i) adapt to an ever-changing working environment, (ii) learn from failure, and (iii) take informed risks to allow innovation. They stressed the need for the Interna- tional Finance Corporation to play a larger role and for the Bank Group to enhance cross-institutional collaboration. Members expressed concern about the lack of reliable data on DTT oper- ations and the impact this may have on monitoring these operations or fostering innovation. In response, management highlighted a sharp increase in Bank Group financing of DTT from $69 million in FY19 to a projected $1.5 billion in FY21 and $2 billion in FY22. Members appreciated the proposed di- agnostic work and pipeline of analytical activities and organizational chang- World Bank Group Independent Evaluation Group    xxxv es to better reflect the role of technology in development. They encouraged management to speed up implementation where there was capacity to deliver technology deployment and bridge the digital gap within and across countries. Acknowledging the impact that technology can have in tradition- al firms, members encouraged the Bank Group to think about how to make technology inclusive, equitable, and an enabler of job creation. 1 | Background and Context Highlights The World Bank Group adopted a new approach to disruptive and transformative technologies (DTT) in 2018, later merging it with the 2019 DTT Mainstreaming approach, and further developing it through the 2020 Mainstreaming Digital and Disruptive Technol- ogies Initiative. The transition to a mainstreaming approach from earlier discrete areas of support marked a significant increase in the Bank Group’s ambition regarding DTT. DTT-related commitments pertaining to the 19th Replenishment of the International Development Association and the capital increase package, as well as the opportunities DTT offer to respond to the disruptions caused by the coronavirus (COVID-19), add significance and urgency to the Bank Group’s DTT work. Consistent with the Bank Group’s use of the term, this evaluation adopts a broad interpretation of DTT and covers (i) support to digital technologies (such as the internet, mobile phones, smart- phones, and Wi-Fi), and to other technologies (such as robotics, artificial intelligence, the internet of things, nanotechnology and biotechnology, 3D printing, batteries, drones, solar panels, and self-driving vehicles); (ii) enabling policies, institutions, and skills; (iii) relevant lending and nonlending operations in all sectors (not just in the digital development and information and communica- tion technology sectors); and (iv) both the “application” of tech- nology (for example, using computers in the classroom) and the “response” to technology (for example, imparting socioemotional skills to equip people for future jobs that machines will not be able to perform). 1  The world is experiencing rapid technological change with far-reaching implications for development. The accelerating pace of technology dif- fusion, the convergence of multiple technologies, and the emergence of global platforms are disrupting traditional development models (Pauwels 2019; World Bank 2019h). Disruptive and transformative technologies (DTT) are already reshaping the way goods and services are produced and consumed and have profound implications for the functioning and dynamics of the global economy (Dobbs, Manyika, and Woetzel 2015; Manyika and others 2017). At the national level, traditional development paths may no longer be viable, at least in some developing countries, as they run out of low-skilled manufacturing opportunities sooner and at much lower levels of income compared with the experience of early Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 1 industrializers (Hallward-Driemeier and Nayyar 2018).1 DTT offer new opportunities for reducing poverty and promoting shared prosperity. Business as usual is unlikely to get countries to the Sustainable Development Goals by 2030 (Kramer, Agarwal, and Srinivas 2019). DTT can improve people’s lives and transform economies, governments, and societ- ies, for example by enhancing connectivity among markets around the globe, opening up new jobs and sources of livelihood, and improving the delivery of social and financial services (Brynjolfsson and McAfee 2014; Dobbs, Man- yika, and Woetzel 2015; Manyika and others 2017). The United Nations Sec- retary General’s High-Level Panel on Digital Cooperation (2018) advanced proposals to strengthen cooperation on digital issues among governments, the private sector, civil society, international organizations, academia, the technical community, and other relevant stakeholders.2 Box 1.1 presents examples of development opportunities created by DTT. 2  isruptive and Transformative Technologies Have Created New Box 1.1. D Development Opportunities Low-Tech Development Solutions: Smartphones and Audio » The World Bank’s Geo-Enabling for Monitoring and Supervision is a client- focused capacity-building program that enables local stakeholders to leverage simple and low-cost digital tools (“pocket science”) for customized monitoring and evaluation, remote supervision, real-time risk monitoring, and portfolio map- ping for coordination across projects and partners (World Bank 2019k). » Amplio Network, a nonprofit organization, is sharing knowledge with the world’s hardest-to-reach communities through an easy-to-use “talking book” audio device de- signed to provide information to people who cannot read and who live in places where there is no electricity or network and collect their feedback (Brodnig and others 2020). High-Tech Development Solutions: Artificial Intelligence » Deep-learning algorithms are diagnosing retinopathy in patients living in rural India, where there is a shortage of ophthalmologists. » Portable genomic sequencers are bringing the lab to the jungle, allowing for the diagnosis of the Ebola virus in “hot zones.” » Open Innovation Lab in Shenzhen, China, young inventors have designed wearable devices that rely on image recognition to help farmers detect diseases on crops. » Companies like Zipline are using artificial intelligence in autonomous drones to World Bank Group Independent Evaluation Group    3 deliver critical medical supplies, such as vaccines, to rural hospitals in Africa. » The United Nations Children’s Fund is collaborating with the Massachusetts Insti- tute of Technology on deep-learning expertise to simulate images of major global cities in ruin to promote empathy and connection with the suffering of those who have experienced bombing, loss, and war. » The combined optimization of biometric, genomic, behavioral, and physical systems data is giving rise to “affective computing”—algorithms that can successfully analyze, nudge, and communicate with people. Affectiva, an affective computing company, has developed Peppy Pals, a series of educational apps that teach children about social and emotional intelligence by learning from situations in an online world. (continued)  isruptive and Transformative Technologies Have Created New Box 1.1. D Development Opportunities (cont.) » The World Development Report 2019 notes that Ant Financial (of the Alibaba Group) makes loans using a “3-1-0” online lending model based on big data and artificial intelligence, which involves a three-minute application process and a one-second processing time, with zero manual interventions (Pauwels 2019; World Bank 2019m). » India’s Mobile-Based Surveillance Quest Using Information Technology is a smartphone-based platform that enables better outbreak monitoring and real- time responses for malaria (Pathways for Prosperity Commission 2019a, 2019b). Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 1 Source: Independent Evaluation Group. DTT also pose risks for reducing poverty and promoting shared prosperity. At the macro level, new technologies may reduce job opportunities for the less skilled. According to recent studies by the International Monetary Fund (2018) and the Organisation for Economic Co-operation and Development (2018), technological change likely has contributed to widening inequali- ties in earnings, shifting labor demand toward high-skilled occupations, and promoting automation over routine work. Concentrating market power in the hands of digital and data service providers, for example, may give rise to new policy and regulatory issues concerning monopolies, competition, account- ability, data privacy, and taxation (Lee 2018). Furthermore, deep-learning sys- tems (systems inspired by the structure and function of the brain or neural networks) can drastically intensify the nature and scope of cyberespionage and cyberattacks within smart cities (Pauwels 2019) and enable cybersur- veillance by governments for political ends. New technologies also bring the risk that a few powerful states will shape the digital future for everyone else (Pathways for Prosperity Commission 2019a). Box 1.2 highlights the risks posed by DTT. 4 Box 1.2. Risks Posed by Disruptive and Transformative Technologies » Skills and income inequality. The World Development Report 2016 pointed out that with the changing nature of jobs, those who can acquire additional skills will switch to better-paid nonroutine occupations; they will be the winners, and others will be the losers (World Bank 2016e). Automation poses a greater risk for women as they tend to be overrepresented in routine work, although this depends on their level of education (Brussevich and others 2018). » Platform firms and income inequality. By operating in regulatory gray zones, platform firms can avoid taxes, creating huge wealth for a few with data collected from the many, thereby creating monopolistic tendencies and exacerbating income inequality. » Algorithms as “weapons of math destruction.” Algorithms can be opaque, unreg- ulated, and uncontestable and may reinforce bias: “If a poor student cannot get a loan because a lending model deems him too risky (by virtue of his zip code), he’s then cut off from the kind of education that could pull him out of poverty, and a vicious spiral ensues” (O’Neil 2016). » “Internet of bodies,” data privacy, and cybercrime. Most corporate artificial intel- ligence platforms already have access to individuals’ online behaviors, relation- ships, health, and emotional states. But soon enough, these platforms will acquire baseline information about people’s vital signs, organs, and genomes. The “internet of bodies” and simpler digital technologies all have implications for data privacy and cybercrime (Pauwels 2019; USAID 2018b). » Internet’s environmental footprint. Observers are increasingly concerned about World Bank Group Independent Evaluation Group    5 the internet’s environmental footprint. They point out, for example, that mining of blockchain-based digital currencies is hugely energy intensive (see Randeep Sudan comments in Stanley 2017). Concerns about the effects of electromagnetic radiation on people, plants, bee colonies, and frogs around the globe were presented at the United Nations 2018 Forum on Science, Technology and Innovation (Singer 2014). » Social media, aspirations, envy, and violence. The World Development Report 2019 notes that technology, in particular social media, affects the perception of rising inequality in many countries and heightens feelings of alienation and envy, which can in turn incite discontent, conflict, and violence (World Bank 2019m). Women and girls are more likely to experience threats of intimidation, harass- ment, defamation, violence, surveillance, or some combination of these than are men and boys (World Wide Web Foundation 2015). The explosion of data distinguishes the current DTT revolution from past technological revolutions (such as the industrial revolution): In 2015 and 2016 alone, more data were created than in all previous years combined (Goldberg 2020). The recent explosion of data has largely come from private sources, such as mobile phones, electronic transactions, and satellites. Such data are collected cost-effectively, with higher frequency, and at fine levels of granularity, typically by private sector actors. Such data has fueled inno- vation in the private sector. According to the World Development Report 2021, development policy has traditionally relied on public data collected by gov- ernments, which offer better coverage of the population of interest but can also be costly, collected infrequently, and lack granularity (World Bank 2021). Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 1 The World Bank Group has adopted a new approach to DTT in recognition of the opportunities DTT offer and the risks they pose. The Bank Group outlined its new approach to DTT in a 2018 Development Committee paper, “Disruptive Technologies and the World Bank Group: Creating Opportunities—Mitigating Risks” (World Bank 2018b), which was subsequently merged with the 2019 Development Committee paper “Mainstreaming the Approach to Disruptive and Transformative Technologies at the World Bank Group” (henceforth called the “Mainstreaming paper”; World Bank 2019h). The new Mainstreaming approach aims to harness the opportunities and mitigate the risks posed by DTT to accelerate progress toward the Sustainable Development Goals and the Bank Group’s twin goals of ending extreme poverty and boosting shared prosperity.3 The Mainstreaming paper was accompanied by the Bali Fintech Agenda, jointly prepared by the World Bank and the International Monetary Fund (World Bank 2018g). More recently, the Bank Group’s 2020 Program Document for the Mainstreaming Digital and Disruptive Technologies (MDDT) Initiative further develops the Mainstreaming approach, mainly outlining the institutional arrangements for DTT (a Network, a Technical Working Group, and a Secretariat) within the Bank Group (World Bank 2020d). The Mainstreaming approach to DTT builds on previous technology strate- gies, while implying a significant increase in ambition. The Bank Group’s past technology engagements encompassed support to science and technology, re- search and development, and innovation. Specific technology strategies have existed since 2002, mainly for the information and communication technology (ICT) sector. In transitioning from discrete areas of support to mainstream- 6 ing, the Bank Group’s Mainstreaming approach extends beyond ICT to other sectors and covers the enabling policies, institutions, and skills. Appendix H provides details of the evolution of the Bank Group’s approach to DTT. The Independent Evaluation Group (IEG) definition of DTT is consistent with that of the Bank Group.4 Box 1.3 provides further detail of the definition, and box 1.4 lists the various programs and units engaged in DTT work across the Bank Group.  orld Bank Group and Independent Evaluation Group Definition Box 1.3. W of Disruptive and Transformative Technologies First, the World Bank Group’s Mainstreaming paper notes that disruptive and transforma- tive technologies (DTT) are “often based on digital technologies and products,” but that these “go far beyond connectivity and the potential of the internet” (World Bank 2019h, 4). Second, beyond digital technologies, DTT include technologies such as robotics, artifi- cial intelligence, the internet of things, nanotechnology and biotechnology, 3D printing, batteries, drones, solar panels, and self-driving vehicles. Third, DTT also include tech- nology’s analog complements (such as enabling policies, institutions, and skills; World Bank 2016e). Fourth, in addition to the “application” of technology to development (for example, using computers in the classroom), DTT also cover the “response” to technol- ogy, as in the case of education (for example, imparting socioemotional skills to equip people for future jobs that machines will not be able to perform). Fifth, the Bank Group recognizes the close relationship between DTT and innovation. Innovation both impacts and influences DTT and results from them: innovation has created DTT solutions such as online education and telemedicine, and DTT have led to innovations such as artificial World Bank Group Independent Evaluation Group    7 intelligence and the internet of things. Furthermore, the links between DTT and innova- tion are recognized in the East Asia and Pacific Region’s DTT framework called Connect, Harness, Innovate, Protect (CHIP). Sixth, DTT cover both new uses of old technologies (for example, mobile phones) and new technologies and business models. Seventh, the magnitude of the “step change” in disruptive technologies or the degree of transfor- mation that defines transformative technologies were not specified in the Bank Group’s Mainstreaming paper, leaving the terms disruptive and transformative ambiguous. Finally, the Mainstreaming paper presented digital connectivity and agile regulation as two of the Bank Group’s five DTT corporate priorities, implying that the Bank Group conceives of DTT as an umbrella term that subsumes digital technologies. Source: Document reviews by the Independent Evaluation Group.  isruptive and Transformative Technologies Programs and Box 1.4. D Units in the World Bank Group Key World Bank programs and units focused on disruptive and transformative tech- nologies (DTT) include the following: • AgObservatory • Artificial Intelligence Lab • Blockchain Lab • Cybersecurity workstream • Data Collaboratives • Data Council • Data Development Group • Data Lab • DECAT • DEC Data • Digital Technology for Devel- opment • Disruptive KIDS (Knowledge, Information and Data Services Helpdesk) • EdTech • Fintech Coordination Group • Gender Innovation Lab • Geo-Enabling Initia- tive for Monitoring and Supervision • Geospatial Operations Support Team • GovTech • Information and Technology Solutions (ITS) Geospatial • ITS Innovation Lab • Open Learning Campus • Social Development Data Lab • Sustainable Development Remote Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 1 Preparation and Supervision Group • Technology and Innovation Lab• World Bank Data Catalog • workstream on data and privacy (piloted in Middle East and North Africa and in Europe and Central Asia) • regional DTT initiatives such as Digital Central Asia–South Asia (Europe and Central Asia), Connect, Harness, Innovate, Protect—CHIP (East Asia and Pacific), Digital Economy Country Diagnostics and advisory services and analytics (South Asia), Digital Economy for Africa Moonshot/Accelerate (Africa), Digital Trans- formation Program (Latin America and the Caribbean), and MNA Tech (Middle East and North Africa) • leveraging digital technologies to support the Jobs and Economic Transformation Agenda in International Development Association countries • several DTT activities linked to COVID-19 • data programs and units covering data collection; data storage, management, and sharing; data analysis; and work and industry. Key International Finance Corporation units and programs focused on DTT include the following: • the Telecoms, Media, and Technology group within the Infrastructure and Natural Resources Department • the Fintech team in the Financial Institutions Group • the Disruptive Technology and Funds Group • TechEmerge • ScaleX. In addition, the International Finance Corporation has created Upstream units for each of the industry departments, including those focusing on DTT, to support market cre- ation initiatives and pipeline development for the International Finance Corporation. Source: World Bank Group data (not evaluated by the Independent Evaluation Group). 8 The Bank Group has adopted the build, boost, and broker value proposition to mainstream its DTT approach and is pursuing five corporate priorities, the Bali Fintech Agenda priorities, and sectoral and regional programs. The five corporate priorities are (i) country diagnostics, (ii) agile regulations, (iii) digital connectivity, (iv) digital government, and (v) skills and capabilities for the new economy and the role of education. Following the Bali Fintech Agenda, the Bank Group is also supporting fintech and digital entrepreneur- ship. Sectoral and regional programs to support the Mainstreaming approach include the Digital Economy for Africa (DE4A) Moonshot/Accelerate and MNA (Middle East and North Africa Tech). Appendix B provides details. The Bank Group is implementing an array of DTT initiatives across all Re- gions. These initiatives include lending and nonlending work: (i) in Europe and Central Asia, through the Digital Central Asia–South Asia (CASA) optical fiber cable program; (ii) in East Asia and Pacific, through the Connect, Har- ness, Innovate, Protect (CHIP) framework; (iii) in South Asia, through the Digital Economy Country Diagnostics; (iv) in Africa, through the DE4A Moon- shot/Accelerate with the African Union; (v) in Latin America and the Carib- bean, through the Digital Transformation Program with the Organization of Eastern Caribbean States countries; and (vi) in Middle East and North Africa, through MNA Tech, focused on the digital transformation of governments and economies as well as regional integration. Bank Group initiatives also include a significant pipeline of country and regional initiatives. World Bank Group Independent Evaluation Group    9 The Bank Group is also engaged in innovative DTT initiatives. Innovative DTT initiatives include the Development Data Partnership (formerly Data Collab- oratives), Development Marketplace to Address Gender-Based Violence (in which the International Finance Corporation [IFC] also participated), Disrup- tive Technologies for Development Trust Fund, Geo-Enabling for Monitoring and Supervision (GEMS), Geospatial Operations Support Team, Information and Technology Solutions (ITS) Technology and Innovation Lab, Lighting Global (including Lighting Africa and Lighting Asia), and Social Develop- ment Data Lab. IFC’s innovative DTT initiatives include TechEmerge and ScaleX (which increases access to funding for women entrepreneurs). DTT-related commitments in the 19th Replenishment of the International Development Association (IDA) and the capital increase commitments add significance to the Bank Group’s DTT support. These include several policy commitments relating to technology that the Bank Group will need to meet, for example boosting broadband penetration and supporting digital financial services and entrepreneurship; supporting women’s digital skills and result- ing access to higher productivity jobs and increased access to ICT services; and supporting the use of field-appropriate digital tools and analysis in frag- ile and conflict-affected situations (IDA 2020). Regarding the capital increase commitments, Bank Group management has undertaken to further leverage smart technology solutions (World Bank 2018e). More recently, by bringing to light the importance of social distancing and travel restrictions to contain pandemic diseases, the coronavirus (COVID-19) Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 1 has elevated the urgency of location-neutral solutions, for which DTT are especially suited. The Organisation for Economic Co-operation and Develop- ment and the World Bank are collaborating to use DTT (for example, mobile apps, artificial intelligence [AI], and big data) to aid the COVID-19 response around the world. The four directly health-related uses of DTT are (i) out- break spread monitoring; (ii) prevention and symptom tracking; (iii) contact tracing; and (iv) enforcement of containment measures (Amaral, Vranic, and Lal Das 2020). Beyond these uses, the Bank Group has experience in other areas—such as fintech;5 digital IDs; technology-enabled operations in fra- gility, conflict, and violence contexts; and distance education and online learning—that can also be deployed to respond to COVID-19. IFC has also sought to leverage its venture capital portfolio in HealthTech and EdTech. The Bank Group has provided $160 billion in financing through June 2021, including to support digital foundations and applications (appendix J). This significant commitment for the pandemic response demonstrates the Bank Group’s preparedness to quickly address exogenous shocks. An IEG evalua- tion is planned to assess the quality and relevance of the response. About This Evaluation This evaluation sought to answer the following question: How well prepared is the Bank Group to help clients harness the opportunities and mitigate the risks posed by DTT? Key aspects of the evaluation’s focus and scope are as follows. First, the focus of the evaluation is on the preparedness of the Bank 10 Group to provide DTT support rather than on measuring the effectiveness or outcomes of that support—a deliberate choice given the newness of the Bank Group’s Mainstreaming approach. Second, the term DTT refers to DTT for development. Third, the evaluation does not include an assessment of the Bank Group’s own use of computer hardware and software or of its own information technology (IT) systems. Fourth, the evaluation’s analysis of education, procurement, and gender covers the World Bank only. Fifth, in this report the term Bank Group is used to cover the World Bank and IFC only, not the Multilateral Investment Guarantee Agency or the International Centre for Settlement of Investment Disputes, which are not part of this evaluation. IEG identified the preparedness dimensions from the organizational effec- tiveness literature and relevant IEG evaluations and organized these into two groupings: (i) organizational ability (in terms of staff skills and mind- sets, as well as internal processes and procedures such as collaboration and procurement) and (ii) organizational willingness (in terms of institutional culture and incentives for risk taking and innovation).6, 7, 8 These dimensions were examined to assess how well prepared the Bank Group was to help clients harness the opportunities and mitigate the risks posed by DTT. The overall framework for analysis informing this evaluation was that preparedness dimensions relating to organizational ability and organizational willingness influence the Bank Group’s support to DTT with regard to both its five DTT corporate priorities, fintech, and digital entre- World Bank Group Independent Evaluation Group    11 preneurship, as well as its sectoral and regional programs. The relevance, timeliness, and effectiveness of the Bank Group’s support for DTT will also be influenced by how well the Bank Group support for DTT is tailored to and affected by country context factors. Figure A.1 in appendix A illustrates this. Specifically for IFC, the assessment of DTT preparedness focused mainly on the corporate priorities of the Mainstreaming paper and interventions from relevant key departments. These key departments were as follows: for the digital connectivity corporate priority, the Telecoms, Media, and Technology (TMT) group in the Infrastructure and Natural Resources Department; for fintech and digital entrepreneurship, the Financial Institutions Group and the Disruptive Technology and Funds group. Activities of other units in IFC’s evolving organizational structure, notably the Upstream units created in fiscal year (FY)20, were too recent to be the subject of this evaluation. The main evaluation instruments included the following: » A review of the organizational effectiveness literature and existing IEG evaluations. » Thematic reviews covering the Bank Group’s DTT corporate priorities, fin- tech, and digital entrepreneurship as well as selected sectoral and regional programs. » A review of project documents and advisory services and analytics (ASA) relating to all DTT-related World Bank projects in education and gender over Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 1 the past five years. » A review of tech-informed Systematic Country Diagnostics (SCDs) and Coun- try Partnership Frameworks (CPFs). » Document reviews of lending and ASA for a purposive sample (selected based on interviews and database analysis) of digital connectivity in the World Bank and IFC (investment projects), electronic government services (especially those pertaining to social protection and digital ID), fintech (in both the World Bank and IFC), and digital entrepreneurship operations. » A review of the Bank Group’s information systems and databases—in collab- oration with the Development Economics, Development Data Group (DEC- DG)—to assess the staff skills for DTT work. » Semistructured interviews covering 105 Bank Group staff, including World Bank and IFC team leaders and investment officers, as well as 57 external stakeholders, including World Bank and IFC clients. The 105 Bank Group staff interviews were analyzed using NVivo, qualitative data analysis computer software. Details of the NVivo analysis and other methodological aspects are presented in appendix A. Evaluation Challenges There is no official Bank Group operational list that identifies DTT-related 12 projects and ASA (except for Digital Development or ICT projects and ASA). Consequently, IEG identified DTT-related projects and ASA in consultation with the relevant Global Practices. It did so for two Bank Group priorities that have strong significance for the twin goals: education (the Bank Group’s DTT corporate priority skills and capabilities for the new economy) and Gen- der (Bank Group Global Theme). Given that the Bank Group’s Mainstreaming approach for DTT is relatively new, this evaluation focused on Bank Group preparedness for providing DTT support rather than on the outcomes of that support. Most of the Bank Group’s DTT-related operations identified in the Development Committee papers are incipient or incomplete, meaning that effectiveness or outcomes, which are among the key measures of Bank Group preparedness, do not yet exist (World Bank 2018b; World Bank 2019h). In this context, the evaluation establishes a baseline and highlights specific areas for the Bank Group’s at- tention, action, and monitoring that can help it enhance the effectiveness of its future DTT support to clients. Bank Group data related to DTT preparedness (such as on staff skills and mindsets, collaboration, procurement, and institutional culture and incen- tives) are, at best, incomplete, requiring IEG to draw on staff interviews. Staff interviews had the advantage of helping assess not only how things looked on paper but also how staff perceived these things in practice. If staff perceived that the Bank Group had not equipped them (for example, with appropriate skills and knowledge), empowered them (for example, through World Bank Group Independent Evaluation Group    13 adequate decision-making power to innovate and chart a new course), or supported them (for example, when they took risks and failed), it would not matter how good things looked on paper. NVivo was used to systematically analyze staff views. Report Organization Chapter 2 focuses on key Bank Group enablers of DTT support and the links between this support and the twin goals. The links with the twin goals were assessed for two select Bank Group DTT corporate priorities: (i) country di- agnostics that help chart the new drivers of growth (corporate priority 1) and (ii) skills and capabilities for the new economy (corporate priority 5) and the role of education. Chapters 3 and 4 examine the Bank Group’s preparedness regarding organizational ability in terms of staff skills and mindsets (chap- ter 3), and internal processes and procedures, specifically collaboration and procurement (chapter 4). Chapter 5 assesses the Bank Group’s preparedness regarding organizational willingness in terms of the Bank Group’s institu- tional culture and incentives for risk taking and innovation. Chapter 6 pres- ents—for the Bank Group’s consideration—directions of travel that can help the Bank Group move forward in strengthening its preparedness for DTT for development. Chapter 7 concludes and makes three recommendations. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 1 14 1  The World Development Report 2013 pointed out how India and other countries set up high- tech enclaves despite poor overall information communication technology infrastructure (World Bank 2012b, 55–57). See also Peralta-Alva and Roitman (2018) for a comparison. 2  The United Nations High-Level Panel on Digital Cooperation, chaired by Jack Ma and Melinda Gates, makes five sets of recommendations: (i) build an inclusive digital economy and society; (ii) develop human and institutional capacity; (iii) protect human rights and human agency; (iv) promote digital trust, security, and stability; and (v) foster global digital cooperation. 3  The twin goals are (i) end extreme poverty: reduce the percentage of people living on less than $1.25 a day to 3 percent by 2030; and (ii) promote shared prosperity: improve the living standards of the bottom 40 percent of the population in every country. 4  Although the 2019 Development Committee mainstreaming paper used the term disruptive and transformative technologies (DTT) (which the Independent Evaluation Group [IEG] uses), the World Bank Group’s most recent mainstreaming paper uses the term DDT (digital and disruptive technologies) with no change in the definition. 5  The scale of mobile money continues to grow, with more than 900 million accounts in 90 countries and $1.3 billion in daily transactions (Pasti 2018). 6  For organizational effectiveness literature, see, for example, Burnard and Bhamra 2011; Christensen 1997; Christensen, Anthony, and Roth 2004; Christensen and Overdorf 2000; Chris- tensen and Raynor 2003; Hatum and Pettigrew 2006; Sheffi 2007. 7  For relevant IEG evaluations, see, for example, World Bank 2011a, 2012a, 2013, 2015a, 2016b. The detailed list of preparedness dimensions based on the literature is (i) organizational World Bank Group Independent Evaluation Group    15 8  ability: knowledge and thought leadership; staff skills; resources; organizational structure, roles, processes and decision-making that allow for flexibility; agility, knowledge sharing and learning, and cross-sectoral coordination and collaboration; ability to identify, adapt, and deploy knowledge on technological developments; and partnerships, global networks, and positioning relative to other actors; and (ii) organizational willingness: leadership, culture, and incentives for innovation, and risk appetite and informed risk taking. 2 | Harnessing DTT for Development: Key Bank Group Enablers and Links with the Twin Goals Highlights Key World Bank Group Enablers The Bank Group’s traditional areas of strength, such as its support for global public goods including development data, honest bro- ker role, production of quality advisory services and analytics, and ability to mobilize International Development Association resources and trust funds, have enabled its support for disruptive and trans- formative technologies for development. Links between Two Bank Group Disruptive and Transformative Technology Corporate Priorities and the Twin Goals Country Diagnostics The Bank Group has given increased attention to digital economy issues in a few recent country diagnostics and strategies. However, the Country Partnership Frameworks are not always informed by the digital economy findings in the Systematic Country Diagnos- tics, and there has also been insufficient effort in the latter in linking the support for digital technologies to the broader development agenda, including the twin goals (for example, when identifying key constraints). Furthermore, the newly introduced digital econ- omy diagnostics do not adequately address issues of poverty and inclusion. 17  Skills and Capabilities for the New Economy and the Role of Education The Bank Group’s top corporate initiatives in education—the Human Capital Project and the Learning Poverty initiative—spot- light “necessary” skills, such as literacy and numeracy. The Bank Group has an opportunity to move the narrative forward from neces- sary skills to also include twenty-first century “sufficient” skills, such as advanced cognitive skills, digital literacy, and socioemotional skills, in particular a growth mindset or the ability to “learn to learn” and adapt. Both necessary and sufficient skills are critical for ensur- ing that a workforce is well prepared for the future labor market. 18   Key Enablers of the Bank Group’s DTT Support The Bank Group’s comparative strength on global public goods has enabled its DTT support. The Bank Group is well positioned to provide leadership on global public goods using a mix of knowledge, financing, and technology. It has created an important niche in global public goods through which it is increasingly deploying DTT (Kanbur 2017). For example, it is applying digital technologies to regulatory harmonization through the Single Digital Mar- ket for East Africa. It is also using new technological tools, such as machine learning, to predict the onset of famines and the displacement of popula- tions resulting from food shortages and conflicts, for example through the Famine Action Mechanism, a global partnership, discussed in appendix G. The Global Facility for Disaster Reduction and Recovery, also a global part- nership, enables DTT support through open-source technology and geo- spatial data-sharing platforms to help people worldwide gain access to risk information.1 In addition, IFC has aimed to be a thought leader in applying technologies to development, with initiatives such as the Investor Guide- lines for Responsible Investing Digital Financial Services (IFC 2017a), reports such as How Technology Creates Markets (IFC 2018), recent analytical work on DTT and COVID-19, and several global upstream platforms (which, given their recent introduction, have not been evaluated for this report). World Bank Group Independent Evaluation Group    19 The Bank Group has established an international reputation in development data, a global public good, which has enabled its DTT support. As the de- mand for development data increases, new technological developments are revolutionizing data production, analysis, and use. For example, the internet of things and social media are contributing to an explosion in the quantity of data for machine learning and AI and are improving the quality of insights from those data. The Bank Group is helping clients take advantage of these opportunities through innovative data initiatives such as the Geospatial Operations Support Team, GEMS, and the Development Data Partnership (formerly Data Collaboratives). Appendix D discusses these initiatives. The Bank Group is an honest broker between the public and private sectors, which has also enabled its DTT support. The Bank Group’s perceived neutral- ity with regard to technology solutions, reputation among global technology players, access to international technological knowledge (academic and other), and seat at the government table underpin its role as an honest broker (Kan- bur 2017; Skoll World Forum 2013). In particular, the Bank Group’s honest broker role in providing clients with technical expertise, especially on poli- cy and regulatory matters related to DTT support, was highlighted by Bank Group staff working on digital connectivity and the GovTech Global Initia- tive who were interviewed for this evaluation. The Bank Group leverages regulatory reform as it expands connectivity infrastructure in lower-income countries and uses its honest broker role to help countries address privacy and other risks. For example, the 2017 Uruguay Improving Service Delivery to Citizens and Businesses through e-Government project aims to address Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 2 DTT risks by issuing guidelines for data architecture platforms on gover- nance structures, security tools, and data privacy protocols. Interviews with clients for IFC projects in Nicaragua and Turkey indicated that they appre- ciated IFC’s knowledge about technology, long-term focus, and ability to connect clients with the right vendors. However, clients also noted ways in which the Bank Group’s honest broker role may be compromised: (i) lengthy bureaucratic processes, especially in procurement (for example, in the Dem- ocratic Republic of Congo, Nicaragua, and Turkey); (ii) lack of understanding of local context and technology, as well as poor coordination, especially in fragility, conflict, and violence contexts (for example, the Central African Republic and Somalia); and (iii) staff who are not trained in the latest tech- nology that clients demand (for example, in Vietnam). In addition, the Bank Group has leveraged its honest broker role to adopt a problem-driven approach in the Identification for Development (ID4D) program. The Bank Group has raised awareness about and encouraged foun- dational ID systems because they can help more efficiently deliver multiple services (such as social protection and taxation) through interoperable sys- tems, although foundational ID systems carry risks (see the Building on the Bank Group’s Existing Strengths section in chapter 6). The ID4D program aims to address long-standing challenges in targeting the right beneficiaries— digitizing unified social registries can reduce the likelihood of inefficiencies in transfers, for example. Appendix I lists key ID4D activities within the World Bank, and box 2.1 presents lessons from the Bank Group’s experience with 20 ID4D on meeting development challenges and the operational needs of clients by taking advantage of the opportunities created by digital solutions. Box 2.1. Lessons from Identification for Development Adoption and scaling of digital technologies work well when the operational needs of clients identified by the Global Practices converge with the opportunities created by digital and technological solutions. This appears to have been achieved, for example, in the case of the World Bank Group’s support for the Identification for Development program. The Bank Group has been well positioned to provide its borrowers (where there is demand) with the knowledge and technical assistance to develop or upgrade existing identification systems that can enable delivery of a wide range of social and e-governance services. Leadership support for Identification for Development has been important. The Jobs and Social Protection Global Practice and the Governance Global Practice were at least partly responsible for developing services that demonstrated the immediate value of improved identification to governments (for safety nets, conditional cash transfers, and government-to-person salaries and pensions), which generated client interest in foundational ID systems. Important for effectiveness has been the flexibility in sequencing the various interventions and services that are digitized and in estab- lishing the links to foundational ID systems when the opportunity and demand arises among country clients. World Bank Group Independent Evaluation Group    21 Source: Independent Evaluation Group interviews. The Bank Group’s DTT support has been enabled by its ability to undertake quality ASA. Bank Group interviewees reported that ASA place the Bank Group at the cutting edge for supporting clients with the needed action- able recommendations to ensure a firm footing for their digital economies. For example, ASA for the Kyrgyz Republic in the wake of the 1990s telecom reforms highlighted the need to move from a fragmented to a consolidated regulatory framework and focused on regional connectivity, attracting pri- vate investment, and building up the demand side (for example, data centers, digital platforms, identification systems, and cybersecurity). Another exam- ple of ASA that can enable World Bank support to DTT is the forthcoming ASA The Converging Technology Revolution and Human Capital: Potential and Implications for South Asia (Bashir and others, forthcoming), which assesses what the World Bank can do to further build, protect, deploy, and empower human capital in the South Asia Region. The World Bank’s ability to mobilize IDA grants, credits, and trust funds has enabled its DTT support. IDA financing facilitated open access to all pro- viders in the Benin telecom sector as part of the West Africa Regional Com- munications Infrastructure Program, making the sector more competitive and potentially enabling Benin to market excess capacity to neighboring landlocked countries. Trust funds, which offer more flexibility, are a frequent and popular source of financing for DTT, especially for innovative and risky Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 2 projects, such as those in fintech and digital entrepreneurship. In the Kenya Industry and Entrepreneurship Project, local start-ups were not producing what local industry needed. Using trust funds, the World Bank launched a competition among the start-ups; the winner would work with an agricul- tural company on concrete solutions. This helped the World Bank team to better understand local issues such as intellectual property rights. Going forward, the World Bank would benefit from monitoring whether its financ- ing for DTT relies too heavily on trust funds, affecting budget predictability or skewing organizational priorities toward what trust fund donors want. IFC’s Disruptive Technologies and Funds department is working with both start-ups and established companies to support innovation in developing countries. IFC seeks to support entrepreneurship ecosystems in developing countries. IFC’s investments cover a broad spectrum of disruptive technolo- gies, including fintech (BKash in Bangladesh and GoPay in Indonesia), e-com- merce (Twiga in Kenya and TradeDepot across Africa), e-logistics (Kobo360, a digital platform connecting truck drivers with freight companies, initially in Nigeria), and HealthTech (Clinicas Azucar in Mexico). IFC advisory services have similarly supported the DTT agenda. For instance, through its Digi- tal2Equal initiative, in close partnership with the European Commission, IFC has sought to bring together technology companies to promote oppor- tunities for women in expanding access to jobs, assets, and business across online platforms. 22 Links between the Bank Group’s DTT Support Relating to Two Corporate Priorities and the Twin Goals These links were assessed for two of the five Bank Group DTT corporate pri- orities, noting that all DTT corporate priorities are relevant for achieving the twin goals: (i) country diagnostics that help chart the new drivers of growth (corporate priority 1) and (ii) skills and capabilities for the new economy (corporate priority 5) and the role of education.2 The corporate priority of country diagnostics was selected because it can provide a basis for designing country level (rather than ad hoc) DTT support for addressing the twin goals. The country priority of skills and capabilities for the new economy and the role of education was selected because it is a powerful driver of develop- ment, and education is one of the strongest instruments for reducing poverty and inequality.3 Country Diagnostics Core country diagnostics such as SCDs, the newly introduced digital economy diagnostics, and ASA can help the Bank Group identify potential DTT op- portunities and risks. Even when there is no specific client demand for such an identification, these diagnostics can allow the Bank Group to effectively World Bank Group Independent Evaluation Group    23 exercise selectivity in programming and to nudge the client toward greater focus on the role DTT can play to further country development agendas and the risks such technologies may present. The Bank Group has begun to address the digital economy in its country diagnostics (in SCDs) and strategies (in CPFs). The Mainstreaming paper ref- erenced the digital economy discussion in the CPF for Morocco and the SCDs for Poland and Senegal (World Bank 2017b, 2018e, 2019h, 2019i). This eval- uation found that both the Morocco and Senegal CPFs addressed the digital economy gaps identified in their respective SCDs. Digital technologies refer- enced in the Morocco CPF were linked to financial inclusion, social protec- tion, and environmental sustainability of agrifood value chains (World Bank 2019i, 23, 26, 33, 35). The SCD had noted the nascent stage of digital pay- ments in Morocco, highlighting that better access to a diversity of financial services, including digital payments, would help vulnerable households man- age emergencies, build up assets, and invest in health and education (World Bank 2018c, 20). Similarly, the 2020 Senegal CPF, following up on the find- ings of the SCD, highlighted the transformative role that digital technologies could play in education, digitizing commercial and financial transactions in the agriculture sector, and improving the transparency and efficiency of delivery mechanisms in social protection (World Bank 2020c, 28).The Sen- egal SCD had noted that many vulnerable people had difficulty accessing basic services and acquiring the necessary skills in school, such as lifelong learning or technical and soft skills (World Bank 2018e, 101). By contrast, the Poland CPF did not mention the digital economy, even though the Po- land SCD had identified the digital economy as a key constraint (World Bank Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 2 2017b, 6, 77; World Bank 2018d, 29, 30). Although the CPF cannot be expected to address all the constraints identified in the SCD, it would be important to ensure that the CPF’s prioritization and selectivity are well justified. IFC, for its part, sees the creation of its Upstream units in FY20 as strengthening the corporation’s ability to interact with the World Bank on country diagnostic and strategy work as well as policy and regulatory interventions. IFC has also created the Country Private Sector Diagnostic, which is a tool to assess opportunities for and constraints to private sector led growth. To date, there has been insufficient discussion in SCDs of the link between digital technologies and the broader development agenda, including the twin goals (for example, when identifying key constraints). The World Development Report 2019 pointed out that digital technologies have been spreading, but their divi- dends have not (World Bank 2019m). This evaluation found that the discussion of poverty in the two SCDs mentioned in the Mainstreaming paper (World Bank 2019h)—the Senegal SCD (World Bank 2018e, 23–38) and the Poland SCD (World Bank 2017b)—did not consider the opportunities and risks posed by DTT. A review of the Bank Group’s newly introduced digital economy diagnostics found that none of the sample pilot countries made poverty reduction and in- clusion a focus for the digital economy strategy (Hanna 2019).4 The Bank Group has recently introduced new digital economy diagnostics, which can help assess client capacity for digital solutions. These include the Digital Economy Coun- try Assessment, DE4A Country Diagnostic, and Digital Government Readiness 24 Assessment. The Bank Group’s review of these digital economy diagnostics found that they often failed to explain the persistence of bottlenecks in achiev- ing inclusion. It also highlighted some of the key questions to address in this regard; for example, What explains nonadoption when there is internet cover- age? Is low usage the result of high cost, lack of local content, lack of hardware, or missing platforms? When there is usage, how effectively does it contribute to poverty reduction and shared prosperity? Such a focus was clearly absent for the Russian Federation. The Russian Federation Digital Economy Country Assessment pilot did not address the effectiveness of internet usage in contrib- uting to poverty reduction and shared prosperity, even when assessing foun- dations or use cases like digital government, small and medium enterprises, finance (micro, payments), and innovation and entrepreneurship (Hanna 2019). This was also true for Malaysia and for low-income countries like Senegal (Hanna 2019). Although the DE4A diagnostic focused on the foundations of the digital economy, sectoral issues (in addition to the digital economy foundations) are important for the twin goals. Recently, the Bank Group has undertaken more di- agnostic work pertaining to the digital economy.5 It has also updated its meth- odology for digital economy diagnostics with version 2.6 The reasons for the inadequate treatment of inclusion and poverty in digital economy diagnostics are twofold (Hanna 2019). First, relevant Global Practices have limited participation in the preparation of digital diagnostics. For exam- ple, the lack of participation by the Social, Urban, Rural, and Resilience Global Practice may have resulted in failing to account for major spatial inequali- World Bank Group Independent Evaluation Group    25 ties (important in a large, diverse country like the Russian Federation, where national averages for digital economy indicators miss local variation). Other Global Practices that also tend to be excluded are the Poverty; Education; Health, Nutrition, and Population; and Governance Global Practices. Partici- pation by the regional chief economist’s office would also be important for the inclusion agenda, which may be neglected if left solely to ICT specialists. Sec- ond, disaggregated data are lacking. The International Telecommunications Union, for example, generates data on the number of households with internet access but not necessarily on internet usage or how this varies across income groups or across dynamic and lagging Regions.7 Given its expertise in develop- ment data, this is a gap the Bank Group could help fill. Finally, there has been insufficient attention to linking the support for dig- ital technologies to the twin goals in two Bank Group regional initiatives: the Bank Group’s MNA Tech program and the DE4A Moonshot/Accelerate. The MNA Tech Concept Note did not mention how it would help contribute to the twin goals or discuss links between digital connectivity and poverty (poverty is mentioned twice in the 61-page Concept Note and is left out of the theory of change) (World Bank 2019b).8 The Concept Note for the DE4A Moonshot/Accelerate did not discuss some of the big risks from DTT, such as income inequality resulting from automation (World Bank 2019a). Skills and Capabilities for the New Economy and the Role of Education The fast pace of technological change likely means that individuals will have Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 2 more than one career in their lifetime, which in turn means that they will need to continually learn new skills. Individuals will need to adopt a growth mind- set, or the ability to “learn to learn” and to adapt to new jobs (box 2.2). The Bank Group’s FY20–22 Human Resources Strategy has recognized the need for staff to embrace a growth mindset for continuous learning. The garment industry illustrates how the nature of jobs will change. Sewbots are likely to replace at least some garment factory jobs, and new jobs of the future will in- volve using and producing sewbots. Those who cannot transition from manual sewing will likely lose out. According to the World Economic Forum (2018), ac- cess to economic opportunities will require individuals to remain competitive so that businesses have the talent they need for the jobs of the future. Imparting skills that allow children to acquire a growth mindset or the ability to “learn to learn” and adapt will be key, given the rapidly changing job market. The Inter-American Development Bank has pointed out that students who begin primary school today will graduate from university in the mid-2030s, and their careers will last through 2060 or beyond (IDB 2018).9 The Bank Group has recog- nized the necessity of students continually acquiring new skills. A recent Tech- nology for Youth Empowerment and Value Exchange (Evoke) project in Colombia takes a “sandbox” approach and uses blockchain and crypto-tokens to improve the process by which students are incentivized to develop the skills necessary to create innovative solutions to local challenges. Three recent World Develop- ment Reports dealing with digital dividends, education, and the changing nature of work identify the need to provide future workers with cognitive, technical, 26 and socioemotional skills (World Bank 2016e, 2018h, 2019m). Other World Bank publications have also contributed to the literature on the importance of socio- emotional skills (see, among others, Cunningham, Acosta, and Muller 2016; Cun- ningham and Villaseñor 2016; Guerra, Modecki, and Cunningham 2014; Puerta and others 2016). In its Step by Step Social and Emotional Learning Program for children and teens, the World Bank outlines six core life skills for children ages 6 through 17 that foster social responsibility, autonomy, and resilience.  he Unpredictability of the Future Job Market and the Box 2.2. T Importance of a Growth Mindset According to historian Yuval Noah Harari: “The crucial problem isn’t creating new jobs. The crucial problem is creating new jobs that humans perform better than algorithms. Since we do not know what the job market will look like in 2030 or 2040, already today we have no idea what to teach our kids. Most of what they currently learn at school will probably be irrelevant by the time they are 40. Traditionally, life has been divided into two main parts: a period of learning followed by a period of working. Very soon this traditional model will become obsolete, and the only way for humans to stay in the game will be to keep learning throughout their lives, and to reinvent themselves repeatedly” (Harari 2016). Psychologist Carol Dweck has observed that “in a growth mindset, people believe that their most basic abilities can be developed through dedication and hard work—brains and talent are just the starting point. This view creates a love of learning and a resil- World Bank Group Independent Evaluation Group    27 ience that is essential for great accomplishment” (Dweck 2006). Dweck argues that when people believe that their intellectual abilities can be devel- oped, they have a desire to learn and, therefore, a tendency to embrace challenges, persist in the face of setbacks, see effort as a path to mastery, learn from criticism, and find lessons and inspiration in the success of others—all of which help people reach even higher levels of achievement. People who operate with a fixed mindset are more likely to stick to activities that use skills they have already mastered rather than risk embarrassment by failing at some- thing new. People focused on growth make it their mission to learn new things, under- standing that they won’t succeed at all of them—or at first. Source: Dweck 2006; Dweck and Yeager 2019; Harari 2016; Yeager and others 2019. DTT have a critical role to play in education to equip workers and children for the jobs of the future. Key intersections between education and DTT include the following: » The “application” of DTT to improve education (for example, by using com- puters in the classroom). » The use of education as a “response” to DTT to prepare people for a world disrupted by DTT, imparting twenty-first century skills, by » Preparing people to develop or produce DTT (for example, by teaching ad- vanced cognitive skills, including higher-order problem solving and creative thinking); Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 2 » Preparing people to use or consume DTT (for example, by teaching digital literacy); and » Preparing people for new jobs in which humans have the comparative ad- vantage (for example, by teaching socioemotional skills, including empathy, curiosity, intrinsic motivation, grit, communication, teamwork and collab- oration, leadership, global awareness, self-regulation, conflict resolution, relationship management, and, in particular, a growth mindset or the ability to “learn to learn”—to be lifelong learners and adapt to change). The World Bank’s education lending and ASA over the past five years have focused more on the application of DTT and less on the response to DTT in preparing children for the twenty-first century (appendix A provides details on the identification of lending projects and ASA):10 » Regarding lending, of the 163 education projects approved during FY15–19, 57 projects (35 percent) had at least one DTT-relevant objective, component, or indicator. » Of these 57 education projects (35 percent) with at least one DTT-relevant objective, component, or indicator, 79 percent provided support for the appli- cation of DTT to improve education, 11 percent provided support for preparing people to develop or produce technology in the new economy, 4 percent pro- vided support for preparing people to use or consume technology in the new economy, and 21 percent provided support for preparing people for new jobs 28 in which humans have the comparative advantage (italics show the key inter- sections between education and DTT discussed in the previous paragraph).11 » Regarding ASA, of the 403 education-related ASA approved during FY15–19, only 15 percent had at least some DTT-relevant discussion in ASA abstracts or ASA documents. The scarcity of DTT-relevant analysis in ASA is of partic- ular note, as it is in such analysis that new areas or innovations reflecting fu- ture educational and labor market needs and the role of DTT can be explored. The Bank Group has an opportunity to spotlight twenty-first century skills and highlight both necessary and sufficient skills. The Bank Group’s flagship Human Capital Project (adopted by 77 countries in 2020 from just 28 early adopters in 2018) and Learning Poverty initiative (which includes an am- bitious new learning target that aims to cut by at least half the global rate of learning poverty by 2030) seek to address fundamental gaps in learning outcomes.12 The Human Capital Index focuses on reading and math and the Learning Poverty initiative on reading and understanding a simple text.13 The Bank Group has an opportunity to move the narrative forward from necessary skills such as literacy and numeracy to include twenty-first century sufficient skills such as advanced cognitive skills, digital literacy, and socioemotional skills, particularly a growth mindset or the ability to “learn to learn” and adapt.14 Both necessary and sufficient skills are critical for ensuring that a workforce is well prepared for the future labor market. There are strong rea- sons for imparting these skills simultaneously—not sequentially—from early World Bank Group Independent Evaluation Group    29 childhood itself, recognizing the profound positive effect on brain develop- ment and lifetime capabilities. Imparting sufficient skills may also serve the nascent demand from developing countries that are aspiring to grow and catch up by becoming knowledge-intensive economies and for reducing the North-South divide (UNESCO 2015). DTT are emerging as an important mechanism in the context of COVID-19, given their relevance for social distancing and online learning. The Bank Group estimated that because of the pandemic, close to 1.6 billion children and youth in 161 countries were out of school as of April 2020. To minimize disruption to learning while schools are closed, countries are using various online learning technologies (for example, computers, tablets, or mobile phones) and off-line distance learning technologies (for example, TV or radio) to reach children remotely. World Bank support in this regard com- prises (i) providing (on its EdTech website) guiding principles for short and long-term distance learning plans; (ii) maintaining a catalog of emerging ap- proaches that documents how education systems are responding around the world; and (iii) maintaining a list of resources and platforms that identifies helpful technological solutions to support remote learning, covering off-line distance learning and online learning. Box 2.3 discusses the Bank Group’s experience with off-line distance learning and online learning. IFC has sup- ported education technology clients through its venture capital investments, such as Byju’s in India, with the objective of providing solutions to adapt education and learning to COVID-19. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 2  everaging World Bank Group Experience with Off-Line Box 2.3. L Distance Learning and Online Learning in Responding to COVID-19 The World Bank Group’s experience with off-line distance learning and online learning provides pointers in the context of the coronavirus (COVID-19). A review of a sample of recent education projects for this evaluation highlighted Bank Group experience with online learning, which has been used so far to (i) improve teacher training; (ii) reach spa- tially diverse populations; (iii) improve assessments in the classroom; and (iv) support blended learning, combining traditional face-to-face teaching methods and resources (textbooks, for example) with digital resources (such as tablets and e-learning). Fol- low-up interviews with key staff working in education and technology provided insights about the Bank Group’s use of both off-line distance learning and online resources and how these may be leveraged in responding to COVID-19. The Bank Group’s experience with off-line distance learning and online learning points to the following challenges, which it will need to address as it ramps up its COVID-19 response and continues to provide ongoing education support: » Developing high-quality digital content in a format that follows pedagogical principles. Bank Group interviewees noted that although some open-source digital content is available, a key challenge is to prepare and make available pedagogical material that captures the attention of all students. They pointed to a shortage of digital curriculum expertise in the Bank Group. 30 (continued)  everaging World Bank Group Experience with Off-Line Box 2.3. L Distance Learning and Online Learning in Responding to COVID-19 (cont.) » Training teachers to effectively deliver digital content. Although online learning has been increasingly deployed for this purpose, this needs to keep in step with the capacity and connectivity of the receiver. » Ensuring a systems approach that develops all the necessary complements for digital education. This includes developing digital content, digital teaching, assessment data, infrastructure and device financing, and connectivity, since just distributing laptops does not improve learning outcomes. » Ensuring equitable access to infrastructure for connectivity through, for exam- ple, access to tablets, computers, wind-up radios, and TVs. This remains an issue in several countries and in several parts of each country. » Effectively delivering blended learning, which combines face-to-face teaching with digital instruction. This may be highly relevant in the short term, as children may be sometimes in and sometimes out of school, depending on progress in controlling the pandemic. » Building back better. The development literature suggests that crises often tend to galvanize support for change and make reforms easier. The COVID-19 pandemic presents the Bank Group with an opportunity to take bold action (for World Bank Group Independent Evaluation Group    31 example, in strengthening curriculums to address job markets of the future), so that the education system that emerges after COVID-19 will be more relevant to twenty-first century needs than it might otherwise have been. » Strengthening networks and partnerships. Great strides in online learning are occurring through governments, educational institutions, the private sector, and others. Collaboration and partnerships can keep the Bank Group’s work state of the art. Source: Independent Evaluation Group review. 1  Details about the Global Facility for Disaster Reduction and Recovery can be found at https://www.gfdrr.org/en/feature-story/using-technology-stay-ahead-natural-disaster-risk. 2  The overall discussion of skills and capabilities for the new economy and the role of edu- cation, especially in the context of the Human Capital Project, covers Bank Group support, whereas analysis of the education lending and advisory services and analytics (ASA) over the past five years covers the World Bank only. 3  For more information about the Bank Group’s education programs, see https://www.world- bank.org/en/topic/education. 4  Hanna (2019) assessed a purposive sample drawn from the Digital Economy Country As- sessment, Digital Economy Country Assessment 2.0, Digital Economy for Africa, and Digital Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 2 Government Readiness Assessment of pilots that were at a sufficiently advanced stage of implementation or drafting to generate important lessons, covered different Regions, includ- ed countries at different levels of development, or adopted different assessment tools and processes. The countries were the Kyrgyz Republic, Malaysia, the Russian Federation, Senegal, and Tunisia; there was also a multicountry assessment of Indonesia, Malaysia, Thailand, and Vietnam. Assessment reports and documentation for several other countries and regional initiatives that were reviewed in less depth included those for Kenya, Lesotho, and the MNA (Middle East and North Africa) Tech Initiative. 5  IEG has not evaluated the recent digital diagnostics. 6  Version 2 is available on the Digital Economy for Africa public webpage: https://www.world- bank.org/en/programs/all-africa-digital-transformation. 7  Household surveys and administrative data of the kind the Bank Group collects will likely be required to address such questions. 8  MNA Tech is part of the Middle East and North Africa Strategy, and the latter emphasizes the achievement of the twin goals. 9  “To prepare for the 2030s and beyond, emerging economies have an opportunity to leapfrog, to prepare their youth for the careers of the future. Our schools should teach the curriculum of the future, not just the curriculum of the past” (IDB 2018). 10  The International Finance Corporation is also engaged in education through venture capital education technology companies, but these were not evaluated here. 32 11  One project could fall into more than one category. 12  The Human Capital Project notes that children in many countries are struggling to learn in school (with nearly 60 percent of primary school children in developing countries failing to achieve minimum proficiency in learning) and that countries often underinvest in human capital (Gatti and others 2018, 2, 14). Furthermore, the learning poverty indicator (the per- centage of children who cannot read and understand at age 10) stood at 53 percent of children in low- and middle-income countries. See also Saavedra 2020. 13  The Human Capital Index is a tool designed to measure how well a country is doing in terms of fulfilling its human capital potential. It converts core indicators on survival, schooling, and health into measures of worker productivity (World Bank 2020b). 14  Recent initiatives of the Bank Group for promoting digital literacy include technology efforts in the Education Global Practice portfolio. There are increasing efforts to support the development of digital skills both in basic and higher education for teachers, principals, and learners. This encompasses support for the enabling environment, including digital infra- structure, data systems, and digital learning resources. Together with the technology and innovation team, the Education Global Practice has developed an external Community of Practice in this area, engaging partners such as the Massachusetts Institute of Technology, Stanford University, University of Wisconsin, Microsoft, Lego, the UK Department for Inter- national Development, and the US Agency for International Development. Furthermore, as part of the Partnership for Skills in Applied Sciences, Engineering, and Technology program, the World Bank and its partners are supporting the development of a Digital Skills Action Plan that will guide Country Action Plans for digital skills in Africa. Digital skills comprise both digital literacy skills, which are required by citizens and workers in many occupations, World Bank Group Independent Evaluation Group    33 and specialized digital skills for the information and communication technology professions. The Digital Skills Action Plan will support five strategies: (i) enabling policies and a digital skills framework; (ii) reforming digital skills programs; (iii) using technology in teaching and learning; (iv) connecting institutions to high-speed internet; and (v) building capacity and reengineering business processes. 3 | Staff Skills and Mindsets for DTT Support Highlights The World Bank Group’s fiscal year 2020–22 Human Resources Strategy presents key insights about the staffing issues that con- front the Bank Group. Reflecting the latest thinking in organization- al psychology, it recognizes the need to build new staff skills and expertise and foster a growth mindset for continuous learning in the face of rapid technological change. This evaluation found that the Bank Group has yet to (i) identify the staff skills needed to harness disruptive and transformative tech- nologies (DTT) opportunities and mitigate DTT risks; (ii) ensure that its information systems and databases provide information on its current DTT-relevant skills; and (iii) take action to fill any gaps. Expertise across institutions and units is not leveraged efficiently, especially in bringing together the DTT expertise of technology staff and the sector knowledge of specialists and task team leaders. The number of existing staff with DTT-relevant skills is insufficient to meet client demand, especially in areas such as regulatory re- form, data privacy, cybersecurity, 5G networks, and artificial intelli- gence; consequently, staff with these skills are often overstretched. A mindset for continuous learning and adaptation is also lacking in the Bank Group. 34   DTT-Relevant Staff Skills and Mindsets Providing DTT support requires specific skills and mindsets. DTT-relevant skills include those that enable Bank Group staff to (i) provide informed advice to clients on technology policy, regulation, and standards; (ii) guide clients about risks such as income inequality, data privacy, or cybersecurity; and (iii) help clients mine large data sets for policy insights. DTT-relevant mindsets include those that ensure that Bank Group staff (i) learn contin- uously and adapt to technological change; (ii) use innovative solutions to tackle development challenges; and (iii) adopt an entrepreneurial (prob- lem-solving) approach to address new development challenges. Bank Group Recognition of Key Issues The Bank Group has recognized the necessity of upgrading staff skills and the need for staff to embrace a growth mindset for continuous learning, giv- en the rapid changes caused by DTT. The Bank Group’s Mainstreaming paper acknowledges the importance of ad- dressing staff skills in providing DTT support (World Bank 2019h). The paper notes that the Bank Group is using the annual business planning process to plan for the impact of disruptive technologies on the emerging skills it will need. The Bank Group’s FY20–22 Human Resources Strategy notes that one of the World Bank Group Independent Evaluation Group    35 “most pressing challenges” is skills gaps that impede delivery of changing Bank Group priorities, including thought leadership on disruptive technol- ogies. The strategy also mentions the need for staff to embrace a growth mindset for continuous learning. Furthermore, the strategy recognizes that the Bank Group will need to build a pipeline of future leaders possessing skills such as emotional intelligence and resilience, collaboration, problem solving, critical thinking, innovation proactivity, and continuous learning. The Bank Group recognizes the need for staff who can innovate, given the interconnectedness of DTT and innovation. A 2017 World Bank report notes the importance of having staff with technical expertise and staff with the right mindset or attitude as a prerequisite for innovation, including technological innovation (World Bank 2017c). Drawing on IEG’s Supporting Transformational Change evaluation (World Bank 2016c), the 2017 report emphasizes the impor- tance of staff with an entrepreneurial attitude and a willingness to take risks. The report also flags hiring and retention challenges at the Bank Group that have bearing on the diverse skills needed for innovative work. The World Bank recognizes that operational involvement in incorporating technology in World Bank–supported projects still tends to be piecemeal. A forthcoming ASA, The Converging Technology Revolution and Human Capital: Potential and Implications for South Asia, notes that although operational involvement in incorporating technology is widespread across many World Bank–supported projects in the South Asia Region, it still tends to be piece- meal and “pilot-focused” (Bashir and others, forthcoming). The ASA explains Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 3 that technology and innovation are rapidly changing, while the skills mix of staff is largely unchanged, with limited basic knowledge and lack of dedicated funds and technical resources for task team leaders to experiment with in- novation. Furthermore, it notes that engagement with local country experts and with global experts is limited by lack of both time and basic knowledge of technology trends and issues. IEG Findings » Based on the IEG-DECDG joint analysis of staff skills for DTT support un- dertaken for this evaluation, the key findings are that the Bank Group has yet to (i) identify the DTT-relevant skills that it needs; (ii) ensure that its information systems and databases contain the necessary information on its current DTT-relevant skills; and (iii) take action to fill any gaps. IEG—in collaboration with DECDG—assessed the state of the Bank Group’s current staff skills in providing DTT support. The assessment asked this question: Within the Bank Group, how widespread are the technical skills and experience that are relevant for DTT support across the Regions? The objec- tive was to gain insight into the Bank Group’s current state of preparedness to help clients harness DTT opportunities and mitigate risks and to generate recommendations. IEG and DECDG developed a two-part approach to the as- sessment (see box 3.1). The approach was impractical, given the findings listed and described in this section. 36 EG and DECDG’s Two-Part Approach to the Staff Box 3.1. I Skills Assessment Part One: Defining terms. The Independent Evaluation Group and the Development Eco- nomics, Development Data Group investigated whether the Bank Group Human Resources Department had created a taxonomy of skills and competencies using consistent defini- tions. Such a taxonomy would facilitate an assessment using standard Bank Group–wide definitions and would help avoid definitional confusion. The existence of SkillFinder sug- gested that such a taxonomy might exist, which would be useful for assessing the current state of skills and competencies relevant for disruptive and transformative technologies (DTT) and for clarifying any recommendations that the assessment might generate. Part Two: Gathering and analyzing relevant data. The Independent Evaluation Group and the Development Economics, Development Data Group worked jointly to devise a list of proxy data (from SkillFinder, People Profile pages, human resources, and other institutional sources), which, although perhaps not a precise account of the current staff skills and experience relevant for providing DTT support, might at least yield ac- tionable insights. This list included the following: — The number (and proportion) of Bank Group staff who have skills and competencies relevant for providing DTT support, and the following information: » How these staff are spread across Regions, Global Practices, grade levels, and geographical locations (headquarters versus country offices) World Bank Group Independent Evaluation Group    37 » The number of years of service they have completed in the Bank Group » Their primary, secondary, and tertiary specializations » Their professional, cross-cutting solutions areas (now called Global Themes), and Global Solution Group affiliations » Their current mapping to career streams and job families » Which talent boards or career advisory panels oversee and advise on their career management » The sector and theme codes associated with the projects and advisory services they have led as a task team leader » Previous experience at other organizations relevant to DTT. — The Bank Group’s Communities of Practice and Yammer groups. These would con- ceivably offer an insight into how many staff were sharing their interest or experiences in working on DTT. Source: Independent Evaluation Group and Development Economics, Development Data Group research. SkillFinder did not provide reliable information for assessing DTT staff skills and gaps. The SkillFinder mechanism was self-reported and optional. At the time this evaluation was being conducted, only 21 percent of World Bank staff had self-reported their skills and competencies.1 SkillFinder contained about 6,200 individual skill entries, but there was no taxonomy of skills, and no definitions had been assigned to any skills. The skills list was simply an open text field into which staff could enter what they chose. There was no oversight or governance of SkillFinder. Given the small percentage of World Bank staff who had opted in and the fully subjective nature of the skills reported, it was determined that the data would not be useful for this evaluation. Although there are some improvements, many of the SkillFinder problems Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 3 persist in the recently updated People Profile pages. The updated People Pro- file pages software now encourages staff to fill in their profile by showing them how much of their profile is incomplete. All official areas of expertise have been merged into a single attribute, eliminating the need for a staff member to specify primary, secondary, or tertiary specializations. The lack of a skills taxonomy and definitions of skills, incomplete staff profiles, and absence of oversight or a governance mechanism, however, are persistent gaps in the up- dated People Profile pages. Coursera and LinkedIn provide working examples of skills taxonomies from which the Bank Group could glean relevant ideas.2 The usefulness of much of the other staff data (identified in box 3.1) is con- tingent on gathering good-quality data on Bank Group staff skills and com- petency. For example, information on geographic locations, specializations, affiliations, career streams, job families, projects and ASA, and previous occupations is moot if good-quality Bank Group staff skills and competency data are missing. Obtaining curriculums vitae (CVs) of Bank Group staff is not straightforward since no electronic database compiles all staff CVs using a standard structure and format, and CVs are also self-reported with no oversight. This means that there is the possibility that CVs may under- or overreport academic qualifications and work experience or report them in various ways. Gathering information on the Bank Group’s Communities of Practice (CoPs) is likewise not straightforward; the only readily available information is an 38 inventory of CoPs but not membership. Moreover, some CoPs are Microsoft email groups and others are Outlook groups, and all manage their member- ship through nonpublic email distribution lists, for which points of contact are not readily identifiable. Of the few CoP focal points that this initial analysis yielded, some were hesitant to share membership details. Further investigation is necessary to canvass CoP members and CoP focal points, demonstrate to them why these data are necessary for analysis, and obtain listserv information. An examination of World Bank Yammer groups also does not provide insights into the range of skills relevant to DTT. IEG and DECDG investigated wheth- er a relationship exists between staff membership in World Bank Yammer groups and thematic relevance to DTT. Analysis of World Bank Yammer groups found that these groups are informal and open to all World Bank staff, based on an enrollment ethos that promotes awareness of and en- gagement with a particular subject. As such, there are no threshold criteria for technical expertise or operational experience to create a group, nor is there a representative number of groups across the World Bank’s thematic areas relevant to DTT work. Although Yammer groups may coalesce around DTT-relevant themes,3 they are not a source of information on the DTT- relevant staff skills of their members. » In Bank Group staff interviews conducted for this evaluation, interviewees reported shortages of staff possessing DTT-relevant skills, pointing out that existing staff with these skills were often overstretched. Interviewees World Bank Group Independent Evaluation Group    39 also reported that mindsets for continuous learning and adaptation were lacking in the Bank Group. IEG’s analysis of the interview data using NVivo suggested that the Bank Group has insufficient DTT-relevant skills or mindsets.4 Of the 70 interviewees who talked about staff skills, 52 interviewees (or 74 percent) reported a lack of DTT-relevant skills. Of the 28 interviewees who talked about staffing mindset, 22 interviewees (or 79 percent) reported a lack of a DTT-relevant mindset. Interviewees generally indicated that too few Bank Group staff have DTT- relevant skills. Although there have been recent initiatives to build DTT-rele- vant skills (box 3.2), interviewees noted that there are not enough ICT special- ists, regulation experts, geospatial specialists, data managers, data scientists, statisticians, and technology lawyers. There being few available DTT-skilled experts means that often those experts are not able to respond to existing client demand in areas such as regulatory reform, data privacy, cybersecurity, 5G networks, intellectual property issues,5 and AI and often become over- stretched. The lack of DTT-skilled experts may also mean that Bank Group policy dialogue is weakened, especially when the client is better versed in the latest technologies than Bank Group staff. Bank Group interviewees also men- tioned a shortage of technical experts on ID4D. Demand from country clients for ID4D support has grown substantially, even as a billion people globally are still unable to prove their identity and millions more have forms of identity that cannot be reliably verified or authenticated. In a country visit undertaken for this evaluation, country clients noted that the Bank Group often provides generic advice, which may suggest a lack of expert knowledge on specific Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 3 technologies. Interviewees noted that staff with cutting-edge digital skills can be forced into more generalist roles, potentially making it difficult for them to keep up with relevant developments in the digital world. Furthermore, successful DTT support depends on staff acquiring and using relevant skills and adopting a growth mindset, but interviewees highlighted some caveats regarding this at the Bank Group. In particular, a mindset that favors standard or normal projects over those requiring new learning or techniques constrains skill acquisition and development. Bank Group interviewees in the Digital Development Global Practice and IFC corroborated a shortage of DTT-skilled staff. Recent IFC hiring has sought to address gaps in DTT-relevant expertise. Interviewees noted that staff dedi- cated to fintech and digital entrepreneurship generally have the technical capabilities and necessary skills and are often able to reach out to qualified consultants to supplement these skills. Nevertheless, World Bank interview- ees working on digital development pointed to insufficient expertise, espe- cially for regulation of technological companies and platform firms, and in data privacy, cybersecurity, and DTT-relevant procurement. Regarding IFC, interviewees noted that staff can respond to much of the demand, but that they could do more if the venture capital and fintech teams had more qual- ified staff. Furthermore, they noted that IFC has good industry specialists in debt financing who understand the sector and keep up with technologi- cal changes, as well as solid investment staff, but it needs more specialists in mezzanine and equity products, which require very different skills than 40 those needed for working on debt. More recently, following IFC’s reorganiza- tion and the creation of the Disruptive Technology and Funds and Upstream units in FY20, IFC has begun to increase staff in DTT-relevant areas, includ- ing specialists in AI and machine learning. However, it is too early to assess the effectiveness of these measures. Box 3.2.  Recent Initiatives to Build Skills Relevant to Disruptive and Transformative Technologies A number of recent initiatives in the World Bank are supporting learning for staff with roles related to disruptive and transformative technologies: » The World Bank’s Open Learning Campus aims to facilitate online university degrees for staff on artificial intelligence, blockchain, internet of things, cyberse- curity, 5G, and regulation. » Information and Technology Solutions has a Technology and Innovation Lab whose goal is to “serve as a catalyst, enabler and accelerator for Bank Group staff to learn about and build expertise around emerging technologies’ potential to support the Bank Group development agenda.” It also has an active staff learning program (Learn2Innovate) that partners with Global Practices for learning sessions on topics such as blockchain, artificial intelligence, and other technologies with potential applications to the World Bank’s work. It aims to apply design thinking principles to the projects it explores and prototypes it develops to ensure that the World Bank Group Independent Evaluation Group    41 solutions reflect end-user preferences. » Sustainable Development has a learning program on the use of technologies for remote preparation and supervision for projects, including remote supervision using virtual reality. Source: World Bank data. Interviewees noted insufficient focus on learning and adaptation. The lack of learning inhibits adaptability, which has been called the “new competitive advantage” (Reeves and Deimler 2011). An adaptability quotient has recently been proposed to measure the extent to which individuals and organizations will thrive in the twenty-first century.6 IEG’s second Learning and Results evaluation noted that a learning mindset is hindered at the World Bank by weak handover arrangements between incoming and outgoing task team leaders, poor on-the-job mentoring, insufficient focus on lesson learning by the peer review system, and inadequate rewards for learning and knowledge sharing (World Bank 2015a). Developing the requisite skills to support DTT for development could come from a combination of new recruitment (of staff, consultants, or both), re- training, secondments, outsourcing, and external partnerships. Bank Group interviewees discussed scenarios in which one or some combination of these might apply. New recruitment would be best in situations where DTT-relevant skills are continuously needed, as in the case of data scientists, for exam- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 3 ple. Education sector staff noted that the EdTech Fellows program, with 25 fellows—including some newly recruited by the Bank Group—could provide lessons, both positive and negative, on instilling openness to new technolo- gies and enhancing specialized skills for DTT for development. Interviewees identified retraining as an option for staff who already had the technical foundation on which to build. Client interviewees noted that Bank Group ad- vice did not always promote the latest relevant technologies,7 and staff inter- viewees perceived insufficient investment in training for DTT-relevant skills and poorly targeted training.8 Secondments offer a viable option for growing lines of business, where the Bank Group could consider hiring seconded staff after having tested them during the secondment period. Outsourcing specific DTT functions may be considered for highly specialized or temporary needs. Finally, external partnerships with other development organizations, the private sector, academia, or nongovernmental organizations could also help access missing skills and spur innovation.9 External partnerships could also help the Bank Group increase the scale of projects, for example in education by partnering with companies whose online teacher training platforms are accessed by millions of teachers. A few recent partnerships provide some lessons (appendix G discusses the Famine Action Mechanism and the Bank Group–LinkedIn partnership). Interviewees noted that for the Bank Group to be at the forefront of DTT for development, Bank Group staff would benefit from opportunities to engage with top-level technology experts. Lokshin and Sajaia (2019) note that to be 42 a trusted adviser, the Bank Group should “support a strong team of top-level development experts integrated into the international tech community.” Furthermore, Bank Group interviewees reported that access to development practitioners in leading technology companies that are taking on develop- ment roles would also encourage the flow of knowledge and expertise across technology and development domains, sparking fresh solutions to complex development challenges. A DTT knowledge platform that combines such expertise could contribute to the Bank Group’s DTT thought leadership and enable it to use cutting-edge DTT knowledge in the pursuit of the twin goals. Bank Group staff and managers pointed to the need to better leverage the DTT expertise of technology staff and the sector knowledge of specialists and task team leaders. A point of convergence in staff and managerial inter- views was that the Bank Group needs staff with technological expertise who are familiar with what the institution does and who could be the technologi- cal connectors with sector specialists, operational task team leaders, clients, and vendors. Furthermore, the Bank Group needs task team leaders who are open to adopting technological solutions, which can be achieved through targeted training sessions. IFC interviewees perceived that DTT expertise in the TMT group initially suffered from the realignment of TMT teams that now report to regional infrastructure units (rather than to TMT at headquarters) and from rotating non-DTT staff into DTT positions, which diluted IFC’s ability to maintain thought leadership in DTT. The workforce planning and restructuring in World Bank Group Independent Evaluation Group    43 recent years has resulted in losing some middle management and knowl- edgeable staff. These individuals had been the ones informing senior man- agement and instilling knowledge in younger staff. Recent initiatives, such as training bootcamps, knowledge sharing, and systematic sector mapping have aimed to improve IFC’s thought leadership and expertise. In addition, some IFC teams, such as those working in fintech and venture capital, have maintained and in some cases added to their expertise. » Previous research undertaken by the Bank Group had identified a short- age of staff with data science and statistical skills—skills required to gen- erate and analyze granular development data for policy insights. There is a shortage of staff with data science and statistical skills in the Bank Group. The Development Economics Vice Presidency and Human Resources undertook a snapshot analysis in 2018 using staff job titles, which determined that 84 staff were working on statistics and data issues. Of these, 43 worked in Development Economics, 13 were in Global Practices, and the remainder were in institutional, governance, and administration units (such as ITS and Trea- sury). IFC also identified a dearth of data scientists and a lack of the mindset and business culture needed to employ these data scientists effectively. At the 2019 Data Day, then Bank Group CEO Kristalina Georgieva noted that there were fewer than 100 operational data scientists and statisticians and that she wanted this number to grow and for these staff to also increase their skills. The newly launched Data Governance body and its supporting Data Gover- nance Steering Committee—set up to provide executive leadership, oversight, and support for data-related matters—is a positive development. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 3 Bank Group data scientists and statisticians have limited institutional visibility, limited opportunities for mobility and advancement, and limited opportunities for career development. The Bank Group is missing an oppor- tunity to benefit from a functioning market of data science and statistical knowledge that could be mobilized to enhance rapid and effective services to internal and external clients. Cutting-edge data science and statistical capa- bility is essential in the Bank Group because such expertise can help in both generating more granular development data and analyzing them in ways that yield more granular policy insights for the twin goals. Collaborative work by the Group Internal Audit (formerly the Internal Audit Department of the Bank Group) and DECDG pointed to the lack of a career stream for data scientists and statisticians and recommended that this be addressed (World Bank 2016d). Bank Group management committed to this as part of the Man- agement Action Record for the Group Internal Audit report. A subsequent 2018 Development Economics study found recruitment of data scientists and statisticians had been ad hoc, with no consistent process to ensure standards of selection, promotion, or skill enhancement to meet institutional needs. This also meant that there were limited opportunities for current data sci- ence and statistics staff (de facto) to interact, share knowledge, or move into appropriate positions. A 2019 analysis by the Group Internal Audit pointed to analogous findings (World Bank 2019j). In a positive development, dis- cussions continue concerning possible next steps for the creation of a Bank Group career stream for data scientists and statisticians. 44 1  SkillFinder covered only World Bank staff, but the People Profile page has been released Bank Group–wide. 2  Coursera is focusing on skill development in three subject areas, given that these domains cover skills that will increasingly become crucial to the future of work. Two of these subject areas are technology and data science (the third is business). Coursera has developed a taxon- omy of over 40,000 skills in these three areas and has developed competencies and skill levels. See Coursera’s Global Skills Index: https://pages.coursera-for-business.org/rs/748-MIV-116/ images/global-skills-index.pdf. 3  At the time this evaluation was being conducted, the following Yammer groups were exam- ined: Digital Transformation Collection—World Bank Group; Digital Transformation of En- ergy; Digital Financial Services in Africa; Digital Agriculture and Food; Digital Development Community of Practice; Digital Governance GOV Global Practice Community of Practice; Information and Technology Solutions Technology and Innovation Lab—Tech Community; MNA Tech Community of Practice; and GovTech Leadership Group—World Bank Group. 4  The phrase “lacks sufficient DTT-relevant skills or mindsets” refers to either an insufficient number of staff with DTT-relevant skills or insufficient DTT-relevant skills or mindsets in staff. 5  For example, equitable and appropriate technology partnerships with the private sector may require new business and legal skills, as well as leveraging the World Intellectual Property Organization relationship, where relevant. 6  See Natalie Fratto’s TED Talk, May 2019: https://www.ted.com/talks/natalie_fratto_3_ways_ World Bank Group Independent Evaluation Group    45 to_measure_your_adaptability_and_how_to_improve_it. 7  Client interviews noted that Bank Group advice on climate change, agriculture, and educa- tion did not necessarily promote the latest relevant technologies such as the digital tracking of water leakages, artificial intelligence for crop monitoring, or smart classrooms. Interviewees cautioned that trainers need to be carefully selected and noted that the Bank Group 8  could reshape its retraining programs to incorporate lessons learned in Silicon Valley. They also noted that it would not be efficient to use the most highly skilled data scientists to provide basic DTT training to generalist task team leaders. They suggested that basic training in DTT could be mandatory for most staff and that more specialized, higher-level training (for example, in artificial intelligence, 5G networks, or blockchain) could also be provided for a smaller group of relevant staff. 9  Engaging in the budding movement of public interest technology on college campuses can be helpful to capture cutting-edge knowledge and harvest fresh ideas. 4 | Internal Processes and Procedures for DTT Support Highlights Collaboration The need for collaboration is underlined by the wide-ranging nature of disruptive and transformative technologies (DTT) projects that cut across sectors and often involve both public and private institutions. This evaluation found that there are examples of effective collabo- ration for DTT support, for example Development Data Partnership (formerly Data Collaboratives), the Digital Central Asia–South Asia program, Geo-Enabling for Monitoring and Supervision, and Identi- fication for Development. However, overall, there was a perception among interviewees from across Global Practices that collaboration is insufficient among the different Global Practices, with instances of competition for task team leadership. Given that contributions from both public and private sectors are necessary for successful DTT-related outcomes, the lack of col- laboration between the World Bank and the International Finance Corporation was also an issue. Procurement 46   Procurement is a major implementation constraint in DTT-related projects. There is only fragmented guidance for complex technology proj- ects, specifically on the risks associated with these projects (such as vendor lock-in, lack of interoperability of systems, potential loss of data ownership and lack of data privacy, limitations in technology- specific legislation and regulation, and mismatch between busi- ness processes and the requirements of the particular technology). There is also insufficient guidance on how the flexibility available for complex technology projects in the World Bank’s procurement systems can be applied in different situations. 47  Collaboration DTT-Relevant Collaboration The need for greater collaboration in providing support in a cross-cutting area like DTT is underlined by the importance of the analog complements to digital technologies. These analog complements include improving the busi- ness climate, investing in people’s education and health, and promoting good governance (World Bank 2016e). Consequently, effectively harnessing DTT in development requires cross-sectoral links and collective (rather than individual) action. When an organization fails to foster sufficient collaboration among relevant departments in its DTT support, the division of labor may become in- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 4 efficient and inhibit innovative solutions (Epstein 2019). The need for collab- oration is also underlined by the role of both public and private institutions in implementing complex technology solutions (World Bank 2011a). Bank Group DTT-related projects are wide ranging and illustrate why cross-sectoral collaboration is necessary. For example, a 2019 Tunisia proj- ect, GovTech Digital Transformation for User-Centric Public Services, aims to improve equitable access to and the quality and accountability of social protection and education services through a GovTech approach, targeting low-income groups, the vulnerable, women, the illiterate, and persons with disabilities. This wide-ranging objective would be addressed best by bringing to bear expertise in each of the identified areas (for example, collaboration across governance, education, poverty, social protection, digital develop- ment, competitiveness, and innovation).1 The importance of collaboration in providing DTT support—not just coordination or cooperation—is also em- phasized in a recent publication: “Rather than the business specifying what was needed and telling the technology department (or vice versa), the two groups needed to think together about the issues, risks, and opportunities of the digital age” (Karacaoglu, Mocan, and Halsema 2018, 26). Bank Group Recognition of Key Issues » The Bank Group has emphasized the importance of collaborative work to solve development challenges in the face of emerging disruptions. 48 The Bank Group FY20–22 Human Resources Strategy recognizes the im- portance of catalyzing a culture of greater collaboration. It calls for the Bank Group to develop leaders who can mobilize resources across the Bank Group to solve increasingly complex development challenges. Furthermore, it emphasizes that the emerging disruptions require emerging leaders to be integrators and coalition builders and notes that “while technical expertise is critical, new leaders must go beyond individual excellence to contribute collaboratively to a shared purpose to help improve staff engagement and catalyze the WBG [World Bank Group] culture of greater collaboration and innovation” (World Bank 2019l, 12). The Bank Group’s Mainstreaming paper recognizes the need to improve inter- nal capabilities, including collaboration (World Bank 2019h). It notes that the Bank Group will ensure a coordinated, Bank Group–wide approach for DTT support on several fronts and support knowledge sharing and collaboration. The Program Document for the World Bank’s 2020 MDDT Initiative refers to insufficient collaboration and coordination in mainstreaming digital and dis- ruptive technology solutions across the Bank Group. High demand for great- er collaboration and coordination was expressed in a series of consultations across the Bank Group. These noted that although there are tech initiatives managed by various units at the Bank Group, there is lack of cross-sectoral knowledge sharing and peer-to-peer learning (World Bank 2020d). World Bank Group Independent Evaluation Group    49 The World Bank is working to understand the state and nature of cross-sectoral collaboration. A forthcoming ASA, The Converging Tech- nology Revolution and Human Capital: Potential and Implications for South Asia, reports that looking across the Human Development portfolio in all Regions, Human Development in South Asia stands out as the most “in- ternally collaborative” (with joint projects between Education, Health, and Social Protection) (Bashir and others, forthcoming). However, it is also the least “externally collaborative” (with non–Human Development Global Practices), as measured by share of portfolio size either led by one Human Development Global Practice and contributed by others, or led by others and contributed by Human Development Global Practices. The level of collab- oration with non–Human Development Global Practices in South Asia has been less than in most other Regions. Furthermore, the ASA points out that following the COVID-19 onset, pipeline projects under preparation indicate that South Asia Human Development has started to work with Global Prac- tices focused on urban development, social development, and private sector development. But, paradoxically, there is very little collaboration with those Global Practices that could complement investments in technology that pro- mote human capital (for example, Digital Development; Governance; Macro- economics, Trade, and Investment; Energy; Water; and Agriculture). IEG Findings » There have been efforts to improve collaboration within the World Bank, and there are some examples of effective collaboration in providing DTT support. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 4 The Bank Group has attempted to improve collaboration for DTT support across Global Practices. For example, in July 2019, the Infrastructure Vice Presidency and the Equitable Growth, Finance, and Institutions Vice Pres- idency issued a joint memorandum agreeing on a mechanism whereby the Digital Development Global Practice would have coordination responsibility for all digital engagements with the country teams. However, the memoran- dum did not address leadership and coordination of DTT work, corporate responsibility for which currently lies with a small team housed in the front office of the Infrastructure Vice Presidency (see box 4.1). In July 2019, a new Disruptive Technology Task Force was set up under this team. The Disruptive Technology Task Force comprises one representative from each operational vice presidency and aims to follow up on the Mainstreaming paper by sup- porting the tracking of corporate priorities, new disruptive technology ini- tiatives, knowledge sharing, and capacity building for staff and clients. New concerns that the internal realignment—which took effect on July 1, 2020, and involves the appointment of regional directors and maps technical staff to Regions—might hinder improved collaboration by recreating old prob- lems of low mobility across Regions and lead to insufficient DTT specialized staff in each Region. The realignment’s effects on collaboration—in terms of quantity and quality—among the various parts of the World Bank will need to be monitored at three levels: (i) among Global Practices within a Practice Group; (ii) among Global Practices across Practice Groups; and (iii) among global directors, regional directors, and country directors. 50 To further foster DTT work, the 2020 MDDT Initiative proposes improved in- stitutional arrangements. A Network and a Technical Working Group aimed at connecting teams and programs across the Bank Group that are relevant for DTT are to be established. In addition, the Bank Group proposes a Sec- retariat, housed in the Digital Development Global Practice, to support the Network and the Technical Working Group and implement an initial set of MDDT activities. As the initiative is rolled out and the detailed roles, respon- sibilities, and activities of the Secretariat in relation to other parts of the Bank Group engaged in DTT work are finalized, the following considerations can help avoid pitfalls and enhance the effectiveness of the Bank Group’s DTT work: » Ensure that decision-making on DTT-relevant issues for sectors (such as agri- culture, education, health, transport, or energy) and/or themes (such as data/ development economics, or governance) continues to be made in the unit in which the expertise for that sector or theme lies. » Closely monitor the extent to which the proposed institutional arrange- ment—Network, Technical Working Group, and Secretariat—is addressing the collaboration problem it was meant to solve. » Ensure that DTT-relevant issues that cut across sectors, themes, or projects are addressed at the institutional level, rather than on an individual project-by-project basis. World Bank Group Independent Evaluation Group    51 » Ensure that the involvement of technology companies in the new institutional arrangement is appropriately ring-fenced to avoid compromising the Bank Group’s honest broker role and to ensure there is no conflict of interest with regard to, for example, procurement reform. nstitutional Arrangements for Digital Technologies and DTT in Box 4.1. I the World Bank Group In the World Bank In 2020, as part of the Mainstreaming Digital and Disruptive Technologies (MDDT) Initiative, the World Bank Group proposed a Network, a Technical Working Group, and a Secretariat housed within the Digital Development Global Practice, to implement an initial set of MDDT activities. Prior to the MDDT Initiative, the World Bank’s Digital Development Global Practice, housed in the Infrastructure Vice Presidency, had corporate responsibility for “digital engagements” within the World Bank. This Global Practice both provided task team Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 4 leadership for projects and performed knowledge and coordination functions. Corporate responsibility for disruptive and transformative technologies (DTT) rested with a small team located in the front office of the Infrastructure Vice Presidency. This group did not provide task team leadership for projects. It played knowledge and coordination roles, mostly through a Disruptive Technology Task Force, which was set up in July 2019 and comprised a representative from each of the operational vice presidencies. Digital engagements and DTT also form part of the work of the Finance, Competitive- ness, and Innovation and Governance Global Practices (and to a lesser degree, the Macroeconomics, Trade, and Investment Global Practice) under the Equitable Growth, Finance, and Institutions Vice Presidency and that of each of the sectoral Global Practices under the Social Development and Human Development Vice Presidencies. Each of these Global Practices provides task team leadership for projects in addition to playing knowledge and coordination roles in their respective sectors. The Technology and Innovation Lab in Information and Technology Solutions provides support to operational teams engaging in technology-related piloting and innovation. Development Data Partnership (formerly Data Collaboratives), Disruptive Technologies for Development Trust Fund, Geo-Enabling for Monitoring and Supervision, Geospa- tial Operations Support Team, and Social Development Data Lab are among the other groups in the Bank Group that also deal with digital engagements and DTT and that sup- port operational teams by playing a knowledge function. Staff managing these groups do not provide task team leadership for projects. (continued) 52 nstitutional Arrangements for Digital Technologies and DTT in Box 4.1. I the World Bank Group (cont.) In the International Finance Corporation The International Finance Corporation (IFC) seeks to support DTT through several core units described below. In addition, IFC’s mainstream industry departments and advisory functions support DTT interventions. IFC restructured the DTT practices in fiscal year 2019. Prior to the reorganization, the Telecoms, Media, and Technology teams, fintech teams, and Private Equity and Ven- ture Capital Funds teams were housed in a single department. Since the reorganiza- tion, the Telecoms, Media, and Technology group is part of the Infrastructure and Nat- ural Resources Department; the fintech team has migrated to the Financial Institutions Group; and the Private Equity and Venture Capital Funds team has moved to a new Disruptive Technology and Funds group headed by a senior director. This new group aims to better integrate disruptive technologies across IFC through joint ventures and knowledge sharing with other departments. In addition, the implementation of IFC’s Strategy 3.0 involved the creation of new IFC Upstream units in fiscal year 2020. Upstream units were created under each of the industry groups, including for infrastructure and disruptive technology, to support mar- ket creation initiatives and pipeline development for IFC. The Upstream units are also expected to be points of interaction with the World Bank. World Bank Group Independent Evaluation Group    53 Source: World Bank data and Independent Evaluation Group interviews. Examples of effective cross–Global Practice collaboration for DTT for devel- opment exist. Interviewees pointed to ID4D, GEMS, the Digital CASA pro- gram, and the Development Data Partnership (formerly Data Collaboratives). Box 4.2 discusses the collaborative aspects of ID4D (box 2.1 provided ID4D lessons) and the Digital CASA program, and appendix D discusses GEMS and the Development Data Partnership. In a recent first, the Vietnam country team collaborated with the Bank Group’s corporate disruptive technology team (in the front office of the Infrastructure Vice Presidency) to review ongoing and pipeline projects for DTT opportunities. Ongoing and pipeline projects in the Vietnam country portfolio were examined to check whether technology was already being de- ployed and whether there were new opportunities to do so. Among ongoing projects, the review identified 35 projects where DTT-relevant opportunities could potentially be harnessed. These 35 projects were in the following sec- tors: education; energy and extractives; environment and natural resources; health, nutrition, and population; social protection and jobs; transport; and water. The next step would be to seize missed opportunities, and the review has identified specific operations for project-level deep dives. Furthermore, the corporate disruptive technology team has developed a general frame- work for examining the opportunities and risks of technology adoption in infrastructure operations—a framework that could be examined for possible Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 4 application to other sectors as well.  ross-Sectoral Collaboration in the ID4D and Box 4.2. C Digital CASA Programs The Identification for Development (ID4D) program, launched in 2014, reports to a senior directors’ group, which includes 10 members who meet regularly to ensure a cohesive and coordinated approach across different sector entry points (for example, financial inclusion, social protection, and gender). The ID4D initiative, led by a program manager and a core team of six full-time staff members, works closely with an ex- tended team of World Bank Group colleagues who are part of a formal cross-sectoral working group, consisting of lead specialists from each of the Global Practices and de- partments involved (Digital Development; Jobs and Social Protection; Health, Nutrition, and Population; Finance, Competitiveness, and Innovation; Governance; Gender; and Legal), and the Disruptive Technologies and Funds industry group of the International Finance Corporation (IFC). These staff provide a significant focus on ID4D and ensure links to the operational work of the Bank Group. ID4D staff provide critical advice and knowledge to operational teams but are not task team leaders of operations. The ID4D work program was developed with input from sector experts, including the leads from the five Global Practices, the Gender Global Theme group, the Consultative Group to Assist the Poor, and IFC. The initiative includes three pillars: World Bank operations/ country engagement, thought leadership and analytics, and global platforms and con- vening. A ID4D high-level advisory council chaired by a World Bank managing director 54 (continued)  ross-Sectoral Collaboration in the ID4D and Box 4.2. C Digital CASA Programs (cont.) and 11 global thought leaders from the private sector and government meets twice a year with the aim of supporting global advocacy and providing a sounding board on emerging issues. In addition, partnerships with donors, the United Nations, and private sector and development partners help leverage one another’s strengths and achieve alignment on good practices in ID4D. The Digital Central Asia–South Asia (CASA) program shows the importance of Country Management Units in eliciting multisectoral collaboration within the sectoral governance structure of the World Bank. Within this context, the Country Management Units need to play an essential role to create adequate room and resources for the appropriate integration of digital technologies in the country program. Such Country Management Unit leadership was a key enabler in facilitating the sample of projects in Kosovo and Nicaragua reviewed for this evaluation. Conversely, in another case, the collaborative nature of the project was not sufficiently appreciated and the budget was assigned to one Global Practice, even as key inputs were expected from lead specialists in three different Global Practices; the budget was depleted quickly with no genuine collaboration. Instead of relying on the usual work program agreement process, which tends to discourage collaboration as teams compete for budgets to cover their own fixed costs, in the Afghanistan Digital CASA project, the practice managers worked together to World Bank Group Independent Evaluation Group    55 prepare a team-based budget, distributing the budget according to tasks and costs. Then they compared this distribution against the work program agreement budget to make sure the teams were covered. This allowed the teams to finance a geospatial data specialist to work with several Global Practices and for the Finance, Competitive- ness, and Innovation and the Social Protection teams to work together. The Digital CASA projects in Afghanistan and the Kyrgyz Republic combined public and private sector engagement. The Digital Development Global Practice served as the repository of technological knowledge. The Finance, Competitiveness, and Inno- vation and the Governance Global Practices provided their experience in the applica- tions of technology, and IFC’s Telecoms, Media, and Technology group contributed its experience with information technology and telecoms in the private sector. Source: Document reviews and Independent Evaluation Group interviews. » Bank Group databases lack sufficient data on collaboration relating to DTT. Adequate data on internal Bank Group collaboration relating to DTT do not exist, despite the cross-cutting nature of DTT. Existing Bank Group data on cross-support among Global Practices from Bank Group databases, or on col- laboration among Global Practices from the Bank Group’s engagement sur- veys, fall short for the following reasons: (i) there is no official Bank Group list that identifies DTT-related projects and ASA (except for Digital Devel- opment or ICT projects and ASA); (ii) disaggregated data on cross-support by Global Practices from Bank Group databases do not provide information on the extent of collaboration specifically in DTT-related projects and ASA; (iii) although the Bank Group’s engagement surveys do capture aspects of Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 4 collaboration, disaggregations specifically for DTT-related projects and ASA are unavailable; and (iv) in any case, none of the above data say anything about the nature and quality of the collaboration in enhancing the outcomes of DTT-related projects and ASA. » In interviews, Bank Group staff reported that collaboration across Global Practices has been insufficient in scale and scope. IEG interviews conducted for this evaluation pointed to insufficient DTT-relevant collaboration among the different Global Practices. NVivo analysis of IEG’s inter- views of Bank Group staff that are part of the reference population knowledge- able about collaboration and coordination (described in appendix A) found that of the 46 interviewees, 37 (or 80 percent) reported insufficient collaboration in providing DTT support. Of the 29 interviewees who talked about internal structures and collaboration, 23 interviewees (or 79 percent) reported that these had been inadequate to support collaboration. There was a perception among interviewees from across Global Practices that collaboration is insuffi- cient among the different Global Practices, with instances of competition for task team leaderships. Interviewees further observed that ICT may be embed- ded in many Bank Group projects in a country, but each project team often has its own IT expert; each expert could give different advice regarding data collection and management. The result is that the systems “do not talk to each other.” Interviewees noted that many issues related to budget allocation and recognition arose when two or more Global Practices or vice presidencies were involved in a multisectoral project.2 Moreover, even in digital economy diag- 56 nostics, which were intended to encourage collaboration across Global Practic- es, collaboration problems persist (Hanna 2019).3 Previous IEG evaluations have flagged general collaboration challenges within the Bank Group. IEG’s Knowledge Flow and Collaboration evaluation pointed to a lack of collaboration for integrated solutions at the Bank Group (World Bank 2019g). Furthermore, IEG’s evaluation Support for Innovation and Entrepreneurship found that the flow of knowledge on innovation and entrepreneurship is limited, especially across networks, sectors, Regions, and the three Bank Group institutions (World Bank 2013). The World Bank and IFC could play separate but complementary and highly coordinated roles in DTT projects. IFC staff highlighted the importance of the World Bank in persuading governments to implement regulatory changes, while the World Bank needs IFC’s investments to incentivize reform on the part of governments. This collaboration works either when the incentives of the two institutions are aligned or in an ad hoc manner largely based on personal relationships among IFC and World Bank staff. The lack of more systematic collaboration is partly due to the discrepancy in the timescales of the two institutions: the regulatory reforms implemented by the World Bank often span a much longer time than IFC’s investment projects. In such cases, the two institutions can have separate but complementary roles rather than committing to joint operations. Even with separate roles, ensuring comple- mentarity requires collaboration. This evaluation found that collaboration World Bank Group Independent Evaluation Group    57 works when IFC and the World Bank engage upstream even before any lend- ing or investment projects are in the pipeline. Such preengagement was a major factor contributing to the success of the Digital CASA program. Con- versely, in DE4A Moonshot/Accelerate, IFC was brought in late in the process, which initially created hurdles, although the teams were eventually able to overcome these. A One World Bank Group approach would also help with funding and data issues relating to areas where IFC is not currently involved (for example, Digital Economy Sandbox funding). Internally in IFC, collabo- ration remains a work in progress since IFC’s realignment of the Global and Regional Mainstream units and the addition of the Upstream units. Although the DE4A country diagnostics are an example of different Global Practices coming together for a more coherent approach to DTT, they illus- trate some shortcomings in collaboration. DE4A country diagnostics mainly involve the Digital Development, the Finance, Competitiveness, and Inno- vation, and the Governance Global Practices. At the review meeting for the initiative, the need for engaging with additional Global Practices such as Macroeconomics, Trade, and Investment and Poverty was noted, but no sub- sequent action was taken (Hanna 2019, 70). The lack of systematic collaboration across Global Practices remains an issue for several reasons. The reasons include (i) linking budgets with task team leadership, often resulting in Global Practices lobbying country directors to secure task team leaderships; (ii) holding regional directors and practice managers accountable for ensuring that each of their staff is fully employed Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 4 and requiring that preset lending volume targets are met; (iii) giving most, if not all, recognition to the task team leader rather than also recognizing co-task team leaders and other team members; (iv) having unclear division of roles and responsibilities for digital technologies and DTT (see box 4.3 for details); and (v) importantly, having a lack of emphasis on the achievement of intended development outcomes.  ivision of Roles and Responsibilities for Digital Technologies Box 4.3. D and DTT Staff interviewed for this evaluation pointed to a lack of clarity with regard to the following issues which, according to them, resulted in unreasonable transaction costs and hampered the efficient delivery of lending and nonlending operations: » Although the coordinating role of the Digital Development Global Practice was clear with regard to information and communication technology and digital tech- nologies, there was ambiguity about the role of this Global Practice for coordina- tion and task team leadership when it came to sectoral operations, such as digital agriculture, energy, transport, water, edtech, and e-health. » World Bank administrative resources and staff time was taken up with competition for task team leaderships among the Digital Development; Finance, Competi- tiveness, and Innovation; and Governance Global Practices. Lack of collaboration among different experts ran the risk of inconsistent advice on the same issue by different parts of the Bank Group (such that the systems “do not talk to each other”). 58 (continued)  ivision of Roles and Responsibilities for Digital Technologies Box 4.3. D and DTT (cont.) » There was insufficient clarity about the role of the Digital Development Global Practice when it came to operations involving disruptive and transformative tech- nologies (DTT) (beyond information and communication technology and digital technologies) such as robotics, artificial intelligence, internet of things, nano- technology and biotechnology, batteries, drones, solar panels, and self-driving vehicles. » Interviewees further noted that there was fragmentation of DTT work within the Regions, with the risk of missed opportunities for lesson learning. They suggested regional focal points—similar to gender focal points—as a possible way to help coordinate DTT work within a Region. Source: Independent Evaluation Group interviews. Procurement (World Bank Only) DTT-Relevant Procurement In DTT-related projects, procurement can raise specific concerns due to (i) an incomplete understanding (by decision makers, client IT specialists, and World World Bank Group Independent Evaluation Group    59 Bank staff) of functional and technical specifications relating to interfaces, protocols, and data structures; (ii) consequent potential difficulties in interop- erability across systems and the potential for continued dependence on specific vendors; (iii) potential need for business process reengineering relating to, for example, the conversion from analog to digital data; (iv) potential loss of data ownership and privacy and the commoditization of user data by vendors; and (v) limitations in technology-specific legislation and regulation. Bank Group Recognition of Key Issues » The World Bank has recognized that procurement is a significant con- straint to innovation within the World Bank and that there is also need for continued innovation in DTT-relevant procurement. According to the Mainstreaming paper, “procurement can play a strategic role in enabling innovation and mainstreaming disruptive technology into Bank operations” (World Bank 2019h, 16). The paper notes the need for continued innovation in DTT-relevant procurement. A 2017 World Bank report found that staff considered procurement to be a significant con- straint to innovation within the World Bank, due to long delays that affect the flexibility needed for innovation, and, in particular, difficulties in hiring young firms (World Bank 2017c). These issues are especially detrimental to DTT-related projects. IEG Findings Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 4 » Procurement is a major implementation constraint in DTT-related projects in the eyes of most World Bank interviewees. Staff interviewed by IEG overwhelmingly reported that the World Bank’s pro- curement systems are not supportive of DTT and constrain the institution’s ability to respond to client needs. NVivo analysis of staff interviews conducted by IEG showed that of the 24 interviewees who talked about procurement systems, 21 interviewees (or 88 percent) reported a lack of DTT-relevant procurement systems conducive to DTT-relevant work. The NVivo analysis is described in appendix A. Interviewees reported that procurement processes do not sufficiently encourage cases where the World Bank or borrower may be unaware up-front of the best solutions available in the market, which is frequently the case with innovative DTT-related projects. For example, regarding procuring services related to AI, clients and the World Bank may not have a priori specifications but may have knowledge only of the problem to be solved. Ideally, vendors would submit their best solutions based on the problem to be solved. World Bank staff interviewees pointed to trade-offs between helping clients to customize specifications to adapt to present needs and maintaining in- teroperability with both legacy and future systems. Interviewees pointed to capacity limitations, not only at the level of the borrower but also at the level of World Bank operational and procurement staff, flagging the need to devel- op ICT procurement skills, encourage principles of innovation, and stream- line processes. Because clients typically need software and an after-sale 60 relationship with the vendor, particular skill is needed to navigate the fine line between customizing specifications on behalf of the client and drawing up specifications that are too generic to be useful. If specifications are too customized, the outcome will tend toward single sourcing, which would put the World Bank in the position of advocating for a single supplier. Several past IEG evaluations have flagged procurement issues in technology projects. For example, IEG’s evaluation Capturing Technology for Development reported that ICT procurement was a major constraint for implementation due to the lack of expertise in ICT procurement, the need to separate the procurement of goods and services, inconsistent advice to borrowers, lim- ited borrower capacity, and the lengthy procurement process (World Bank 2011a). Management accepted that procurement was an issue, and procure- ment guidelines were revised in 2011 to include ICT-specific approaches, procurement methods that reflected the complexities of ICT system design and development, and the review of bidding documents for ICT projects and guidelines for task team leaders. IEG’s evaluation The World Bank Group and Public Procurement confirmed a series of improvements to ICT procurement, including revisions to the standard bidding documents, acceptance of two- stage bidding, the use of framework agreements for software maintenance or servicing, and other means of flexibility, although legacy difficulties remain (World Bank 2014b). In the following year, IEG’s second evaluation on learn- ing and results noted that World Bank staff sometimes lacked the knowledge World Bank Group Independent Evaluation Group    61 needed to check whether procurement specifications for high-technology data systems were adequate, making them vulnerable to companies seeking to sell expensive systems that exceeded client needs (World Bank 2015a). » IEG’s expert review undertaken for this evaluation found that there is cur- rently only fragmented guidance for complex technology projects, spe- cifically on the risks associated with complex technology projects (such as vendor lock-in, lack of interoperability of systems, potential loss of data ownership and lack of data privacy, limitations in technology-specific leg- islation and regulation, and mismatch between business processes and the requirements of the particular technology). IEG’s expert review of the World Bank’s procurement regulations from the perspective of DTT-related operations found that World Bank procurement regulations and guidance do acknowledge a range of DTT-relevant factors. These include (i) the need for procurement to consider the speed of tech- nological change; (ii) the need for transferability and security of informa- tion across past and future systems and across different providers; (iii) the dangers of continued dependency on particular bidders; and (iv) possible constraints on eligible technical solutions due to limited internet access or mobile bandwidth and coverage. The review also referred to the importance of specifying functional over technical requirements and allowing borrow- ers a wide range of market approaches to procurement, including of new technologies. Although offering a range of selection options and market approaches to procurement, the World Bank’s procurement regulations and guidance only offer a generic guide to risk management, which is specific to Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 4 neither IT nor DTT. Current procurement guidance has limited relevance for risk factors that greatly influence the success or failure of complex technology projects. These factors include the following: » Borrower capacity issues, including technology staff capacity 4 » Functional and technical specifications that may not be well understood by decision makers or developers5 » Interoperability, not only with legacy systems but also with potential future directions » Business process reengineering, for example converting from analog to digi- tal systems and processes6 » Uncertain costs arising from unclear buyer or supplier transparency of re- sponsibilities or ongoing license or maintenance costs, or both » Legal challenges (box 4.4) Procurement requirements for technology are partly dealt with in the stan- dard procurement documents, but critical aspects are only briefly mentioned, such as process reengineering, the significance of open standards, and sys- tems piloting. 62 The World Bank does permit the use of two-stage bidding for complex ICT and DTT procurement, for example in cases where the exact solutions are not known up-front. In such cases, unpriced technical proposals are invited, based on performance specifications or on design concepts. Bidding docu- ments may be adjusted based on information acquired during the process. Even this approach may not be sufficiently open ended for certain areas of DTT, such as AI, where it may be possible to specify only the problem to be solved. Electronic-reverse options are another flexibility that has been introduced. In principle, the World Bank also offers a “competitive dialogue” process that allows for wide flexibility. Box 4.4. Legal Challenges in Procurement One example of legal issues arising in complex technology was the failure to specify ownership and usage rights for an application programming interface in Estonia, lead- ing to litigation years later between the government and the private vendor. Interviewees pointed out that copyright of applications purchased from the private sector can also lead to downstream legal issues. Moreover, as data become digital, the distinction between content and data blurs, leading to issues of ownership, privacy, and data use. Country clients also mention possible cross-border legal issues, for example those related to cloud storage of information in servers in other countries or with certain large- World Bank Group Independent Evaluation Group    63 scale private data companies. Cloud storage in the United States or elsewhere could be problematic if, for example, political sanctions are imposed that deny countries access to their data or compromise privacy. Storage with some private providers could open the door to questions about the ownership or confidentiality of information. One solution could be that the World Bank advises clients on how to classify their data. Risk-free data (such as that collected from the web) could be given to private company servers, while sensitive information, such as health or security-related information, could be hosted on in-country servers. But data sovereignty issues will need to be resolved. Source: Independent Evaluation Group interviews for the expert review undertaken for this evaluation. In addition, the World Bank’s procurement systems provide some flexibility in the thresholds for bidders’ years of experience. This facilitates bidding by relatively young start-ups but may still deny entry to the youngest firms, which could be the most innovative. However, thresholds are needed to safe- guard the buyer. Task team leaders are reluctant to use the available flexibility as this is perceived to add to the time and complexity of an operation. Interviewees attribute the delays and complexity to insufficient guidance on how to apply the available flexibility for complex technology projects in the World Bank’s procurement systems in different situations. Providing guidance can en- courage project teams to use the available flexibility for complex technology Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 4 projects.7 This will require that procurement staff are well versed in the guid- ance to help ensure that projects proceed smoothly and without undue delay. Reluctance to use the available flexibility may also result from misaligned incentives that encourage risk aversion. 64 1  The six Global Practices would be Governance, Jobs and Social Protection, Education, Pov- erty, Digital Development, and Finance, Competitiveness, and Innovation. In practice, the Poverty and Finance, Competitiveness, and Innovation Global Practices are not involved in the project at this time. 2  Normally, the country director only gives the budget to the Global Practices it chooses for task team leadership. If a Global Practice is not providing task team leadership for a project, it cannot show in the system that it is working on a project (it is only shown as “cross-support”). 3  Hanna (2019) noted that in digital economy diagnostics, a preoccupation with capturing budget has come at the cost of sufficient collaboration, a lack of analytical depth, insufficient attention to the twin goals, and a failure to connect to the analog foundations stressed by the World Development Report 2016 (World Bank 2016e). He concluded, “The underlying factors for these gaps go beyond having common assessment tools. The current operating model tends to inhibit collaboration and cause fragmentation, internal competition.…These gaps are not in- surmountable, within the World Bank and in country clients and there are a few encouraging examples. But overall assessment of all pilots suggests there is much room for improvement” (Hanna 2019). 4  Staff pointed out that the World Bank could better use its convening power to identify spe- cific country risks for different stakeholders. 5  Technical specifications relate to interfaces, protocols, data structures, system language, and standards that ensure interoperability across vendors, legacy systems, programming lan- guages, data formats, and the like. Efforts should be made to adopt or specify open standards World Bank Group Independent Evaluation Group    65 that are approved by recognized standards organizations and available for public use free of royalties. These can help eliminate the problem of being tied to a specific vendor. 6  Converting analog to digital information to enable numeric analysis will commonly require a review and, often, the overhaul of all systems, processes, methods, rules and regulations, and legislation. Stakeholders often resist this. World Bank staff said that the digitization rate is as low as 30 percent in some client countries where most financial systems are manual and point out that the World Bank can introduce clients to a variety of alternative digitization tools. 7  Two draft World Bank analytical reports—GovTech Procurement Strategy and Disruptive Tech- nologies in Public Procurement—are in preparation. 5 | Institutional Culture and Incentives for Risk Taking and Innovation Highlights The World Bank Group recognizes the importance of institutional in- centives and a culture of learning and innovation for achieving a more effective Bank Group and acknowledges risk aversion as an issue. This evaluation found that although harnessing disruptive and transformative technologies for development often demands inno- vation, which by definition is without precedent and inevitably risky, there were insufficient incentives for staff to take risks and innovate. The levers at the disposal of the Bank Group (such as leadership signaling, questions asked in operational review meetings, and the performance management system) were not yet being effectively used to facilitate informed risk taking and innovation for disruptive and transformative technologies for development. 66   DTT-Relevant Risks and Innovation DTT-related operations are inherently risky and often call for innovative approaches. The riskiness of DTT-related operations results from the many complex reforms that need to be pursued—often simultaneously—and that involve multiple public and private sector stakeholders. This is illus- trated by the World Bank’s 2013 Ghana eTransform project, which aimed to improve the coverage and transparency of government service delivery by digitizing several parts of government. This required the project to support significant legislation (such as a data protection act and an elec- tronic transactions act) and regulations (such as for electronic signatures, electronic investigations and interception, electronic payments medium, and electronic waste). Furthermore, upgrading the country’s biometric ID system was essential for facilitating electronic transactions and called for sophisticated and innovative management information systems for the proper handling of citizens’ personal data. In addition, collaboration with the private sector was required to set up medical call centers to provide a first-response mechanism to citizen inquiries about medical and health care issues in remote and rural parts of the country. Finally, the success of the project also depended on procuring computer hardware and software and ensuring affordable high-speed internet access. More recently, the COVID-19 pandemic has brought into stark relief World Bank Group Independent Evaluation Group    67 DTT-relevant risks and the need for innovation. The Organisation for Eco- nomic Co-operation and Development–World Bank team collaborating to use DTT to aid the COVID-19 response points out that “the use of new tech- nologies to cope with COVID-19 creates a fundamental tension between the opportunities brought by digital tools and the risks that the use of data pose to privacy rights and security” (Amaral, Vranic, and Lal Das 2020). Almost overnight, the pandemic disrupted the centuries-old model of “children going to school,” requiring it to be blended with “schools going to children,” and thereby forcing public and private sector organizations to instantly step up to deliver innovative DTT solutions for learning, including in situations of low capacity and connectivity. Bank Group Recognition of Key Issues » The Bank Group has acknowledged the existence of risk aversion as a constraint for innovation and the need for a culture change in favor of a new learning culture, especially in the context of the changing nature of work. The Bank Group’s FY20–22 Human Resources Strategy states that incentiviz- ing behaviors that create a more effective, results-oriented, innovation-driven culture are key to achieving a more effective Bank Group. Drawing on the technology-related discussion of the changing nature of work in the World Development Report 2019 (World Bank 2019m), the Bank Group’s Human Resourc- es Strategy emphasizes learning agility and conscious on-the-job learning over Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 5 mastery (World Bank 2019l). Recognizing the need for a culture change, the strat- egy emphasizes that the new learning culture aims to treat learning as a journey rather than an event and places increased focus on innovation and results. A 2017 World Bank report recognizes that there are barriers to innovation and that there is risk aversion at the World Bank (World Bank 2017c). About half of the interviewees for the 2017 report opined that there would be negative professional consequences if innovative elements failed and that often man- agement did not sufficiently recognize innovative projects due to their small size. The same report also notes that certain task team leaders argued that they were evaluated, measured, and promoted based on their outputs or disburse- ments rather than on results, impact, and learning. Furthermore, according to the report, although some interviewees discussed how their managers were happy to recognize their staff if they successfully used innovative methods, and others discussed how managers can provide support when navigating the World Bank’s bureaucracies, most management support to enable innovation ap- peared to come in the form of “not impeding” their teams (World Bank 2017c). IEG Findings » There are a few instances of greater risk appetite. The West Africa Unique Identification for Regional Integration and Inclu- sion Program is an example of the World Bank demonstrating greater risk appetite. This is a 10-year operation in six countries—Côte d’Ivoire, Guinea 68 (Phase 1); Benin, Burkina Faso, Niger, and Togo (Phase 2)—and the Eco- nomic Community of West African States Commission to build foundational ID platforms that are regionally interoperable. As such, the program faces technical risks (data privacy and data security), policy risks (weak implemen- tation that could lead to further marginalization of vulnerable populations), and organizational risks (cooperation with external organizations such as the Economic Community of West African States and coordination within the World Bank). To manage these risks, a director-level mechanism was established within the World Bank to review periodically the project’s risk management actions. The World Bank also employed an agile multiphase programmatic approach, under which countries could join when they were ready. The World Bank deemed the potential benefits of providing social assistance to vulnerable groups in West Africa to be greater than the risks. » In general, however, Bank Group staff interviewees perceived a lack of sufficient institutional incentives for risk taking and innovation. IEG interviews of Bank Group staff cited the institutional culture and incen- tives for risk taking and innovation as challenges for DTT support. NVivo anal- ysis of staff interviews conducted by IEG showed that of the 41 interviewees who talked about incentives for DTT support, 37 interviewees (or 90 percent) reported these to be insufficient. Of the 19 interviewees who talked about incentives for risk taking, 18 interviewees (95 percent) reported these to be insufficient. Furthermore, of the 23 interviewees who talked about incentives World Bank Group Independent Evaluation Group    69 for innovation, 18 interviewees (78 percent) reported these to be insufficient. The NVivo analysis is described in appendix A. A past IEG evaluation found that staff experience pressure to avoid risks. An IEG survey and interviews conducted for IEG’s first Learning and Results evaluation found that staff experienced pressure to avoid risks and not to admit failure and that in the view of many, there was limited learning from mistakes (World Bank 2014a). That evaluation also found that staff per- ceived the lack of institutional incentives to be among the biggest obstacles to learning and knowledge sharing in the World Bank. Moreover, the World Bank’s internal organizational structure has been subject to many reforms, but serious reforms of internal incentives have not been undertaken. Interviewees noted that the willingness to attempt innovative solutions for the twin goals depends on an institutional culture that supports risk taking and acknowledges that failure is part of this. Failing is essential, because if one is not “prepared to be wrong,” one will never produce “anything orig- inal” (Robinson 2009). Interviewees noted that incentives to learn from mistakes would require that the Bank Group convey clearly and explicitly how failure would be treated. Every project document in the Bank Group is required to discuss potential risks and mitigation measures, but interviewees reported that the “real” risks are sometimes downplayed so as not to hold up the operational review process. The implicit assumption staff make is that the institution has a low tolerance for risk and failure. Ensuring a more realistic up-front assessment of risks would require staff to feel psychologi- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 5 cal safety in speaking up about potential risk and failure and for operational review meetings to also promote speaking candidly about them. Google X holds meetings termed “premortems,” where teams are encouraged to speak their mind from the start about all the things that could go wrong. This enables decision makers to be fully aware of risks and make calculated bets. Furthermore, projects that are expected to fail are not viewed negatively: “We see killing projects as a normal part of doing business because it means we can go faster and take on ideas that are more promising” (Teller 2016). » Interviewees reported that the Bank Group’s institutional incentives do not yet sufficiently encourage staff to keep up with DTT trends, despite the fluid and fast-moving nature of DTT. Interviewees reported a lack of institutional incentives for staff to keep up with DTT trends. An institutional barrier cited in the interviews was that Bank Group staff were not encouraged to keep up with technological ad- vances to enable them to deploy frontier DTT for development. Interviewees reported a lack of active support and enthusiasm for DTT-related external training, secondment of staff to and from organizations leading in DTT, en- couragement of participation in relevant conferences, and partnership with external organizations that are pioneers in specific DTT areas. Interviewees saw the Bank Group treating DTT in the same way as other sectors and themes, with little awareness that DTT are fluid and fast moving. World Bank interviewees in the Digital Development and the Finance, Com- 70 petitiveness, and Innovation Global Practices and in IFC said that leadership did not provide sufficient incentives for innovation and smart risk taking. Such incentives are especially needed in DTT-related operations since the Bank Group needs to move with greater agility (by reducing processing com- plexity) compared with traditional infrastructure sectors. Interviewees noted that IFC is often the last to sign deals, which can be a particular issue in a fast-moving area such as DTT. Although IFC and the World Bank tend to take more risks in venture capi- tal investments in DTT and fintech projects than in more traditional proj- ects, their risk tolerance is still below that of other players in the area. Risk aversion is apparent in broad managerial comfort with what has been tried before rather than something that breaks new ground. In DTT, and fintech and digital entrepreneurship specifically, there are often no past examples, but interviewees confirmed managerial preference for precedents even in these areas. Nevertheless, interviewees noted that there is a marked increase in risk appetite for venture capital investments in DTT and fintech projects compared with a couple of years ago. » Interviewees reported that the Bank Group was not making the best use of its levers to create incentives for risk taking and innovation and that pi- lots were insufficiently monitored, precluding the expansion of successful initiatives or the scaling back of unsuccessful ones. World Bank Group Independent Evaluation Group    71 The levers at the disposal of the Bank Group (such as leadership signaling, questions asked in operational review meetings, and the performance man- agement system) were not yet being effectively used to facilitate informed risk taking and innovation for DTT for development. The organizational effectiveness literature suggests that leading on an overarching agenda typ- ically requires a strong mandate and signaling delivered from the top.1 The housing of Singapore’s Smart Nation program in the prime minister’s office is a case in point (Lin 2016). Interviewees saw the operational review process as constraining innovation and creative solutions, given the general sup- port for continuing on a familiar path rather than for breaking new ground. Interviewees reported that legal, procurement, and external relations de- partments often asked for precedents and could not always be counted on to support first-time initiatives. Furthermore, interviewees reported that the operational review meetings did not consistently ask questions focused on informed risk taking, learning from successes and failures, and innovation for DTT for development. Interviewees also noted pressure to lend, saying that it took a lot to be at the cutting edge and that it was easier to just keep things simple. Interviewees noted that some innovation occurs despite the system. As one interviewee put it: “There are fearless innovators. . . . It takes a lot of self-motivation.” Other interviewees noted that a lot of innovation is being driven by intrapreneurs, but they are not being fully supported. There is tremendous pressure for them to conform to more traditional paths and ways of operating. Economist Gary Pisano identifies a number of behaviors necessary for creating a culture of risk taking and innovation (appendix F). Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 5 The Bank Group has supported pilots for innovation, but the systematic monitoring of pilots, which can allow subsequent upward or downward scal- ing, is not sufficiently undertaken in the Bank Group. Pilots for innovation are supported through three initiatives: (i) the Disruptive Technologies for Development Fund, which finances pilots that are proposed by Bank Group staff and connects them to experts within and outside the Bank Group; (ii) the ID4D Mission Billion innovation challenge to source innovation for providing digital ID for vulnerable groups; and (iii) the Development Mar- ketplace (gender-based violence). However, interviewees noted that there has been insufficient monitoring of the pilots supported through these three mechanisms and, consequently, insufficient follow-up. The Bank Group’s stated goal of “thought leadership on what is changing, why it matters, and where the new opportunities lie” (World Bank 2019h, iii) would suggest greater emphasis on research and development, piloting, incubation, eval- uation, and the scaling of new ideas. In this regard, the Bank Group could explore “sandboxing” and “real-life” piloting in addition to the initiatives noted above. Sandboxes can allow Bank Group teams to temporarily waive procedures in particular areas to freely experiment, affording staff the psy- chological safety to innovate, take calculated risks, and fail without fear of repercussion. Real-life piloting can allow new approaches to be implemented in different country contexts where there are willing clients, enabling inno- vation to be scaled quickly, taking it from the “lab to the last mile” (Watkins 2018). Even when pilots are well monitored and suggest potential for scaling, challenges may arise. Disruptive technologies in the energy sector, such as 72 the Bank Group’s experience with Lighting Global (which includes Lighting Africa and Lighting Asia) and Scaling Solar, were covered in a recent IEG evaluation and are discussed in appendix K. World Bank Group Independent Evaluation Group    73 1  For example, the United Kingdom’s Department for International Development notes that a well-networked central hub “will ensure coordinated implementation of the digital strategy. It will set strategic direction for digital policy and programming; build digital capability and capacity; inspire transformation through visible digital demonstration; and ensure impact is maximised through partnerships and collaborative working with those who share our digital vision” (DFID 2018). Similarly, the 2018 United Nations survey on frontier technologies for sustainable development found that having a central hub group was important to success (United Nations Department of Economic and Social Affairs 2018). The United States Agency for International Development has its Global Development Lab, and Singapore has the Gov- ernment Technology Agency and the Smart Nation Programme Office, which is charged with building innovation capabilities and mindsets across the entire public service. It is housed in the Public Service Division, a central human resources agency under the prime minister’s of- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 5 fice. These technological innovation hubs inside of organizations provide links among actors within their innovation systems. 74 6 | Moving Forward: Directions of Travel Highlights The Independent Evaluation Group presents directions of travel for the Bank Group’s consideration: Mining development data generated through disruptive and transfor- mative technologies (DTT) to create even greater social value; leverag- ing the Bank Group’s honest broker role in advising clients, especially on DTT risks; and using explicit corporate metrics to track the imple- mentation of the Bank Group’s DTT Mainstreaming approach. Enhancing Bank Group capabilities to address the differential gen- der effects of DTT, beginning with advisory services and analytics. Fostering a growth mindset among Bank Group staff to adapt to the rapidly changing technological landscape, and applying fore- sight and anticipation to proactively foresee DTT opportunities and identify looming threats, enabling the Bank Group to regroup and take on challenges in real time and thrive even in markedly changed circumstances. 75  IEG presents—for the Bank Group’s consideration—directions of travel, including examples of each, that can help the Bank Group move forward in strengthening its preparedness for DTT for development. Building on the Bank Group’s Existing Strengths Mining DTT-generated development data to create even greater social value. The Bank Group is in a unique position to ramp up the deployment of DTT in the generation, analysis, and use of development data in achieving the twin goals. According to Fu (2019), “High quality development data is Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 6 the foundation for meaningful policy-making, efficient resource allocation, and effective public service delivery.” DTT elements such as mobile phones, electronic transactions, and satellites have led to an explosion of data that are collected and analyzed cost-effectively, with high frequency, and at fine levels of granularity (Goldberg 2020). This granularity can allow for the per- sonalization of services, for example tailoring learning to individual student needs in poor rural areas and potentially reducing high dropout rates. The World Development Report 2021 on data offers an opportunity to explore how DTT can revolutionize public and private data collection, analysis, and use and better inform policy making for addressing the twin goals (World Bank 2021). Furthermore, the recently established (managing director-level) Data Governance body and its supporting (vice president–level) Data Governance Steering Committee, which aim to provide executive leadership, oversight, and support for data-related matters, can boost the World Bank’s ability to generate even more social value from (DTT-generated and analyzed) devel- opment data. Optimizing the use of public and private data—those made available by DTT but also those that can be analyzed more effectively by using DTT—can help the Bank Group become a data-driven organization. Leveraging the honest broker role by advising clients, especially on DTT risks. The Bank Group’s perceived neutrality positions it well for flagging DTT risks. Fieldwork for this evaluation found that country clients expressed a desire for greater Bank Group advice on DTT risks such as job losses, lack of privacy, and data breaches.1 Attention to DTT risk is particularly import- 76 ant in ID4D systems, which are especially vulnerable to inadvertent data spills or willful misuse.2 Another area of risk is the fragmentation of digital data across different proprietary system silos within government. The Bank Group should help prevent vendor lock-in and improve government efficiency and transparency while speeding up innovation in the public sector.3 Where a local tech ecosystem exists or is nascent, Bank Group DTT support can strength- en this and use it as an opportunity to foster inclusive growth and job creation. Addressing DTT risks and adhering to the do no harm principle may require the Bank Group to navigate sensitive ethical and political issues and advise clients on such issues.4 AI governance—ensuring that AI is explainable, transparent, and ethical—is critical.5 Ethical issues are also prominent in the use of personal data and vaccine distribution, for example.6 After due vetting, the Bank Group could use its convening power to bring together relevant experts in areas where it may lack the necessary technical expertise to help clients mitigate DTT risks.7 Using explicit corporate metrics to track the implementation of the Main- streaming approach. The DTT Mainstreaming approach applies across Bank Group institutions, Global Practices, and Regions. Explicit metrics—defined at the corporate level—can help track progress and enable timely course corrections, thereby facilitating learning and the successful implementation of the Mainstreaming approach. Two Bank Group initiatives—the work on Gender and the Human Capital Project—may provide lessons for the main- streaming of the DTT agenda. World Bank Group Independent Evaluation Group    77 Enhancing Bank Group Capabilities That Are Not Yet Its Forte Addressing gender-differential impacts of DTT, beginning with ASA. The Bank Group’s 2015 Gender Strategy highlighted gender gaps in the use of technology and women’s lower participation in technical jobs and technical occupations (World Bank 2015b). It notes these gender gaps were often due to off-line factors like poverty, gender discrimination, and gender stereotypes that prevent girls and women from benefiting from digital technologies. At the same time, increased access to credit, knowledge, and markets made possible by DTT may place women at higher risk for domestic violence (McDougal and others 2019; see appendix C). Analysis of gender and DTT for this evaluation found that although 76 percent of the 51 World Bank ICT projects (ICT sector percentage weight of 50 percent or more) approved during FY15–19 had one or more gender-relevant flags, just 37 percent had all three gender flags. Of particular concern was that just 7 percent of the 90 World Bank ICT ASA initi- ated during FY15–18 were gender relevant—a particular concern since it is in ASA that new opportunities to address the gender-differential impacts of DTT can be explored. Beyond gender considerations, it will be important to ensure that other excluded groups (such as the LGBTQ community, people with dis- abilities, ethnic and religious minorities, and displaced communities) benefit from the opportunities of DTT and are protected from its risks. Developing New Strengths Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 6 Fostering a growth mindset among Bank Group staff. Given the uncertain- ties created by technological shifts, the Bank Group could enhance its pre- paredness for DTT by ensuring a growth mindset that promotes continuous learning and adaptation among its staff. Dweck (2006) explains that leaders of organizations that possess a growth mindset constantly try to improve by setting out to hire highly capable staff, endeavoring to learn from their own mistakes and shortcomings, and addressing openly the skills that they and the company will need for future success. In this way, continuous learning becomes part of the organization’s culture. IEG’s first Learning and Results evaluation noted that organizations that do not learn, die (World Bank 2014a) and pointed to the need for the World Bank to embrace learning and adaptation. Box 6.1 shows how Microsoft fosters a growth mindset. Systematically applying foresight and anticipation to help inform and guide the Bank Group’s future support of DTT for development.8 Foresight and anticipation can help the Bank Group to proactively foresee DTT opportunities and identify looming threats for the organization, its operations, and its clients (appendix E provides details). As new players enter the DTT for development field, the Bank Group will need to dynamically build new areas of comparative advantage even as it maintains specific old ones. Foresight and anticipation can enable organizations to regroup and take on challenges in real time and thrive even in markedly changed circumstances. For example, Galloway (2020) notes that the landscape of higher education will change “seismically” in the wake of big technological companies entering education during the COVID-19 pandemic, 78 seeking to expand enrollment by offering hybrid online–off-line degrees. The Bank Group can use foresight and anticipation to help clients adjust in a timely way to such developments. Thinking systematically about tomorrow and being prepared is particularly important in the face of rapid change and uncertainty (Conway 2015; Pauwels 2019; van de Pol 2017).9 A forthcoming ASA, The Con- verging Technology Revolution and Human Capital: Potential and Implications for South Asia, notes that the World Bank needs to undertake “normative technolo- gy foresight exercises” (Bashir and others, forthcoming).  it Refresh: Concrete Actions Taken by Microsoft to Acquire a Box 6.1. H Growth Mindset Old Microsoft. By 2014, the Microsoft that CEO Satya Nadella had inherited was fading toward irrelevance as the technology industry shifted from desktop computers to smartphones. New CEO’s vision. In July 2015, seizing on the ideas in Carol Dweck’s book Mindset: The New Psychology of Success, Nadella sought to forge a culture change at Microsoft such that staff would believe that “everyone can grow and develop; potential is nur- tured, not predetermined; and anyone can change their mindset,” and that they would shift from being “know-it-alls” to “learn-it-alls.” Microsoft followed up this vision with several concrete actions, including the following: World Bank Group Independent Evaluation Group    79 Concrete action #1: CEO issuing monthly videos on his top learnings. Nadella has taken to issuing monthly videos reviewing his top few learnings. This has prompted groups within the company to review their learnings too. Concrete action #2: closing meetings with a reflection. Nudges and small remind- ers engage all 125,000 of Microsoft’s employees with the new culture; for example, leaders close meetings with a reflection, asking questions such as, Was this a growth mindset or fixed mindset meeting? Why or why not? Microsoft today. Microsoft is once again a magnet for top engineering talent, rated as one of five best artificial intelligence companies for employees, and Nadella has a Glassdoor employee approval rating of 95 percent. Source: Adapted from Ibarra, Rattan, and Johnston 2018. Specifically, applying foresight and anticipation to identifying three sources of uncertainty and risk can help guide the Bank Group’s DTT support (UNDP Global Centre for Public Service Excellence 2018): » “Known knowns” (where it is known that something will change and, at least to an extent, what will change; for example, it is known that the nature of jobs will change, and it is also known that routine jobs will be lost to automa- tion), requiring the Bank Group to anticipate the changes and ensure com- plete preparedness to be able to address them as soon as they occur. » “Known unknowns” (where it is known that something will change, but the nature and extent of change is unknown; for example, it is known that new skilled jobs will be created, but what is unknown is the net change in the total Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 6 number of jobs due to technological advances), requiring the Bank Group to anticipate, constantly monitor, plan for possible scenarios, and adapt to the evolving scenario as further information becomes available. » “Unknown unknowns” or “black swan events” (where events cannot be fore- seen, and therefore their nature and scale are also unknown) requiring the Bank Group to develop strong organizational foundations by investing in a growth mindset such that it can regroup, learn, and adapt with agility. Foresight and anticipation are not completely new to the Bank Group. The Bank Group’s CPF is an anticipatory document that outlines the Bank Group’s country strategy four to six years out. More specifically for DTT, the recent exploration of foresight and anticipation by the Bank Group’s Tech- nology and Innovation Lab is a promising step. 80 1  One possible avenue to explore in addressing job losses and income inequality is universal basic income (see Gentilini and others 2020). 2  So far, the Bank Group’s attention to DTT risks appears inadequate. IEG received a list of 24 Identification for Development projects and 32 Identification for Development ASAs approved during fiscal years 2010–20. Of the 24 projects, 16 had at least one keyword (privacy, security (including cybersecurity, data security), foundational, surveillance, governance (data governance), ownership (data ownership), and risk) in the project abstract, objectives, components, or indi- cators as per the analytics run by the IEG team. A further desk review to remove false positives identified 12 (50 percent) out of 24 projects with relevant keywords. Similarly, of the 32 ASAs, 21 had at least one keyword in the objective or activity summary. A further desk review to remove false positives identified 16 (50 percent) out of 32 ASAs with relevant keywords. Goh, Kaiser, and Wright (2020) emphasize the importance of promoting open-source software. 3  4  A forthcoming ASA, The Converging Technology Revolution and Human Capital: Potential and Implications for South Asia, notes that the World Bank needs to operationalize a theory of do no harm when it comes to technology risks (Bashir and others, forthcoming). A further DTT risk pointed out by Abdalla and Abdalla (2020) is that Big Tech (similar to Big Tobacco) may co-opt academic research. They note that avoiding conflict of interest between Big Tech and its funding of academic work is vital. They advocate deeper discussion of the appropriateness and trade-offs of accepting funding from Big Tech and imposing necessary limitations and conditions on this funding. 5  Gasser and Almeida (2017) note the information asymmetries that exist between the devel- World Bank Group Independent Evaluation Group    81 opers and users (government and consumers) of artificial intelligence. 6  Pate (2020) notes that “we cannot afford to let high-income countries monopolize the global supply of COVID-19 vaccines, as happened during the 2009 influenza A/H1N1 pandemic.” 7  Some organizations working on DTT risks include Center for Humane Technology, AI for Good, the Institute of Electrical and Electronics Engineers, and the Stanford Institute for Human-Centered Artificial Intelligence. A 2020 docudrama film, The Social Dilemma, aims to raise awareness of the opportunities and risks of the rise of social media. 8  The evaluation does not advocate any particular brand for analyzing foresight or anticipa- tion. The evaluation sees a critical role for looking ahead and for the Bank Group not to be blindsided or caught off guard by technological developments or changes in country contexts. Some tools for conducting foresight and anticipation include foresight analysis, horizon scan- ning, scenario planning, and drivers and trends impact analysis (Conway 2015; Pauwels 2019; van de Pol 2017). 9  The importance of foresight and anticipation is evident in the remarks of a technological entrepreneur: None of us should be focused on the current quarter because that quarter was “baked three years ago.” We need to be thinking right now about the quarter that will reveal itself in three years (Investment Masters Class 2018). Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 6 82 7 | Conclusion and Recommendations Conclusion This evaluation sought to answer the question, How well prepared is the Bank Group to help clients harness the opportunities and mitigate the risks posed by DTT? The answer to that question is, Given the accelerat- ing pace and complexity of technological change, the Bank Group is not yet sufficiently well prepared, despite some areas of strength. The Bank Group’s traditional areas of strength are what have enabled its DTT support. The Bank Group has a commanding reputation for supporting global public goods, a support that now also extends to DTT. Furthermore, the Bank Group’s honest broker role has enabled the Bank Group to provide clients with technical expertise, especially in policy and regulatory matters related to DTT. The Bank Group is perceived as a neutral voice advising on different technological solutions. In addition, the Bank Group’s ability to of- fer quality ASA has equipped it with actionable recommendations to support clients in putting their digital economies on a robust footing. Finally, the Bank Group’s ability to mobilize trust funds and IDA grants and credits has also enabled its DTT support. Many areas vital to the Bank Group’s preparedness for DTT are, however, either still in development or developing too slowly to match the acceler- ating technological change occurring in every field. The Bank Group has given increased attention to the digital economy in country diagnostics and strategies, but the discussion in SCDs has not always carried over into CPFs. Although the CPF cannot be expected to address all the constraints identi- fied in the SCD, it would be important to ensure that the CPF’s prioritization and selectivity are well justified. There has also been insufficient effort in SCDs in linking the support for digital technologies to the broader develop- ment agenda, including the twin goals (for example, when identifying key 83 constraints). Furthermore, the newly introduced digital economy diagnostics focus on a narrow agenda with room to consider the broader development agenda and the twin goals. The Bank Group’s strategic documents emphasize the need to invest early in twenty-first century skills and the World Development Report 2019 notes that, in view of the changing nature of work, lack of education is likely to be one of the strongest mechanisms for transmitting inequalities from one generation to the next (World Bank 2019m). The Bank Group’s top corporate initiatives in education—the Human Capital Project and the Learning Pov- erty initiative—spotlight necessary skills such as literacy and numeracy. The Bank Group has an opportunity to spotlight twenty-first century skills and Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 7 move the narrative from a focus on necessary skills to also include twen- ty-first century sufficient skills such as advanced cognitive skills, digital lit- eracy, and socioemotional skills, in particular a growth mindset or the ability to “learn to learn” and adapt. Both necessary and sufficient skills are critical for ensuring that a workforce is well prepared for the future labor market. There are strong reasons for imparting these skills simultaneously—not sequentially—from early childhood, recognizing the profound positive effect on brain development and lifetime capabilities. Imparting sufficient skills may also serve (i) the nascent demand from developing countries that are aspiring to grow and catch up by becoming knowledge-intensive economies and (ii) reduction of the North-South divide (UNESCO 2015). The Bank Group’s FY20–22 Human Resources Strategy presents key insights about the staffing issues that confront the Bank Group (World Bank 2019l). Reflecting the latest thinking in organizational psychology, it recognizes the need to build new staff skills and expertise and foster a growth mindset for continuous learning in the face of rapid technological change. The Bank Group has yet to (i) identify the DTT-relevant skills that it needs; (ii) ensure that its information systems and databases contain the necessary informa- tion on its current DTT-relevant skills; and (iii) take action to fill any gaps. Bank Group staff noted in interviews that the number of existing staff with DTT-relevant skills is insufficient to meet client demand, especially in areas such as regulatory reform, data privacy, cybersecurity, 5G networks, and AI, and, consequently, staff with these skills are often overstretched. They also 84 reported that mindsets for continuous learning and adaptation in the Bank Group fall short of what is needed. Despite the cross-cutting nature of DTT, there was a perception among interviewees from across Global Practices that collaboration was insufficient among the different Global Practices, with instances of competition for task team leadership. There was also insufficient collaboration between the World Bank and IFC, even though DTT typically require the public and private sec- tors to work together. Furthermore, interviewees noted that procurement was a major con- straint to the smooth implementation of DTT-related projects. There is only fragmented guidance for complex technology projects, specifically on the risks associated with such projects (such as vendor lock-in, lack of interoperability of systems, potential loss of data ownership and lack of data privacy, limitations in technology-specific legislation and regu- lation, and mismatch between business processes and the requirements of the particular technology). There is also insufficient guidance on how the flexibility available for complex technology projects in the World Bank’s procurement systems can be applied in different situations. In addition to insufficient guidance, misaligned incentives may contribute to the reluctance to use the available flexibility. Harnessing DTT for development often demands innovation, which by World Bank Group Independent Evaluation Group    85 definition is without precedent and inevitably risky. However, interviewees reported that DTT were treated in the same way as other sectors or themes, despite their fluid and fast-moving nature, and that relevant staff were not encouraged to keep at the cutting edge of technological advances and DTT trends. Furthermore, the levers at the disposal of the Bank Group (such as leadership signaling, questions asked in operational review meetings, and the performance management system) were not yet being effectively used to facilitate informed risk taking and innovation for DTT for development. Interviewees saw the operational review process as constraining innovation and creative solutions, given the general support for continuing on a familiar path rather than for breaking new ground. Interviewees reported that legal, procurement, and external relations departments often asked for precedents and could not always be counted on to support first-time initiatives. Finally, IEG presents—for the Bank Group’s consideration—directions of travel (including examples under each) that can help the Bank Group move forward in strengthening its preparedness for DTT for development. These include (i) building on the Bank Group’s existing strengths (by, for example, mining DTT-generated development data to create even greater social value, leveraging the honest broker role by advising clients especially on DTT risks, and using explicit corporate metrics to track the implementation of the Mainstreaming approach); (ii) enhancing Bank Group capabilities that are not yet its forte (by, for example, addressing the gender-differential effects of DTT, importantly in ASA); and (iii) developing new strengths (by, for example, fostering a growth mindset among staff to adapt to technological change and applying foresight and anticipation to proactively foresee and Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Chapter 7 address DTT opportunities and challenges). Recommendations This evaluation makes three recommendations: Recommendation 1: Where DTT offer opportunities to make progress on the twin goals more effectively or efficiently, ensure that the Bank Group avails itself of those opportunities and addresses, in particular, the risks posed by DTT. This will entail, for example: » Systematically identifying in both country diagnostics and digital economy diag- nostics the opportunities and risks posed by DTT for achieving the twin goals; » Consistently asking in operational review meetings (from Concept Note to approval and implementation stages) whether and how the use of DTT in operations can bring (or is bringing) effectiveness and efficiency gains in addressing the twin goals; and » Strengthening cross-sectoral linkages and synergies between DTT and sec- toral issues, including through enhanced collaboration across Global Prac- tices and between the World Bank and IFC, particularly where different units support technological solutions with the same client. 86 Where appropriate, a complement to country and digital economy diagnos- tics could be a review of ongoing and pipeline projects for DTT opportuni- ties, such as the one undertaken recently in Vietnam. Recommendation 2: Build a Bank Group workforce with the skills re- quired to harness DTT opportunities and mitigate DTT risks by identi- fying DTT-relevant skills, determining gaps in these skills, and filling these gaps. This will entail: » New recruitment of staff or consultants to fill specific skills gaps, retraining of existing staff, outsourcing, secondments, external partnerships, or a com- bination of these; » Ensuring a more efficient use of the skills of available specialized staff and a better bridging of technology expertise with that of sector specialists and task team leaders; and » Supporting a growth mindset for continuous learning as noted in the Bank Group’s FY20–22 Human Resources Strategy. Recommendation 3 (World Bank only): Improve the effectiveness and efficiency of World Bank procurement for complex technology projects. This will entail, for example: » Strengthening procurement guidance for staff and borrowers, for early-stage project scoping, as well as subsequent capacity development, on the identi- World Bank Group Independent Evaluation Group    87 fication, prioritization, and mitigation of the risks associated with complex technology projects. The guidance could cover: availing of existing flexibili- ties for complex technology projects such as two-stage bidding; monitoring the extent to which World Bank staff teams adopt these flexibilities; and institutionalizing arrangements for technology project management. » Ensuring that teams benefit from effective and efficient innovation while protecting the World Bank and the borrower from procurement-related reputational risks. » Preparing a roster of the world’s leading experts on the procurement of com- plex technology projects and encouraging market consultations on technical requirements in the preparation of bidding documents. Bibliography Abadie, Daniel. 2019. “Part 8: Argentina: Building a Civic Tech Startup within Government (2019).” digital HKS (blog), June 8, 2019. https://medium.com/digi- talhks/argentina-building-a-civic-tech-startup-within-government-cebe861d- bcc7. Abdalla, M., and M. Abdalla. 2020. “The Grey Hoodie Project: Big Tobacco, Big Tech, and the Threat on Academic Integrity.” https://arxiv.org/abs/2009.13676. Amaral, Miguel, Goran Vranic, and Prasanna Lal Das. 2020. “Emerging Technologies Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography to Fight Covid-19: Practices across the World.” White Paper, Organisation for Economic Co-operation and Development, Paris; World Bank, Washington, DC. Ansell, C., and A. Gash. 2018. “Collaborative Platforms as a Governance Strategy.” Journal of Public Administration Research and Theory 28 (1): 16–32. Aron, D., and B. Murray. 2018. “Are Platform Organizations a Wicked Problem?” Leading Edge Forum (blog), March 18, 2018. https://leadingedgeforum.com/re- search/are-platform-organizations-a-wicked-problem. Athena Infonomics, Lirneasia, and ResearchICTAfrica.net. 2018. Engendering ICT: A Toolkit for Task Team Leaders. Washington, DC: World Bank. Baller, Silja, Soumitra Dutta, and Bruno Lanvin, eds. 2016. The Global Information Technology Report 2016: Innovating in the Digital Economy. Geneva: World Eco- nomic Forum/INSEAD. http://www3.weforum.org/docs/GITR2016/WEF_GITR_ Full_Report.pdf. Barder, O., and T. Talbot. 2015. “Guarantees, Subsidies, or Paying for Success? Choosing the Right Instrument to Catalyze Private Investment in Developing Countries.” Working Paper 402, Center for Global Development, Washington, DC. http://www.cgdev.org/publication/guarantees-subsidies-or-paying-suc- cess-choosing-rightinstrument-catalyze-private. Bashir, S., C. Dahlman, N. Kanehira, and K. Tilmes, forthcoming. The Converging Technology Revolution and Human Capital: Potential and Implications for South Asia. South Asia Development Matters, World Bank, Washington DC. 88 Bezanson, K. A., and P. Isenman. 2012. “Governance of New Global Partnerships: Challenges, Weaknesses and Lessons.” Policy Paper 014, Centre for Global De- velopment, Washington, DC. https://www.cgdev.org/sites/default/files/1426627_ file_Bezanson_Isenman_FINAL.pdf. Borins, S. 2001. “Encouraging Innovation in the Public Sector.” Journal of In- tellectual Capital 2 (3): 310–19. https://www.emeraldinsight.com/doi/pdf- plus/10.1108/14691930110400128. Bower, J., and C. M. Christensen. 1995. “Disruptive Technologies: Catching the Wave.” Harvard Business Review 73 (1): 43–53. Brodnig, G., E. Chee, A. Raina, Z. True, and A. Rahim. 2020. “Inclusive Disruption: Harnessing the Social Power of Technology.” Governance for Development (blog), March 26, 2020. https://blogs.worldbank.org/governance/inclusive-disrup- tion-harnessing-social-power-technology. Brook, J. 2020. “At Microsoft, Sharon Lo ’16 Ponders How Products Can Harm Soci- ety.” Brown Alumni Magazine, October 23, 2020. https://www.brownalumnimag- azine.com/articles/2020-10-23/the-ethics-team. Brussevich, Mariya, Era Dabla-Norris, Christine Kamunge, Pooja Karnane, Salma Khalid, and Kalpana Kochhar. 2018. “Gender, Technology and the Future of Work.” Staff Discussion Note, International Monetary Fund, Washington, DC. Brynjolfsson, E., and A. McAfee. 2014. The Second Machine Age: Work, Progress, and World Bank Group Independent Evaluation Group    89 Prosperity in a Time of Brilliant Technologies. New York: W. W. Norton. Bryson, J. M., B. C. Crosby, and M. Middleton Stone. 2006. “The Design and Imple- mentation of Cross-Sector Collaborations: Propositions from the Literature.” Public Administration Review 66 (6 [Supplement]): 44–55. Burnard, Kevin, and Ran Bhamra. 2011. “Organisational Resilience: Development of a Conceptual Framework for Organizational Responses.” International Journal of Production Research 49 (18): 5581–99. Burns, Mary, and Peter Orne. 2020. “Will the COVID-19 Pandemic Speed a Global Embrace of Online Learning?” ICTworks, June 10, 2020. https://www.ictworks. org/covid19-digital-response-online-learning. Callan, M., and R. Davies. 2013. “When Business Meets Aid: Analyzing Public-Private Partnerships for International Development.” Development Policy Centre Dis- cussion Paper 28, Crawford School of Public Policy, Australian National Univer- sity, Canberra. Cancialosi, C. 2017. “What Is Organizational Culture?” Gotham Culture, July 17, 2017. https://gothamculture.com/what-is-organizational-culture-definition. Christensen, C. M. 1997. The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail. Boston, MA: Harvard Business Review Press. Christensen, C. M., S. D. Anthony, and E. A. Roth. 2004. Seeing What’s Next: Using the Theories of Innovation to Predict Industry Change. Boston, MA: Harvard Business Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography School Press. Christensen, C. M., and M. Overdorf. 2000. “Meeting the Challenge of Disruptive Change.” Harvard Business Review 78 (2): 66–77. Christensen, C. M., and M. E. Raynor. 2003. The Innovator’s Solution: Creating and Sustaining Successful Growth. Boston, MA: Harvard Business School Press. Ciborra, C. U. 1996. “The Platform Organization: Recombining Strategies, Structures, and Surprises.” Organization Science 7: 103–18. Cohen, J. L., and H. Kharas. 2018. “Using Big Data and Artificial Intelligence to Accel- erate Global Development.” Brookings Institution, November 15, 2018. https:// www.brookings.edu/research/using-big-data-and-artificial-intelligence-to-ac- celerate-global-development. Conway, M. 2015. Foresight: An Introduction. Melbourne: Thinking Futures. https:// thinkingfutures.net/downloads/foresightintro. Cull, Bob, and David McKenzie. 2020. “Implementing Successful Small Interventions at a Large Scale Is Hard.” Let’s Talk Development (blog), March 19, 2020. https:// blogs.worldbank.org/developmenttalk/implementing-successful-small-inter- ventions-large-scale-hard. Cunningham, Wendy, Pablo Acosta, and Noël Muller. 2016. Minds and Behaviors at Work: Boosting Socioemotional Skills for Latin America’s Workforce. Directions in Development Series. Washington, DC: World Bank. 90 Cunningham, W., and P. Villaseñor. 2016. “Employer Voices, Employer Demands, and Implications for Public Skills Development Policy Connecting the Labor and Education Sectors.” Policy Research Working Paper 7582, World Bank, Washing- ton, DC. Deltek. 2011. “Mastering Your Government Business: The 6 Critical Elements for Improving Visibility and Control.” White Paper, Deltek, Herndon, VA. https:// cordiapartners.com/media/deltek_whitepaper_visibility_control_6elements.pdf. DFID (UK Department for International Development). 2017. DFID Technology Strate- gy 2017–2020: Technology to Support Effective Development. London: Department for International Development. https://assets.publishing.service.gov.uk/govern- ment/uploads/system/uploads/attachment_data/file/698192/DFID-Technolo- gy-Strategy-2017-2020.pdf. DFID (UK Department for International Development). 2018. “DFID Digital Strategy 2018 to 2020: Doing Development in a Digital World.” Policy Paper, Department for International Development, London. https://assets.publishing.service.gov. uk/government/uploads/system/uploads/attachment_data/file/701443/DFID- Digital-Strategy-23-01-18a.pdf Dobbs, R., J. Manyika, and J. Woetzel. 2015. No Ordinary Disruption: The Four Global Forces Breaking All Trends. New York: Public Affairs. Dweck, C. S. 2006. Mindset: The New Psychology of Success. New York: Random House. World Bank Group Independent Evaluation Group    91 Dweck, C. S., and D. S. Yeager. 2019. “Mindsets: A View from Two Eras.” Perspectives on Psychological Science 14: 481–96. Economist. 2012. “The Last Kodak Moment?” January 14, 2012. https://www.econo- mist.com/business/2012/01/14/the-last-kodak-moment. Epstein, David. 2019. Range: Why Generalists Triumph in a Specialized World. New York: Riverhead Books. Evans-Greenwood, P., and D. Leibowitz. 2017. “Your Next Future: Capitalising on Disruptive Change.” Deloitte Insights, May 1, 2017. https://www2.deloitte. com/insights/us/en/focus/disruptive-strategy-patterns-case-studies/capitalis- ing-on-disruptive-change.html. Fu, Haishan. 2019. “Data for Development Impact: Why We Need to Invest in Data, People and Ideas.” Voices (blog), February 13, 2019. https://blogs.worldbank.org/ voices/data-for-development-impact-why-we-need-to-invest-in-data-people- and-ideas. Galloway, Scott. 2020. “The Pandemic Will Reshape the Future of Higher Education— And Many Universities May Not Make It.” Business Insider, April 3, 2020. https:// www.businessinsider.com/scott-galloway-coronavirus-pandemic-reshap- ing-higher-education-universities-may-close-2020-4. Gasser, U., and V. A. F. Almeida. 2017. “A Layered Model for AI Governance.” IEEE Internet Computing 21 (6): 58–62. doi:10.1109/mic.2017.4180835. http://nrs. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography harvard.edu/urn-3:HUL.InstRepos:34390353. Gatti, R. V., A. C. Kraay, C. Avitabile, M. E. Collin, R. Dsouza, and N. A. P. Dehnen. 2018. The Human Capital Project. Washington, DC: World Bank. http://docu- ments.worldbank.org/curated/en/363661540826242921/The-Human-Capi- tal-Project. Gentilini, Ugo, Margaret Grosh, Jamele Rigolini, and Ruslan Yemtsov, eds. 2020. Exploring Universal Basic Income: A Guide to Navigating Concepts, Evidence, and Practices. Washington, DC: World Bank. http://documents.worldbank. org/curated/en/993911574784667955/pdf/ Exploring-Universal-Basic-In- come-A-Guide-to-Navigating-Concepts-Evidence-and-Practices.pdf. George, D. 2016. “Trust and Growth in the Workplace: An Analysis of Leadership in Flat Organizations.” PhD thesis, Portland State University. https://doi. org/10.15760/honors.341. Goh, Lesly, Kai Kaiser, and Nataliya Wright. 2020. “The Rise of Open Source to Spur Agile Digital Government.” Governance for Development (blog), February 26, 2020. https://blogs.worldbank.org/governance/rise-open-source-spur-agile-dig- ital-government. Goldberg, P. 2020. “World Development Report 2021—Data for Development.” Let’s Talk Development (blog), January 7, 2020. https://blogs.worldbank.org/develop- menttalk/world-development-report-2021-data-development. Goler, L. 2017. “I’m Facebook’s Head of People—Here’s What We’re Hiring for 92 Right Now (and Why).” Fast Company, May 1, 2017. https://www.fastcompany. com/40408179/im-facebooks-head-of-people-heres-what-were-hiring-for- right-now-and-why. Goodwin, Tom. 2015. “The Battle Is for the Customer Interface.” TechCrunch, March 3, 2015. https://techcrunch.com/2015/03/03/in-the-age-of-disintermediation-the- battle-is-all-for-the-customer-interface. Green, D. 2016. “Where Are the ‘Digital Dividends’ from the ICT Revolution? The New World Development Report.” From Poverty to Power (blog), February 22, 2016. https://oxfamblogs.org/fp2p/where-are-the-digital-dividends-from-the- ict-revolution-the-new-world-development-report. Guerra, N., K. Modecki, and W. Cunningham. 2014. “Developing Social-Emotional Skills for the Labor Market: The PRACTICE Model.” Policy Research Working Paper 7123, World Bank, Washington, DC. Hallward-Driemeier, M., and G. Nayyar. 2018. Trouble in the Making? The Future of Manufacturing-Led Development. Washington, DC: World Bank. Hammond, A., and D. Young. 2018. “Can Blockchain Disrupt Gender Inequality?” Private Sector Development (blog), March 7, 2018. https://blogs.worldbank.org/ psd/can-blockchain-disrupt-gender-inequality. Hanna, N. 2019. “Lessons from Digital Economy Assessment Engagements.” Digital Development Global Practice, World Bank, Washington, DC. World Bank Group Independent Evaluation Group    93 Hanna, N. 2020. “Assessing the Digital Economy: Aims, Frameworks, Pilots, Results, and Lessons.” Journal of Innovation and Entrepreneurship 9 (1): 1–16. Harari, Y. N. 2016. Homo Deus: A Brief History of Tomorrow. New York: HarperCollins. Hatum, A., and A. M. Pettigrew. 2006. “Determinants of Organizational Flexibility: A Study in an Emerging Economy.” British Journal of Management 17 (2): 115–37. Huawei Technologies and Oxford Economics. 2017. Digital Spillover: Measuring the True Impact of the Digital Economy. Shenzhen: Huawei. https://www.huawei. com/minisite/gci/en/digital-spillover/files/gci_digital_spillover.pdf. IAD (Internal Affairs Division). 2016. Audit of the Process for the Collection of Data and Production of Indices to Report on the Twin Goals. Report IBRD FY16-01. Washington, DC: World Bank Group. Ibarra, H., A. Rattan, and A. Johnston. 2018. “Satya Nadella at Microsoft: Instilling a Growth Mindset.” London Business School, London. https://krm.vo.llnwd.net/ global/public/resources/WIN_Engage/143/LBS128p2_SR_CS_20181024.pdf. IDA (International Development Association). 2020. “Additions to IDA Resources: Nineteenth Replenishment. IDA19: Ten Years to 2030: Growth, People, Re- silience.” International Development Association, Washington, DC. http:// documents.worldbank.org/curated/en/459531582153485508/pdf/Additions-to- IDA-Resources-Nineteenth-Replenishment-Ten-Years-to-2030-Growth-People- Resilience.pdf. IDB (Inter-American Development Bank). 2018. “Emerging Economies Should Teach Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography the Curriculum of the Future.” Enfoque Educación (blog), July 26, 2018. https:// blogs.iadb.org/educacion/en/curriculumofthefuture. IFC (International Finance Corporation). 2017a. “Investor Guidelines for Respon- sible Investing Digital Financial Services.” International Finance Corporation, Washington, DC. IFC (International Finance Corporation). 2017b. “Opportunities and Risks in Digi- tal Financial Services: Protecting Consumer Data and Privacy.” International Finance Corporation, Washington, DC. IFC (International Finance Corporation). 2018. How Technology Creates Markets. Washington, DC: International Finance Corporation. https://www.ifc.org/wps/ wcm/connect/6616fd9f-854a-45bd-8588-6c3d57bec589/IFC-EMCompass-Tech- MarketsReport_FIN+2018-ForWeb.pdf?MOD=AJPERES&CVID=mdwBXRb. IMF (International Monetary Fund). 2018. “Technology and the Future of Work.” IMF Working Paper WP/18/207, IMF, Washington, DC. https://www.imf.org/en/Publi- cations/WP/Issues/2018/09/28/Technology-and-the-Future-of-Work-46203. Investment Masters Class. 2018. “Jeff Bezos Interviewed by David Rubenstein.” Investment Masters Class (blog), October 3, 2018. http://mastersinvest.com/new- blog/2018/9/29/jeff-bezos-interviewed-by-david-rubenstein. Kanbur, R. 2017. “What Is the World Bank Good For? Global Public Goods and Global Institutions.” Discussion Paper DP12090, Centre for Economic Policy Research, London. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2988848. 94 Kapur, D. 2015. “Escape from the World Bank.” CGD Policy Blogs (blog), October 27, 2015. https://www.cgdev.org/blog/escape-world-bank. Karacaoglu, Y., S. Mocan, and R. A. Halsema. 2018. “The World Bank Group’s Tech- nology and Innovation Lab, from Concept to Development: A Case Study in Leveraging an IT Department to Support Digital Transformation.” Innovations: Technology, Governance, Globalization 12 (1–2): 18–28. Kharas, H., A. Prizzon, and A. Rogerson. 2014. Financing the Post-2015 Sustainable Development Goals: A Rough Roadmap. London: Overseas Development Insti- tute. https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opin- ion-files/9374.pdf. Kotter, J. P. 2014. Accelerate: Building Strategic Agility for a Faster Moving World. Bos- ton, MA: Harvard Business Review Press. Kramer, Mark R., Rishi Agarwal, and Aditi Srinivas. 2019. “Business as Usual Will Not Save the Planet.” Harvard Business Review, June 12, 2019. https://hbr. org/2019/06/business-as-usual-will-not-save-the-planet. Lee, K. 2018. AI Superpowers: China, Silicon Valley and the New World Order. Boston, MA: Houghton Mifflin Harcourt. Lee, N. 2017. “Billions to Trillions? Issues on the Role of Development Banks in Mo- bilizing Private Finance.” Center for Global Development, November 17, 2017. https://www.cgdev.org/sites/default/files/billions-trillions-issues-role-develop- World Bank Group Independent Evaluation Group    95 ment-banks-mobilizing-private-finance.pdf. Lin, D.-Y. 2016. “What Makes Singapore’s Government a Digital Leader—And How It Might Do Better.” LinkedIn, December 5, 2016. https://www.linkedin.com/pulse/ what-makes-singapores-government-digital-leaderand-how-diaan-yi-lin. Lokshin, M. M., and Z. Sajaia. 2019. “We Have Been Here Before: Development Agen- cies and Disruptive Technologies.” Let’s Talk Development (blog), April 5, 2019. https://blogs.worldbank.org/developmenttalk/we-have-been-here-develop- ment-agencies-and-disruptive-technologies. Luo, X. 2019. “Digital Technologies Allow People to Take on Risks and Explore New Opportunities.” Let’s Talk Development (blog), January 29, 2019. https:// blogs.worldbank.org/developmenttalk/digital-technologies-allow-peo- ple-take-risks-and-explore-new-opportunities. Mansharamani, V. 2012. “All Hail the Generalist.” Harvard Business Review, June 4, 2012. https://hbr.org/2012/06/all-hail-the-generalist. Manyika, J., M. Chui, M. Miremadi, J. Bughin, K. George, P. Willmott, and M. De- whurst. 2017. A Future That Works: Automation, Employment, and Productivity. McKinsey Global Institute. https://www.mckinsey.com/featured-insights/digi- tal-disruption/harnessing-automation-for-a-future-that-works. Mazzucato, M. 2013. The Entrepreneurial State: Debunking Public vs. Private Sector Myths. London: Anthem Press. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography Mazzucato, M. 2018. Mission-Oriented Research & Innovation in the European Union: A Problem-Solving Approach to Fuel Innovation-Led Growth. Directorate-General for Research and Innovation. Brussels: European Commission. https://ec.euro- pa.eu/info/sites/info/files/mazzucato_report_2018.pdf. McDougal, L., J. Klugman, N. Dehingia, A. Trivedi, and A. Raj. 2019. “Financial Inclu- sion and Intimate Partner Violence: What Does the Evidence Suggest?” PLOS ONE 14(10): e0223721. https://journals.plos.org/plosone/article?id=10.1371/ journal.pone.0223721 Merluzzi, J., and D. J. Phillips. 2015. “The Specialist Discount: Negative Returns for MBAs with Focused Profiles in Investment Banking.” Administrative Science Quarterly 61 (1): 87–124. https://doi.org/10.1177/0001839215610365. Mühleisen, M. 2018. “The Long and Short of the Digital Revolution.” Finance & Development 55 (2). https://www.imf.org/external/pubs/ft/fandd/2018/06/im- pact-of-digital-technology-on-economic-growth/muhleisen.htm. OECD (Organisation for Economic Co-operation and Development). 2015. Dig- ital Security Risk Management for Economic and Social Prosperity: OECD Recommendation and Companion Document. Paris: OECD. http://dx.doi. org/10.1787/9789264245471-en. OECD (Organisation for Economic Co-operation and Development). 2016. “Public Governance Reviews: Skills for a High-Performing Civil Service.” OECD Public 96 Governance Reviews, OECD, Paris. https://www.oecd.org/gov/pem/Skills-High- lights.pdf. OECD (Organisation for Economic Co-operation and Development). 2017. The Next Production Revolution: Implications for Governments and Business. Paris: OECD Publishing. https://doi.org/10.1787/9789264271036-en. OECD (Organisation for Economic Co-operation and Development). 2018. Achiev- ing Inclusive Growth in the Face of Digital Transformation and the Future of Work. Paris: OECD. https://www.oecd.org/g20/summits/buenos-aires/OECD_Achiev- ing%20inclusive%20growth%20in%20the%20face%20of%20FoW.pdf. Olken, B. A. 2007. “Monitoring Corruption: Evidence from a Field Experiment in Indonesia.” Journal of Political Economy 115: 200–49. O’Neil, C. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. New York: Crown Publishing. O’Reilly, T. 2010. “Government as a Platform.” Innovations 6 (1): 13–40. https://www. mitpressjournals.org/doi/pdf/10.1162/INOV_a_00056. Pasti, F. 2018. State of the Industry Report on Mobile Money. GSMA. https://www. gsma.com/mobilefordevelopment/wp-content/uploads/2019/02/2018-State-of- the-Industry-Report-on-Mobile-Money.pdf. Pate, M. A. 2020. “COVID-19 (Coronavirus): Ensuring Fair Global Allocation of World Bank Group Independent Evaluation Group    97 Vaccines.” Investing in Health (blog), April 27, 2020. https://blogs.worldbank.org/ health/covid-19-coronavirus-ensuring-fair-global-allocation-vaccines. Pathways for Prosperity Commission. 2018. Digital Lives: Meaningful Connections for the Next 3 Billion. Oxford: Pathways for Prosperity Commission. Pathways for Prosperity Commission. 2019a. Digital Diplomacy: Technology Gover- nance for Developing Countries. Oxford: Pathways for Prosperity Commission. Pathways for Prosperity Commission. 2019b. Positive Disruption: Health and Educa- tion in a Digital Age. Oxford: Pathways for Prosperity Commission. Pauwels, E. 2019. The New Geopolitics of Converging Risks: The UN and Prevention in the Era of AI. New York: United Nations University Centre for Policy Research. https://collections.unu.edu/eserv/UNU:7308/PauwelsAIGeopolitics.pdf. Pauwels, E. 2020. The Anatomy of Information Disorders in Africa. Geostrategic Po- sitioning and Multipolar Competition Over Converging Technologies. New York: Konrad Adenauer Stiftung. Pazarbasioglu, C., J. L. Irigoyen, and A. Mehta. 2018. “African Leaders Committed to Building a Digital Economy.” Voices (blog), May 30, 2018. https://blogs.world- bank.org/voices/african-leaders-committed-to-building-digital-economy. Peralta-Alva, A., and A. Roitman. 2018. “Technology and the Future of Work.” Work- ing Paper 18/207, IMF, Washington, DC. Pisano, G. 2019. “The Hard Truth about Innovative Cultures.” Harvard Business Re- view (January–February): 62–71. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography Puerta, Sánchez, Maria Laura, Alexandria Valerio, and Marcela Gutiérrez Bernal. 2016. Taking Stock of Programs to Develop Socioemotional Skills: A Systematic Review of Program Evidence. Directions in Development Series. Washington, DC: World Bank. Ramalingam, B. 2016. “10 Frontier Technologies for International Development.” From Poverty to Power (blog), November 1, 2016. https://oxfamblogs.org/fp- 2p/10-frontier-technologies-for-international-development. Reeves, M., and M. Deimler. 2011. “Adaptability: The New Competitive Advan- tage.” Harvard Business Review (July). https://hbr.org/2011/07/adaptabili- ty-the-new-competitive-advantage. Reid, S., J. P. Hayes, and D. T. Stibbe. 2014. “Platforms for Partnership: Emerging Good Practice to Systematically Engage Business as a Partner in Development.” The Partnering Initiative, Oxford. https://www.thepartneringinitiative.org/ wp-content/uploads/2015/03/PLATFORMStealcoverallpages.pdf. Ribeiro, F., A. Giacoman, and M. Trantham. 2016. “Dealing with Market Disruption: Seven Strategies for Breaking Down Silos.” PriceWaterhouseCooper. https:// www.strategyand.pwc.com/gx/en/insights/2016/dealing-market-disruption/ dealing-with-market-disruption.pdf. Robinson, K. 2009. The Element: How Finding Your Passion Changes Everything. New York: Penguin Books. 98 Rodrik, D. 2018a. “New Technologies, Global Value Chains, and the Developing Econo- mies.” Background Paper Series 1, Pathways for Prosperity Commission, Oxford. Rodrik, D. 2018b. “Will New Technologies Help or Harm Developing Countries?” GP Opinion (blog), November 16, 2018. https://www.globalpolicyjournal.com/ blog/16/11/2018/will-new-technologies-help-or-harm-developing-countries. Rosenblatt, M. 2011. “The Use of Innovation Awards in the Public Sector: Indi- vidual and Organizational Perspectives.” Innovation: Management, Policy & Practice 13 (2): 207–219. https://www.tandfonline.com/doi/abs/10.5172/ impp.2011.13.2.207. Runde, D. F. 2013. “The Future of Public-Private Partnerships: Strengthening a Powerful Instrument for Global Development.” Center for Strategic and International Studies, October 25, 2013. https://www.csis.org/analysis/fu- ture-public-private-partnerships-strengthening-powerful-instrument-glob- al-development. Saavedra, J. 2020. “Educational Challenges and Opportunities of the Coronavirus (COVID-19) Pandemic.” Education for Global Development (blog), March 30, 2020. https://blogs.worldbank.org/education/educational-challenges-and-opportuni- ties-covid-19-pandemic. Sahni, N. R., M. Wessel, and C. M. Christensen. 2013. “Unleashing Breakthrough Innovation in Government.” Stanford Social Innovation Review (Summer): 27–31. World Bank Group Independent Evaluation Group    99 https://ssir.org/articles/entry/unleashing_breakthrough_innovation_in_govern- ment. Sheffi, Y. 2007. The Resilient Enterprise: Overcoming Vulnerability for Competitive Ad- vantage. Cambridge, MA: MIT Press. Singer, Katie. 2014. An Electronic Silent Spring: Facing the Dangers and Creating Safe Limits. Great Barrington, MA: Portal Books. Skoll World Forum. 2013. “How Does the World Bank Think about Scaling In- novation?” Forbes, September 4, 2013. https://www.forbes.com/sites/ skollworldforum/2013/09/04/how-does-the-world-bank-think-about-scaling- innovation/#655a69654c06. Smedlund, A., A. Lindblom, and L. Mitronen, eds. 2018. Collaborative Value Co-creation in the Platform Economy. Translational Systems Sciences 11. Singapore: Springer. Smith, B., and H. Shum. 2018. “The Future Computed: Artificial Intelligence and Its Role in Society.” Official Microsoft Blog (blog), January 17, 2018. https://blogs.mic- rosoft.com/blog/2018/01/17/future-computed-artificial-intelligence-role-society. Stanley, A. 2017. “‘End Poverty, Restore Trust’: World Bank Dives into Blockchain with Lab Launch.” Coindesk, June 28, 2017. https://www.coindesk.com/end-pov- erty-restore-trust-world-bank-dives-into-blockchain-with-lab-launch. Stibbe, D. 2013. A New Global Partnership with Business: Building a Post-2015 Devel- opment Framework to Achieve Sustainable Prosperity in Africa. Business Action for Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography Africa, Harvard Kennedy School, and The Partnering Initiative. https://www.hks. harvard.edu/sites/default/files/centers/mrcbg/programs/cri/files/CSRI_BAA_TPI_ New_Global_Partnership_With_Business_Sept2013.pdf. Teller, A. 2016. “The Head of ‘X’ Explains How to Make Audacity the Path of Least Resistance.” Wired, April 15, 2016. https://www.wired.com/2016/04/the-head- of-x-explains-how-to-make-audacity-the-path-of-least-resistance. Tetlock, P. E. 2006. Expert Political Judgment: How Good Is It? How Can We Know? Princeton, NJ: Princeton University Press. UNDP Global Centre for Public Service Excellence. 2018. Foresight Manual: Empow- ered Futures for the 2030 Agenda. Singapore: UNDP. UNESCO (United Nations Educational, Scientific, and Cultural Organization). 2015. UNESCO Science Report: Towards 2030. Paris: UNESCO. United Nations Department of Economic and Social Affairs. 2018. World Economic and Social Survey 2018: Frontier Technologies for Sustainable Development. New York: United Nations. USAID (US Agency for International Development). 2018a. “Risk Appetite State- ment.” USAID, Washington, DC. https://www.usaid.gov/sites/default/files/docu- ments/1868/USAID_Risk-Appetite-Statement_Jun2018.pdf. USAID (US Agency for International Development). 2018b. “USAID and AI— Reflecting the Past, Shaping the Future: Making AI Work for International De- 100 velopment.” USAID, Washington, DC. https://www.usaid.gov/sites/default/files/ documents/15396/AI_ExecutiveSummary-Digital.pdf. van de Pol, P. 2017. “Africa and Foresight: Better Futures in Development.” Working Paper, Global Centre for Public Service Excellence, United Nations Development Programme, Singapore. https://www.undp.org/content/dam/undp/library/ca- pacity-development/English/Singapore%20Centre/UNDP-GCPSE_Foresight_for_ Africa.pdf. van Tulder, R., M. M. Seitanidi, A. Crane, and S. Brammer. 2016. “Enhancing the Impact of Cross-Sector Partnerships: Four Impact Loops for Channeling Part- nership Studies.” Journal of Business Ethics 135: 1–17. doi:10.1007/s10551-015- 2756-4. Vota, W. 2018. “Wow! Mobile Money Transactions Are Now Larger Than Kenya’s GDP.” ICTworks, May 16, 2018. https://www.ictworks.org/mobile-money-larg- er-kenya-gdp. Walker, R. 2017. “Insourcing vs. Outsourcing: How to Strike the Right Balance.” Deluxe (blog), September 5, 2017. https://fi.deluxe.com/community-blog/finan- cial-marketing-insights/insourcing-vs-outsourcing-strike-right-balance. Watkins, A. 2018. “From Lab to the Last Mile: Technology Deployment Business Models for the SDGs.” Presentation for United Nations Commission on Science and Technology for Development, Geneva, May 16, 2018. https://unctad.org/ World Bank Group Independent Evaluation Group    101 meetings/en/Presentation/enc162018p13_Watkins_en.pdf. Watkins, A. 2019. “Disruptive Inclusive Innovation: The Imperative of Tech- nology Deployment.” Presentation for United Nations Global Solutions Summit, New York, May 13, 2019. http://www.globalsolutionssummit.com/up- loads/3/1/5/5/31554571/watkins.pdf. Weiss, C. 2006. “Science and Technology at the World Bank, 1968–83.” History and Technology 22 (1): 81–104. Whittle, D. 2013. “How Feedback Loops Can Improve Aid (and Maybe Governance).” Center for Global Development, August 13, 2013. https://www.cgdev.org/publi- cation/how-feedback-loops-can-improve-aid-and-maybe-governance. World Bank. 2011a. Capturing Technology for Development: An Evaluation of World Bank Group Activities in Information and Communication Technologies. Indepen- dent Evaluation Group. Washington, DC: World Bank. World Bank. 2011b. Learning for All: Investing in People’s Knowledge and Skills to Pro- mote Development. Education Strategy 2020. Washington, DC: World Bank. World Bank. 2012a. The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2012b. World Development Report 2013: Jobs. Washington, DC: World Bank. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography World Bank. 2013. World Bank Group Support for Innovation and Entrepreneurship. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2014a. Learning and Results in World Bank Operations: How the Bank Learns. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2014b. The World Bank Group and Public Procurement: An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2015a. Learning and Results in World Bank Operations: Toward a New Learning Strategy. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2015b. World Bank Group Gender Strategy (FY16–23): Gender Equality, Poverty Reduction, and Inclusive Growth. Washington, DC: World Bank. World Bank. 2015c. World Bank Group Support to Electricity Access, FY2000–2014. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.world- bankgroup.org/sites/default/files/Data/Evaluation/files/Electricity_Access.pdf. World Bank. 2016a. Forward Look—A Vision for the World Bank Group in 2030. Washington, DC: World Bank. http://documents1.worldbank.org/curated/ en/620311493344873007/pdf/03-14-17-COW-SUMMARY-FINAL-03222017.pdf. World Bank. 2016b. Reliable and Affordable Off-Grid Electricity Services for the Poor: Lessons from the World Bank Group Experience. Learning Product. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup. org/sites/default/files/Data/reports/lp_off-grid_electricity_1116.pdf. 102 World Bank. 2016c. Supporting Transformational Change for Poverty Reduction and Shared Prosperity: Lessons from World Bank Group Experience. Independent Eval- uation Group. Washington, DC: World Bank. World Bank. 2016d. World Bank Group. Internal Audit Vice Presidency. Annual Report. Washington, DC: World Bank. World Bank. 2016e. World Development Report 2016: Digital Dividends. Washington, DC: World Bank. World Bank. 2017a. “Maximizing Finance for Development: Leveraging the Private Sector for Growth and Sustainable Development.” Development Committee Pa- per DC2017-0009, World Bank, Washington, DC. https://www.devcommittee.org/ sites/dc/files/download/Documentation/DC2017-0009_Maximizing_8-19.pdf. World Bank. 2017b. Poland—Toward a Strategic, Effective, and Accountable State. Systematic Country Diagnostic. Washington, DC: World Bank. http://documents. worldbank.org/ curated/en/877211502112969688/Poland-Systematic-Coun- try-Diagnostic-Toward-a-Strategic-Effective-and-Accountable-State. World Bank. 2017c. “Summary, Unearthing Innovation in the World Bank: A Report from the Innovation Labs.” Innovation Acceleration Program, World Bank, Washington, DC. World Bank. 2018a. Competing in the Digital Age: Policy Implications for the Russian Federation. Washington, DC: World Bank. https://openknowledge.worldbank. World Bank Group Independent Evaluation Group    103 org/bitstream/handle/10986/30584/AUS0000158-WP-REVISED-P160805-PUB- LIC-Disclosed-10-15-2018.pdf. World Bank. 2018b. “Disruptive Technologies and the World Bank Group: Creating Opportunities—Mitigating Risks.” Development Committee Paper DC2018- 0010, World Bank, Washington, DC. https://www.devcommittee.org/sites/www. devcommittee.org/files/download/Documents/2018-09/DC2018-0010%20Dis- ruptive%20Technologies.pdf. World Bank. 2018c. Morocco—Governing Towards Efficiency, Equity, Education and Endurance. Systematic Country Diagnostic. Washington, DC: World Bank. World Bank. 2018d. Poland—Country Partnership Framework for the Period FY19–24. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/ 209111528428654498/Poland-Country-partnership-framework-for-the-period- of-FY19-24. World Bank. 2018e. Senegal. Systematic Country Diagnostic. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/336611539873310474/Sene- gal-Systematic-Country-Diagnostic. World Bank. 2018f. “Sustainable Financing for Sustainable Development: World Bank Group Capital Package Proposal.” Paper prepared for Development Com- mittee Meeting, April 21, 2018, Washington, DC. World Bank. 2018g. “The Bali Fintech Agenda: A Blueprint for Successfully Harness- ing Fintech’s Opportunities.” Press Release, October 11, 2018. https://www. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography worldbank.org/en/news/press-release/2018/10/11/bali-fintech-agenda-a-blue- print-for-successfully-harnessing-fintechs-opportunities. World Bank. 2018h. World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank. World Bank. 2019a. Digital Economy for Africa (DE4A) Moonshot (ID: P169935). Con- cept Note. Washington, DC: World Bank. World Bank. 2019b. MNA Tech Initiative (ID: P171209). Concept Note. Washington, DC: World Bank. World Bank. 2019c. Digital Economy for Africa (DE4A). Country Diagnostic of Sene- gal. Washington, DC: World Bank. http://documents1.worldbank.org/curated/ en/814111559645179494/pdf/Country-Diagnostic-of-Senegal.pdf. World Bank. 2019d. “E-Commerce Can Boost Job Creation and Inclusive Growth in Developing Countries.” Press Release, November 23, 2019. https://www.world- bank.org/en/news/press-release/2019/11/23/e-commerce-can-boost-job-cre- ation-and-inclusive-growth-in-developing-countries. World Bank. 2019e. Identification for Development (ID4D) 2019 Annual Report. Wash- ington, DC: World Bank. World Bank. 2019f. “Internal and Client Digital Transformation: An Update.” Techni- cal Briefing, Corporate Secretariat, World Bank, Washington, DC. 104 World Bank. 2019g. Knowledge Flow and Collaboration under the World Bank’s New Operating Model. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019h. Mainstreaming Paper. “Mainstreaming the Approach to Dis- ruptive and Transformative Technologies at the World Bank Group.” Develop- ment Committee Paper DC2019-0002, World Bank, Washington, DC. https:// www.devcommittee.org/sites/www.devcommittee.org/files/download/Docu- ments/2019-04/DC2019-0002-Mainstreaming%20disruptive%20%204-13.pdf. World Bank. 2019i. Morocco—Country Partnership Framework for the Period FY19– FY24. Washington, DC: World Bank. World Bank. 2019j. “Review of the Bank’s Management of Corporate Data Used in Operations.” Internal Audit Vice Presidency, World Bank, Washington, DC. World Bank. 2019k. “Using ‘Pocket Science’ to Monitor and Supervise Operations in Fragile and Conflict-Affected Settings.” World Bank, November 11, 2019. https://www.worldbank.org/en/news/feature/2019/11/11/using-pocket-sci- ence-to-monitor-and-supervise-operations-in-fragile-and-conflict-affect- ed-settings. World Bank. 2019l. World Bank Group HR Strategy FY20–22. Washington, DC: World Bank. World Bank. 2019m. World Development Report 2019: The Changing Nature of Work. Washington, DC: World Bank. World Bank Group Independent Evaluation Group    105 World Bank. 2020a. Mainstreaming Digital and Disruptive Technologies Initiative (ID: P175132). Concept Note, World Bank, Washington, DC: World Bank. World Bank. 2020b. Human Capital Project HCI Compass. Washington, DC: World Bank. World Bank. 2020c. Senegal—Country Partnership Framework for the Period FY20– FY24. Washington, DC: World Bank. https://openknowledge.worldbank.org/han- dle/10986/33439. World Bank. 2020d. “The Final Program Document for Mainstreaming Digital and Disruptive Technologies (‘MDDT’) Initiative.” World Bank, Washington, DC. World Bank. 2020e. World Development Report 2020: Trading for Development in the Age of Global Value Chains: Overview. Washington, DC: World Bank. World Bank. 2021. World Development Report 2021: Data for Better Lives. Washington, DC: World Bank. World Economic Forum. 2018. The Future of Jobs Report 2018. Geneva: World Eco- nomic Forum. http://www3.weforum.org/docs/WEF_Future_of_Jobs_2018.pdf. World Wide Web Foundation. 2015. Women’s Rights Online: Translating Access into Empowerment. Washington, DC: World Wide Web Foundation. http://webfoun- dation.org/docs/2015/10/womens-rights-online21102015.pdf. Yeager, D. S., P. Hanselman, G. M. Walton, J. Murray, R. Crosnoe, C. Muller, E. Tipton, B. Schneider, C. S. Hulleman, C. P. Hinojosa, D. Paunesku, C. Romero, K. Flint, A. Roberts, J. Trott, R. Iachan, J. Buontempo, S. Man Yang, C. M. Carvalho, P. R. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Bibliography Hahn, M. Gopalan, P. Mhatre, R. Ferguson, A. L. Duckworth, and C. S. Dweck. 2019. “A National Experiment Reveals Where a Growth Mindset Improves Achievement.” Nature 573: 364–69. 106 APPENDIXES Independent Evaluation Group Mobilizing Technology for Development Appendix A. Methodological Approach Evaluation Question The evaluation sought to answer the following question, How well prepared is the World Bank Group to help clients harness the opportunities and mitigate the risks posed by disruptive and transformative technologies (DTT)? Thus, the evaluation’s focus was on the Bank Group’s preparedness to sup- port clients rather than on an assessment of the effectiveness of Bank Group support to clients based on ex post evaluation methods and metrics or the Bank Group’s adoption of technology for its own internal information tech- nology systems. Concept of Disruptive and Transformative Technologies The evaluation adopted the broad concept of DTT in line with the 2018 and 2019 Development Committee papers (World Bank 2018, 2019b [the latter World Bank Group Independent Evaluation Group    109 henceforth called the “Mainstreaming paper”]). The Mainstreaming paper states that disruptive technologies can be defined as emerging technologies that “result in a step change in the cost or access to products or services, or that dramatically change how we gather information, make products, or interact” (World Bank 2019b, iv). The Mainstreaming paper outlines five corporate priorities for mainstreaming the Bank Group’s approach to disrup- tive technologies: (i) country diagnostics; (ii) agile regulation; (iii) digital connectivity; (iv) digital government services; and (v) skills and capabilities for the new economy and the role of education. The Mainstreaming paper also adds sectoral and regional programs and programmatic approaches to the five corporate priorities. The Bank Group’s approach to DTT is discussed in more detail in appendix B. The evaluation is aligned with the concept and scope of DTT introduced in the Mainstreaming paper. Accordingly, in addition to digital technologies, it covers support to new and existing technologies (and their use) and the enabling policies, institutions, and skills and the lending and nonlending operations with significant DTT components or that are a response to the implications of DTT. In particular, the evaluation examined the preparedness of the World Bank and the International Finance Corporation (IFC) in each of the five Bank Group corporate priorities and their preparedness with regard to sectoral and regional programs for mainstreaming DTT. It also reviewed fintech and digital entrepreneurship, whose relevance for the DTT approach was highlighted in the Bali Fintech Agenda. The Challenges section in this Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A appendix discusses issues arising from the Bank Group’s definition of DTT. Overarching Principles Apart from the concept of DTT, two central principles motivated the evalua- tion design. First, the evaluation was grounded in a conceptual framework to assess the Bank Group’s preparedness to help clients harness the opportuni- ties and mitigate the risks posed by DTT, based on the preparedness dimen- sions emerging from the organizational effectiveness literature and relevant Independent Evaluation Group (IEG) evaluation findings. These dimensions are critical for organizations to adjust under conditions of uncertainty. The IEG team constructed the conceptual framework (figure A.1) through an iterative process with the IEG methods team and validated it with key stakeholders. Second, the evaluation followed a mixed-methods approach, combining a range of methods for data collection and analysis, and applied systematic triangulation to ensure the robustness of the findings. 110 Figure A.1. Framework for Analysis Source: Independent Evaluation Group; Development Committee papers (World Bank 2018, 2019b); organizational effectiveness literature. Note: MNA = Middle East and North Africa. World Bank Group Independent Evaluation Group    111 Evaluation Components Table A.1 lists the evaluation components. The rest of the section provides more detail on each component and how it relates to the thematic reviews undertaken by IEG for the Bank Group’s DTT corporate priorities. The sec- tion that follows it provides details on how the validity of the findings was ensured. The last section delves into the challenges faced by this evaluation. Table A.1. Evaluation Components Evaluation Component Description Literature review Several reviews were undertaken: (i) literature relating to disruptive technologies and their effects; (ii) literature on Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A organizational effectiveness; (iii) existing IEG evaluations; (iv) Bank Group documents on technology and innovation; and (iv) policy papers from other development institutions. Document and database A review of (i) the Bank Group’s information systems and review databases in the five corporate priorities, as well as fintech and digital entrepreneurship, and (ii) selected sectoral and regional programs. Review of project documents for a purposive sample of World Bank and IFC operations for corporate priority reviews in digital connectivity, digital government services, and fintech and entrepreneurship. Extensive review of all DTT-related ASA interventions in education and gender and of SCDs and CPFs for coun- tries referred to in the Mainstreaming paper (World Bank 2019b). Assessment of staff skills A collaborative effort between IEG and DEC to assess the extent of Bank Group preparedness with regard to staff skills to help clients harness the opportunities and miti- gate the risks posed by DTT. Semistructured interviews Semistructured interviews of 105 Bank Group staff, includ- ing staff and managers at headquarters and in country offices. Qualitative analysis using Qualitative analysis of the 105 semistructured Bank Group NVivo staff interviews based on a coding protocol and keyword search to extract information on staffing, internal col- laboration, and incentives (the findings are presented in relevant chapters of this report). External stakeholder inter- Interviews with 57 clients and external stakeholders, views including government officials, staff in nongovernmental organizations, private sector stakeholders, other donor agency staff, and researchers. Source: Independent Evaluation Group. 112 Note: ASA = advisory services and analytics; CPF = Country Partnership Framework; DEC = Development Economics Vice Presidency; DTT = disruptive and transformative technologies; IEG = Independent Eval- uation Group; IFC = International Finance Corporation; SCD = Systematic Country Diagnostic. Literature Review The evaluation conducted a literature review to (i) understand and concep- tualize the impact of DTT on inclusive development, including the interre- lationships among disruptive technologies, digital technologies, the digital economy, and inclusive development; (ii) identify the key dimensions of organizational effectiveness, particularly as they pertain to DTT; (iii) ex- plore the Bank Group’s preparedness to address DTT through reviewing past strategies, policy papers, analytical internal World Bank documents, and relevant IEG evaluations; and (iv) review the experience of development or- ganizations relating to DTT, especially in distilling good practices for dealing with change, uncertainty, innovation, and risk, based on reports and policy documents from other development partners, including the Organisation for Economic Co-operation and Development, United Nations, Internation- al Monetary Fund, UK Department for International Development, United States Agency for International Development, consulting firms, and the scholarly literature from academia and think tanks. Document and Database Reviews The document and database review was conducted on two levels: (i) cor- porate level to allow for an assessment of the organizational enablers and constraints for harnessing opportunities and mitigating risks posed by World Bank Group Independent Evaluation Group    113 disruptive technologies and (ii) intervention level to draw conclusions about the Bank Group’s current preparedness based on a purposive sample of in- terventions in the corporate priorities, weighted toward more recent inter- ventions. On the corporate level, the review of documents included Bank Group country and regional diagnostics, program documents (for example, Digital Economy for Africa Moonshot/Accelerate), and IEG higher-level evaluations. The database review covered Bank Group databases, such as for human resources, budgets, skills mix, incentives, and rewards (covered in the respective chapters of this report). For interventions, relevant Bank Group documents were reviewed: (i) all advisory services and analytics (ASA) for education and gender for the past five years (support to skills and capabili- ties and education is one of the five corporate priories, and Gender is a Bank Group Global Theme—details are presented in appendix C); (ii) selected project documents for China, India, and Vietnam as part of country valida- tion visits; and (iii) project approval, supervision, and closing or evaluation documents, as well as IEG evaluations for selected projects in digital con- nectivity, digital government, and fintech and digital entrepreneurship. The review of documents and databases was used to distill contextualized find- ings from case-based analysis on both the corporate and intervention level to illustrate specific dimensions of preparedness and to explore the Bank Group’s comparative advantage in DTT. Assessment of Staff Skills Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A IEG and the Development Economics, Development Data Group collaborated to assess the extent to which the Bank Group was prepared with regard to its staff skills to help clients harness the opportunities and mitigate the risks posed by DTT. To do so, IEG and the Development Economics Vice Presi- dency examined the Bank Group’s taxonomy and definitions of staff skills, together with relevant data from SkillFinder, People Profile pages, staff cur- riculums vitae, human resources, and other institutional sources. The assess- ment also examined the Bank Group’s Communities of Practice and Yammer groups since these would conceivably offer an insight into how many staff were sharing their interest or experiences in working on DTT. Semistructured Interviews of Bank Group Staff Semistructured interviews included the following stakeholders: (i) World Bank staff in Global Practices, Regions, country offices, and IFC; (ii) in-coun- try stakeholders, such as Bank Group clients, government, and the private sector through selected country visits and remote interviews of stakeholders in selected Bank Group projects (discussed further in the External Stakehold- er Interviews section in this appendix); and (iii) donors, partner agency staff, and private sector stakeholders. The purpose of the semistructured inter- views was to gain insights into what Bank Group staff, clients, and external stakeholders perceived as organizational enablers and constraints within 114 the Bank Group for harnessing opportunities and mitigating risks posed by disruptive technologies. Qualitative Analysis Using NVivo Qualitative analysis was conducted using NVivo qualitative data analysis software of interviews of 105 Bank Group staff (including task team leaders or investment officers, managers, and directors from Global Practices or IFC departments and Country Management Units) in headquarters and country offices (tables A.2 and A.3). The team formulated a coding scheme to identify views expressed on each of the following issues: staffing, internal collabora- tion, and incentives (which reflect three chapters in this report). The coding scheme consisted of three levels of codes: first-level codes (parent codes) and second- and third-level subcodes (child codes). The team then used the coding scheme and keywords identified for each code to analyze the inter- views. The team consulted with sectoral experts to ensure that the keywords captured the language and terminology they used for DTT-related topics. Finally, the team analyzed the codes using NVivo features like cross-tabs and matrix coding to evaluate the responses. The interviewees were categorized by their position title—whether they were staff or management (directors and managers). The complete coding information for the NVivo analysis can be found in table A.8 in this appendix. World Bank Group Independent Evaluation Group    115 Table A.2. Interview Respondents by Position Title Title Count Percent Nonmanagerial staff 86 82 Managers (directors + managers) 19 18 Total 105 100 Source: Independent Evaluation Group. Table A.3. Interview Respondents by Location Location Count Percent Headquarters 58 55 Country office 47 45 Total 105 100 Source: Independent Evaluation Group. External Stakeholder Interviews Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A The evaluation team engaged with external stakeholders on several levels. First, the team interviewed a wide array of stakeholders, including gov- ernment officials, staff in nongovernmental organizations, private sector stakeholders, other donor agency staff, and researchers to supplement the literature review in assessing (i) the impact of DTT on inclusive develop- ment, including the interrelationships among disruptive technologies, digital technologies, the digital economy, and inclusive development, and (ii) the perceived role of the Bank Group at the nexus of technology and devel- opment. Second, the interviews of external stakeholders also aimed to (i) determine the extent and nature of client demand for Bank Group support for disruptive technologies and (ii) identify organizational good practices in specific areas that the Bank Group can benefit from and enablers for benefi- cial partnerships with external organizations or companies. Thus, the team conducted in-person interviews during three country valida- tion visits (in India, China, and Vietnam) to triangulate its findings. It also conducted remote client interviews related to intervention-level analysis for projects in digital connectivity, digital government services, and fintech and digital entrepreneurships. This ensured better triangulation of the findings per corporate priority and allowed the team to reach a point of theoretical saturation with the number of interviews it was able to conduct. This point was reached after an iterative process of triangulation between interview findings and findings from desk reviews until additional interviews did not add any new perspectives or cast any doubts on the triangulated findings. In some cases where a point of saturation was not reached (for example, due to 116 low numbers of interviews or high levels of divergence across interviews), the team took this into account in the formulation of findings (by using more tentative language) or decided that not enough evidence was available to assess a particular issue. DTT Corporate Priority Reviews The team conducted thematic reviews in the five corporate priorities iden- tified in the Mainstreaming paper (World Bank 2019b) and in fintech and digital entrepreneurship. The thematic reviews were conducted based on a combination of the components discussed in the previous section. The tem- plates and protocols for conducting the document reviews, semistructured interviews, and qualitative analysis using NVivo are included in this appen- dix (tables A.5–A.11), and some of the projects reviewed for the interven- tion-level analysis are listed in tables A.12–A.15. Corporate Priority 1: Country Diagnostics Preparation of the Country Diagnostics thematic review involved, first, a re- view of the Systematic Country Diagnostics and Country Partnership Frame- works for Morocco, Poland, and Senegal (all three countries were showcased in the Mainstreaming paper); second, a review of the Digital Economy Diag- nostics paper produced for the Digital Development Global Practice; third, World Bank Group Independent Evaluation Group    117 a review of approaches to country benchmarking, assessing the new digital business indicators and relevant lessons from the Doing Business series of studies; and, fourth, interviews with five World Bank staff working on coun- try and digital economy diagnostics to extend IEG’s review of documents, which formed the main analysis in determining the links between country diagnostics and the twin goals. Corporate Priority 2: Agile Regulation Preparation of the agile regulation thematic review involved, first, an ex- tensive review of academic and think tank literature from outside the Bank Group; second, two separate searches of the Bank Group database: identi- fying all lending projects and ASA marked as digital markers for fiscal years (FY)15–19 using keywords approved by the Global Practices, and identifying projects and ASA listed in Image Bank for FY19 using the terms digital and regulat; third, a detailed examination of project documents to examine prog- ress with regulatory reform in a series of West and Central African telecoms projects; and, fourth, interviews of 11 Bank Group staff and (for bioengineer- ing) staff from the International Food Policy Research Institute. Corporate Priority 3: Digital Connectivity (World Bank and IFC) The assessment of the Bank Group’s preparedness to scale up universal affordable digital connectivity included a literature review, review of World Bank and IFC corporate databases, review of project documents, staff inter- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A views at the corporate and project level, and interviews of internal stake- holders. IEG selected a purposive sample of 50 interventions to include different modalities of support and geographic diversity (25 in each of the World Bank and IFC). The list of projects is available in tables A.12 (World Bank) and A.13 (IFC). The sample included both ASA and lending operations. For the World Bank, the sample was drawn from a list of operations provid- ed by the Digital Development Global Practice and identified from Business Intelligence, comprising operations approved between FY11 and FY19 that either were led by the Digital Development Global Practice or included a significant component of the digital development theme. The sample in- cluded completed and evaluated interventions, mature interventions (with at least one Implementation Status and Results Report or progress report on file) and recent approvals. IEG oversampled more recent approval years to account for the evolving nature of Bank Group support. IEG then interviewed a selection of the World Bank’s senior managers, corporate partners, 10 of the task team leaders involved with the selected sample of projects, and 8 client counterparts. For IFC, the team looked into the four digital infrastruc- ture “verticals” supported by IFC: mobile network operators, shared infra- structure (that is, towers), data centers, and broadband. IEG interviewed IFC managers, corporate partners, and 10 investment officers involved with the selected sample of projects. 118 Corporate Priority 4: Digital Government Services The priority review of the World Bank’s readiness to support the provision of transparent, efficient, and accountable government services focused on support to digital ID as one of the key disruptive technologies that enable access to services and development opportunities. The review examined how digital IDs either catalyzed or interacted synergistically with related services to allow transformative interventions in World Bank operations. Since social protection services were the most frequent (although not the only) users of digital ID systems, the review focuses mainly on the adoption of digital tech- nologies in social protection programs. The assessment of the World Bank’s readiness included a literature review, a review of project and program doc- uments, and semistructured interviews of managers and task team leaders in three Global Practices (Digital Development, Jobs and Social Protection, and Governance) and the Identification for Development program. IEG selected a purposive sample of six operations involving digital systems for Identifica- tion for Development and/or the delivery of social protection services from a universe of 18 relevant operations approved between FY12 and FY20 identi- fied from World Bank documents and staff interviews. Four operations were in the Sub-Saharan Africa Region and one each in the Middle East and North Africa and South Asia Regions. The desk review was complemented by 12 semistructured interviews (including the task team leaders for the reviewed operations, managers, and Identification for Development program staff). World Bank Group Independent Evaluation Group    119 Corporate Priority 5: Skills Development and Capabilities (Education) Preparation of the education thematic review involved, first, an extensive review of academic and think tank literature from outside the Bank Group; second, a review of education projects and ASA approved during FY15–19 that were DTT-related (the methodology is described in detail in the next para- graphs and in tables A.9–A.11 of this appendix); third, a review of key Bank Group documents on the Human Capital Project; and, fourth, interviews of EdTech fellows and World Bank staff in the Education Global Practice. Since the World Bank does not maintain a list of DTT-related projects and ASA in education, IEG identified these in collaboration with the Education Global Practice. The relevant projects and ASA had at least one DTT-relevant objective, component, or indicator (or discussion) that met the following criteria: 1. They “applied” DTT to improve education (for example, by using computers in the classroom). 2. They used education to “respond” to DTT to prepare people for a world disrupted by DTT by imparting twenty-first century skills, such as (i) devel- oping or producing DTT (for example, by teaching advanced cognitive skills, including higher-order problem solving and creative thinking); (ii) using or consuming DTT (for example, by teaching digital literacy); (iii) preparing people for new jobs in which humans have the comparative advantage (for Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A example, by teaching socioemotional skills, including empathy, curiosity, intrinsic motivation, grit, communication, teamwork and collaboration, leadership, global awareness, self-regulation, conflict resolution, relation- ship management, and, in particular, a growth mindset or the ability to “learn to learn”—to be lifelong learners and to adapt to change). The evaluation team identified 217 keywords (see table A.10) based on a structured literature review and World Bank and IEG reports (see table A.9 for documents reviewed) and consultation with education experts in the Global Practice. The evaluation team then conducted a multiple keyword search using R analytics on all 163 lending projects and 403 ASA approved in the past five fiscal years (FY15–19), identifying 119 lending projects and 332 ASA with at least one keyword. The evaluation team then conducted a desk review of the 119 projects and 332 ASA using a coding protocol (see table A.11). The desk review screening resulted in 57 lending projects and 62 ASA, which were validated with education experts in the Global Practice (table A.4).1  ducation and DTT-Related Lending Projects and ASA Identified Table A.4. E Total Projects and ASA Projects and ASA Product (approved FY15–19) after Keyword Search after Desk Review Projects 163 119 57 ASA 403 332 62 120 Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; DTT = disruptive and transformative technologies; FY = fiscal year. Fintech and Digital Entrepreneurship (World Bank and IFC) For this assessment of the Bank Group’s preparedness in fintech and digital entrepreneurship, the IEG team selected, after consultations with manage- rial and operational staff from the Finance, Competitiveness, and Innova- tion Global Practice and the Fintech and Payments unit at IFC, a purposive sample of 23 interventions. The sample included both lending or investment operations and analytical work approved between FY11 and FY18, covering different modalities of support and geographic diversity (the list of projects is in table A.15). IEG then conducted document reviews and semistructured interviews of 10 staff based on a standardized template (table A.7) and sever- al clients and donors of the same interventions. The inputs and additional information obtained from the literature review were used to complete the standardized template (table A.6) and produce a background paper, whose findings fed into the chapters of this report. Ensuring the Validity of Findings IEG took several steps to ensure a consistent approach across evaluation team members, for example using interview templates and protocols to ensure a common framework and evaluative lens across studies. Similarly, IEG secured World Bank Group Independent Evaluation Group    121 interrater reliability across team members charged with coding interview tran- scripts. Furthermore, the team applied triangulation at multiple levels, first by cross-checking evidence sources within a given methodological component. Within the thematic reviews, for example, the team compared and contrasted evidence from interviews (used in the NVivo qualitative analysis) with project document reviews, reviews of strategies and policy notes, previous IEG evalua- tions, and interviews of external stakeholders, such as clients. In addition, the team applied triangulation across evaluation components to ensure a multi- tude of independent references, for example cross-validating findings from the thematic reviews for each of the corporate priorities (which included semi- structured interviews and document and database reviews) with the literature review, interviews of clients and donors during country validation visits, and remote intervention-level client interviews. The evaluation team also applied external validation mechanisms at various intervals during the evaluation process. For example, the team identified activities to review and staff to interview through an iterative process in dialogue with staff involved in DTT. The team also organized workshops with key Bank Group stakeholders at the beginning of the evaluation process to validate the scope and the approach, and at the end to ensure the relevance and feasibility of the evaluation recommendations. Ensuring the validity of findings for the analysis of collaboration and coordi- nation included the following five factors: » Reference population. The reference population for the collaboration anal- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A ysis was Bank Group staff with an informed view on DTT and collaboration, rather than all Bank Group staff. Although the exact number of staff in the reference population was not known, IEG deemed 45 interviews to be an ap- propriate number in absolute terms. » Strategic selectivity. The strategic selectivity exercised in identifying the interviewees added to the validity of the findings. The 105 interviewees, of whom 45 expressed an opinion on collaboration, were selected using the following criteria: (i) those with corporate responsibility for digital develop- ment and DTT, including those involved in preparing the two DTT Develop- ment Committee papers; (ii) those working both at the corporate or strategic level of Global Practices and at the level of implementing DTT lending and nonlending operations (covering relevant staff, task team leaders, investment officers, practice managers, country directors, regional directors, and Global Practices directors); (iii) those managing World Bank-wide DTT networks or initiatives (for example, Geo-Enabling for Monitoring and Supervision, Geospatial Operations Support Team, Development Data Partnership, and Technology and Innovation Lab in Information and Technology Solutions); and (iv) champions or experts on specific DTT topics (for example, regulatory reform or socioemotional skills). » Convergence level. Of the 46 interviewees who expressed an opinion on collaboration, 37 interviewees—or 80 percent—reported “insufficient collabo- ration,” which indicates a high degree of convergence. 122 » Triangulation. Triangulating with Bank Group corporate reports, notably the FY20–22 Human Resources Strategy (World Bank 2019c) and the Mainstream- ing paper (World Bank 2019b)—both of which have acknowledged the need for greater collaboration with respect to emerging technologies and DTT—added to the validity of IEG’s findings from the interview data. Several IEG evaluations have also pointed to the need for greater collaboration within the Bank Group, two specifically with regard to work on technology. IEG’s evaluation Capturing Technology for Development highlighted the importance of institutional coor- dination for the information and communication technology agenda, recom- mending “building incentives mechanisms for collaboration, coordination, and joint approaches for innovation between Bank Group units, reflecting the thematic nature of ICT [information and communication technology]” (World Bank 2011, xvii). It also noted the need for supporting “cross-sectoral enablers, including… policies and standards that would apply across agencies… to effec- tively lead the ICT [information and communication technology] agenda across sectors” (World Bank 2011, xviii). IEG’s evaluation Support for Innovation and Entrepreneurship found that the flow of knowledge on innovation and entrepre- neurship was limited, especially across networks, sectors, Regions, and the three Bank Group institutions (World Bank 2013). More recently, IEG’s evaluation Knowledge Flow and Collaboration found that general collaboration issues still affect the work of the Bank Group (World Bank 2019a). Furthermore, the World Development Report 2016 on digital dividends emphasized the importance of analog complements such as policies, institutions, and skills, which inevitably World Bank Group Independent Evaluation Group    123 require collaboration across sectors (World Bank 2016). Externally too, the liter- ature points to collaboration challenges relating to DTT across other public and private organizations, suggesting that the present evaluation’s finding of the lack of sufficient collaboration within the Bank Group is not an outlier. » Review of the qualitative interview data. Beyond the NVivo analysis, the interview data provided a wealth of experiential insights into the nature of collaboration challenges, including illustrations, that added further validity to IEG’s findings on collaboration. For example, interviewees observed that tech- nology and DTT may be embedded in many Bank Group projects in a country, but each project team often has its own information technology expert, and each expert could give different advice regarding data collection and manage- ment. The result was that the systems do not communicate with each other. Challenges Notwithstanding these steps, the evaluation faced the following challenges relating to design and methodology that broadly fell into three categories. The first set of challenges were the result of definitional or conceptual issues (see the Concept of Disruptive and Transformative Technologies section in this appendix). The definition of DTT used in the Mainstreaming paper (World Bank 2019b) discussed at the beginning of appendix A under the section “the Concept of Disruptive and Transformative Technologies” does not define the magnitude of the step change that makes a technology qualify as disruptive and does not also define transformative. The Bank Group lacks Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A an operationally relevant definition to clearly distinguish projects it consid- ers disruptive or transformative versus those that it does not. As such, there is no authoritative list distinguishing disruptive from nondisruptive inter- ventions, beyond individual interventions that are referenced in the Main- streaming paper as examples of DTT. The second set of challenges was the result of conscious choices about scope, and the third set of challenges was due to other methodological and data availability issues. Challenges in scope mostly resulted from the choice to evaluate the Bank Group’s preparedness (as opposed to effectiveness) given the newness of the Mainstreaming approach and lack of sufficient data on outcomes. Although the conceptual framework used in this evaluation was not designed to allow for an assessment of outcomes and intended impacts of the Bank Group’s new approach, it did yield insights on the extent to which the Bank Group is prepared to deliver on the DTT corporate priorities. Finally, Bank Group data related to DTT preparedness (such as on staff skills, collab- oration, procurement, and institutional culture and incentives) are, at best, incomplete, requiring IEG to draw on staff interviews. As noted, in cases in which a point of saturation was not reached due to low numbers of interviews or high levels of divergence across interviews, the team took this into account in the formulation of findings (using more tentative language) or decided that not enough evidence was available to assess a particular issue. Given these challenges, the evaluation team drew on the Mainstreaming paper and crowdsourced information from Bank Group staff to identify and 124 validate DTT engagements and to review and distill findings for this evalua- tion. The team also conducted careful analysis based on standardized tem- plates, document reviews, and NVivo analysis. Templates and Protocols Tables A.5–A.11 contain the templates and protocols for conducting the corporate priority reviews, and tables A.12–A.15 list the sampled projects selected for reviews relating to digital connectivity, digital governance, and fintech and digital entrepreneurship. Table A.5. Outline for Corporate Priority Reviews Outline Source 1. Introduction 1.1 World Bank Group’s rationale for identifying and engaging in Mainstreaming paper this theme; IEG’s rationale for inclusion of the theme in the eval- uation 1.2 Based on the literature and World Bank and IFC diagnostic Literature review and work, what are the key issues related to the theme? document reviews (i) What are the main disruptive or transformational technologies (internal/external), with implications and opportunities for the corporate priority, interviews (corporate program, or cross-cutting area? level) (ii) What is the impact of the application of the technology on key stakeholders? Consider private sector, consumers, workers, gov- ernment, society, and different social and income groups. (iii) (How) has the Bank Group been involved in the development or adoption or adaptation of the technology or technologies? World Bank Group Independent Evaluation Group    125 (iv) What are other key players or partners offering assistance in this corporate priority? Briefly explain their roles. 2. Evolution of Bank Group support to disruptive technologies in the selected theme 2.1 Summary of the evolution of Bank Group corporate strategies Corporate/sector and objectives with respect to (disruptive) technologies in the strategies/main- selected priority streaming 2.2. Description of the focus of Bank Group interventions in sup- Databases (Business port of this corporate priority (such as type of interventions, evo- Warehouse), inter- lution over time) including financing, ASA, trust-funded activities, views pilots, partnerships, and other initiatives 2.3 Alignment of World Bank or IFC support over the past six Document reviews— years (FY14–19) and pipeline with the corporate priority? (Rele- intervention tem- vance) plates, TTL interviews (continued) Outline Source 3. Bank Group preparedness (in responses, differentiate between different country or client contexts such as IDA countries versus IBRD, FCV; and reflect the views expressed by different stake- holders such as TTLs, Bank Group management, CMUs, Bank Group partners, clients, and beneficiaries, where applicable) 3.1 Areas of demand for Bank Group support to assist clients in Document reviews— the corporate priority? (Consider different income levels, country intervention tem- contexts) plates, TTL interviews 3.2 Areas in which the World Bank has been responding to client Intervention tem- demand effectively plates, interviews 3.3 Organizational enablers that have helped the World Bank or Intervention tem- IFC to respond effectively? plates, interviews Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A 3.4 Areas in which the World Bank or IFC has been unable to respond to demand? 3.5 Organizational factors that have constrained the response? 3.6 Conclusions regarding preparedness dimensions in the cor- porate priority: Leadership, culture, and incentives for innovation Risk appetite and informed risk taking Expertise, knowledge, and thought leadership Staff skills Resources Instrument mix Organizational structure, roles, and processes Ability to identify, adapt, and deploy knowledge Partnerships and global networks and positioning versus other actors Attention to cross-cutting issues and implications (jobs, gender, Intervention tem- inclusion, fragility) plates, interviews Attention to cross-sectoral issues and complementary invest- Intervention tem- ments in, for example, skills, physical infrastructure, business plates, interviews environment, governance 4. Conclusions and implications for World Bank or IFC pre- Intervention tem- paredness plates, interviews 4.1 Emerging conclusions and implications on how well prepared the World Bank and IFC are in helping clients harness the oppor- tunities and manage the risks in this corporate priority? Ways to enhance its preparedness. Source: Independent Evaluation Group. Note: This template was used for the corporate priority thematic reviews on digital connectivity (for both the World Bank and IFC), digital government services, and fintech and digital entrepreneurship, which relied on intervention-level reviews (document reviews, interviews of World Bank staff, and interviews of clients and donors) and not on country validation visits as noted in the section External Stakeholder 126 Interviews in this appendix. ASA = advisory services and analytics; CMU = Country Management Unit; FCV = fragility, conflict, and violence; FY = fiscal year; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IEG = Independent Evaluation Group; IFC = International Finance Corporation; TTL = task team leader. Table A.6. Intervention-Level Review Template Desk Review of (Name of Intervention) Intervention data Project ID Activity title: Country: Type of activity: TTL(s): Approval/start date: Closing date: Borrower/client counterparts: Partnerships involved with this activity (with donors, private sector, CSOs): World Bank Group funding/type (IDA/IBRD/ Actual Bank TF) at appraisal ($ millions): Group dis- bursement to date: ($, millions) Borrower/client funding at appraisal ($ millions): Actual client disbursement to date: ($, millions) Partner/cofinancier funding at appraisal Actual partner ($ millions): disbursement to date: ($, millions) Overall development objective(s): Disruptive/transformative technology-specific objective(s): Relevance Explanation Rating Rating scale: high, substantial, modest, negligible World Bank Group Independent Evaluation Group    127 1: To what extent is activity aligned with corporate priorities and sectoral programs highlighted in the Mainstreaming paper? 1.1: Support country diagnostics that help chart the new drivers of growth? 1.2: Support formulation and implementation of agile regulations for the new economy? 1.3: Scale up universal, affordable digital con- nectivity? 1.4: Support the provision of transparent, efficient, and accountable digital government services? 1.5: Support the development of skills and capabilities for the new economy? 1.6: Improve gender equality and create inclu- sive markets for women? 1.7: Address development challenges in fragile and conflict contexts? (continued) Desk Review of (Name of Intervention) 1.8: Brokering technology solutions through sectoral programs (for example, fintech, digital entrepreneurship) Implementation status and results (from latest ISR) Explanation Rating Rating scale: highly satisfactory, moderately satisfactory, satisfactory, moderately unsatisfactory, unsatisfactory, highly unsatisfactory 2.1: Progress toward achievement of PDO 2.2: Overall implementation progress 2.3: Overall risk rating Systematic operations risk ratings tool (from latest ISR) Explanation Rating Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A Rating scale: high, substantial, moderate, low 3.1: Political and governance 3.2: Macroeconomic 3.3: Sector strategies and policies 3.4: Technical design 3.5: Institutional capacity 3.6: Fiduciary 3.7: Environment and social 3.8: Stakeholders 3.9: Other Interview Name: Position: Date of interview: Interviewer(s): 1: How does this intervention relate to the Bank Group approach to DTT articulated in the Mainstreaming paper and/or as implemented by your Global Practice/Country Management Unit/unit? 2: When and in what way did the (country) cli- ent(s) approach the Bank Group about this ac- tivity? Why did they approach the Bank Group? 3: To what extent has the Bank Group been able to respond to this client request in a full and timely manner? 4: Which organizational enablers have helped the Bank Group respond effectively to this request? 128 (continued) Desk Review of (Name of Intervention) 5. To which areas of the (original) client request has the Bank Group been unable to respond? Why? Which organizational factors have con- strained its response? 6. What recommendations do you have for the Bank Group to better support clients in this area? Interviewer’s Concluding Thoughts on Preparedness Dimensions Preparedness dimension Explanation Rating Rating scale: high, substantial, modest, negligible, N/A » Leadership, culture, and incentives for inno- vation » Risk appetite and informed risk taking » Expertise, knowledge, and thought leader- ship » Staff skills » Resources » Instrument mix » Organizational structure, roles, and process- es » Ability to identify, adapt, and deploy knowl- edge » Partnerships, global networks, and position- ing versus other actors Emerging implications on how well prepared is the Bank Group to help clients harness the World Bank Group Independent Evaluation Group    129 opportunities and mitigate the risks posed by disruptive technologies? Sources of information: Additional information needed/Issues to be fol- lowed up during interviews of Bank Group staff, client representatives, or country case studies: Source: Independent Evaluation Group. Note: CSO = civil society organization; DTT = disruptive and transformative technologies; IBRD = Inter- national Bank for Reconstruction and Development; IDA = International Development Association; ISR = Implementation Status and Results Report; N/A = not applicable; PDO = project development objective; TF = trust fund; TTL = task team leader.  emistructured Interview Questionnaire: Corporate-Level Table A.7. S Preparedness for DTT 1. Please highlight your key points of engagement with the World Bank Group’s work on DTT (including, with the “response” to DTT, for example, through education, social protection, or inclusive development, as relevant). Kindly refer us to (i) main documents, including Bank Group documents and external ones; (ii) main interventions, including good practice interventions and interventions where the Bank Group learned from failure. 2. Stepping back, what would the features of an organization that is a world leader in “technology for development” be? (for example with regard to staff skills mix, process- es, incentives, organizational structure, and so on). Are there any distinctive features of DTT (magnitude of change, pace of change, and so on) make specific organizational features desirable? Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A 3. Which organization do you consider the world leader in “technology for develop- ment”? Where does the Bank Group rank? (at the top, near the middle, at the bottom) 4. Which key organizational enablers or strengths within the Bank Group have en- hanced its preparedness with regard to DTT? How? 5. Which key organizational constraints or weaknesses within the Bank Group have reduced its preparedness with regard to DTT? How? 6. Overall, how well prepared is the Bank Group to help clients to harness the op- portunities offered by DTT? (i) Very well prepared (ii) Well prepared (iii) Somewhat prepared (iv) Barely prepared 7. Overall, how well prepared is the Bank Group to help clients to mitigate the risks posed by DTT technologies (for example, job losses, global monopolies, rising inequality, cybersurveillance, security, privacy, fake news, algorithmic biases, and so on)? (i) Very well prepared (ii) Well prepared (iii) Somewhat prepared (iv) Barely prepared 8. What are the specific areas of comparative advantage of the Bank Group in tech- nology for development? What specific role should the Bank Group play? (continued) 130 9. What approaches or lessons relating to organizational effectiveness from other organizations or the private sector that you are familiar with could be applicable for the Bank Group to enhance its effectiveness? Please explain why. 10. What three recommendations do you have for the Bank Group to enhance its organizational effectiveness in helping clients harness the opportunities and mitigate the risks posed by DTT? Probe for: (i) Skills mix and how new knowledge is acquired for DTT (ii) Internal decision-making, collaboration, and budget processes (iii) Incentives for risk taking and learning from failure (iv) Areas of comparative advantage in DTT (v) Experience with eternal partnerships Source: Independent Evaluation Group. Note: This semistructured interview questionnaire was used for the corporate priority thematic reviews on country diagnostics, agile regulation, and education. DTT = disruptive and transformative technologies. Table A.8. NVivo Coding Scheme Level Keywords Stems and synonyms 1 2 3 Code Description were used 1.0000 DTT incentives 1.05 Incentives Yes Instances that identify the Incentive, presence of incentives for incentives, World Bank Group Independent Evaluation Group    131 DTT-relevant work, for ex- incentivize, ample accessibility to data incentivized science classes 1.0505 Incentives_In- Instances that identify the Innovate, novation_yes presence of incentives for innovation, innovation in DTT-relevant inventive, work incentive, incentives, incentivize 1.0510 Incentive_risk Instances that identify the Risk, In- taking_yes presence of incentives for centive, risk taking in DTT-relevant incentives, work incentivize 1.10 Incentives No Instances that identify Incentive, the lack of incentives for incentives, DTT-relevant work incentivize, incentivized (continued) Level Keywords Stems and synonyms 1 2 3 Code Description were used 1.1005 Incentives_In- Instances that identify the Innovate, novation_no lack of incentives for innova- innovation, tion in DTT-relevant work inventive Incentive, incentives, incentivize 1.1010 Incentive_risk Instances that identify the Risk, in- taking_no lack of incentives for risk centive, taking in DTT-relevant work incentives, Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A incentivize 2.0000 Staffing 2.05 Sufficient skills Instances that identify pres- Skills, train- ence of adequate staffing ing, staff, with respect to DTT-relevant experts work 2.10 Insufficient skills Instances that identify lack Skills, train- of adequate staffing with re- ing, staff, spect to DTT-relevant work experts 2.15 Sufficient Instances that identify pres- Mind- mindset ence of adequate staffing set, think, with respect to a DTT-rele- thought, vant mind-set thinking, leadership, leader, lead 2.20 Insufficient Instances that identify lack Mind- mindset of adequate staffing with set, think, respect to a DTT-relevant thought, mind-set thinking, leadership, lead 3.0000 Collaboration 3.05 Sufficient col- Instances that identify the Collabora- laboration presence of adequate col- tion, collabo- laboration for DTT-relevant rate, support work (continued) 132 Level Keywords Stems and synonyms 1 2 3 Code Description were used 3.0505 Sufficient col- Instances that identify the Collabora- laboration due presence of adequate tion, collabo- to structures collaboration for DTT-rel- rate, support, evant work due to internal Global Prac- structures, for example tice, sector, cross–Global Practice col- cross laboration 3.0510 Sufficient col- Instances that identify the Collabora- laboration due presence of adequate tion, collabo- to budget collaboration for DTT-rel- rate, budget, evant work due to budget allocate, allocation allocation, assign, assignment, money, cross 3.10 Insufficient Instances that identify the Collabora- collaboration lack of adequate collabora- tion, collabo- tion for DTT-relevant work rate, support 3.1005 Insufficient col- Instances that identify the Collabora- laboration due lack of adequate collabora- tion, collabo- to structures tion for DTT-relevant work rate, support, due to internal structures, Global Prac- for example cross–Global tice, sector, Practice collaboration cross World Bank Group Independent Evaluation Group    133 3.1010 Insufficient col- Instances that identify the Collabora- laboration due lack of adequate collabora- tion, collabo- to budget tion for DTT-relevant work rate, budget, due to budget allocation allocate, allocation, assign, assignment, money, cross 4.0000 Processes and Procedures 4.05 Procurement 4.0505 Sufficient pro- Instances that identify the Procurement, curement presence of procurement specification, systems conducive to capacity, DTT-relevant work, for ex- investment ample adequate guidance for complex technology investments (continued) Level Keywords Stems and synonyms 1 2 3 Code Description were used 4.0510 Insufficient Instances that identify the Procurement, procurement lack of procurement sys- specification, tems conducive to DTT-rel- capacity, evant work. For example, investment adequate guidance for complex technology invest- ments 4.10 Budget Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A 4.1005 Adequate Instances that identify the Budget, allo- budget presence of adequate cate, alloca- budget for prioritizing DTT tion, assign, support assignment, money 4.1010 Inadequate Instances that identify the Budget, allo- budget lack of adequate budget for cate, alloca- prioritizing DTT support tion, assign, assignment, money Source: Independent Evaluation Group. Note: All third-level codes were aggregated at second level. DTT = disruptive and transformative technologies. 134  ducation Corporate Priority Methodology— Box A.1. E Section I: Literature Review Armstrong, P. n.d. “Bloom’s Taxonomy.” Center for Teaching, Vanderbilt University. https://cft. vanderbilt.edu/guides-sub-pages/blooms-taxonomy. Chernyshenko, O., M. Kankaraš, and F. Drasgow. 2018. “Social and Emotional Skills for Stu- dent Success and Well-Being: Conceptual Framework for the OECD Study on Social and Emotional Skills.” OECD Education Working Papers 173, OECD Publishing, Paris. https://doi. org/10.1787/db1d8e59-en. Dweck, C. S. 2016. Mindset: The New Psychology of Success. New York: Ballantine. Firestone, C. M., and P. K. Kelly. 2019. Artificial Intelligence and the Good Society. Washington, DC: Aspen Institute. http://csreports.aspeninstitute.org/documents/AI2019.pdf. Gomes, P. 2016. “How PISA Is Changing to Reflect 21st Century Workforce Needs and Skills.” EdSurge Assessments, April 26, 2016. https://www.edsurge.com/news/2016-04-26-how- pisa-is-changing-to-reflect-21st-century-workforce-needs-and-skills. Grand-Clement, S. 2017. Digital Learning: Education and Skills in the Digital Age. Santa Mon- ica, CA: Rand Corporation; Cambridge, UK: Corsham Institute. https://www.rand.org/con- tent/dam/rand/pubs/conf_proceedings/CF300/CF369/RAND_CF369.pdf. Holzer, H. 2017. “Will Robots Make Job Training (and Workers) Obsolete? Workforce Devel- opment in an Automating Labor Market.” Brookings, June 19, 2017. https://www.brookings. World Bank Group Independent Evaluation Group    135 edu/research/will-robots-make-job-training-and-workers-obsolete-workforce-develop- ment-in-an-automating-labor-market. Kanodia, S., and N. Dalmia. 2019. “Life Skills to Tackle 21st Century Challenges: Why We Invested in Dream a Dream.” Omidyar Network (blog), January 27, 2019. https://www. omidyar.com/blog/life-skills-tackle-21st-century-challenges-why-we-invested-dream- %C2%A0dream. Lambert, D. 2018. “California Adopts First Computer Science Standards for K–12 Students.” EdSource, September 5, 2018. https://edsource.org/2018/californias-first-computer-sci- ence-standards-set-for-approval/601985. Levy, F., and R. Murnane. n.d. “Dancing with Robots: Human Skills for Computerized Work.” Third Way. http://content.thirdway.org/publications/714/Dancing-With-Robots.pdf. (continued)  ducation Corporate Priority Methodology— Box A.1. E Section I: Literature Review Martin, L. 2015. “The Promise of the Maker Movement for Education.” Journal of Pre-College Engineering Education Research (J-PEER) 5 (1). https://docs.lib.purdue.edu/cgi/viewcontent. cgi?article=1099&context=jpeer. Soland, J., L. Hamilton, and B. Stecher. 2013. “Measuring 21st Century Competencies: Guide for Educators.” Rand Corporation and Asia Society. https://asiasociety.org/files/gcen-mea- suring21cskills.pdf. Talreja, V., K. Krishnamurthy, D. J. W. Sanchez, and V. Bhat. 2018. “Mapping Life Skills in India: Research, Policy and Practice.” Dream a Dream. http://www.dreamadream.org/reports/ Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A mappinglifeskillsinindia.pdf. Wing, J. M. 2014. “Computational Thinking Benefits Society.” Social Issues in Computing (blog), January 10, 2014. http://socialissues.cs.toronto.edu/index.html?p=279.html. World Bank. 2016. World Development Report 2016: Digital Dividends. Washington, DC: World Bank. World Bank. 2018. “Disruptive Technologies and the World Bank Group: Creating Opportuni- ties—Mitigating Risks.” Development Committee Paper DC2018-0010, World Bank, Washington, DC. https://www.devcommittee.org/sites/www.devcommittee.org/files/download/Docu- ments/2018-09/DC2018-0010%20Disruptive%20Technologies.pdf. World Bank. 2019. “Mainstreaming the Approach to Disruptive and Transformative Technolo- gies at the World Bank Group.” Development Committee Paper DC2019-0002, World Bank, Washington, DC. https://www.devcommittee.org/sites/www.devcommittee.org/files/down- load/Documents/2019-04/DC2019-0002-Mainstreaming%20disruptive%20%204-13.pdf. World Bank. 2019. “World Bank Group HR Strategy FY20–22.” World Bank: Washington, DC. World Bank. 2019. World Development Report 2019: The Changing Nature of Work. Washing- ton, DC: World Bank. World Economic Forum. 2016. “Future Workforce Strategy.” In The Future of Jobs: Employment, Skills and Workforce Strategy for the Fourth Industrial Revolution. Geneva: World Economic Forum. http://reports.weforum.org/future-of-jobs-2016/future-workforce-strategy. 136 Source: Independent Evaluation Group.  ducation Corporate Priority Methodology—Section II: Table A.10. E Keywords for Identifying Education and DTT-Related Projects Keywords 1. achievement striving 74. modesty 147. compute 2. adapt 75. modules 148. CS 3. adult learning 76. motivation 149. cyber 4. adventurous 77. negotiation 150. debug 5. aesthetics 78. neuropsychological 151. design 6. agreeableness 79. neuroticism 152. digi 7. altruism 80. OER 153. disrupt 8. anxiety 81. open educational 154. drone resources 9. assertiveness 82. openness 155. electric 10. attention 83. optimism 156. electronic 11. attitude 84. parenting 157. e-logistics 12. autonomous learning 85. perseverance 158. engineering 13. behavior 86. persistence 159. fablab 14. bootcamps 87. personality 160. Facebook 15. brain development 88. positivism 161. facial recognition 16. build relationships 89. PowerPoint 162. Google 17. building consensus 90. problem-solving 163. hack 18. caregiving 91. psychosocial 164. ICT 19. caring 92. reflect 165. information processing 20. century skills 93. resilience 166. information technology 21. character 94. responsibility 167. internet 22. cognition 95. scientific models 168. invent World Bank Group Independent Evaluation Group    137 23. collaboration 96. search engines 169. IOT 24. communication 97. self-consciousness 170. iPad 25. compassion 98. self-control 171. iPhone 26. conscientiousness 99. self-discipline 172. IT 27. cooperation 100. self-efficacy 173. laptop 28. counseling 101. self-esteem 174. leapfrog 29. creative 102. self-management 175. LinkedIn 30. creativity 103. self-perception 176. machine 31. critical thinking 104. sensitivity 177. mechanized 32. curiosity 105. smart 178. Microsoft 33. delay of gratification 106. sociability 179. mobile 34. distance learning 107. social competence 180. moonshot 35. dutifulness 108. social connections 181. network 36. ed-tech 109. social intelligence 182. new business models 37. efficiency 110. social skills 183. new economy (continued) Keywords 38. e-learning 111. socialization 184. new jobs 39. electronics 112. soft skills 185. new services 40. e-library 113. stimulation 186. new skills 41. emotion 114. straightforwardness 187. online 42. empathy 115. summative scores 188. optimiz 43. enthusiasm 116. surveillance 189. pattern recognition 44. e-resources 117. sympathy 190. phone 45. excitability 118. teamwork 191. platform 46. extraversion 119. time management 192. privacy 47. fixed abilities 120. tolerance 193. programming 48. flexibility 121. trust 194. prototype Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A 49. foundational skills 122. unconventionality 195. reskill 50. future skills 123. vulnerability 196. retraining 51. game 124. 2G 197. robot 52. goal-driven 125. 3D 198. satellite 53. gregariousness 126. 3G 199. screen 54. grit 127. action labs 200. sensors 55. guidance 128. AI 201. simulat 56. happiness 129. algorithm 202. social media 57. higher-order thinking 130. Amazon 203. software 58. humility 131. analog design 204. tablet 59. imagination 132. analytics 205. tech 60. imaginative 133. artificial intelligence 206. telecom 61. impulse 134. automate 207. teleworking 62. initiative 135. automation 208. terabytes 63. integrity 136. big data 209. time series 64. interpersonal 137. blockchain 210. tinkering 65. intrapersonal 138. bots 211. transform 66. irritability 139. broadband 212. web 67. leadership 140. cable 213. WeChat 68. life satisfaction 141. cellular 214. Whatsapp 69. life skills 142. circuits 215. workplace readiness 70. lifelong learning 143. code 216.  YouTube  71. logical rules 144. coding 72. makerspace 145. complex 73. mindset 146. computation Source: Independent Evaluation Group. Note: AI = artificial intelligence; CS =computer science; DTT = disruptive and transformative technologies; 138 ICT = information and communication technology; IT = information technology; IOT = internet of things, OER = Open Educational Resources.  ducation Corporate Priority Methodology—Section III: Table A.11. E Coding Protocol for Categorizing Projects and ASA for Education Question World Bank of Interest Code Definition Examples Identify Data Use this code to identify Guinea-Bissau: The projects that management projects and ASA that use Quality Education for All apply DTT using big data or digital platforms Project establishment of to improve technology to aid data management and online portal and key ed- education data sharing among stake- ucation information (that holders. is, national policies, laws, statistics, and others), publicly available. ICT-based Use this code to identify Democratic Republic of learning projects and ASA that use Congo: The Education enhancement ICT to aid learning. Quality Improvement Project delivers early childhood education using Interactive Audio Instruction in the project provinces. Teacher training Use this code to identify Dominican Republic: projects and ASA that: The National Education » Employ technologies to Pact Project supports enhance teacher train- the Ministry of Educa- ing, for example online tion to leverage ICT for teacher training strengthening in-service World Bank Group Independent Evaluation Group    139 » Enhance STEM-related teacher training through skills of teachers online courses as part of the activities to improve » Impart socioemotional recruitment and training training to teachers to of primary and second- aid students’ interper- ary school teachers. sonal skills or intraper- sonal skills (continued) Question World Bank of Interest Code Definition Examples Identify Teaching Use this code to identify Africa Fund for Science— projects relevant projects that teach science, Africa Regional Scholar- that prepare skills to aid engineering, and mathemat- ship and Innovation Fund people to development ics or technology skills to for Applied Sciences, develop of new students that aid develop- Engineering, and Tech- technology technologies ment of new technologies. nology. Development in the new Note: Only higher education objective: To strengthen economy (master’s level and above). the institutional capacity for quality and sustain- able doctoral training, research, and innovation in transformative tech- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A nologies in Sub-Saharan Africa. Note: One example might be relevant for more than one code. For instance, this example applies to “teaching technolo- gy skills,” but it is also relevant for the category “teaching/ enhancing cognitive competencies.” Identify Provide Use this code to identify projects digital tools to projects and ASA that pro- that prepare students vide digital tools like tablets, people computer, and phones to aid to use or students’ digital literacy. consume Teaching Use this code to identify technology students to use projects and ASA that teach in the new digital tools students digital tools like tab- economy lets, computer, and phones to aid digital literacy. (continued) 140 Question World Bank of Interest Code Definition Examples Identify Teaching/ Academic mastery: Learn- Africa Fund for Science— projects enhancing ing academic content is Africa Regional Scholar- that prepare cognitive fundamental to educa- ship and Innovation Fund people for competencies: tion, and mastery of such for Applied Sciences, new jobs academic content serves as the basis Engineering, and Tech- in which mastery, for higher-order thinking nology. Development humans critical thinking, skills and the impetus for objective: To strengthen have a creativity improved interpersonal and the institutional capacity comparative intrapersonal competencies. for quality and sustain- advantage Academic content includes able doctoral training, instruction in subjects such research, and innovation as mathematics, science, in transformative tech- reading, global studies, and nologies in Sub-Saharan foreign languages. Africa. Note: “21st-century Note: Only higher education skills” included for all (master’s level and above). four codes—Teaching/ Critical thinking: critical enhancing cognitive thinking includes inductive competencies, Teaching/ and deductive reasoning, enhancing interpersonal and making correct analy- competencies, Teaching/ ses, inferences, and evalua- enhancing intrapersonal tions. These competencies competencies, and are important for deeply Socioemotional skills. understanding academic content, and they also relate to later career performance. Creativity: unusualness, World Bank Group Independent Evaluation Group    141 appropriateness, and transformation in varying combinations characterize a large proportion of those things we are willing to call creative. (continued) Question World Bank of Interest Code Definition Examples Teaching/ Communication and collab- enhancing oration: Communication and interpersonal collaboration are identified competencies: as vital 21st-century com- communication petencies. For example, and Pellegrino and Hilton (2012) collaboration, suggest that communication leadership, is vital to facilitate teamwork global and lies at the core of empa- awareness thy, trust, conflict resolution, and negotiation. Leadership: More broad- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A ly, leadership is not just a competency but a set of competencies. For example, a study conducted across Asian countries suggested that leadership involves having initiative, building consensus, innovating new strategies, and implement- ing policies and programs in collaboration with or under the direction of others. Global Awareness: Global awareness has grown in im- portance in the 21st century as economic, social, and cultural connections among countries have increased. Perhaps the best-studied in- terpersonal competency that underlies global awareness is empathy. (continued) 142 Question World Bank of Interest Code Definition Examples Teaching/ Growth mindset: Students enhancing with a growth mindset see intrapersonal intellectual abilities as mal- competencies leable and as a function of (growth mind- effort, good strategies, and set, learning help from others, whereas how to learn, those with a fixed mindset intrinsic motiva- treat intelligence as an innate tion, and grit) ability, immune to the efforts of the individual to improve it. Learning how to learn: Learning how to learn, or “metacognition,” refers to a student’s ability to determine how to approach a prob- lem or task, monitor their own comprehension, and evaluate progress toward completion. Intrinsic motivation: Moti- vation refers to the process that prompts people to take action to attain particular goals, and psychologists distinguish between two types of motivation. Intrinsic motivation refers to forces within the individual that World Bank Group Independent Evaluation Group    143 activate behavior. Socioemotional Jordan Education Re- skills form Support Program: Schools with a high proportion of Syrian ref- ugees implemented the socioemotional learning program. Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; DTT = disruptive and transformative technologies; ECE = early childhood education; ICT = Information communication technology; STEM = science, technology, engineering, and mathematics  orporate Priority Review of Digital Connectivity (World Table A.12. C Bank): Reviewed Projects Project Country or ID Project Name FY Exit FY Region P106589 Mexico IT Industry Development Project 2009 2016 Mexico P115647 E-Society and Innovation for Competi- 2011 2016 Armenia tiveness P116273 3A West Africa Regional Communica- 2011 2017 Western tions Infrastructure Program Africa P116542, Central African Backbone Program— 2011 2013 Africa P122777, APL 1, 3, 5 P122398 P121231 Moldova Governance eTransformation 2011 2017 Moldova Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A Project P121755 Afghanistan ICT Sector Development 2011 2018 Afghanistan Project P128013 First ICT Sector Development Operation 2013 2013 Marshall Islands P130184 West Africa Regional Communications 2013 2017 Western Infrastructure Program APL 1C—Benin Africa P148588 ICT Sector Support in Somalia 2014 2016 Somalia P152358 Somalia ICT Sector Support—Phase II 2015 ongoing a Somalia P155235 Caribbean Regional Communications 2017 ongoinga Nicaragua Infrastructure Program —Nicaragua P160533 Digital Malawi I 2017 ongoinga Malawi P156894 Afghanistan Digital CASA Project 2018 ongoing a Afghanistan P160230 Digital CASA—Kyrgyz Republic 2018 ongoinga Kyrgyz Re- public P160418 Côte d’Ivoire E-Agriculture Project 2018 ongoinga Côte d’Ivoire P162044 Smart Africa Alliance/Korea Partnership 2018 ongoing a Africa P162123 Implementing Open Data in Serbia 2018 ongoinga Serbia P164188 Kosovo Digital Economy 2019 ongoinga Kosovo P164525 Second Multisectoral Structural Reform 2019 ongoinga Senegal P164824 Enabling Digital Governance Project 2019 ongoinga Serbia P168587 Financial Inclusion and Digital Economy DPF 2019 ongoing a Morocco P118670 P4 Pacific Regional Regulatory Re- 2014 ongoinga Pacific source Center P150543 Support to Open Data in Tanzania 2014 ongoinga Tanzania P151210 Mexico RAS: IFT—Shared Wholesale 2015 ongoing a Mexico Network Source: Independent Evaluation Group. 144 Note: APL = adaptable program loan; CASA = Central Asia–South Asia; DPF = development policy financ- ing; FY = fiscal year; ICT = information and communication technology; IFT = Federal Telecommunica- tions Institute of Mexico; IT = information technology; RAS = reimbursable advisory services. a. Ongoing as of August 26, 2020.  orporate Priority Review of Digital Connectivity (IFC): Table A.13. C Reviewed Projects Project Country or Project Name Fiscal Year Status ID Region 35939 Zenium 2015 Closed Turkey 33645 Logo 2014 Closed Turkey 32367 Protelindo 2014 Closed Indonesia 30603 Grameen Phone V 2013 Active Bangladesh 39752 IGT Equity 2017 Active Myanmar 34980 Tikona Broadband 2015 Active India 30402 Altobridge 2011 Active World Region 29741 IHS Nigeria 2011 Closed Nigeria 30136 EMC 2012 Closed Africa Region 32782 IHS Africa 2013 Closed Africa Region 34454 IHS Rwanda 2014 Closed Rwanda 35725 IHS Nigeria 2 2015 Closed Nigeria 27526 Wind Telecom 2011 Active Dominican Republic 31812 VMLA—Chile 2013 Closed Chile 36722 Phoenix Tower Brazil 2016 Closed Brazil 37968 Cabo Telecom II 2018 Active Brazil 41688 Cabo Telecom III 2019 Active Brazil 33142 On Telecom 2014 Active Brazil 31170 IXcellerate 2012 Active Russian Federation World Bank Group Independent Evaluation Group    145 39715 Zain Iraq II 2018 Active Iraq 35289 Nxtgen 2015 Active India 34170 Ooredoo Myanmar 2016 Active Myanmar 36136 Robi Axiata II 2016 Active Bangladesh 32630 TeamNet Romania 2014 Active Romania 35725 IHS Nigeria 2 2015 Closed Nigeria 35932 BTS Towers 2016 Active Peru Source: Independent Evaluation Group. Note: IFC = International Finance Corporation.  orporate Priority Review for Digital Government Services: Table A.14. C Reviewed Projects Country or Region Project ID Project Name West Africa P161329 West Africa Unique Identification for Regional Integration and Inclusion (WURI) Program Bangladesh P132634 Bangladesh: Safety Net Systems for the Poorest Morocco P155198 Identity and Targeting for Social Protection Tanzania P124045 Tanzania: Productive Social Safety Net Ethiopia P151712 Urban Productive Safety Net Madagascar P172202 Madagascar ID User Research and Outreach Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A Source: Independent Evaluation Group.  eview of Fintech and Digital Entrepreneurship: Reviewed Table A.15. R Projects Project Fiscal Country or ID Project Name Year Region P161317 Kenya Industry and Entrepreneurship 2018 Kenya P168897 Lao PDR Payment Systems (FIRST Initiative) 2018 Lao PDR P151544 MSME Growth Innovation and Inclusive Finance 2015 India Project P133789 CO Sound Financial Sector Development 2018 Colombia (com- pletion of ASA) P128048 Afghanistan Access to Finance 2014 Afghanistan P167435 Disruptive Technologies for Development Pro- 2018 World gram P131091 Supporting the Ecosystem for Fostering a Dy- 2012 Middle East namic Entrepreneurship in MENA and North Africa P128167 How to Finance Innovation Technology and 2012 Middle East Entrepreneurship and North Africa P157826 Mongolia: Financial Sector Development Support 2020 Mongolia Program 37402 Net1 2016 South Africa 36516 Remitly 2017 World 146 40491 Konfio Debt 2018 Mexico (continued) Project Fiscal Country or ID Project Name Year Region 32942 Earthport 2013 South Asia 37584 Afluenta 2015 Argentina 31958 bKash 2013 Bangladesh 38728 CompareAsia 2017 East Asia and Pacific Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; MENA = Middle East and North Africa; MSME = micro, small, and medium enterprise. References Pellegrino, J. W., and M. L. Hilton, eds. 2012. Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. Washington, DC: National Academies Press. World Bank. 2011. Capturing Technology for Development: An Evaluation of World Bank Group Activities in Information and Communication Technologies. Washing- ton, DC: World Bank. World Bank. 2013. World Bank Group Support for Innovation and Entrepreneurship. Washington, DC: World Bank. World Bank Group Independent Evaluation Group    147 World Bank. 2016. World Development Report 2016: Digital Dividends. Washington, DC: World Bank. World Bank. 2018. “Disruptive Technologies and the World Bank Group: Creating Opportunities—Mitigating Risks.” Development Committee Paper DC2018- 0010, World Bank, Washington, DC. https://www.devcommittee.org/sites/www. devcommittee.org/files/download/Documents/2018-09/DC2018-0010%20Dis- ruptive%20Technologies.pdf. World Bank. 2019a. Knowledge Flow and Collaboration under the World Bank’s New Operating Model. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019b. “Mainstreaming the Approach to Disruptive and Transfor- mative Technologies at the World Bank Group.” Development Committee Paper DC2019-0002, World Bank, Washington, DC. https://www.devcommit- tee.org/sites/www.devcommittee.org/files/download/Documents/2019-04/ DC2019-0002-Mainstreaming%20disruptive%20%204-13.pdf. World Bank. 2019c. “World Bank Group HR Strategy FY20–22.” World Bank, Wash- ington, DC. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix A 148 1  Tertiary education projects target post-secondary (12th grade) students and includes techni- cal and vocational education and training projects. Roughly three-quarters (73 percent) of ter- tiary education projects are housed in the Education Global Practice, and these 200 projects are included in the analysis undertaken for this evaluation in addition to 31 projects from Jobs and Social Protection, 9 projects from Macroeconomics, Trade, and Investment, 4 projects from Agriculture, and 32 projects from other Global Practices. The bulk (61 percent) of the World Bank’s support for tertiary education has been through analytic and advisory activities, 35 percent of which has been advisory services and analytics. World Bank Group Independent Evaluation Group    149 Appendix B. The World Bank Group’s DTT Mainstreaming Approach The World Bank Group’s new approach to disruptive and transformative technologies (DTT) comprises three Bs—build, boost, and broker—five corporate priorities, and sectoral and regional programs as outlined in the 2018 Development Committee paper (World Bank 2018a), subsequently Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix B merged with the 2019 Development Committee paper (and also called the Mainstreaming paper; World Bank 2019c). The new approach also compris- es support to fintech and digital entrepreneurship, as outlined in the 2018 Bali Fintech Agenda (World Bank 2018b). Recently, the Bank Group’s DTT approach has been further updated through the 2020 Mainstreaming Digital and Disruptive Technologies (MDDT) Program Document described at the end of this appendix. Three Bs: Build, Boost, Broker The Bank Group’s Mainstreaming paper indicates that the new Mainstreaming approach aims to support countries to create opportunities and mitigate risks associated with DTTs through the build-boost-broker value proposition: » Build digital infrastructure and regulatory foundations. » Boost the capacity of institutions, communities, firms, and individuals by developing digital skills, foundational cognitive and socioemotional skills, and social protection. » Broker partnerships between public and private sectors and support global coalitions to ensure that disruptive technologies are harnessed for accel- erating progress toward the Sustainable Development Goals and the Bank Group’s twin goals. 150 Five Corporate Priorities The Bank Group’s Mainstreaming paper describes five corporate priorities (World Bank 2019c, iv–v): » Corporate Priority One: The [Bank Group] is supporting country diagnos- tics to help countries maximize the opportunities and mitigate the risks posed by disruptive technologies. It is developing a new set of diagnos- tics to benchmark countries’ readiness to absorb disruptive technologies by assessing what foundational elements need to be built, what capacities need to be boosted, and what sectors offer opportunities for disruptive technology to be brokered. The [Bank Group] is developing new indicators to measure the digital economy and using disruptive technologies for improved data collec- tion. These tools will inform the [Bank Group’s] regular high-level country as- sessments. In addition, global and regional flagship reports present evidence on how new sources of data, technology, and expertise can support inclusive growth strategies in specific country contexts. » Corporate Priority Two: The [Bank Group] is supporting the formulation and implementation of agile regulations that promote innovation and mobilize the private sector while addressing risks associated with tech- nological disruptions of sectors and markets. To build regulatory founda- tions and boost capacity of firms, particularly local entrepreneurs, the [Bank Group] is scaling up technical assistance and analytical work in three areas: World Bank Group Independent Evaluation Group    151 disruptions of traditional sectors, new business models and sources of market power, and risks related to privacy and data security. And it is participating in international forums and partnering with other international organizations to ensure that the experiences and concerns of developing countries are well represented in areas such as digital taxation and international data flows. » Corporate Priority Three: The [Bank Group] is prioritizing support for uni- versal and affordable digital connectivity. To build digital infrastructure, it is working with public and private partners across the value chain of broadband networks, leveraging the Maximizing Finance for Development and Cascade approaches in addressing policy and institution reforms, implementing large regional projects, and developing new business models that mobilize private investment such as cross-sector infrastructure sharing. By promoting univer- sal, affordable connectivity it is ensuring that people in fragile, conflict-af- flicted, rural, and remote areas are not left behind. And to prime demand for digital connectivity, it is providing underserved populations with online access to markets and services. » Corporate Priority Four: The [Bank Group] is supporting the provision of transparent, efficient, and accountable digital government services through a new GovTech global initiative. GovTech is a [Bank Group] whole-of-govern- ment approach that will develop new tools and interventions to help govern- ments harness DTT to boost capacity in core government functions, in public service delivery, particularly for poor people, and in technology-enabled citizen engagement. It will improve the functioning of the public sector and catalyze Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix B private sector engagement. » Corporate Priority Five: The [Bank Group] is supporting investment in human capital to develop skills and capabilities for the new economy. The World Development Report 2019 and the Human Capital Project highlighted how the changing nature of work is accelerating the need to boost the capaci- ty of people by investing in human capital, especially in skills and capabilities (World Bank 2019[d], Gatti and others 2018). The [Bank Group] is engaging with public and private clients to promote investments in skill development over the life cycle, from early childhood to adult learning. It is also working to boost the managerial capabilities of firms to absorb new technologies and create jobs. Sectoral and Regional Programs The Mainstreaming paper notes that the Bank Group “is developing sectoral and regional programs that leverage the five corporate priorities.” These programs, such as the sectoral programs listed in the next paragraph and the Digital Economy for Africa (DE4A) Moonshot/Accelerate and MNA (Middle East and North Africa) Tech are expected to allow the Bank Group “to be selective, minimize duplication, explore synergies between projects and dis- ruptive technologies, and leverage public and private partnerships to build, boost, and broker. Partnerships are essential to identify and scale up poten- tial applications of disruptive technologies to solve development challenges” 152 (World Bank 2019c, 5). Several Bank Group sectoral strategies and programs recognize the impor- tance of DTT (World Bank 2020): » The Bank Group Sustainable Energy Strategy relies on disruptive technol- ogies such as renewable energy off-grid solutions, which are identified as the most rapid means of providing cost-effective energy services in rural, remote, and isolated areas. » The World Bank Outlook 2050: Strategic Directions Note on Supporting Coun- tries to Meet Long-Term Goals of Decarbonization prioritizes the vision to use digital technologies to achieve decarbonization and resilience while pursuing carbon neutrality for the digital sector itself. » Accelerating Battery Storage for Development is a global program to accel- erate investments in battery storage for electric power systems. The program is intended to increase developing countries’ use of wind and solar power, and improve grid reliability, stability, and quality, while reducing carbon emissions. The program includes $1 billion from the Bank Group and will mo- bilize an additional $1 billion of concessional climate finance and $3 billion from the private sector. The International Finance Corporation is also invest- ing in emerging storage technology for vehicles and assessing opportunities to deploy grid-scale storage technology. » The Bank Group Strategy for Health, Nutrition and Population prioritizes a portfolio shift from traditional systems toward health systems based on the World Bank Group Independent Evaluation Group    153 latest technologies in areas such as telemedicine, artificial intelligence, big data, machine learning, diagnostic advances, e-health, and m-health with the aim of making health services accessible and affordable for all patients, including from their own homes. » The Bank Group’s Education 2020 Strategy highlights that technological advances are changing job profiles and skills, while offering possibilities for accelerated learning. » The Bank Group 2030 Sustainable Development Agenda notes the impor- tance of innovation and technology in building sustainable economies and expanding the capacity of people and institutions to thrive in the rapidly evolving environment. » A forthcoming report, Digital Acceleration of Agricultural Transformation 2020, recognizes the need for accelerating the digital transformation of the agriculture sector, which can disrupt the level and allocation of physical cap- ital (for example, agricultural equipment), labor, and natural capital (such as geology, soil, air, and water) and raise productivity. » The Jobs and Economic Transformation framework for the 19th Replen- ishment of the International Development Association emphasizes digital transformation as a key priority. During the 19th Replenishment, the Bank Group will support the creation of dynamic digital economies to speed up transforma- tion of both the private and public sectors. Digital development cuts across all facets of the Jobs and Economic Transformation agenda, supporting productivity Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix B growth in the agricultural sector and facilitating the structural transformation process by enabling technological leapfrogging, development of new high-pro- ductivity sectors, and new forms of market connectivity. » The Bank Group Strategy for Fragility, Conflict, and Violence emphasizes the need for systematizing the use of digital solutions in fragility, conflict, and violence setting, such as Geo-Enabling for Monitoring and Supervision and remote-sensing technologies to enhance supervision and implementation. Digital Economy for Africa The DE4A Moonshot/Accelerate aims to digitally connect every individual, business, and government in Africa by 2030 and ensure each is digitally en- abled and ready to thrive in the digital economy (World Bank 2019a). DE4A Moonshot/Accelerate would commit all African countries, development part- ners, and the private sector to policies, actions, and programs that catalyze digital transformation to achieve the DE4A Moonshot/Accelerate targets and to massively expand efforts and dedicate resources to build the foundations of a thriving digital economy. It will take leadership and vision to push the frontiers of innovation, collaboration, and integration so that Africa can own the twenty-first century. Achieving the DE4A Moonshot/Accelerate objec- tives will require strong, sustained leadership and a massive expansion of resources dedicated to building the foundations of the digital economy in every African country. 154 The DE4A Moonshot/Accelerate is underpinned by five principles and de- pends on five prerequisite foundations for its implementation. For dynamic, inclusive, and safe digital economies to emerge, African countries will need to follow a holistic approach to building strong foundations for allowing use cases to flourish across the different sectors (e-agriculture, e-health, e-gov- ernment or GovTech applications, and the like). Examination of the experi- ences of successful digital economies shows that five foundations need to be in place: 1. Digital infrastructure. For the digital economy, good connectivity through the internet or broadband is a critical foundation. Broadly, digital infra- structure consists of connectivity (through, for example, high-speed in- ternet and internet exchange points), the internet of things (connecting mobile devices, computers, sensors, voice-activated devices, geospatial in- struments, machine-to-machine communications, vehicle-to-vehicle com- munications), and data repositories (such as data centers and clouds). 2. Digital platforms. Digital platforms offer products and services, accessible through digital channels such as mobile devices, computers, and the internet, for all aspects of life. One foundational platform for the digital economy is dig- ital ID systems and trust services. Digital ID systems and services, such as elec- tronic signatures, underpin trust in online transactions and create opportuni- ties to innovate how products and services are delivered. Other foundational platforms include government-operated digital platforms that offer people-fac- World Bank Group Independent Evaluation Group    155 ing government services (such as online facilities to pay taxes, renew a driver’s license, and validate a digital ID), share information (such as with open data or reusable public sector data), and run back-office systems (such as by digitally managing government accounting and human resource information). Global connectivity allows users to use services and access information regardless of geographic location, leading to global services such as Google Search, Facebook, or Amazon Web Services. 3. Digital financial services. Access to affordable and appropriate digital fi- nancial services is critical for the participation of individuals and business- es in the digital economy. Firms can leverage digital financial services to transact with their customers and suppliers more easily and to build digi- tal credit histories and seek financing. Governments can use digital finan- cial services to increase efficiency and accountability in various payment streams, including for the disbursement of social transfers and receipt of tax payments. Digital payments are often the entry point for digital finan- cial services and provide the infrastructure or “rails” through which addi- tional products and use cases can be developed, as has been demonstrated by the evolution of M-PESA in Kenya, and Alipay/Tenpay in China. A digital financial services ecosystem requires forward-looking and proportionate legal and regulatory frameworks (for example, to allow market entry and in- novation), robust financial infrastructures (for example, national payment systems and credit reporting systems), and development and deployment of low-cost delivery channels (for example, agents, point of sale devices, Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix B automated teller machines, and mobile phones). 4. Digital entrepreneurship. Digital entrepreneurship and innovation create an ecosystem to bring the digital economy to life—with new, growth-ori- ented ventures, and transformation of existing businesses—contributing to net employment growth and helping enhance competitiveness and pro- ductivity of an economy. Digital entrepreneurship offers new products and services, leverages new technologies and business models, and opens new markets. Vibrant digital entrepreneurship ecosystems encompass skill de- velopment (through, for example, business mentoring networks), ecosys- tem support infrastructure (such as accelerators, incubators, innovation hubs, and coworking spaces), and access to markets and early-stage financ- ing (such as seed financing and venture capital); they require a conducive and enabling business environment that motivates the creation and use of novel digital technologies. 5. Digital skills. Economies require a digitally savvy workforce to build robust digital economies and competitive markets. Digital skills constitute tech- nology skills, together with business skills for building or running a start- up or enterprise. Greater digital literacy further enhances adoption and use of digital products and services among the larger population. Advanced digital skills to create local content and drive made-in-Africa solutions are needed to ensure an inclusive digital economy where Africa not only is on the consumer side of the digital revolution but also plays an important role in producing technology. 156 MNA Tech Initiative According to the Bank Group’s MNA Tech Concept Note, this initiative is advisory services and analytics (World Bank 2019b). It was launched in 2019 and will run until 2023. It is led by the Finance, Competitiveness, and In- novation Global Practice, with support from the following Global Practices: Digital Development, Education, Governance, and Macroeconomics, Trade, and Investment. The objective of the MNA Tech Initiative is to support Bank Group teams and engage stakeholders in the Region to achieve the Middle East and North Africa Vice Presidency goals on the digital economy. During the 2019 regional update to the Board of Executive Directors, the Middle East and North Africa Vice Presidency adopted a set of digital transformation priority targets to be achieved by the annual meeting in Marrakesh in 2021; these pertain to digital infrastructure and digital finance: doubling broad- band connectivity from 32 percent to 60 percent and significantly increasing cashless payments in the Region, from its current level of 34 cashless trans- actions per capita per year to 116 by 2021 (also called the Marrakesh 2021 goals). The Middle East and North Africa Strategy includes a commitment to promote inclusive growth and quality jobs by improving contestability in markets through disruptive technologies and digital payments. Bali Fintech Agenda World Bank Group Independent Evaluation Group    157 Fintech holds promise for reducing costs and frictions, increasing efficiency and competition, narrowing information asymmetry, and broadening access to financial services, especially in low-income countries and for underserved populations (IMF 2018). At the same time, national authorities are con- cerned about potential risks. These include consumer and investor protec- tion, the consistency of regulatory and legal frameworks, the adequacy of existing financial safety nets, and potential threats to financial integrity. To address market trends and future opportunities and concerns, the Interna- tional Monetary Fund and the World Bank approved the Bali Fintech Agenda in 2019 (box B.1). An update on the experience with the agenda (IMF 2019) measured the global impact of fintech on the provision of financial services. Technologies ranging from artificial intelligence to mobile applications are providing new solutions that seek to increase the efficiency, accessibility, and security of financial services provision. For example, slow, costly, and hard-to-track traditional payment and remittances services are being sup- plemented with new solutions, built on cloud computing, digital platforms, and distributed ledger technologies. Borrowing services are affected by new algorithms, such as smart contracts or artificial intelligence or machine learning applied to large volumes of data. This improves credit risk modeling and allows lending to new borrowers. Likewise, advances in artificial intelli- gence, digital ID, and cybersecurity are enabling new models for managing risk for individuals, financial institutions, and regulators (IMF 2019). Box B.1. Principles of the Bali Fintech Agenda Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix B The Bali Fintech Agenda outlines 12 high-level considerations: » Embrace the promise of fintech. » Enable new technologies to enhance financial service provision. » Reinforce competition and commitment to open, free, and contestable markets. » Foster fintech to promote financial inclusion and develop financial markets. » Monitor developments closely to deepen understanding of evolving financial systems. » Adapt regulatory framework and supervisory practices for orderly development and stability of the financial system. » Safeguard the integrity of financial systems. » Modernize legal frameworks to provide an enabling legal landscape. » Ensure the stability of domestic monetary and financial systems. » Develop robust financial and data infrastructure to sustain fintech benefits. » Encourage international cooperation and information sharing. » Enhance collective surveillance of the international monetary and financial system. Furthermore, the International Finance Corporation has identified four key determi- nants for the speed of adoption of fintech that to a large degree reflect the 12 ele- ments of the Bali Fintech Agenda. First is universal, affordable, and secure mobile connectivity (Andrews and Schmitz 2019). Second, operators must agree to technical standards that support interoperability. Third, regulators need to adopt new frame- works. Fourth, users must be able to identify themselves online reliably and securely. Source: IMF 2018 158 Mainstreaming Digital and Disruptive Technologies The development objective of the 2020 Program Document for the MDDT Initiative is to support units and teams in the scaling-up of appropriate high-quality digital and disruptive technology solutions in World Bank proj- ects. An immediate focus of the initiative will be supporting the COVID-19 emergency response and recovery and knowledge sharing and supporting the development of knowledge products. This will be achieved through estab- lishing a Network and a Technical Working Group aimed at connecting teams and programs across the Bank Group, and by establishing a Secretariat aimed at supporting the Network, the Technical Working Group and the implemen- tation of an initial set of activities. The focus will be on technology “appli- cations,” and the initiative will not include the foundations of the digital economy that are covered by other programs (such as DE4A). References Andrews, S., and K. M. Schmitz. 2019. “FinTech as Disruptive Technology in Emerg- ing Markets.” In Reinventing Business Through Disruptive Technologies: Sector Trends and Investment Opportunities for Firms in Emerging Markets, edited by S. Andrews, 68–75. Washington, DC: International Finance Corporation. World Bank Group Independent Evaluation Group    159 Gatti, R. V., A. C. Kraay, C. Avitabile, M. E. Collin, R. Dsouza, and N. A. P. Dehnen. 2018. The Human Capital Project. Washington, DC: World Bank. http://docu- ments.worldbank.org/curated/en/363661540826242921/The-Human-Capi- tal-Project. IMF (International Monetary Fund). 2018. “The Bali Fintech Agenda.” IMF Policy Pa- per, IMF, Washington, DC. https://www.imf.org/en/Publications/Policy-Papers/ Issues/2018/10/11/pp101118-bali-fintech-agenda IMF (International Monetary Fund). 2019. “Fintech: The Experience so Far.” IMF Policy Paper, IMF, Washington, DC. https://www.imf.org/en/Publications/Poli- cy-Papers/Issues/2019/06/27/Fintech-The-Experience-So-Far-47056 World Bank. 2018a. “Disruptive Technologies and the World Bank Group: Creating Opportunities—Mitigating Risks.” Development Committee Paper DC2018- 0010, World Bank, Washington, DC. https://www.devcommittee.org/sites/www. devcommittee.org/files/download/Documents/2018-09/DC2018-0010%20Dis- ruptive%20Technologies.pdf. World Bank. 2018b. “The Bali Fintech Agenda: A Blueprint for Successfully Harness- ing Fintech’s Opportunities.” Press Release, October 11, 2018. https://www. worldbank.org/en/news/press-release/2018/10/11/bali-fintech-agenda-a-blue- print-for-successfully-harnessing-fintechs-opportunities. World Bank. 2019a. Concept Note. Digital Economy for Africa (DE4A) Moonshot (ID: P169935). February 8. Washington, DC: World Bank. World Bank. 2019b. Concept Note. MNA Tech Initiative (ID: P171209). December 17. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix B Washington, DC: World Bank. World Bank. 2019c. “Mainstreaming the Approach to Disruptive and Transfor- mative Technologies at the World Bank Group.” Development Committee Paper DC2019-0002, World Bank, Washington, DC. https://www.devcommit- tee.org/sites/www.devcommittee.org/files/download/Documents/2019-04/ DC2019-0002-Mainstreaming%20disruptive%20%204-13.pdf. World Bank. 2019d. World Development Report 2019: The Changing Nature of Work. Washington, DC: World Bank. World Bank. 2020. “The Final Program Document for Mainstreaming Digital and Disruptive Technologies (‘MDDT’) Initiative.” World Bank, Washington, DC. 160 Appendix C. Gender and Disruptive and Transformative Technology Technology offers the potential to improve gender equality in multiple do- mains. More flexible work arrangements made possible by technology may reduce the gender gaps in labor force participation. Use of the same tech- nical skills may reduce gender wage gaps. Increased access to information made possible by technology can affect gender norms and aspirations faster than expected, for example through social media.1 Digital economy innova- tions such as digital payments, mobile money, and taxi-sharing rides can all increase agency and control over economic resources and safety. Compared with manual cash transfers, greater privacy and control of mobile transfers by female recipients in Niger shifted intrahousehold decision-making in their favor (World Bank 2016, 134). Digitized social protection systems help bring low-income women into the financial sector by enabling conversion of social benefits from cash to electronic accounts (World Bank 2015, 52). Bio- metric identity assists women with limited proof of identity to open savings accounts (World Bank 2015, 52). Mobile phones can be an effective way to improve health system performance in areas of low internet coverage. Health World Bank Group Independent Evaluation Group    161 extension workers remind registered women of key appointments through automated short message service (World Bank 2016, 190). At the same time, technology may pose a risk for women. In poor and remote areas, the internet is more likely to be accessed outside the home, where social norms for socializing or safety concerns are a barrier for women. Increased access to credit, knowledge, or markets made possible by disrup- tive and transformative technologies (DTT) can also place women at higher risk for domestic violence (McDougal and others 2019). Women and girls are more likely than men and boys to experience threats of intimidation, ha- rassment, defamation, violence, surveillance, or some combination of these (World Wide Web Foundation 2015). Gender-differential effects of DTT are manifest in the use of and access to technology. More men than women have access to mobile phones and the internet, especially in many parts of Africa and South Asia. Over 1.7 billion women in low- and middle-income countries do not own mobile phones. Women in these countries are 14 percent less likely to own a mobile phone than men on average, and 38 percent of women in South Asia are less likely to own a phone (World Bank 2016, 134). In Africa, women are 50 percent less likely than men to use the internet (World Bank 2016, 134). More men than women also have a bank account, and the gender gap in account ownership has remained consistent since 2011 (World Bank 2015, 52). The main barrier for women, according to the World Bank Group Gender Strategy (2015) is Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix C that relative to men, females have less access to the financial and education- al capabilities required to adopt technology. Owing to childcare and do- mestic duties, women have less time than men to improve their technology skills. Indeed, it is often the off-line factors associated with poverty, gender discrimination, and gender stereotypes that prevent women from benefiting from technology to the same extent as men. Automation in the workplace poses a greater risk for women with less formal education, who tend to be overrepresented in routinized work (Brussevich and others 2018). Men were almost eight times as likely as women to work in information and communi- cation technology (ICT) jobs in 30 emerging markets (World Bank 2019b, 18). Bank Group strategic documents recognize the gender-differential effects of DTT. Both the Bank Group Gender Strategy and the Mainstreaming paper note that harnessing technology for gender equality will require overcoming lower mobile phone ownership by women (World Bank 2015; World Bank 2019b). The “improving human endowments” objective of the Gender Strat- egy states that technology is one of many factors that can help increase girls’ enrollment in science, technology, engineering, and mathematics and advo- cates female take-up of technical and vocational education and training to address the sociocultural barriers at secondary level and increase participa- tion in these subject areas (World Bank 2015, 37). Networking and increased access to technology, markets, and finance may alleviate the constraints experienced by women-owned enterprises, which tend to be smaller, more informal, and less high-tech (World Bank 2015, 52). Use of technology to pro- mote universal identification can help remove barriers to women’s ownership 162 of and control over assets and prioritize financial inclusion to close gender gaps.2 Harnessing technology will require overcoming entrenched social mores and gender sorting patterns across occupations and sectors differentially affected by disruptive technologies (World Bank 2019b, 15). For example, oc- cupational sex segregation keeps women in public sector occupations (World Bank 2015, 46), and female labor force participation in the ICT sector is low (World Bank 2016, 134). Table C.1 highlights initiatives being pursued by the World Bank to address the gender-differential effects of DTT. Output from the portfolio review for this evaluation suggested that further efforts are needed. Of the 51 projects with ICT sector percentage weight at or above 50 percent, 37 percent had all of the gender-relevant flags, and 76 per- cent had one or more gender flags (they identified specific actions to address gender gaps, included analysis to identify project relevant gaps between males and females, and had indicators to monitor project relevant gender actions).3 Of the 90 ICT advisory services and analytics (ASA) approved during fiscal years (FY)15–18, just 7 percent were gender relevant. This low presence of gender in ICT-related ASA is a particular concern given that it is in ASA that new opportunities to address the digital gender divide can be explored. World Bank Group Independent Evaluation Group    163  orld Bank Group Initiatives Addressing Gender Differentials Table C.1. W in Technology Gender Differentials Initiative(s) Source(s) Access, adop- World Bank Group ICT Toolkit identifies oppor- World Bank tion, and use of tunities for using ICT to empower women and 2018a ICT tools suggests a set of actions, drawing from existing examples with the potential to address barriers. Digital economy The Moonshot Africa 2030 program’s five founda- World Bank tions (digital infrastructure, digital platforms, digital 2019a financial services, digital entrepreneurship, and digital skills) will address gender gaps. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix C Female partici- Regional initiatives such as African Centers of World Bank pation in STEM Excellence and Partnership for Applied Sciences, 2019b Engineering, and Technology are improving higher education opportunities in Africa, including for girls. Social protection In Zambia, 75,000 girls and women in remote World Bank areas receive their cash transfers through a mobile 2019b wallet account or a prepaid card. Financial A specific prior action in the DPF for Morocco Financial Inclu- inclusion makes it mandatory for banks and mobile pay- sion and Digital ment companies to report gender-disaggregated Economy DPF data. Entrepreneur- Data from this report are being tested in the Kenya World Bank ship Industry and Entrepreneurship Project. 2018b, 2019b IFC’s Digital2Equal cross-sectoral initiative brings together leading online technology companies to boost opportunities for women in emerging markets. IFC’s investment in Andela helps companies and start-ups in Sub-Saharan Africa meet their de- mand for 21st-century skills and retain women in coding bootcamps and talent pipelines.a Source: Independent Evaluation Group. Note: DPF = development policy financing; ICT = information and communication technology; IFC = Inter- national Finance Corporation; STEM = science, technology, engineering, and mathematics. a. Andela is an organization that cultivates information technology talent in Africa and provides leading global technology companies with access to a high-skilled resource pool. IFC is currently supporting Andela as an equity investor. See IFC (n.d.). 164 Gender and DTT-Related Projects and ASA The gender and DTT-related projects and ASA were identified in the follow- ing manner. The first stage included identifying DTT-related projects using sector codes. The team identified 49 World Bank projects by scanning the Business In- telligence database for projects approved during FY15–19 with ICT sector percentage weight at or above 50 percent. The Digital Development Global Practice suggested two more technology projects be included, resulting in 51 DTT-related projects. Regarding ASA, the team identified 90 DTT-relat- ed World Bank ASA approved during FY15–18 with ICT sector percentage weight at or above 50 percent. The team then used gender flags from the World Bank’s Business Intelli- gence database to identify gender and DTT-related projects. There are three kinds of gender flags: (i) a gender action flag signifies whether the project identifies specific actions to address gender gaps; (ii) a gender analysis flag signifies whether the project includes any analysis to identify gaps between males and females; and (iii) a gender mechanism flag signifies whether the project includes indicators in the results framework to monitor gender out- comes. The team reviewed the 51 DTT-related projects identified in stage 1 and found that 39 of them had one or more gender-relevant flags. World Bank Group Independent Evaluation Group    165 The World Bank does not assign gender flags to ASA. Therefore, the evalua- tion team used a keyword search to identify gender ASA from the 90 DTT-re- lated ASA identified in the first stage. First, the evaluation team identified 38 keywords using World Bank documents. The team also consulted with gender experts in the Gender Global Practice to ensure that the keywords captured the language and terminology used by World Bank gender experts for DTT-related topics. Second, the team searched the 38 keywords among objectives and abstracts of 90 ASA approved over FY15–18 using R analytics. The keyword search yielded 59 ASA with at least one keyword (table C.2). The evaluation team then conducted a desk review of the 59 ASA to elim- inate false positives. ASA objectives and abstracts and selected ASA docu- ments were scanned, yielding 6 ASA. Table C.2. Project Types Projects/ Technology Projects/ ASA after Projects/ Approval ASA (ICT sector Gender Flag ASA after Product Fiscal percentage weight at or or Keyword Desk Type Year above 50%) Search Review Lending 2015–19 51 39 n.a. ASA 2015–18 90 59 6 Source: Independent Evaluation Group. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix C Note: ASA = advisory services and analytics; ICT = information and communication technology; n.a. = not applicable. References Brussevich, Mariya, Era Dabla-Norris, Christine Kamunge, Pooja Karnane, Salma Khalid, and Kalpana Kochhar. 2018. “Gender, Technology and the Future of Work.” Staff Discussion Note, International Monetary Fund, Washington, DC. IFC (International Finance Corporation). n.d. “Case Studies: Andela.” In Digi- tal Skills in Sub-Saharan Africa: Spotlight on Ghana, 100–106. Washington, DC: IFC. https://www.ifc.org/wps/wcm/connect/1f44c711-c6ed-42c9- 9590-9ea53013f927/Digital+Skills_Final_WEB_Andela.pdf?MOD=A- JPERES&CVID=mGk7cgt. McDougal, L., J. Klugman, N. Dehingia, A. Trivedi, and A. Raj. 2019. “Financial Inclu- sion and Intimate Partner Violence: What Does the Evidence Suggest?” PLOS ONE 14(10): e0223721. https://journals.plos.org/plosone/article?id=10.1371/ journal.pone.0223721. World Bank. 2015. World Bank Group Gender Strategy (FY16–23): Gender Equality, Poverty Reduction, and Inclusive Growth. Washington, DC: World Bank. World Bank. 2016. World Development Report 2016: Digital Dividends. Washington, DC: World Bank. World Bank. 2018a. Engendering ICT: Toolkit for Task Team Leaders. Washington, DC: 166 World Bank. World Bank. 2018b. Women Wavemakers: Practical Strategies for Recruiting and Re- taining Women in Coding Bootcamps. Washington, DC: International Bank for Reconstruction and Development and World Bank. World Bank. 2019a. Concept Note. Digital Economy for Africa (DE4A) Moonshot (ID: P169935). February 8. Washington, DC: World Bank. World Bank. 2019b. “Mainstreaming the Approach to Disruptive and Transfor- mative Technologies at the World Bank Group.” Development Committee Paper DC2019-0002, World Bank, Washington, DC. https://www.devcommit- tee.org/sites/www.devcommittee.org/files/download/Documents/2019-04/ DC2019-0002-Mainstreaming%20disruptive%20%204-13.pdf. World Wide Web Foundation. 2015. Women’s Rights Online: Translating Access into Empowerment. Washington, DC: World Wide Web Foundation. http://webfoun- dation.org/docs/2015/10/womens-rights-online21102015.pdf. World Bank Group Independent Evaluation Group    167 1  “‘My Dress, My Choice’ in Kenya, a social media movement against female violence that mobilized thousands of Kenyans, including through street protests, eventually led to changes in relevant laws” (World Bank 2016, 134). 2  The gender gap in account ownership has remained consistent since 2011, with more men than women owning a bank account, a disparity that the World Bank’s commitment to univer- sal financial inclusion by 2020 seeks to address (World Bank 2015, 52). 3  “Gender flag” data were used rather than “gender tag” data, because the latter begins in fiscal year 2016 and was not available for the entire period of this analysis (fiscal years 2015–19). Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix C 168 Appendix D. World Bank Data Initiatives Geo-Enabling for Monitoring and Supervision The Geo-Enabling for Monitoring and Supervision (GEMS) initiative builds capacity in using simple open-source technology to create customized digi- tal monitoring and evaluation systems. It also provides platforms for remote supervision, real-time risk and safeguards monitoring, and portfolio map- ping for coordination across projects and partners. Particularly noteworthy is GEMS’s focus on local ownership and capacity building, made possible by its use of tools and methods that are simple, field appropriate, open source, low cost, and directly applicable. The remote supervision capability of GEMS makes it particularly attractive in situations of fragility, conflict, and vio- lence and during the coronavirus (COVID-19) pandemic, which has imposed social distancing requirements and travel restrictions. GEMS has benefited from a grant from the Korean Trust Fund for Economic and Peacebuilding Transitions based on its support to innovation in fragili- World Bank Group Independent Evaluation Group    169 ty, conflict, and violence contexts. The skills of the 15 GEMS team members involve a mix of operational knowledge, experience in working in fragility, conflict, and violence and low-capacity settings, and understanding of tech- nology. As of March 2020, GEMS has been implemented in over 450 World Bank Group–funded projects in 35 countries, and more than 2,000 project implementation units and government employees have participated in ca- pacity-building training. Constraints have included a Bank Group culture that encourages outsourc- ing tasks that relate to remote supervision to external companies that offer high-tech platforms. Often, those high-tech approaches are not only very costly and unsustainable but also not always appropriate for low-capacity environments and do not adequately feed into decision-making processes. Therefore, the GEMS team has opted to only use open-source tools and an innovative application of low-tech solutions that can be sustainably operat- ed in weak-capacity conditions. Another challenge is that Bank Group staff do not have sufficient time to engage in the various uses of a technology, which results in missed opportu- nities (for example, that GEMS not only produces eye-catching maps but en- ables real-time monitoring of almost any kind of operational engagement). Not having enough time may be a symptom of an incentive regime that does not adequately prioritize engaging with innovative in-house solutions and deploying them to their full potential in projects. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix D GEMS has partnered with several external organizations: Harvard Human- itarian Initiative, provider of the primary open-source technology used by GEMS; United Nations High Commissioner for Refugees; the European Space Agency for remote-sensing support; and bilateral donors, who often benefit from GEMS capacity-building training. Within the World Bank, GEMS partners include the Joint Data Center for Forced Displacement; the Geospatial Operations Support Team; the Disruptive Technology for Social Development initiative; the Middle East and North Afri- ca remote supervision working group; gender-based violence service provider mapping; the Poverty Global Practice, which oversees the Iterative Beneficiary Monitoring approach; the Smart Supervision App team; and the Geolab in the Information and Technology Solutions Department (ITS). Geospatial Operations Support Team The main aim of the Geospatial Operations Support Team is to help the World Bank make better use of geospatial data. This team offers World Bank task teams a variety of services, including advice on how geospatial data can help solve specific problems, support for purchasing data and imagery, and direct technical assistance. The Analytics and Geospatial Working Group, set up by the Development Data Council, is both the governing body and coordinating body for geospatial operations at the World Bank. It consists of representatives of every Global Practice. 170 Development Data Partnership (Formerly Data Collaboratives) The Development Data Partnership is a partnership between international organizations and companies, created to facilitate the use of third-party data in research and international development. The partnership currently sup- ports more than 60 World Bank operations. The Development Data Partnership has benefited from the World Bank’s support of innovation and open-ended objectives and use of development assignments to empower staff to create new initiatives. Constraints have included: » Difficulty in setting up a streamlined, formal system for receiving annual membership dues from partnership members, which are necessary to ensure the long-term sustainability of the initiative; » General lack of coordination in data management and analytics among ITS, the Development Economics Vice Presidency, the Global Practices, and the Regions; » High barriers to use of ITS compute resources (confusing procedures, long turnarounds, unavailability of industry-standard data science tools, among other factors); World Bank Group Independent Evaluation Group    171 » The ITS charge-back system for data compute and storage resources, which is difficult to interpret and not streamlined; and » General lack of guiding vision from senior management and lack of a strong data science and data engineering community to make use of the invaluable data resources brought to the World Bank through the partnership. The continued ability to raise trust funds and funding from external partners will be critical to continuation of the partnership. A long-term institutional home also must be found. Key technical lessons from the experience so far include the following: » Developing template data license agreements and memorandums of under- standing among organizations can save time and resources. » Developing data governance principles can guide best practices for responsi- ble and ethical data use. » Working groups consisting of data engineers, data scientists, sector domain experts, legal counsel, communications specialists, and procurement special- ists across organizations have made the platform possible. » Centralized information technology architecture and processes for ingesting, storing, and accessing data, and for coding and collaboration, create econo- mies of scale among partners and facilitate secure data use. » A web-based partnership management platform helps meet the value propo- sition for data partners. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix D » Managed, accessible repositories for derived data products and algorithms broaden the impact of the partnership. The Development Data Partnership, supported in part by the Bill and Melin- da Gates Foundation, the International Monetary Fund, and Inter-American Development Bank, includes major data partners such as Facebook, Google, Mapbox, Uber, Waze, and Cuebiq. Within the Bank Group, collaborating de- partments include ITS, the Development Economics Vice Presidency, and the Global Practices. 172 Appendix E. Foresight and Anticipation Foresight and anticipation mean thinking systematically about tomorrow to inform decision-making today; they can help in planning for development challenges such as climate change or income inequality in the face of the technological revolution. Tools for conducting foresight and anticipation include foresight analysis, horizon scanning, scenario planning, and drivers and trends impact analysis (Conway 2015, 2; Pauwels 2019; van de Pol 2017). Eleonore Pauwels has pointed out how the convergence of artificial intel- ligence (AI) with other technologies such as those in the biological world (such as genomics or biotechnology), physical world (such as robotics or 3D printing), and digital world (such as cybersecurity, the internet of things, or quantum computing) raises huge potential for human well-being alongside the risk of large-scale challenges for health, security, economics, and gov- ernance in different geopolitical contexts. Pauwels points out that foresight analysis can help anticipate and prevent emerging risks, and emphasizes the need to act collectively because of the spread of AI convergence with other technologies across a wide range of state, nonstate, and transnational actors and entities. World Bank Group Independent Evaluation Group    173 Role of the United Nations and World Bank Group. According to Pauwels, the multilateral system of the United Nations can enable collective action, but to do so, the system will need to strengthen its anticipation and preven- tion capacity relating to emerging and potential risks from technological convergence. AI-driven technologies are spreading beyond the state into the hands of corporations, groups, and individuals. Some major corporations may see little value in bringing multilateral approaches to bear on lucrative, proprietary technologies, while powerful United Nations member states ap- pear more focused on crystallizing their own comparative strengths than in working together to build responsible governance systems in the cyber and biotech spheres. Especially in relation to peace and security, member states may resist attempts to involve the multilateral system in what they see as a fierce competition over powerful new systems of influence. At the same time, there is a clear understanding that the kinds of risks cre- ated by AI convergence do not occur neatly within state boundaries. They manifest globally and must be dealt with across and among states. Against this background, the multilateral system urgently needs to help build a new social contract to ensure that converging technologies, in particular AI, are deployed safely and aligned with the ethical needs of a globalizing world. This new social contract must create opportunities and incentives for gov- ernments, citizens, experts, and, in particular, the private sector to commit to a serious, inclusive, and responsible technological future. References Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix E Conway, M. 2015. Foresight: An Introduction. Melbourne: Thinking Futures. https:// thinkingfutures.net/downloads/foresightintro. Pauwels, E. 2019. The New Geopolitics of Converging Risks: The UN and Prevention in the Era of AI. New York: United Nations University Centre for Policy Research. https://collections.unu.edu/eserv/UNU:7308/PauwelsAIGeopolitics.pdf. van de Pol, P. 2017. “Africa and Foresight: Better Futures in Development.” Working Paper, Global Centre for Public Service Excellence, United Nations Development Programme, Singapore. https://www.undp.org/content/dam/undp/library/ca- pacity-development/English/Singapore%20Centre/UNDP-GCPSE_Foresight_for_ Africa.pdf. 174 Appendix F. Creating a Culture of Risk Taking and Innovation Economist Gary Pisano (2019) writes that a culture of risk taking and innova- tion can be created by counterbalancing the following seemingly contradic- tory behaviors: A tolerance for failure must be counterbalanced by an intolerance for incom- petence. Given that innovation involves the exploration of uncertain and unknown terrain, it is not surprising that a tolerance for failure is an import- ant characteristic of innovative cultures. Successful innovative organizations that tolerate failure must also set exceptionally high performance standards for their people. The goal should be to create a culture that simultaneously values learning through failure and outstanding performance. Building a culture of competence requires senior leaders and managers to communicate expected standards for performance clearly and regularly. A willingness to experiment must be counterbalanced by rigorous discipline. Organizations that embrace experimentation must also be comfortable with uncertainty and ambiguity. They experiment to learn rather than to produce an immediately marketable service. A willingness to experiment, though, requires discipline. Discipline-oriented cultures select experiments carefully World Bank Group Independent Evaluation Group    175 based on their potential learning value, and they design them rigorously to yield as much information as possible relative to the costs. They establish clear criteria at the outset for deciding whether to move forward with, modi- fy, or kill an idea. Psychological safety must be counterbalanced by comfort with brutal candor. Psychological safety is an organizational climate in which individuals feel they can speak truthfully and openly about problems without fear of repri- sal. But psychological safety also means that individuals need to be open to receiving criticism when others speak their mind. Unvarnished candor is crit- ical to innovation because it is how ideas evolve and improve. Senior leaders need to set the tone through their own behavior by inviting critique of their own ideas and proposals. Individual accountability must counterbalance collaboration. Well-function- ing innovation systems need information, input, and significant integration of effort from a diverse array of contributors. People who work in a collabo- rative culture have a sense of collective responsibility. But collaboration does not mean consensus. Consensus can be poison for rapid decision-making and innovation. Someone must decide and be accountable for it. An account- ability culture is one where individuals are expected to make decisions and own the consequences. Strong leadership must counterbalance organizational flatness. In cultur- ally flat organizations, people are given wide latitude to take actions, make decisions, and voice their opinions. They tend to generate a richer diversity Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix F of ideas than hierarchical ones, because they tap the knowledge, expertise, and perspectives of a broader community of contributors. Lack of hierarchy, though, does not mean lack of leadership. Paradoxically, flat organizations require stronger leadership than hierarchical ones, and flatness allows lead- ers to be closer to the action. Reference Pisano, G. 2019. “The Hard Truth about Innovative Cultures.” Harvard Business Re- view (January–February): 62–71. 176 Appendix G. External Partnerships Famine Action Mechanism Famine Action Mechanism is a large partnership program involving about 40 partners from United Nations agencies, bilateral donors, nongovernmen- tal organizations, civil society organizations, private sector firms (Amazon, Google, and Microsoft), academic institutions, and agencies engaged in food security analysis. The kinds of contributions partners make include data and analytics to inform modeling of famine risks, financing for crisis funding, and operations support to develop Anticipatory Action Plans in pilot coun- tries (Afghanistan, Chad, Somalia, and South Sudan). Famine Action Mechanism seeks to make financing more predictable and strategic by linking, for the first time, famine early warnings with prear- ranged financing to ensure that funds are released before a crisis emerges. It also seeks to tackle the root causes of famine and help build livelihoods, safety nets, and stronger coping skills in local communities. The World Bank Group has brought significant technical expertise to the World Bank Group Independent Evaluation Group    177 partnership. The experience so far suggests the following areas for improve- ment. First, since humanitarian partners work under different mandates and timelines than the Bank Group, greater investment of time and energy by the Bank Group in consulting and coordinating with them would ensure that the Bank Group benefits more fully from their expertise. Such consultation and coordination would also give partners a chance to provide feedback to the Bank Group and make links to their own work, building ownership among them. Second, a clearly defined operational link with Bank Group operations would help ensure that the partnership results in action in the field. This may require Bank Group leadership to signal the importance it attaches to the partnership in enhancing the Bank Group’s presence and effectiveness in the field. Finally, Bank Group staff capacity has been a constraint, with technical teams often being overstretched. Once again, signaling by Bank Group leadership on the importance it attaches to the partnership in helping achieve common goals would be helpful. Bank Group–LinkedIn Partnership The Bank Group–LinkedIn partnership was designed to make aggregated LinkedIn profile data available to the development community for bench- marking and evaluating the human capital assets and emerging (digital) industries and technologies used in the labor force in more than 100 coun- tries, especially in the context of adopting (disruptive) technology for eco- nomic growth. These aggregate data can be combined with other third-party Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix G or official data for analytical exercise to maximize policy insights that are related to job and technology trends and upskilling needs. The joint website (linkedindata.worldbank.org) with visuals of the latest industry and skills trends was launched internally at the Bank Group in November 2018 and made publicly available in April 2019. A peer-reviewed “World Bank Group–LinkedIn Data Insights” methodology paper highlighting the strengths, limitations, and potential of the LinkedIn data was released to the public (Zhu, Fritzler, and Orlowski 2018). All under- lying data sets were accessible by the public via the World Bank’s Develop- ment Data Hub as a public good. A more detailed aggregated data set that covers additional metrics, such as LinkedIn occupation data, was also made available to World Bank projects on request. In April 2020 these more de- tailed internal official-use-only data were also being uploaded to the Devel- opment Data Hub but required login from Bank Group computers. So far, the Organisation for Economic Co-operation and Development, the United Nations Development Programme, the United Nations Environ- ment Programme, and individual researchers in academia seem more ready to adopt the data generated through LinkedIn than Bank Group staff. The Organisation for Economic Co-operation and Development, for example, is using the skills penetration rate for artificial intelligence and talent migra- tion data in its Artificial Intelligence Observatory, and the United Nations Environment Programme is using the data to measure “green skills migra- tion” in its Higher Education Sustainability Initiative. Website visit numbers 178 and data set download numbers were also higher than the ex ante estimates of similar efforts by LinkedIn. Within the Bank Group, the initial fear was that researchers could misuse the data without fully understanding the caveats, but evidence from more than 10 World Bank projects that piloted use of the data seems to show this risk is manageable, as good researchers tend to use differ- ent sources of data to triangulate information and can identify which LinkedIn indicators are most representative and telling for the questions at hand. LinkedIn’s interest in the partnership comes from the Bank Group’s abil- ity to encourage governments to use data from alternatives to traditional data sources, such as household surveys. The major challenges for the Bank Group in this partnership include reliability of data sources not of its own provenance and staff reluctance to take risks with new types of data, espe- cially since LinkedIn data tend to be biased toward places and people who are digitally literate and have internet access. The reluctance may also partly arise from the Bank Group’s lack of experience or training with big data and other digital data. Overall, the lack of awareness and profile to the partner- ship internally within the Bank Group may also have been an issue. It would be important to understand the motivations of each party in entering a partnership and to determine if the partnership can deliver what each party wants and the data refresh sustainability plan. So far, LinkedIn is commit- ted to renewing the memorandum of understanding to keep refreshing the data as the indicators mature, and the need for such indicators is still strong. Hosting of the partnership is transitioning to the Development Economics World Bank Group Independent Evaluation Group    179 Vice Presidency Data Group to achieve economies of scale. Reference Zhu, Tingting Juni, Alan Fritzler, and Jan Orlowski. 2018. “World Bank Group—Linke- dIn Data Insights: Jobs, Skills and Migration Trends Methodology and Validation Results.” Working Paper 132009, World Bank, Washington, DC. Appendix H. Evolution of the World Bank Group’s Approach to DTT Although the technology engagements of the World Bank Group have been broad—encompassing support to science and technology, research and devel- opment, and innovation—specific strategies covering technology have existed mainly for the telecommunications and information and communication tech- Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix H nology (ICT) sectors. And the strategies have evolved over time. The pre-2002 approach emphasized expanding communications infrastructure and services and the role of the private sector in unleashing investments in infrastructure, while the public sector would put in place appropriate regulation to promote competition and ensure access. The 2002 ICT strategy reinforced these as- pects with a broader mandate for the public sector to support institutional and sectoral reforms. Also, for the first time, it identified support for ICT applica- tions and use in other sectors, and for ICT skills. The 2012 strategy retained focus on earlier priorities but emphasized the use of technology to transform the functioning of governments and service delivery. It also shifted focus from mobile telephony to broadband access and introduced a new pillar to promote innovation and technology entrepreneurship in the private sector, which also subsumed enhancing ICT skills. In 2018, the Bank Group adopted a new approach to disruptive and trans- formative technologies (DTT), later merging it with its 2019 DTT Main- streaming approach. The transition to “mainstreaming” from earlier discrete areas of support marked a significant increase in the Bank Group’s ambition regarding DTT. Accordingly, the Bank Group’s definition of DTT broadened beyond digital technologies to include other technologies (such as robotics, artificial intelligence, the internet of things, biotechnology, 3D printing, solar power, and batteries) and the analog complements (such as policies, institutions, and skills). It also extended beyond the “application” of tech- nology to development challenges (for example, using computers in the classroom) and the “response” to technology (for example, imparting socio- 180 emotional skills to equip people for future jobs that machines will not be able to perform). More recently, the development objective of the 2020 Program Document for the Mainstreaming Digital and Disruptive Technologies Initiative is to support units and teams in the scaling-up of appropriate high-quality digital and disruptive technology solutions in World Bank projects. An immediate focus of the initiative will be supporting the COVID-19 emergency response and recovery, and on knowledge sharing and supporting the development of knowledge products. This will be achieved through (i) establishing a Network and a Technical Working Group aimed at connecting teams and programs across the Bank Group, (ii) establishing a Secretariat aimed at supporting the Network and the Technical Working Group, and (iii) implementing an initial set of activities. The focus will be on technology applications, and the initia- tive will not include the foundations of the digital economy covered by other programs (such as Digital Economy for Africa). World Bank Group Independent Evaluation Group    181 Appendix I. ID4D: Three Pillars of the World Bank Group’s ID4D Support Based on information from the World Bank’s Identification for Development (ID4D) group, World Bank support for ID4D has been spread across the three pillars listed below. The initiatives are fairly new, and no evaluation exists yet of the impact of World Bank–supported ID4D projects on the twin goals or of the extent to which the risks and challenges of ID4D (for example, Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix I exclusion, privacy and security violations, vendor or technology lock-in, cybersurveillance, and unsuitable or unsustainable technology and design choices1) have been avoided. 1. Thought leadership a. Data on ID system coverage and characteristics: i. The ID4D Dataset, published every two to three years, compiles in- formation on global ID coverage for more than 190 economies. ii. The ID4D-Findex 2017 Survey (which added additional questions to the Findex 2017 Survey) provided the first sample survey across 90+ countries on ID coverage, usage, and barriers to access. b. Practitioner’s Guide: i. The Practitioner’s Guide is a comprehensive guide on how to imple- ment inclusive and trusted ID systems. ii. The Procurement Guide and Checklist for Digital Identification Sys- tems was developed by the ID4D group in collaboration with World Bank procurement specialists. 2. Global platforms and convening a. Principles on ID for Sustainable Development: 182 i. The ID4D group convened and facilitated the development of the ten principles for what makes a good ID system, which are endorsed formally by 25 organizations. These principles have been reflected in country policy documents, including Ethiopia’s Digital Strategy and Somalia’s ID policy. ii. The principles cover data protection and privacy. b. The World Bank has helped shape open-source and open standards platforms to address vendor lock-in and scalability challenges. The first open-source ID platform is being implemented in Morocco and the Philippines and piloted in Guinea. 3. Country and regional action a. Nigeria: ID4D’s technical assistance contributed to (and International Development Association financing is now supporting implementation of) the government: i. Integrating a privacy-by-design approach into the foundational ID system; and ii. Adopting an ecosystem approach to increasing its registration, en- gaging other government agencies and the private sector to widen availability of registration centers. World Bank Group Independent Evaluation Group    183 b. Philippines: ID4D’s technical assistance contributed to the government: i. Adopting an inclusive registration approach for the new founda- tional ID system; ii. Using a COVID-safe registration approach, which allowed the launch of the registration in October 2020; iii. Adopting of tokenization of the unique ID number so that it is not printed on the physical card and therefore protected from misuse; and iv. Adopting open-source software to promote country ownership of the technology. 1  https://id4d.worldbank.org/guide/creating-good-id-system-presents-risks-and-challenges- there-are-common-success-factors. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix I 184 Appendix J. Select Examples of the World Bank Group’s COVID-19 Activities Across Sectors  elect Examples of how COVID-19 Responses are Accelerating Figure J.1. S WBG Engagement on Digital Foundations and Applications Source: World Bank 2020, 6 World Bank Group Independent Evaluation Group    185 Reference World Bank. 2020. “The Final Program Document for Mainstreaming Digital and Disruptive Technologies (‘MDDT’) Initiative.” World Bank, Washington, DC. Appendix K. Scaling Access to Solar Energy: Lighting Global and Scaling Solar Two nondigital disruptive and transformative programs supported by the World Bank Group have aimed to expand access to solar energy. The Lighting Africa Program is a regional Bank Group project that has sup- ported the rapid scale-up and delivery of affordable, quality lighting products Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix K that are predominantly solar lanterns used for household lighting—nearly 7 million people had been reached as of 2015 (World Bank 2015). Subsequent- ly, the World Bank provided similar support outside of Africa—in Bangladesh, India, and Papua New Guinea, with more programs being developed in Indo- nesia and Pakistan. The World Bank and International Finance Corporation’s Lighting Africa and Lighting Asia programs and the successor program, Light- ing Global,1 pioneered well-designed and targeted technical assistance for improving quality assurance and service delivery (World Bank 2016). Scaling Solar, an International Finance Corporation–led Bank Group pro- gram, provides governments with electricity supply solutions. Scaling Solar brings together several Bank Group services under a single engagement.2 Scaling Solar offers advice, simple and rapid tendering, fully developed templates, competitive financing and insurance, and risk management and credit enhancement. It is designed to address the challenges of scaling up; for example, limited institutional capacity, lack of scale, lack of competition, high transaction costs, and high perceived risk. Future efforts by the Bank Group to increase the use of disruptive and transformative technologies can benefit from studying these programs and identifying ways to overcome the challenges that scaling up typically poses. World Bank (2016) notes that “despite some significant pilot examples of micro-grid and mini-grid project components supported by the World Bank Group and in the private sector, scaling up continues to pose a challenge – in terms of institutional arrangements and for commercially viable business 186 models.” More generally, Cull and McKenzie (2020) summarize scaling-up challenges as follows: (i) small-scale pilots may concentrate efforts on those who benefit most, but scaling up may involve a loosening of eligibility cri- teria, and/or smaller effects on new areas; (ii) implementation and political economy issues can arise as programs grow (monitoring and implementation becomes more difficult at a larger scale, possibly subject to lower capacity and vested interests); and (iii) general equilibrium effects can further reduce some impacts; for example, training a small number of workers to find new jobs may be successful, while expanding training to additional workers may generate competition among the workers for the available jobs with a lower impact on overall employment. References Cull, Bob, and David McKenzie. 2020. “Implementing Successful Small Interventions at a Large Scale Is Hard.” Let’s Talk Development (blog), March 19, 2020. https:// blogs.worldbank.org/developmenttalk/implementing-successful-small-inter- ventions-large-scale-hard. World Bank. 2015. World Bank Group Support to Electricity Access, FY2000–2014. In- dependent Evaluation Group. Washington, DC: World Bank. https://ieg.world- bankgroup.org/sites/default/files/Data/Evaluation/files/Electricity_Access.pdf. World Bank. 2016. Reliable and Affordable Off-Grid Electricity Services for the Poor: World Bank Group Independent Evaluation Group    187 Lessons from the World Bank Group Experience. Learning Product. Independent Evaluation Group. Washington, DC: World Bank. https://ieg.worldbankgroup. org/sites/default/files/Data/reports/lp_off-grid_electricity_1116.pdf. 188 Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness  Appendix K 2  1  https://www.scalingsolar.org. https://www.lightingglobal.org/about. The World Bank 1818 H Street NW Washington, DC 20433