Learning in World Bank Lending An Independent Evaluation © 2025 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: World Bank. 2025. Learning in World Bank Lending. Independent Evaluation Group. World Bank. COVER PHOTO “Avenue of Indian Rubber Trees at Peradeniya, Ceylon” by Marianne North. The Board of Trustees of the Royal Botanic Gardens, Kew. CC BY-NC. EDITING AND PRODUCTION Amanda O’Brien GRAPHIC DESIGN Vonda Witley This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Learning in World Bank Lending An Independent Evaluation March 24, 2025 Contents Abbreviations v Acknowledgments vi Overview vii Management Response xiii Background and Context���������������������������������������������������������������������������������������� 1 1.  Approach and Scope 4 Conceptual Framework and Definitions 5 Methods 8 Report Structure and Overarching Finding 9 Knowledge and Learning in the Linear Model��������������������������������������������������� 10 2.  Project Preparation 12 Project Appraisal and Approval 17 Project Implementation 22 Completion and Lesson Learning 29 The World Bank’s Ecosystems for Knowledge and Learning���������������������������� 32 3.  Resources 34 Clients and Partner Contributions 35 Opportunities and Capacities 39 Conclusions and Recommendations������������������������������������������������������������������ 53 4.  Areas for Management Attention 54 Recommendations 58 Bibliography�������������������������������������������������������������������������������������������������������������� 64 ii Boxes Key Definitions for Learning in Lending Box 1.1.  6 Knowledge Reorganization of the Health, Nutrition, and Box 3.1.  Population Global Practice 40 Business Units with Systematic Approaches to Link Producers Box 3.2.  and Users of Knowledge 50 Examples of Lending Processes Where World Bank Management Box 4.1.  Could Infuse Learning Opportunities  59 Figures Regional Results of the Two-Minute Client Survey Figure 1.1.  2 The Linear Model: Knowledge Entry Points in the Project Figure 1.2.  Life Cycle 5 Methods Figure 1.3.  9 Timing of Peer Reviewers’ Advice Before Project Concept Figure 2.1.  Review Meetings 19 Operations with a Quality Enhancement Review by Figure 2.2.  Lending Instrument Type and Lead Global Practice 20 Share of Peer Review Advice from the Peer Reviewer Database Figure 2.3.  21 Task Team Leader Turnover Rates Figure 2.4.  25 Reviewed Multiphase Programmatic Approach Learning Figure 2.5.  Agendas That Met Select Quality Criteria 28 Ecosystem for Learning in Lending Figure 3.1.  33 Processes That Knowledge and Learning Professionals Support Figure 3.2.  48 Tables Overview of World Bank Financing Instruments Table 2.1.  11 Examples of Instrumental, Strategic, and Conceptual Knowledge Table 2.2.  by Lending Instrument 13 Distribution of All Project and Program References by Authorship Table 2.3.  14 Core and Extended Core Advisory Services and Analytics Table 2.4.  Referenced in Project Appraisal Documents 16 Purposes of Knowledge in Multiphase Programmatic Table 2.5.  Approach Programs 27 iii Functional Comparison Table 3.1.  43 Approaches to Country, Sector, and Operational Knowledge Table 3.2.  45 Knowledge and Learning Shortcomings Identified by Table 4.1.  the Evaluation 56 Appendixes Appendix A. Methods 70 Appendix B. Past Independent Evaluation Group Evaluations and Research 87 Appendix C. Deep Dive into Multiphase Programmatic Approach 97 Appendix D. Analysis of References in Project Design Documents 100 Appendix E. Analysis of Two-Minute Client Satisfaction Survey Data 110 Appendix F. Analysis of Project Task Team Leader Turnover 120 Appendix G. Analysis of Project Peer Reviewers 134 iv Abbreviations ASA advisory services and analytics DLI disbursement-linked indicator DPF development policy financing GP Global Practice ICR Implementation Completion and Results Report IEG Independent Evaluation Group IPF investment project financing MPA Multiphase Programmatic Approach PAD Project Appraisal Document PforR Program-for-Results TTL task team leader All dollar amounts are US dollars unless otherwise indicated.     v Acknowledgments This evaluation was prepared by a team from the Independent Evaluation Group led by Rasmus Heltberg (lead evaluation officer). The core team included Farhana Sonia Ahmad, Harsh Anuj, Stuti Sachdeva, and Pascal Saura. Gaby Loibl, Ma. Roanna Sayes, and Munib Qasim Zia provided admin- istrative, graphic, and data collection support. Maximillian Ashwill was the developmental editor and content coach. Estelle Raimondo, Onno Ruhl, and Thomas Delahais provided methods and conceptual guidance. The evaluation was conducted under the guidance and supervision of Galina Sotirova (manager), Timothy Johnston (manager), and Theo Thomas (director) and the overall direction of Sabine Bernabè (Director General, Evaluation, and vice president). Ellen Goldstein, Ede Ijjasz-Vasquez, and Ruth Levine were the peer reviewers. vi Overview This evaluation assesses the World Bank’s approach to knowledge and learning in its lending operations. The evaluation is guided by the high-level question, How can the World Bank create optimal conditions for learning in and from its financing operations? The evaluation’s analysis covers all financing instruments; country, sectoral, and operational knowledge; and tacit and explicit knowledge (box 1.1 defines these and other key terms). The evaluation applies a mixed methods, phased approach that combines case studies of 34 lending projects with desk reviews, interviews, a questionnaire, and quantitative analyses. It aims to support World Bank management’s ongoing efforts to improve the institution’s performance as a “Knowledge Bank” as outlined in The Knowledge Compact for Action: Transforming Ideas into Development Impact—For a World Free of Poverty on a Livable Planet and Realizing the World Bank Group’s Knowledge Potential for Effective Development Solutions: A Strategic Framework (World Bank Group 2021, 2024). Overall, the evaluation finds that the World Bank’s knowledge ecosystem has a formal, linear model for embedding knowledge in its lending processes that creates learning moments at specific points of the project cycle, especially during the preparation stage. The World Bank also has a strong informal learning culture that often helps project teams identify actionable sector and country knowledge. However, this ecosystem has imbalances, weak lesson learning, and fragmented institutional support. Knowledge and Learning in the Linear Model The World Bank uses a linear model to embed knowledge in its financing through key entry points in a project’s preparation, approval, implementa- tion, and completion phases. The linear model excels at producing explicit technical knowledge sourced from both inside and outside the World Bank to inform project designs. Consequently, sampled project designs used diverse knowledge sources that predictably included much World Bank–generated knowledge, including project diagnostics and various types of advisory ser-     vii vices and analytics, but also frequently turned to knowledge from clients, multilateral institutions, and other external sources. These projects rarely used the World Bank’s “core” advisory services and analytics, which are a specific set of World Bank reports prioritized by the Knowledge Compact that serve to inform decision-making and policy making at strategic levels for the World Bank, its clients, and other partners and stakeholders in a way that tran- scends specific operations. Indeed, only 2 percent of all references in Project Appraisal Documents are to any of the core or extended core advisory services and analytics report series. Development policy financing operations more often cited these reports. The linear model is not ideal for fostering learning during the project im- plementation phase and underemphasizes country knowledge. The project preparation phase has budgets and formal requirements to produce specific knowledge inputs and technical assessments, which the implementation phase lacks. Learning diminishes once a project is launched. Moreover, the linear model focuses its explicit knowledge work on generating sectoral knowledge rather than country knowledge, which tends to be tacit and shared unsystematically. Lending instruments vary in what type of knowledge input they emphasize. Investment project financing operations use studies to identify or justify new projects, to design a project’s implementation arrangements, or to identify the specific pieces of infrastructure it will finance. Development policy financing operations drew on policy studies, and Program-for-Results (PforR) financ- ing operations used integrated risk assessments to bolster projects’ technical soundness and support for client systems. Learning in World Bank Lending Overview PforR and Multiphase Programmatic Approach operations have knowledge mandates that better suit them to learning. The investment project financing and development policy financing instruments have formal policies, proce- dures, and guidance to acquire or use knowledge in the operations’ design phase but not during the implementation phase. PforR operations, by contrast, have explicit incentives and mechanisms for learning during implementation because of their annual validation of disbursement-linked indicators (DLIs). The DLI process creates a formal, mandated space for knowledge sharing and learning among World Bank teams and clients. This space is supported by DLI data on PforR’s progress toward achieving project outcomes. The Multiphase Programmatic Approaches similarly focus on knowledge because they require viii learning agendas that span the Multiphase Programmatic Approaches’ mul- tiple phases and operations. However, these learning agendas are not always of the same quality, with some being comprehensive and fully implemented and others not. World Bank operations have a formal lesson learning mandate during Implementation Completion and Results Reports, but these are un- derused by teams who frequently view them as compliance tools rather than learning tools. Learning from tacit knowledge and informal exchanges is deeply embedded in the World Bank’s organizational culture. Yet the World Bank has not optimized the transfer of its tacit knowledge. Indeed, some of the most valuable and widespread learning moments are the least formal ones, often on the margins of the linear model. For example, the World Bank has a structured and valuable peer review process for improving project design quality. Quality enhance- ment reviews are also valuable because they occur earlier and are less formal than other review meetings. These processes create the most value for project teams when they result in early and frank advice and dialogue in an informal safe space. Certain forms of knowledge are better preserved by the World Bank for later use than others. The evaluation found that review and decision meetings, peer reviews, and studies underpinning project designs were well document- ed, although that does not ensure use. In contrast, trust-funded studies, project-commissioned innovations, and informal client learning during Independent Evaluation Group World Bank Group    ix implementation were not often documented, making such knowledge hard to reuse or build on. Further, the World Bank does not document the lessons from dropped and canceled operations, hindering learning from these expe- riences and from mistakes and failures more broadly. Additionally, existing tools to capture and learn from country knowledge have shortcomings, mak- ing country office staff the primary repository for tacit country knowledge (World Bank 2020d). Ecosystems for Knowledge and Learning The World Bank relies on a broader decentralized knowledge ecosystem that goes beyond the formal linear learning model for lending. This ecosystem includes the resources, opportunities, capacities, and client engagements that create effective learning conditions for staff outside of mandated project-related requirements. It is within this broader knowledge ecosystem that successful teams and units create the most valuable learning opportunities using tacit knowledge, regular exchanges within and beyond the team, safe spaces, and sustained client engagements. Trust funds often support them in this process. In sum, the linear model for embedding knowledge in financing relies on formal knowledge mandates in the project design and approval stages, whereas the broader knowledge ecosystem relies on tacit knowledge, informal ap- proaches, and staff’s and managers’ motivation to drive learning. Global units’ knowledge production often focuses on corporate and global unit priorities and not on operational needs or country knowledge. The absence of a centralized knowledge system means business units must develop their own knowledge systems, with some thriving and some falling behind. The World Bank has no minimum standards and has few incentives and mechanisms to ensure excellence in knowledge sharing and use. Learning is often driven by task team leaders’ motivation and external trust funds. The evaluation finds that in the absence of formal learning mandates, it is the staff’s motivation to pursue knowledge and learning that often de- termines the level of knowledge and learning the World Bank generates. Moreover, without internal administrative budgets for knowledge and learn- ing, teams must turn to trust funds to finance these activities. These become incredibly valuable for new projects that do not have prior operations from which to learn. However, this also means that much of this important work depends on the availability of trust fund resources. Learning in World Bank Lending Overview Tacit learning is preferred by staff and embraced by some managers because it is a trusted and easy-to-access source of actionable knowledge. This infor- mal learning stems from candid conversations in informal safe spaces. These spaces are valuable because of their confidentiality, the access they provide to trusted experts, and their ability to generate actionable knowledge that teams can directly apply in their projects’ contexts. Approximately half of the sampled World Bank teams collaborated with development partners on learning initiatives, and nearly all task teams shared knowledge with clients and fostered joint learning. Yet informal sharing of tacit knowledge also has risks and limitations, such as narrow diffusion of knowledge, bias toward x perceived successes, the tendency to ignore failures, and inaccuracies. Despite its prevalence, the transfer of tacit knowledge is not systematized within the institution. For example, the World Bank’s few knowledge and learning staff primarily focus on explicit knowledge sharing but do not regularly support in- formal knowledge exchanges nor have the remit to support World Bank–wide knowledge management. Conclusions The World Bank’s approach to learning (which typically encompasses a linear, formal project-driven model and multidimensional, informal systems) has three major imbalances. First, the linear model creates a focus on generating sectoral knowledge and producing reports but often neglects applying this knowledge to operations and country-level processes. It also steers teams’ and management’s attention to the design phase, leading to less attention to knowledge and learning during the implementation phase. Second, the World Bank’s lesson and country learning often relies on unpacking tacit experiences in prior or parallel operations. However, this approach is not always reliable because it depends on chance discoveries and shortchanges smaller countries and new engagement areas that typically lack prior or parallel operations to learn from. This is part of a larger issue of a culture that greatly values infor- mal knowledge exchanges but organizes them with little methods and support. Third, the World Bank’s knowledge ecosystem is generally fragmented, discon- nected, and underresourced. This lack of consistency and minimum standards leads to inefficiencies in the knowledge flows. Independent Evaluation Group World Bank Group    xi Recommendations Recommendation 1. Make better use of the learning opportunities that are already embedded in the lending processes. Operations Policy and Country Services should revise the procedures and guidance for lending to incen- tivize more consistent learning throughout the lending cycle. Specifically, Operations Policy and Country Services should set clear expectations to the type of knowledge and learning, the moments and processes for learning activities, and the level of client engagement required from project teams (ex- amples of what such process tweaks could entail are offered in chapter 4). At the same time, managers at all levels should create spaces for knowledge and learning, including via quality enhancement reviews and informal and early meetings with peer reviewers. Managers should also role model attention to knowledge and learning and openness to discussing failures. Recommendation 2. The Knowledge and Learning Department should ensure core knowledge management capacity and set World Bank–wide standards and processes for knowledge capture, storage, sharing, and access. The World Bank should rationalize the core capacity to organize key learning events with instructional designers and knowledge management profession- als. The essential knowledge management capacities, standards, and processes should have oversight by senior management. Implementing this recommen- dation could entail: » Setting standards for how knowledge is tagged, classified, stored, and shared across units. » Supporting the Global Practices in adopting these standards, for example, by insisting that knowledge from key events such as Knowledge Weeks, client workshops, and long-term learning engagements is tagged, classified, stored, and shared across units. » Professionalizing and enhancing the capacities of knowledge management staff. Supporting career management for knowledge management staff. » Working with the Global Practices and Regions to better leverage professional knowledge management staff and thereby relieve task team leaders. » Working with the Global Practices and Regions to bring enhanced methods Learning in World Bank Lending Overview and support to knowledge sharing activities, including enhanced attention to instructional design. » Bringing methods and support to enhance informal knowledge exchanges. Investing also in communities of practice and technical help desks. » Periodically surveying staff’s need for, and satisfaction with, knowledge management. xii Management Response Management of the World Bank welcomes the Independent Evaluation Group’s report, Learning in World Bank Lending, and thanks the team for addressing the management comments provided earlier. The report examines how knowledge is used in the World Bank’s project design, implementation, and completion phases. The evaluation is timely considering the launch of the Knowledge Compact and the opportunity through new “horizontals” to strengthen coordination and adoption of more systematic and modern ap- proaches to leverage knowledge and learning (K&L) to improve impact. The insights provided by the report are valuable for improving design, implemen- tation, evaluation, and overall K&L practices. Management thanks IEG for its continued collaboration and notes that IEG plans a follow-on report for K&L topics not covered or only lightly covered in the current evaluation. Management welcomes the report’s recognition of the World Bank’s long- term commitment to being a “knowledge bank” in producing, disseminating, and applying knowledge through its established partnerships and the work of the teams and managers in the knowledge ecosystem. Management appreciates that the evaluation considered how the World Bank embeds knowledge in its financing. Further, the evaluation recognizes the World Bank’s strong reputation as a provider of development knowledge that builds on its deep relationships with clients and cross-sectoral expertise and Independent Evaluation Group World Bank Group    xiii in contributing policy advice with financing. Management thanks IEG for providing important evidence, which will assist management in follow-up discussions on the role of K&L across the World Bank—and the wider World Bank Group. This includes the helpful presentation of summarized anal- yses and guidance in tables. Additionally, management appreciates IEG’s acknowledgment of recent efforts to enhance its K&L practices and the seriousness with which management believes in the importance of this work. This includes, among other things, the launch of the Knowledge Compact associated with the Evolution and Better Bank process, the establishment of the K&L Department in the Senior Managing Director’s office, the updating of the country engagement arrangements that involve K&L, among others. Management acknowledges the report’s findings of areas to strengthen the World Bank’s approaches to K&L, and the World Bank is taking action to address them. Management recognizes the value in evolving from lin- ear knowledge inputs to an open knowledge ecosystem, noting that the Knowledge Compact and establishment of the Senior Managing Director’s K&L Department is key to addressing this. Management recognizes the mismatch between the supply side of knowledge generation (especially from global units) and the demand side of knowledge needs (from the re- gions). Management recognizes various other challenges in K&L, including resourcing involving staffing and reliance on trust funds; projects spending more resources in the design phase and giving less attention to K&L during implementation; learning processes that can be better employed to im- prove formal lesson learning (such as with Implementation Completion and Results Reports [ICRs] and ICR Review), which are being addressed with the ICR reform process, with, among other things, plans for distilling lessons from these documents into a knowledge tool for staff preparing projects). Management notes that IEG identified several areas for future research relat- ed to K&L topics that were out of scope for this evaluation or provided only partial analysis. These include, as IEG described in the evaluation, advisory services and analytics, client learning and capacity development outcomes, and culture and incentives related to K&L. Management appreciates that Learning in World Bank Lending Management Response future research and evaluations could consider covering these topics. Recommendations Management broadly agrees with both recommendations. The first recommendation emphasizes making better use of learning op- portunities that are already embedded in the lending process. Management broadly agrees with this recommendation and is committed, as appropriate, to identifying ways to further strengthen the implementation of learning opportunities already embedded in the lending process; and, as appropriate, amend or issue necessary operational guidance to staff. Management will also explore more opportunities for learning that exist in the project cycle, beginning with accreditation of task team leaders, and throughout the vari- ous stages of project preparation and on to implementation and completion, xiv including through the preparation of ICRs (as mentioned previously). Management emphasizes the need to support these efforts of learning from operations with agility as part of the ongoing operational efficiency and effectiveness initiatives. For the second recommendation, work is already underway to ensure core knowledge management capacity and World Bank–wide standards and processes for knowledge capture, storage, sharing, and unpacking—with senior management oversight. This involves developing more systematic ap- proaches to harnessing K&L for greater development impact. The new K&L Department under in the Senior Managing Director’s unit has been created to bring order, structure, and professionalism to the K&L agenda. Efforts include strengthening the knowledge ecosystem, including access to tacit knowledge, through intentional, systematic, and impact-driven approaches that leverage advanced technology. Management appreciates IEG’s sugges- tions on implementation and monitoring, noting that the K&L Department is collaborating with partners to better coordinate access and use of knowl- edge across the World Bank Group. The K&L Department is already working closely with IEG to inform these efforts, building on the findings of this evaluation. Independent Evaluation Group World Bank Group    xv xvi Learning in World Bank Lending 1 | Background and Context The World Bank Group considers itself a “Knowledge Bank.” It pursues its vision of a world free of poverty on a livable planet by supporting developing countries with funding and knowledge. Since the 1990s, it has increasingly cast itself as a Knowledge Bank, even more so than as a lending institution. The World Bank’s knowledge work contributes to development outcomes both by informing clients in various policy and reform areas and by inform- ing staff and clients on the selection, design, and implementation of lending operations. The Bank Group’s knowledge work has strengths and limitations. External assessments, client surveys, and Independent Evaluation Group (IEG) reports consistently show that member countries and development partners value the depth and breadth of the World Bank’s knowledge work. However, client sur- veys highlight a gap in tailoring its knowledge to country contexts (figure 1.1). Moreover, the World Bank’s lending delivery model does not readily support projects’ adaptation and course corrections to circumstances that emerge during implementation (World Bank 2019a, 2020b). The transaction costs of designing solutions from the best available evidence remain high, which is part of a bigger limitation—namely, that the World Bank does not systemati- cally and consistently learn from its own projects and programs (World Bank 2016, 2020d; World Bank Group 2021). A Knowledge Bank’s greatest asset is its ability to embed, create, and share knowledge. The World Bank achieves this but has room to improve the methods, predictability, and consisten- cy with which it does so, as discussed in Realizing the World Bank Group’s Knowledge Potential for Effective Development Solutions: A Strategic Framework (World Bank Group 2021). Over the years, the World Bank has sought to improve its approach to being a Knowledge Bank. These efforts focused on two themes. First, the World Bank optimized its organizational structure to remove the geographical silos that impeded the flow of knowledge. There have been various initiatives, includ- ing more recently in 2014 with the creation of Global Practices (GPs). These     1 GPs were designed to provide globally integrated technical knowledge and oversee various mechanisms to move staff and knowledge across countries. The World Bank revised the GP model in 2019 and 2020 by moving most GP staff to the Regions. Second, the World Bank enacted strategies, action plans, mechanisms, and incentives to better carry out its knowledge man- date. Two prominent recent initiatives demonstrate this. The 2021 Strategic Framework for Knowledge aimed to improve the Bank Group’s knowledge contributions by adjusting its knowledge systems, creating staff incentives, and enhancing staff expertise through hiring practices and other means (World Bank Group 2021). Similarly, the 2024 Knowledge Compact for Action is part of a larger Bank Group reform known as the evolution (World Bank Group 2023). This Knowledge Compact emphasizes using new technolo- gies to deliver knowledge to clients with greater speed, quality, and impact (World Bank Group 2024). It proposes specific changes to the Bank Group’s knowledge products, processes, budgeting, systems, partnerships, and capac- ity building.  egional Results of the Two-Minute Client Survey Figure 1.1. R Eastern and Southern Africa Latin America and the Caribbean South Asia Region East Asia and Pacific Western and Central Africa Learning in World Bank Lending Chapter 1 Europe and Central Asia Middle East and North Africa 70 75 80 85 90 95 Share of survey responses (%) The World Bank brought global expertise to support this activity. The World Bank's expertise was tailored to my country context. Source: Independent Evaluation Group, based on the two-minute client surveys the World Bank does for all lending projects. Note: The number of projects in the sample is 1,374 for country context question and 1,378 for global expertise question. The data are for projects approved during FY14–23. 2 Four issues affected the World Bank’s past efforts to improve the Knowledge Bank. First, senior management largely left decisions about the World Bank’s knowledge architecture to the GPs. Second, the World Bank’s past efforts lacked continuity because management discontinued certain strategies and approaches, as well as a previous central knowledge management team. This instability limited the time that business units and the organizational culture had to adjust to new knowledge management approaches. Third, these past efforts have often focused on creating databases and other for- mal knowledge management mechanisms and have not seriously embraced World Bank staff’s revealed preference for informal knowledge sharing. Fourth, the World Bank has room to improve how it manages the balance between its traditional country-based model and its newer and growing role as a convener and knowledge provider on global issues. The World Bank engages primarily through a country-based model, conducting country diag- nostics, working with government to establish a program based on strategic priorities and Bank Group comparative advantage, and using lending and knowledge to pursue country development goals. This engagement model has several advantages, chiefly in promoting a country-owned development agenda, which is critical for development effectiveness. However, the model faces challenges in addressing regional and global issues because it focuses the program on those national priorities for which governments demand support, which do not necessarily align with global and regional challenges and global public goods. This evaluation is timely because it supports ongoing efforts to improve Independent Evaluation Group World Bank Group    3 the Knowledge Bank. Similar to the Strategic Framework for Knowledge and the Knowledge Compact, this evaluation’s broad purpose is to help the World Bank improve learning in (and from) World Bank–financed opera- tions. Specifically, it aims to identify actions that the World Bank could take to ensure impactful learning in all its financing operations. This includes improving learning with clients and partners, enhancing staff incentives and managerial signals for knowledge, revitalizing the World Bank’s knowledge and learning staff, and generating the knowledge required for scaling and replicating impactful projects. The evaluation’s purpose and scope were informed by the Strategic Framework for Knowledge and the Knowledge Compact; the internal discussions surrounding the preparation of these documents, in which IEG participated; and discussions with managers and directors from the Regions, Practice Groups, and Development Economics Vice Presidency. Approach and Scope This evaluation is guided by the high-level question, How can the World Bank create optimal conditions for learning in and from its financing operations? Past IEG evaluations, described in appendix B, have covered different aspects of the Knowledge Bank and established the positive links between knowledge and outcomes. This evaluation examines the “production function,” or the enablers and processes, that World Bank teams and managers use to ensure that knowledge is impactful and contributes to granular insights and action- able recommendations to systematically foster multidimensional learning to improve operations and, ultimately, development outcomes. The evaluation’s scope is knowledge and learning in and from World Bank financing. First, the evaluation covers the World Bank, which includes the International Bank for Reconstruction and Development and the International Development Association. It centers more on learning for its staff and management and less on learning for clients and partners. Second, the evaluation focuses on the World Bank’s unique value proposition as an institution that functions between global dynamics and local development. This unique value proposition comes together in lending operations, from project preparation to completion (figure 1.2). The evaluation therefore looks at the timing, purpose, and types of knowledge surrounding develop- ment projects and assesses how “learning enablers”—or factors associated Learning in World Bank Lending Chapter 1 with more knowledge inputs and better conditions for learning—fit and extract knowledge during certain milestones in the project life cycle and how these pieces of knowledge fit together to contribute to learning in lending. Third, the evaluation covers both country knowledge and sector knowledge, formal and informal knowledge processes, and tacit and explicit knowledge (for definitions, see the Conceptual Framework and Definitions section). Fourth, the evaluation period is 2014–23, coinciding with the creation of the GPs. Fifth, the evaluation considers knowledge as both an input to action and an output or result from action. Consequently, the evaluation focus- es on how the World Bank applies and expands its knowledge (knowledge as a process) rather than on how it compiles and organizes its knowledge 4 (knowledge as a resource). For example, the evaluation examines World Bank projects’ use of advisory services and analytics (ASA) but not the volume, topic, or quality of ASA.  he Linear Model: Knowledge Entry Points in the Project Figure 1.2. T Life Cycle Source: Independent Evaluation Group. This evaluation’s underlying assumption is that effective management of knowledge resources and processes facilitates continuous learning, contrib- utes to performance, and ultimately yields better outcomes. Managing for outcomes requires having credible, operationally relevant knowledge that can feed into the choice and design of operations and inform course cor- rection during project implementation. Past research and evaluations have found a positive role of ASA, staff quality, monitoring and evaluation quality, and other aspects of knowledge on measures of the World Bank’s effective- ness (appendix B). The link has also been documented in other industries (Davenport and Prusak 1998; Goldhar et al. 1976; Orpen 1985). Strategy doc- uments over the years have repeatedly highlighted that knowledge generated by World Bank–financed operations is an important part of the World Bank’s Independent Evaluation Group World Bank Group    5 value proposition. For example, the 2018 Capital Increase Package highlight- ed the vast amount of knowledge embedded in operations and the World Bank’s role in creating and transferring knowledge as part of its financing, and framed knowledge as a way to create greater impact. This evaluation focuses on the factors associated with knowledge inputs and learning condi- tions and does not attempt to link knowledge inputs to projects’ outcomes. Conceptual Framework and Definitions The evaluation defines learning as the ability of World Bank staff to iden- tify, transfer, and transform knowledge to benefit development outcomes. This definition supports the evaluation’s examination of the processes that inform lending operations and capture the lessons of implementation in a continuous learning loop. The focus is on staff learning. Box 1.1 defines the terms related to multidimensional learning in lending.  ey Definitions for Learning in Lending Box 1.1. K Knowledge ecosystem: References, resources, capacities, opportunities, and cli- ent engagements that support effective learning conditions for staff and projects. The knowledge ecosystem enables or hampers learning through factors within the control of the World Bank (processes and management activities, capacity of staff and systems, and staff motivation) and outside of the World Bank’s control (social contexts, economic trends, political situations, and so on). Knowledge forms: » Explicit: Codified and digitized in books, documents, reports, memos, and so on. It is documented information that can facilitate action and knowledge that is easily identified, articulated, shared, and used. » Tacit: Unspoken, intuitive, and experiential insights that individuals accumulate over time. Unlike explicit knowledge, tacit knowledge is personal and often chal- lenging to articulate or document. Knowledge inputs: Knowledge resources and processes that influence project design and implementation. Knowledge inputs have multiple dimensions—types, forms, sources, and uses. Knowledge sources: » Internal: From World Bank Group staff and publications. Learning in World Bank Lending Chapter 1 » External: From country clients, development partners, academia, and others. Knowledge types: » Country knowledge: Specific to the development challenges and opportunities within an individual country—for example, knowledge related to the country’s stakeholders, political economy, cultural context, or implementing agencies. » Global knowledge: Sectoral knowledge generalized across countries. » Operational knowledge: Specific to the World Bank’s policies, procedures, and standard practices around financing. 6 (continued)  ey Definitions for Learning in Lending (cont.) Box 1.1. K » Sectoral knowledge: Specific to the issues, sectors, and interventions at the core of lending operations. Knowledge uses: » Conceptual: Informs thinking and promotes shared understanding of an issue. » Instrumental: Solves a problem, such as how to design a specific project compo- nent, and requires that task teams have access to the right knowledge inputs at the right time. » Strategic: Builds a business case for a project or motivates a decision. Learning: The ability of World Bank staff and management to identify, transfer, and transform knowledge into better development outcomes. Learning occurs when knowledge is transformed so it can be applied to practice. Multidimensional learning: Learning proceeding from and leading to several or all types and forms of knowledge, originating from internal and external sources, and adapting to various uses during the project life cycle. Source: Independent Evaluation Group. Multidimensional learning results from using multiple dimensions of Independent Evaluation Group World Bank Group    7 knowledge. Learning is elicited through the lending project life cycle by the knowledge ecosystem. This knowledge ecosystem provides positive or negative learning enablers, some within the World Bank’s control and some not. Learning comes from and leads to knowledge, which spans multiple dimensions, including operational, sectoral, and country knowledge, coming in two forms: explicit or tacit. This knowledge can come from inside or out- side the World Bank. The World Bank can use this knowledge for strategic, instrumental, and conceptual purposes. Hence, multidimensional learning encompasses all types and forms of knowledge. Effective learning in lending can lead to faster project preparation, faster adaptation to changing cir- cumstances and emerging issues, faster scaling of projects and acceleration of results, more systematic replication of successes and innovations, better avoidance and building on past failures and mistakes, and more effective capacity building and joint learning with clients and partners. In other words, effective learning in lending is closely linked to managing for outcomes. Methods The evaluation applies a mixed methods, phased approach to the conceptu- al framework. IEG used case-based methods to review a sample of 34 World Bank operations approved between 2014 and 2023 (appendix A). IEG selected these cases—20 investment project financing (IPF), 8 development policy financing (DPF), and 6 Program-for-Results (PforR) operations mapped to 10 GPs and covering all Regions—through a combination of stratified random and purposive sampling. The sample included 16 active and 18 closed op- erations to allow for analysis of knowledge inputs used through the project cycle, from design and appraisal to completion. IEG collected data on the 34 case operations’ knowledge inputs from project documents and semistruc- tured interviews with past and current task team leaders (TTLs). The case study data collection focused on analytic work (including ASA, to the extent these were used to inform projects), lessons from other projects, review processes, team composition, learning with clients and partners, and other internal knowledge management mechanisms. In addition, the evaluation conducted a desk review of 34 learning agendas in a sample of Multiphase Programmatic Approach (MPA) operations approved between 2017 and 2024. The evaluation did not deeply analyze learning from the self-evaluation sys- tems and the country engagement cycle but drew on prior IEG evaluations on this. Learning in World Bank Lending Chapter 1 IEG generated and further tested the emerging findings. As shown in figure 1.3, IEG’s case analysis generated findings and hypotheses about what types of knowledge inputs are used, in what stage of the project cycle, how they are used depending on the financing instrument, and what factors enable learn- ing. IEG further explored the emerging findings and hypotheses using quantitative analysis of project peer reviews, project document references, and TTL turnover; reviews of financing instrument guidance and other documents; direct observations of review meetings; a questionnaire for knowledge and learning professionals; and interviews with practice manag- ers from both sides of the matrix. The evaluators discussed early findings 8 emerging from triangulating findings across all qualitative and quantitative methods in a series of meetings with staff and managers.  ethods Figure 1.3. M What types of knowledge inputs inform the design and implementation of World Bank financing operations? Case-based analysis Mixed methods for hypothesis testing 34 Cases Reference analysis Client survey Turnover analysis Text mining and Analysis of Document Semistructured information Generative AI two-minute Observations Peer review review TTL interviews extraction client survey algorithms analysis Within- and cross-case analysis Set of hypotheses MPA analysis Have different operational units put effective knowledge enablers in place? World Bank staff Questionnaire to the Benchmarking interviews K&L community with MDBs What would it take to have more consistent learning across World Bank financing operations? Workshops Source: Independent Evaluation Group. Note: AI = artificial intelligence; K&L = knowledge and learning; MDB = multilateral development bank; MPA = Multiphase Programmatic Approach; TTL = task team leader. Report Structure and Overarching Finding Independent Evaluation Group World Bank Group    9 The report is structured as follows. Chapter 2 describes how the World Bank structures knowledge inputs along major project milestones. Chapter 3 discusses how the World Bank’s knowledge ecosystems enable effective learning beyond the standard project processes. Chapter 4 presents conclu- sions and recommendations. Overall, the evaluation finds that the World Bank’s formal, linear approach to embedding knowledge in its lending processes strongly supports knowledge inputs and learning opportunities at specific entry points of the project cycle, especially at the preparation stage, but that it also has imbalances, weak lesson learning, and fragmented insti- tutional support. 2 | Knowledge and Learning in the Linear Model Highlights The World Bank uses a linear knowledge model that excels at pro- ducing technical knowledge sourced from both inside and outside the World Bank to inform project designs. However, the linear model is not ideal for fostering learning during the project implementation phase and underemphasizes country knowledge. Some of the most valuable learning comes from early, less formal opportunities and from unpacking of tacit knowledge. This in- cludes early and informal discussions with project peer reviewers and early, low-stakes (“safe space”) reviews. There is much varia- tion in the extent to which managers create such early, low-stakes opportunities to share knowledge. Case study projects relied far more on tacit knowledge from previous or ongoing projects than on formal lessons from Implementation Completion and Results Reports. The World Bank has little learning from canceled and dropped operations and from failure. 10   This chapter evaluates the World Bank’s successes and failures in em- bedding knowledge in projects. It shows that the World Bank uses a linear model to embed knowledge in its financing through key entry points in a project’s preparation, approval, implementation, and completion phases and that the policies, procedures, and guidance for lending create defined learn- ing moments for staff that vary depending on the financing instruments’ design (table 2.1). It finds that the World Bank’s linear approach to embed- ding knowledge into lending has weak lesson learning and imbalances that emphasize producing reports over applying knowledge, steers its knowledge work toward a project’s design phase rather than toward learning during the project implementation phase, and focuses its explicit knowledge work on generating sectoral knowledge rather than country knowledge, which tends to be tacit and unsystematic. This chapter examines each of the project phases in turn, from project preparation to completion.  verview of World Bank Financing Instruments Table 2.1. O Aspect IPF PforR DPF Support of Ring-fenced project A wider government A set of policy and activities program institutional actions Disbursement Borrower incurring Verified achievement Achievement of based on eligible investment of program’s DLIs with development policy project expenditures no tracing of specific actions expenditures Disbursement Designated accounts General budget General budget Independent Evaluation Group World Bank Group    11 to or specific accounts (exceptionally to for reimbursement implementer account) Implementation Bank IPF rules and Borrower program Country policy mechanism procedures systems processes Knowledge that Technical At design: Integrated Analysis of economy- underpins assessments of risk assessments to wide or sectoral projects’ technical help projects address policies and design, approach, technical, fiduciary, institutions, and the and appropriateness environmental, and poverty and social social risks impacts of proposed policies Annually during implementation: Data to verify achievement of DLIs Source: Independent Evaluation Group, based on Operations Policy and Country Services guidance. Note: DLI = disbursement-linked indicator; DPF = development policy financing; IPF = investment project financing; PforR = Program-for-Results. Project Preparation All project teams in the evaluation’s sample generated and used diagnostic studies to inform project designs during project preparation. This analytic work was highly diverse and had strategic, instrumental, and conceptual applications depending on the lending instrument (table 2.2). Different financing instruments generate and use analytic work differently during project design and, in so doing, adhere to guidance for these instruments: » The 20 IPF operations in the evaluation’s sample used studies to identify new projects or design specific pieces of infrastructure and financial arrange- ments. This approach adheres to the World Bank’s corporate guidance on IPF operations that recommends technical assessments on a project’s technical design, approach, and appropriateness. Teams often used needs assessments or global sector studies strategically to justify projects—at least 53 percent of IPF in the evaluation’s cases had such strategic knowledge use. For example, a needs assessment in Benin built the business case for investing in nutri- tion. Teams also often used project-commissioned studies for highly specific instrumental purposes—at least 80 percent of IPF in the evaluation’s sample used studies this way. For example, a study helped integrate nature-based solutions into a drainage and solid waste project in Côte d’Ivoire, and another study assessed flood risk and climate scenarios for embankment specifica- tions in Viet Nam (table 2.2). » All eight DPF operations in the evaluation’s sample conducted policy impact modeling, studies on the benefits of specific policy actions, and other reform Learning in World Bank Lending Chapter 2 analyses. Some of the DPF operations also had conceptual knowledge uses, for example, to understand policy reforms’ distributional consequences. This conforms to corporate guidance for DPF, which recommends analysis on a country’s economy-wide or sectoral policies and institutions, and the poverty and social impacts of any proposed policies. » The six PforR operations in the sample used integrated risk assessments to bolster these projects’ technical soundness and their support for clients’ systems and capacity within a sectoral program. In accordance with corporate guidance, these integrated risk assessments focused on ensuring that projects addressed technical, fiduciary, environmental, and social risks. According to 12 the team leaders, the PforR assessments’ focus on technical soundness and reinforcing country systems helped orient teams toward achieving outcomes.  xamples of Instrumental, Strategic, and Conceptual Table 2.2. E Knowledge by Lending Instrument Financing Instrumental Strategic Conceptual Instrument Knowledge Knowledge Knowledge IPF Studies of future flood Studies showing A study of global risk and how to build the benefits of brownfield remediation embankments to immunization and and redevelopment withstand long-term vaccine uptake best practices helped climatic changes for promotion (Pakistan’s conceptualize risk- infrastructure resilience National Immunization based approaches and (Viet Nam). Support Project). legal liability issues (China). DPF Monitoring of policy Studies demonstrated Studies on air pollution, reform implementation the benefits of green urban infrastructure, (all sample DPF). growth, covering social housing, carbon fisheries, energy, trading, and urban pollution, and climate forest conservation change (Morocco helped identify and Green Growth). prioritize environmental and urban resilience issues (Mexico). PforR Dialogues with clients on Cities’ urban mobility Studies helped the annual verification plans, done as part conceptualize of the DLIs led to of the DLIs, led to and incorporate inclusion of additional the development environmental institutions (Kenya) and of a national urban objectives in the other outcome-oriented transport strategy country’s Green changes (all PforR (Morocco Urban Agricultural and Rural projects in the sample). Transport Program). Revitalization (China). Source: Independent Evaluation Group’s case studies conducted for this evaluation. Independent Evaluation Group World Bank Group    13 Note: Refer to chapter 1 for the definitions of the terms. DLI = disbursement-linked indicator; DPF = development policy financing; IPF = investment project financing; PforR = Program-for-Results. Project teams use a broad spectrum of knowledge, not just from the World Bank sources, to inform project preparation. The sample projects’ techni- cal assessments were often conducted collaboratively with clients, at times incorporating the clients’ own research. For example, in Viet Nam, the World Bank’s technical expertise supplemented what government agencies lacked in knowledge capacity. Conversely, in China, where government agencies possess more expertise, the World Bank used analytics to inform project designs and integrate global knowledge and best practices into country strategies. IEG used artificial intelligence and text mining to examine the sources of the explicit knowledge cited in the Project Appraisal Documents (PADs; appendix D). The analysis found that one-third of these citations were authored by the World Bank, another third were individually listed authors (internal and external to the World Bank), 13 percent were client government sources, 9 percent were United Nations agencies, and the re- maining were other multilateral institutions and other sources (table 2.3). The relatively frequent use of client and partner sources is encouraging. Overall, IEG’s analysis found that 41 percent, or 1,271, of the total 3,102 project document citations were published by the World Bank or included current World Bank staff as the author or coauthor. Many cited references were also from World Bank knowledge collaborations. Twenty-five of the 1,271 World Bank references were coauthored by the World Bank and client governments, another 20 were coauthored by the United Nations agencies, and 237 were coauthored by a mix of individual authors from the World Bank and other institutions.  istribution of All Project and Program References by Table 2.3. D Authorship Author Type References (no.) World Bank Group 1,033 Individual authors 994 Client government 396 United Nations agency 284 Other multilateral institutions 87 Private sector 79 Learning in World Bank Lending Chapter 2 International Monetary Fund 79 Nongovernmental organization 77 Bilateral donor 52 Multilateral development bank 46 University 22 Research organization 20 News organization 6 Organization type could not be assigned 76 Total 3,102 Source: Independent Evaluation Group. 14 Note: The number of projects in the sample is 1,020. PADs cite a variety of World Bank documents, but few cite core ASA. World Bank management has designated specific categories of country-focused reports as core and extended core ASA—a specific set of World Bank reports meant to inform country programs and help clients advance development objectives. The Knowledge Compact prioritizes core ASA production for all client countries. Project documents refer to Enterprise Surveys, global flagship reports, partnership strategies, sector studies, economic updates, and project-related technical assessments. However, as shown in table 2.4, project documents only sparingly cite core ASA. Only 2 percent of all refer- ences and 5 percent of Bank Group references are to core and extended core ASA. The project case studies show that DPF used core ASA more than IPF, which more often used project-commissioned technical assessments. This is unsurprising: DPF teams are expected to use and be familiar with core ASA. Core ASA inform decision-making and policy making at strategic levels for the World Bank, its clients, and other partners and stakeholders, in a way that transcends specific operations, and their broad scope and shortened format may not match the specific needs of projects. Therefore, core ASA may still hold value for operations in indirect ways even if project teams cite them sparingly, perhaps because they are less familiar with them. Regarding the types of knowledge, the project teams in the evaluation’s sample relied mostly on tacit knowledge to understand country dynamics. In stark contrast to the explicit sectoral knowledge that informed the proj- ect designs, teams relied on talking to the right individuals to understand a country’s context, political economy, implementing agency capacity, and Independent Evaluation Group World Bank Group    15 so on. The cases found little documentation or dissemination of this coun- try knowledge except in the Systematic Country Diagnostics, which are no longer mandatory. A handful of team leaders acknowledged receiving tacit country knowledge from Country Management Unit staff. National staff of- ten maintained strong connections with key national figures to stay abreast of country knowledge, which was useful for teams. A few national staff even alternated between positions as World Bank staff and counterpart staff. For example, one of the World Bank’s national staff in Sri Lanka became the project director for the Sri Lanka biodiversity project. Projects also relied on partners for country knowledge; for example, the Moldova tax project team relied on a resident European Union official as a peer reviewer and informal adviser, even after leaving the country.  ore and Extended Core Advisory Services and Analytics Table 2.4. C Referenced in Project Appraisal Documents ASA Type References (no.) Core 55 Country Climate and Development 6 Report Country Economic Memorandum 8 Country Private Sector Diagnostic 9 Poverty Assessment 11 Public Expenditure Review 21 Extended core 11 Agriculture Sector Review 1 Financial Sector Assessment Program 2 Infrastructure Sector Assessment 1 Program Public Expenditure and Financial 3 Accountability Risk and Resilience Assessment 4 Total 66 Source: Independent Evaluation Group. Note: The number of projects in the sample is 1,020. Teams composed of global and in-country staff helped make projects tech- nically sound and anchored in country contexts. Project teams often had Learning in World Bank Lending Chapter 2 members who specialized in different technical areas. For example, the Jordan PforR had one team member for each of its three disbursement-linked indicators (DLIs). Many project teams had TTLs or co-TTLs stationed in country offices, which made them well-placed to engage with counter- parts and grasp the nuances of the country context. For example, in Brazil, a local TTL for the Amazonas Fiscal and Environmental Sustainability Programmatic DPF—the World Bank’s first engagement with the state—fa- cilitated the integration of state-specific insights into the project design. In the Philippines, having a financial sector DPF co-led by country office staff helped underpin policy reforms with a deep understanding of the country’s context. This helped keep the project relevant and prevented the reforms 16 from being reversed after the government changed. Similarly, in Pakistan, an immunization project’s success was partly due to the leadership of na- tional staff who were well-versed in the political dynamics surrounding the country’s ongoing devolution efforts. In some instances, travel restrictions or the limited physical presence of in-country staff in smaller countries impeded the team’s ability to forge client relationships and devise optimal implementation strategies. For example, the World Bank’s ability to coordi- nate multiple projects in Eswatini was constrained by having only one staff member in the country. The Marshall Islands encountered similar obstacles because of World Bank staff’s limited on-the-ground presence. Project Appraisal and Approval The most valuable learning opportunities during project approval and ap- praisal processes come at the least formal moments. IEG analyzed project approval meeting minutes and comments, observed some concept and deci- sion meetings, interviewed TTLs and peer reviewers, and examined the selection of peer reviewers quantitatively. The structured nature and hierar- chical dynamics of internal Concept Note and decision meetings limited the free flow of ideas and lessons learned, according to IEG’s direct observations and case study interviews. Decision meetings, which are the final step in the approval process, determine whether the project is ready for appraisal and negotiations but are not designed for and do not provide learning opportuni- ties. That said, the decision meeting’s agenda and the specific guidance sought by teams can enhance these meetings’ usefulness. Similarly, early-stage peer reviews and quality enhancement reviews, which are the least formal parts of Independent Evaluation Group World Bank Group    17 the design and approval processes, often contribute the most valuable knowl- edge and learning to teams. This section further discusses these dynamics. The World Bank has a structured peer review process. The appointed peer reviewers of project Concept Notes and appraisal documents impart instru- mental and conceptual knowledge during the project approval process. They endorse or critique project objectives, results frameworks, and technical designs, contributing to coherent project designs. Managers rely on peer reviewers to add credibility to projects’ technical design quality as evidenced by IEG interviews. The World Bank’s introduction of a more streamlined peer review process in 2017 improved the quality of peer review feedback by mak- ing it more targeted and succinct, as shown by case studies. When peer reviewers provide timely advice, they are more likely to add value; however, many comment shortly before meetings. The case studies’ assessment of peer review advice stored in the Operations Workspace shows that timely reviews, especially those provided well ahead of project approval at the concept review meetings or the decision review meetings, contributed to improved project designs. For example, peer review comments at the con- cept review meetings led the Côte d’Ivoire Urban Resilience and Solid Waste Management Project to remove a risky landfill component and Türkiye Climate and Disaster Resilient Cities Project to adjust its design to concen- trate on earthquake risk instead of multiple hazards, thereby simplifying its design. By contrast, in the Panama water and sewerage project, peer review- ers advised the project team to promote a tariff scheme instead of relying on expected government subsidies for the system’s financial viability. However, the project team ignored this advice and subsidies never materialized, even- tually contributing to the project’s cancellation. IEG’s analysis of the timing of peer review advice at projects’ concept review stage shows that 25 percent of comments arrived the same day or one day before, and 16 percent of com- ments only two days before the concept review meeting (figure 2.1). When teams receive such late review comments, it can be hard for them to fully absorb and act on the advice before meetings, according to the case studies. In interviews, TTLs complained about receiving the reviews late. Sectors vary in the timeliness of peer review advice. The median lead time for peer review advice for projects mapped to the Environment, Natural Resources, and Blue Economy GP was seven and a half days, whereas for projects mapped to the Social Protection and Jobs GP it was only two days, and other GPs falling in Learning in World Bank Lending Chapter 2 between those extremes. Early and informal knowledge inputs maximize the project teams’ learning. Informal discussions well before projects are ready for approval are highly valuable, with many TTLs appreciating in-depth, off-the-record conversa- tions with project peer reviewers and others on project design. For example, the TTL for the West Africa Unique Identification for Regional Integration and Inclusion project commended the benefits of informally engaging with peers for technical advice. One reason it was so valuable is that peer review- ers often provide more frank and candid feedback in these informal settings. In addition, the evaluation’s case studies found that less formal meetings, 18 such as quality enhancement reviews, led by practice managers are more conducive to open learning because they focus on technical details that may not be addressed in more formal settings.  iming of Peer Reviewers’ Advice Before Project Concept Figure 2.1. T Review Meetings 30 25 25 Share of observations (%) 21 20 16 15 10 7 6 5 4 3 3 2 2 2 2 1 1 1 1 1 1 1 1 0 1 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 20+ Number of days peer review advice received before the concept review meeting Source: Independent Evaluation Group. Note: The number of projects in the sample is 725. 20+ = more than 20. There is much variation in the extent to which managers have embraced the value of early, low-stakes processes to unpack tacit knowledge. The World Bank has a standard early review meeting, quality enhancement reviews, and Independent Evaluation Group World Bank Group    19 nonstandard informal meetings variably referred to as preproject Concept Note meetings, clinics, and safe space meetings. These meetings, held at the discretion of managers, are all opportunities for a small group of ad- visers to provide candid feedback for projects under preparation. Managers use quality enhancement reviews for 91 percent of PforR, 72 percent of IPF, and 1 percent of DPF operations under preparation (figure 2.2). The early, low-stakes meetings represent managers’ intentional efforts to unpack tacit knowledge. For example, the Social, Urban, Rural, and Resilience GP began organizing such safe spaces. The Social, Urban, Rural, and Resilience GP director at the time explained the value of these safe space sessions, “These brainstorming sessions offered a forum to explore new operational frontiers by leveraging global tacit knowledge from experts working in different re- gions on similar issues. As the reviews were designed to influence the design of new operations under preparation, the connection between knowledge and solutions was strong” (Ijjasz-Vasquez et al. 2024). The Governance GP has also created informal spaces, referred to as clinics, to brainstorm technical project designs before approval.  perations with a Quality Enhancement Review by Lending Figure 2.2. O Instrument Type and Lead Global Practice Water Digital Development Agriculture and Food Transport Environment, Natural Resources, and Blue Economy Education Lead Global Practice and lending instrument type Energy and Extractives Urban, Disaster Risk Management, Resilience, and Land Social Sustainability and Inclusion Governance Social Protection and Jobs Finance, Competitiveness, and Innovation Health, Nutrition, and Population Poverty and Equity Macroeconomics, Trade, and Investment Learning in World Bank Lending Chapter 2 PforR IPF DPF All operations 0 10 20 30 40 50 60 70 80 90 100 Share of operations (%) Lead Global Practice Lending instrument All operations Source: Independent Evaluation Group. Note: The number of projects in the sample is 1,942 approved since FY19. DPF = development policy financing; IPF = investment project financing; PforR = Program-for-Results. 20 The World Bank has curated a peer reviewer database, but managers do not always use it when selecting reviewers. Setting up the peer reviewer data- base was part of the World Bank’s action plan to implement its Strategic Framework for Knowledge. The database includes World Bank staff vetted as qualified to be peer reviewers. Sixty-four percent of lending project peer reviews in FY 2018–23 were conducted by peer reviewers in the database, according to IEG’s analysis of project peer reviews. GPs use the database unevenly—the share of reviewers from the GPs in People (previously Human Development) and Prosperity (previously Equitable Growth, Finance, and Institutions) was higher than for Infrastructure and Planet (previously Sustainable Development; figure 2.3). Some interviewed managers readily admitted to never using the database.  hare of Peer Review Advice from the Peer Reviewer Figure 2.3. S Database Education Social Protection and Jobs Governance Health, Nutrition, and Population Environment, Natural Resources, and Blue Economy Energy and Extractives Lead Global Practice Digital Development Independent Evaluation Group World Bank Group    21 Transport Agriculture and Food Finance, Competitiveness, and Innovation Water Urban, Disaster Risk Management, Resilience, and Land Social Sustainability and Inclusion Macroeconomics, Trade, and Investment Poverty and Equity 0 10 20 30 40 50 60 70 80 90 100 Share of peer review advice provided by persons in PRDB (%) Source: Independent Evaluation Group. Note: The number of projects in the sample is 1,011. PRDB = peer reviewer database. Project Implementation During project implementation, staff use operational and other knowledge to improve projects via restructurings. The case studies revealed examples of teams using Mid-Term Reviews for IPF and PforR to adjust or restruc- ture projects. Several IPF teams conducted assessments that contributed to improving project results frameworks, scaling up operations, or reallocating funds among project components. Past IEG studies found that when project teams engage effectively in adaptive learning, they can overcome implemen- tation challenges—for example, by identifying risks early, eliciting support from managers, and acting quickly to restructure projects or mitigate these risks in other ways—and linked such adaptive learning to improved project performance (World Bank 2020b, 2023). However, IEG studies and research also concluded that the World Bank’s incentives, results measurement systems, and risk-averse corporate culture do not support adaptive manage- ment well. Incentives for project staff are sometimes focused on checking the box—that is, meeting targets and feeding the demands for corporate monitoring data—more than on promising learning (Honig 2018, 2020; World Bank 2016, 2020b, 2020d). Routine project documents do not report on staff’s knowledge use or learn- ing. Adhering to the World Bank guidelines, all PforR and IPF operations in the sample conducted biannual in-country visits or missions that are docu- mented in external management letters, aide-mémoire, and Implementation Status and Results Reports. DPF operations, for their part, function different- ly because all prior actions need to be completed before these operations are Learning in World Bank Lending Chapter 2 approved, but they also have missions, Implementation Status and Results Reports, and aide-mémoire. Interviewed TTLs for IPF and PforR reported that they gained valuable tacit knowledge from missions, often on practical operational matters, such as procurement, financial management, and imple- mentation specifics, including strengths and weaknesses of client-appointed project managers. The evaluation’s case studies had many examples of teams collaborating with clients and using knowledge to solve implementation chal- lenges and build capacity, which often solved project implementation challenges and showed positive shifts in supervision ratings. However, Implementation Status and Results Reports and aide-mémoire did not docu- 22 ment this knowledge and learning but instead focused on the project’s status and compliance with the World Bank policies and procedures. In this, teams adhered to these documents’ standardized reporting templates, which do not have fields to capture learning. Project teams are often too busy or underresourced to carry out knowledge work and promote learning during implementation. The project prepara- tion phase has formal requirements to produce specific knowledge inputs and technical assessments as shown in table 2.1, but the implementation phase has few such requirements. In addition, teams rarely have the bud- gets to produce studies during implementation. As a result, the case studies show that project teams develop far fewer knowledge inputs and obtain less learning during implementation than during preparation. Among lending instruments, IPF tended to carry out the least knowledge work. Twenty- seven percent of the TTLs managing IPF projects, whom IEG interviewed as part of the case studies, stated that there were barriers—such as work pressures, compliance requirements, immediate problem-solving needs, and budget constraints—to conducting or using analytic work during the imple- mentation phase. These TTLs described project implementation challenges as barriers to generating and using knowledge, but, ironically, the urgency to address implementation challenges, and ultimately pursue outcomes, is precisely why relevant knowledge is so valuable. That said, some projects from the sample collected data or conducted informal studies, often financed by trust funds, to help them address operational challenges or introduce Independent Evaluation Group World Bank Group    23 innovations in pursuit of development outcomes. For example, the Morocco Urban Transport Program PforR commissioned a gender survey to improve women’s access to public transport. Pakistan’s National Immunization Support Project developed advocacy plans and innovative mechanisms to improve vaccination coverage. The Sri Lanka Ecosystem Conservation and Management Project produced and shared publications on landscape ap- proaches and managing human–elephant conflicts. However, the World Bank lacks a systematic way to store, share, or reuse studies and innovations produced by operations, and project teams lack time, budgets, and incentives to do so. International staff rotations have pros and cons. The World Bank Human Resources Career Development and Mobility Framework mandates that most international staff rotate positions every three to four years. This is a deliberate strategy by the World Bank to transfer knowledge across Regions, among other reasons. Indeed, the case studies showed examples where staff rotations facilitated knowledge sharing. For example, some case study TTLs recently rotated out of global units known as centers of expertise, such as the Global Facility for Disaster Risk Reduction and Recovery, where they had gained knowledge and subsequently used that knowledge in projects in their next position. However, there is also a downside to this strategy, as IEG’s evaluation of the World Bank’s global footprint found (World Bank 2022a). TTLs accumulate a wealth of tacit sector and country knowledge and estab- lish trusting client relationships when in a certain position or geographic location, and, therefore, when they rotate out of these positions, it tends to create gaps in knowledge and disrupt client relationships. Econometric studies have correlated TTL turnover with negative project performance as measured by IEG outcome ratings (Ashton et al. 2023). TTL turnover rates are high and increasing, which can potentially exacer- bate tacit knowledge losses. IEG developed an indicator to measure annual TTL turnover per project using panel data on project team composition for all projects within the evaluation period (appendix F). IEG’s analysis shows that TTL turnover averaged 0.85 rotations per year over the FY14–23 period. The turnover increased threefold from 0.4 in FY14 to 1.2 per year in FY23 (figure 2.4). Since the average World Bank operation had 2.2 TTLs in FY23, this means that 1.2 of an average operation’s co-TTLs rotate out per year while 1 remains. This is an alarming statistic. The case studies found several instances of discontinuity among successive TTLs. In addition, TTLs ex- Learning in World Bank Lending Chapter 2 plained that handover notes were of uneven quality and that they found tacit exchanges more helpful and better suited to convey sensitive information about clients. An increase in overlapping co-TTL arrangements may have mitigated the associated disruptions. Co-TTL arrangements can offer valuable mentor- ing opportunities between senior and junior co-TTLs, observed in several of the case studies. Overlapping transition periods between outgoing and incoming TTLs and co-TTL arrangements can also reduce knowledge gaps. IEG adjusted the TTL turnover indicator to account for the presence of overlapping co-TTLs. The adjusted indicator shows that TTL-out-rotations 24 that were not mitigated by a co-TTL overlap were much lower, at about 0.1 TTL-out-rotations per year. Moreover, this rate has remained stable since FY18. In effect, the World Bank’s growing use of co-TTL arrangements has kept pace with the increase in staff rotations. Among instruments, DPF op- erations had marginally higher adjusted TTL turnover rates compared with IPF and PforR projects. Basic TTL turnover rates were notably higher for the countries affected by fragility, conflict, and violence, likely due to shorter rotation cycles, but the adjusted turnover rates showed minimal difference (appendix F). These findings suggest that the World Bank is proactively managing the trade-off between maintaining knowledge continuity within projects and the desire to rotate staff for knowledge transfer and other rea- sons. That said, relationships of trust between staff and clients are critical to achieving results, and the high and increasing TTL turnover rates should be cause for concern.  ask Team Leader Turnover Rates Figure 2.4. T 1.4 1.2 Turnover per year 1.0 0.8 0.6 0.4 0.2 0.0 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 Independent Evaluation Group World Bank Group    25 Project approval fiscal year Average TTL turnover per year of project duration Linear (Average TTL turnover per year of project duration) Average co-TTL-ship–adjusted TTL turnover per year of project duration Linear (Average co-TTL-ship–adjusted TTL turnover per year of project duration) Source: Independent Evaluation Group. Note: The number of projects in the sample is 2,785. TTL = task team leader. PforR operations used learning to pursue project results. In all six case study PforR projects, teams used integrated reviews and annual verifications to maintain a strong focus on learning during implementation because of the need to verify DLIs every year. Learning from these projects strengthened country fiduciary systems, built client capacity, refocused the government’s sector strategy, or enhanced the project’s focus on results and sustainability. For example, in Tanzania, workshops on the DLI verification’s findings that involved the World Bank, the government, and an independent verification agency generated feedback and reflection on the project’s results, according to aide-mémoire and interviews. In other PforR operations, annual verifi- cations informed course correction initiatives. For example, in a PforR in Kenya, the verification process brought in additional institutions to enhance environmental and social management midway through the project’s imple- mentation. Moreover, by disbursing funding against outcomes rather than project inputs, PforR operations give implementing agencies more room to reflect, learn, adapt, and innovate. In the West Africa Unique Identification for Regional Integration and Inclusion PforR project, ongoing learning in the five participating countries promoted peer-to-peer learning across countries and led to adjustments in the program’s direction, policies, and protocols. Some of the sample DPF supported policy monitoring. Much explicit knowl- edge goes into preparing DPF operations, as already mentioned. During implementation, half of the evaluation sample’s eight DPF operations used monitoring and evaluation systems to gain knowledge on the programs’ implementation. In some cases, the focus was on monitoring whether gov- ernment agencies complied with the agreed policy reforms. In some other cases, the focus extended to assessing the policy reforms’ impacts. For exam- ple, in the Mexico Environmental Sustainability and Urban Resilience DPF, the World Bank used trust funds to design monitoring frameworks, gener- ate estimates of policies’ expected distributional effects, collect new data, and assess policy impacts, thereby bringing positive policy impacts to light. The learning continued after the operation closed and eventually informed Learning in World Bank Lending Chapter 2 Mexico’s Country Climate and Development Report and other products. MPA programs’ learning agendas hold promise. MPAs are not a lending instrument but rather an approach to sequencing or combining projects over multiple years. MPA programs foster learning during implementation by mandating learning agendas that cover the lifespan of phased, long-term, or multicountry programs. When well-designed, these learning agendas identify knowledge gaps, monitor progress, and use adaptive learning to make program adjustments (table 2.5). IEG assessed 34 MPA projects’ learn- ing agendas against seven criteria of a well-designed learning agenda.1 IEG found that 4 out of 34 learning agendas met just one of these criteria, anoth- 26 er 4 met two criteria, 23 met three criteria or more, 13 met four criteria or more, 5 met five criteria or more, and 2 met six criteria. None met all seven criteria. Only 2 of the learning agendas identified learning outcomes—the least commonly met criterion (figure 2.5). The review of these 34 learning agendas’ designs concluded that MPAs offer a structured approach to gener- ating and using knowledge extensively; however, not all MPA programs have comprehensive learning agendas in place at project approval.  urposes of Knowledge in Multiphase Programmatic Table 2.5. P Approach Programs MPA Program Instrumental Strategic Conceptual Aspect Knowledge Knowledge Knowledge MPA overall Use cross-fertilization Provide continuous An ethnographic of lessons to support for institutional study of beneficiaries’ problem-solve, development, needs and concerns develop standardized capacity building was used to documentation, of implementing conceptualize the and increase agencies, and design (West Africa harmonization of stakeholder coalitions Unique Identification regulations (India River Basin for Regional (regional MPAs). Development). Integration and Inclusion). MPA learning Continually improve Prepare a feasibility Use pilot programs agenda implementation study for a statewide in private sector quality through flash flood forecasting development to periodic M&E system during phase 1, create approaches assessments that which is then piloted that are suitable, feed back into design in phase 2, and, based fair, and effective in of activities on insights, expanded supporting women Independent Evaluation Group World Bank Group    27 (Kenya Digital sitewide (India River entrepreneurs Economy Basin Development). (Fiji Tourism Acceleration). Development). Source: Independent Evaluation Group’s review of 34 MPA learning agendas. Note: M&E = monitoring and evaluation; MPA = Multiphase Programmatic Approach. Early indications suggest that learning during MPA program implementa- tion is below potential. MPAs do not monitor the learning agendas—MPA operational systems and guidance do not require or support such monitor- ing—and the majority of MPA programs are still early in implementation (see also IEG’s evaluation of the MPAs; World Bank 2024). As a result, it is premature to reach firm conclusions on how well MPA programs implement their learning agendas, support learning with clients, and use this learning to make changes to the program. IEG reviewed three MPAs that were at an advanced stage of implementation: the West Africa Unique Identification for Regional Integration and Inclusion, Western Balkans Global COVID-19, and Madagascar Health projects. Only one of these, West Africa Unique Identification for Regional Integration and Inclusion, generated robust learning across countries and across project phases. West Africa Unique Identification for Regional Integration and Inclusion did so by creating frequent opportunities for tacit learning across the program’s countries, both for clients and World Bank staff, on different topics, such as how to conduct know-your-customer compliance during pandemic lockdowns. In the Madagascar Health MPA, the implementing agency’s capacity shortcom- ings limited its ability to apply the knowledge in the reports produced by the learning agenda, and COVID-19 travel restrictions hindered the World Bank team’s ability to support learning. As a result, the project team added capacity building to the learning agenda for the project’s second phase. In the Western Balkans regional MPA, the implementing agencies’ insufficient readiness to execute the project, coupled with World Bank management’s urgency to advance swiftly to the program’s second phase, did not allow enough time to generate and incorporate lessons from the first phase.  eviewed Multiphase Programmatic Approach Learning Figure 2.5. R Agendas That Met Select Quality Criteria Data sources for learning identified 28 Adaptive learning mentioned 26 Plans for feedback from 19 Quality criteria clients or partners Learning in World Bank Lending Chapter 2 Capacity building for clients 10 to participate in learning Knowledge gaps identified 10 Types of knowledge identified 5 Learning outcomes identified 2 0 5 10 15 20 25 30 MPA learning agendas (no.) Source: Independent Evaluation Group. Note: N = 34. See appendix C. MPA = Multiphase Programmatic Approach. 28 Completion and Lesson Learning The case study projects seldom used formal lessons from the Implementation Completion and Results Reports (ICRs). The World Bank has a long-established self-evaluation system. After a project closes, project teams complete an ICR, which is validated by IEG through the ICR Review; compliance with the pro- cess’ requirements’ is high. However, the case studies’ document reviews and TTL interviews found that new projects rarely used lessons from previous projects’ ICRs. The case studies showed that many staff perceive the for- mal self-evaluation of projects as administrative tasks rather than valuable knowledge inputs. This observation is consistent with the findings from IEG’s evaluation of the Bank Group’s self-evaluation systems (World Bank 2016). ICRs tend to focus on ratings, and the lessons they capture vary in quality, validity, and relevance. Past World Bank initiatives to create a data- base of project “delivery challenges,” termed DeCODE (Delivery Challenges in Operations for Development Effectiveness), and to provide an automated, curated “knowledge package” with lessons and other information to TTLs were discontinued, but the ongoing reform of the ICRs provides an oppor- tunity to improve the quality and use of lessons. Extraction, synthesis, and application of lessons need judgment to apply well to context and are hard to automate. Many evaluations have documented weaknesses in how the World Bank evaluates and learns from its projects (Ravallion 2016; World Bank 2023). The weak formal lesson learning limits outcome orientation Independent Evaluation Group World Bank Group    29 because staff use process or informal evidence of lower quality instead. In contrast, the World Bank often uses tacit knowledge from previous or ongoing projects, particularly sector- and country-specific knowledge. The case studies found that World Bank teams far more often used tacit lessons than they did explicit lessons from ICRs, Completion and Learning Reviews, or other sources. Tacit lesson learning from prior projects and peers’ experi- ences contributed instrumental, conceptual, and strategic knowledge in the early stages of project development. For example, project teams applied tacit lessons from a previous IPF operation to different financial instruments, leading the World Bank to adopt a hybrid PforR and IPF model in Tanzania’s water sector. This model allowed for a more sustainable and results-focused sector strategy. Similarly, in China’s agriculture sector, the World Bank’s stra- tegic approach was shaped by tacit past experiences, leading to a PforR that aligned with the government’s Green Agricultural and Rural Revitalization program. Cross-country lesson learning is less common, but multicountry MPAs stood out as an exception by facilitating peer-to-peer learning across countries. The World Bank has no established system or safe space for capturing les- sons from canceled and dropped operations, hindering learning from these experiences. Political and institutional sensitivities lead to some approved projects being fully or partially canceled or dropped before approval, some- times after years of preparation effort. The understanding of such political and institutional sensitivities is an important tacit knowledge. For example, in the Iraq Emergency Social Stabilization and Resilience Project, manage- ment urged the project team to include a component that the team thought was ill-suited to the context. Given the team’s clearer understanding of the on-the-ground realities, the component was ultimately deemed unsuit- able and had to be dropped during implementation. However, the World Bank has no space for sharing such experiences except through occasional IEG evaluations and cursory notes in the project files. IEG’s Nepal Country Program Evaluation similarly found that the World Bank’s country team had no mechanism to learn from the political economy obstacles that led the World Bank to drop or cancel several projects and components. Such a mechanism could potentially have been valuable to identify the reasons for the repeated droppages and cancellations in the Nepal program and mitigate these. This is part of a larger organizational culture in the World Bank and other multilateral development banks that sometimes focuses on compli- Learning in World Bank Lending Chapter 2 ance, disbursements, and meeting targets and that tends to project progress and success. Such a corporate culture can induce risk aversion and reduce openness about problems, mistakes, failures, and shortcomings (EBRD 2021; World Bank 2020b). Yet mistakes and failures are important for learning and innovation, perhaps more so than successes. Some foundations and civil society organizations actively promote learning from mistakes. BRAC pub- lishes an annual Failure Report with examples of programs that did not scale, did not meet beneficiaries’ needs, or failed to make a dent in big problems (BRAC 2024). The World Bank has tried to promote learning from failure but with limited success. 30 1  IEG identified the following seven criteria for a comprehensive learning agenda: (i) setting explicit goals and outcomes for learning, (ii) identifying knowledge gaps to understand what is missing, (iii) characterizing data sources to use in the learning process or to assess learn- ing progress, (iv) listing mechanisms to capture lessons and use these to improve and inform subsequent phases, (v) providing capacity building for clients to participate in learning, (vi) identifying types of knowledge generated and needed for progressing through the different phases, and (vii) specifying the clients and partners who will be part of feedback loops or support the learning. Independent Evaluation Group World Bank Group    31 3 | The World Bank’s Ecosystems for Knowledge and Learning Highlights The absence of a centralized knowledge system means that busi- ness units must develop their own knowledge systems, with some thriving and some falling behind. Operational knowledge is the best organized, and country knowledge is the least organized, with sector knowledge in between. Global units’ knowledge production often lacks strong operational applications. Teams and managers must go above and beyond the linear mod- el’s formal knowledge requirements by using informal exchanges of tacit knowledge and creating their own spaces for learning. Teams rely on trust funds because there are no dedicated budgets for knowledge outside of project design requirements. Tacit knowledge and informal exchanges are preferred by many staff and managers because it is a valuable and easy-to-access source of practical knowledge. The World Bank’s few knowledge and learning staff focus on explicit knowledge sharing but do not support transfer of tacit knowledge and do not have the remit to support World Bank–wide knowledge management. Certain instruments, such as Multiphase Programmatic Approaches and Program-for-Results, have knowledge mandates that better suit them to learning. Other instruments, such as development policy financing and investment project financing, lack knowledge mandates and rely on motivated task team leaders to drive learning. 32   This chapter examines the broader World Bank knowledge ecosystem’s inputs and assesses how well they support effective learning. The chapter identifies key knowledge enablers and reviews the World Bank’s knowledge ecosystem. The knowledge ecosystem refers to the resources, opportunities, capacities, and client engagements that support effective learning condi- tions for staff and projects. The ecosystem for knowledge is nonlinear and multidimensional, with many more entry points for knowledge and spaces for learning than in the formal linear model (figure 3.1). The chapter is orga- nized around several of these inputs: it examines the knowledge ecosystem’s resources, its client and partner contributions, and its learning opportunities and capacities. The chapter finds that some teams and units run knowledge systems that go above and beyond the linear model’s formal knowledge re- quirements to create effective learning conditions. However, the World Bank has no minimum standards, provides little guidance, and creates few incen- tives and mechanisms to ensure excellence in knowledge sharing and use.  cosystem for Learning in Lending Figure 3.1. E Opportunities E NABLIN G E N V I R O N M E N T and Capacities nt ie ity Cl ac p Ca K N OW L E D G E Kn and Why? M ow ate ts Cli ing Uses of Knowledge en led s wi earn ge th L Independent Evaluation Group World Bank Group    33 Strategic Instrumental Conceptual Knowledge Sharing with Partners Learning How? Who? Approaches THE LINEAR MODEL Form of Knowledge Source of Knowledge Preparation Appraisal Implementation Completion and Approval Tacit Explicit Internal External What? Type of Knowledge St fo Mo aff r Clients and Le tiva Partners ar tio nin n Sectoral Country Operational g s Knowledge Human Trust Base Resources Funds Resources Source: Independent Evaluation Group. There are several inputs, or knowledge enablers, that create effective learn- ing conditions within the World Bank. These enablers, which also define this chapter’s subsections, include knowledge ecosystem’s resources, its client and partner contributions, and its learning opportunities and capacities. Two of the evaluation’s case studies illustrate how these knowledge enablers contribute to successful knowledge management in practice: » The China Green Agricultural and Rural Revitalization PforR capitalized on knowledge from three decades of prior engagements and the client’s learning culture. The World Bank spent an unusually large amount of admin- istrative budget on a series of studies that recommended investments and policy reforms to make Chinese agriculture more sustainable. The program design incorporated these recommendations. During implementation, the World Bank collaborated with the Agricultural Development Bank and other Chinese institutions to leverage local expertise, ensure client learning, and scale up lessons into a national program, largely financed by the government, thereby multiplying the influence of the World Bank–supported learning. World Bank researchers collaborated with the Chinese institutions to publish a joint report on China’s experiences in rural poverty reduction (World Bank and Development Research Center of the State Council of China 2022). » The Tanzania Sustainable Rural Water Supply and Sanitation Program, also a PforR, used knowledge to incentivize results by linking payments to veri- fied outcomes. During intense discussions on program outcomes, the World Bank team and the local governments explored implementation lessons and identified performance improvements. Lessons from the past World Bank– Learning in World Bank Lending Chapter 3 and United Kingdom–funded projects informed institutional reforms aimed at systemic change within the water sector. The proactive TTL used trust fund–supported studies that explored approaches to promoting private sec- tor participation in sanitation service delivery. The program used the studies’ findings to define a model for engaging private sector service providers in support of community-based water organizations. Resources Trust fund financing boosts knowledge and learning. Trust funds allow World Bank staff to carry out analytic work, deliver technical assistance, 34 engage clients, and organize joint learning with clients when administrative budgets are insufficient. This analytic work informs project, program, and prior action designs; builds capacity and facilitates other engagements with clients, partners, and civil society organizations; and enables the World Bank to provide technical assistance and collect data. At least 60 percent of the evaluation’s case studies—that is, 19 out of the 32 projects for which the team collected trust fund information—used one or more World Bank–executed trust funds to support project preparation and implementation. Nine projects had three or more trust funds. The Madagascar Nutrition MPAs had nine World Bank–executed trust funds. DPF operations generally do not rely on trust funds, although much of the analytic work that helped teams identify policy actions had been financed through such funds. According to TTLs, trust-funded studies helped inform projects that lacked prior operations to learn from. Larger country programs tend to have access to more human and knowledge resources than smaller programs. Teams in countries with larger programs, including China, Kenya, Mexico, and Tanzania, could draw on knowledge from several prior or parallel operations, many in-country staff, and existing analytic work. IEG’s global footprint evaluation found that the World Bank concentrates the largest share of field-based professional staff, including managers and program leaders, in a few countries with a country director presence. It also reported that “case studies from smaller countries without country directors almost always revealed examples of an important proj- Independent Evaluation Group World Bank Group    35 ect or business line in a Country Partnership Framework underachieving or being delayed because the right expertise was not available in the country” (World Bank 2022a, xvi). Small countries in this evaluation’s sample, such as Belize and the Marshall Islands, lacked such knowledge resources. For ex- ample, Belize’s Climate Resilient and Sustainable Agriculture Project would have benefited from cross-support from another GP to design a project com- ponent, but that GP had neither lending nor ASA in that sector from which to draw knowledge and was unable to assist with cross-support. Clients and Partner Contributions Learning with clients is a common practice and useful for building trust. Nearly all task teams in the cases shared knowledge with clients and fostered joint learning, often using informal approaches to share tacit knowledge, such as safe spaces, calls and meetings, workshops, and so on. TTLs ex- plained that this learning often built on prior relationships with clients that had been established during previous projects or technical assistance. Study tours were one method for client learning, adding value to project identification and preparation. For example, a case study project used a mul- tiyear, sequenced learning approach with study tours and a report on global best practices to help Chinese clients identify a new approach to cleaning heavily contaminated sites, eventually leading to a financing project that scaled up this approach nationally. Similarly, Moldova’s Tax Administration Modernization Project held workshops with taxpayer associations and others to identify reforms for an IPF project to support. In Benin, missions enabled the monitoring and evaluation specialist to build local capacity and stron- ger management information systems for nutrition. However, much of this learning is not formalized. For example, joint learning at the project level is rarely documented—review meeting minutes, project Concept Notes, and PADs rarely mention it. However, experienced TTLs emphasized that learning with clients helps them develop close personal and professional relationships built on trust. PforR and MPAs have explicit mandates for continuous World Bank–client learning, although budget norms do not reflect this. The PforR’s annual DLI verification process uses data about progress toward achieving the DLIs to foster dialogue and learning between World Bank teams and their counter- parts. This occurred in all six samples PforR operations. The sample’s three regional, or horizontal, MPAs involved active World Bank–client exchanges Learning in World Bank Lending Chapter 3 on lessons and implementation challenges. The cross-country nature of MPAs facilitated such exchanges. As a result, some Regions are promot- ing MPAs and structuring client-oriented learning agendas around them. However, budget norms for MPAs and PforR are the same as for regular IPF, despite MPAs’ and PforR’s larger learning mandates. TTLs for regional MPAs argued during interviews that they should be allocated more budget because they cover more countries. Some of the MPAs used trust funds to compen- sate for the scarcity of administrative budget. The level of a client’s capacity can affect knowledge generation and use. At least eight of the cases showed low client capacity hindering knowledge and 36 learning during implementation. A team leader for projects in Côte d’Ivoire and Cameroon noted that clients with higher capacity implemented project activities faster and more efficiently, leaving time to develop a knowledge program with trainings, innovations, and study tours. The two Chinese cases stood out as examples of this, leading the World Bank to produce widespread analytic work and organize study tours and workshops with global experts. Lower-capacity clients spend more time on troubleshooting implementa- tion challenges, leaving neither the client nor the World Bank the time to promote knowledge—for example, in the small island state, the Marshall Islands, the limited client capacity reduced the World Bank team’s ability to pursue learning with the clients. Nevertheless, there were cases of strong learning with clients in several low- and lower-middle-income countries (for example, Kenya’s Devolution Support PforR) and, conversely, of challeng- es to learning in upper-middle-income countries. For example, a project in Türkiye produced many diagnostic reports but found that it was hard for the client to absorb these. Many GPs and some Regions have formal client capacity development initiatives to overcome these challenges. These are structured approaches, often financed through trust funds. The World Bank units invest in capacity development because it complements their lending programs and promotes development outcomes. The Health, Nutrition, and Population GP runs the Joint Learning Network for Universal Health Coverage—a network of practitioners and policy makers who codevelop knowledge and solutions Independent Evaluation Group World Bank Group    37 to country-specific health challenges. Health, Nutrition, and Population also has a flagship client learning program on health systems. The Social Protection and Jobs GP has a long-standing Global Forum on Social Protection issues and core courses on safety nets at the global and region- al levels. The Water GP has a field leadership training course to help water utility managers drive cultural change in their organizations. The Transport GP has Leaders in Urban Transport Planning and other capacity develop- ment initiatives. The East Asia and Pacific Region has knowledge hubs in the Republic of Korea and Malaysia that work with clients to build their capacity to manage World Bank projects. These hubs also collaborate with the Asian Development Bank on client capacity-building projects in Viet Nam, Myanmar, and other Asian countries. The Knowledge Compact seeks to elevate capacity development as a business line on par with lending and set up a new World Bank Group Academy (World Bank Group 2024). Previously, the World Bank Institute had a similar mandate. Academy programs have the potential to elevate capacity development and lead to greater recognition and incentives for knowledge management and instructional design. Nearly half of sampled World Bank teams collaborate with development partners on learning initiatives. All GPs have formal knowledge partner- ships with reputable organizations in their sectors, according to interviews with GP global unit managers and the GPs’ intranet pages. For example, the Education GP partners with the United Nations Children’s Fund; the United Nations Educational, Scientific, and Cultural Organization; Bill & Melinda Gates Foundation; the Foreign, Commonwealth and Development Office; and the International Labour Organization. The case studies show that devel- opment partners often provide instrumental knowledge to World Bank teams for project design and some strategic knowledge. The partners can also raise co-financing. The reference analysis discussed in chapter 2 and appendix D shows that a healthy two-thirds of PADs’ citations are to sources authored by individuals, clients, and organizations other than the Bank Group. The case studies also show that the World Bank learns with bilateral donors through multidonor trust funds and, in some instances, collaborated with the International Monetary Fund and other multilateral development banks. For example, the Amazonas case study demonstrates that the World Bank col- laborated with the Inter-American Development Bank in Brazil on learning. The World Bank has few formal requirements on external collaboration, such as with the International Monetary Fund on policy lending and debt issues, with associated review processes. Apart from this, much knowledge collabo- Learning in World Bank Lending Chapter 3 ration appears to happen organically or in the context of trust funds, outside the formal lending process requirements. Approximately 45 percent of the case studies showed the World Bank partnerships with think tanks, academ- ic institutions, and development organizations as enablers for knowledge generation and use. In Tanzania, the lessons from a United Kingdom–funded PforR operation informed the World Bank’s own PforR operation. In Belize, the World Bank collaborated with the International Center for Tropical Agriculture and Belize’s Ministry of Agriculture to identify climate-smart agriculture practices and investment opportunities. 38 Opportunities and Capacities The World Bank has many business units with strong knowledge capacity. All units in World Bank operations produce knowledge to some extent, and knowl- edge generation is a core mandate for GP global units, IEG, cross-cutting theme groups, Gender Innovation Labs, the Development Economics Vice Presidency, many trust-funded partnerships, and others. Most of the World Bank’s knowl- edge generation focuses on sectoral knowledge packaged as reports, data, and tools. Many of the GPs organize this knowledge in a logical structure with clear priorities and focus areas that correspond to the GPs’ lending business lines. GPs are sometimes helped in this process by the World Bank’s umbrella trust fund reforms that consolidated financing for knowledge into strategi- cally aligned funds, thereby reducing knowledge fragmentation. For example, the Health, Nutrition, and Population GP consolidated 187 financing streams into five broad funds that align with the focus areas of Health, Nutrition, and Population (box 3.1). IEG’s Knowledge Flow and Collaboration Under the World Bank’s New Operating Model continues to ring true; it found that the creation of the GPs in 2014 improved knowledge flow and staff mobility, mobilized expertise for clients, and sometimes deepened expertise in relevant areas (World Bank 2019a). The World Bank’s knowledge generation is not necessarily aimed at op- erational needs. This is most clearly the case for global units’ knowledge generation. These units often have strategic motives for their knowledge Independent Evaluation Group World Bank Group    39 production, for example, to inform a new or growing business line or re- spond to corporate priorities, such as climate change. A review of GPs’ intranet pages shows that they produce valuable notes and reports on many operationally relevant topics, such as inclusion, citizen engagement, Paris Alignment, and climate co-benefits. Many of these knowledge resources respond directly to current management priorities. According to IEG’s inter- views with managers, responding to implementation challenges or demands from clients and regional staff were infrequent motives for global units’ knowledge production. As a result, global units’ knowledge often lacks strong operational appli- cations. Managers in “knowledge-producing” units are concerned about weak uptake of the knowledge they produce, according to IEG’s interviews. In interviews, many managers were concerned that much of the knowledge their units produce is too supply driven and unused by regional staff who seem to be unaware of this available knowledge. Most interviewed managers were aware that the traditional model of producing reports and disseminat- ing them to regional staff and clients through webinars, newsletters, briefs, blogs, and intranet pages is not leading to much application of this knowl- edge in operations. Yet this approach remains prevalent around the World Bank, including in many GP global units. There were exceptions, of course, including in the Water GP and the Africa Gender Innovation Lab (box 3.2). A project on early childhood education in Honduras benefited from knowledge support from the Global Partnership for Education. Similarly, an experienced TTL of environmental projects listed the trust funds that are useful knowl- edge sources and others that are not. However, overall, the evaluation’s project case studies did not show much use of the global units’ knowledge. TTLs interviewed for the case studies were partially aware of the global units’ sectoral expertise but were not aware of the entire range of global unit knowledge and expertise available to them.  nowledge Reorganization of the Health, Nutrition, and Box 3.1. K Population Global Practice The Health, Nutrition, and Population Global Practice reorganized itself to have a more coherently organized model for knowledge generation. It used a functional review by an external consultant to inform a reorganization, which resulted in five focus areas, each with its own technical teams led by the global leads, complemented by central functions, including staff support for training, partnerships, and knowledge manage- Learning in World Bank Lending Chapter 3 ment. The global leads have defined knowledge responsibilities with budgets and technical teams they supervise. The unit also consolidated its production of adviso- ry services and analytics into five umbrella programs. Consolidating 187 financing streams of Health, Nutrition, and Population assisted in this process. The Global Practice revamped its staff learning by improving new staff onboarding, instituting basic accreditation, initiating mentoring and peer groups, and mandating knowledge sharing as part of staff’s work program. The unit uses multiple approaches to promote operations’ uptake of its knowledge. It provides global evidence, produces diagnostic tools intended for clients’ use, assists task teams in the Regions in using the tools, and runs client-facing training courses and networks. 40 Source: Independent Evaluation Group. Staff’s motivation to pursue knowledge and learning often determines the level of knowledge and learning the World Bank generates. The World Bank holds staff accountable for project design and disbursements more than for fostering knowledge use, engaging clients in learning, and achieving influ- ence. IEG interviewed many TTLs who saw themselves as part of a learning culture and, seemingly driven by intrinsic motivation, went beyond their official duties and the linear model’s requirements to learn and share knowl- edge. The World Bank gives TTLs discretion over the extent to which their lending projects harness and apply knowledge, subject to meeting certain minimum requirements at design as mentioned in chapter 2. Proactive, motivated TTLs did this by securing trust funds and other resources, partic- ipating in knowledge exchanges and communities of practice, and engaging with clients and partners to conduct analytic work, share knowledge, and learn together. In addition, IEG interviewed many managers who champi- oned knowledge sharing and use by setting up knowledge sharing modalities. The Knowledge Compact seeks to bolster this staff motivation and the organization’s knowledge culture, although the pathways it intends to follow remain unclear. IEG’s evidence and experience from World Bank practi- tioners suggest that giving rewards and prizes is not the best way to boost staff’s intrinsic motivation, but creating incentives, managerial signals, and systematic and predictable opportunities to share knowledge can be much more valuable (Ijjasz-Vasquez et al. 2024). Independent Evaluation Group World Bank Group    41 The World Bank’s knowledge ecosystems for country and technical knowledge are largely decentralized. The lack of structure and support for knowledge ecosystems stands in sharp contrast to the linear model’s prescribed processes and the World Bank’s well-structured approach to orga- nizing operational knowledge. In the linear model, well-defined entry points and assigned roles ensure that operational knowledge—such as instrument choice or design of DLIs—is given due consideration at specific moments. The Operations Policy and Country Services Vice Presidential Unit has codi- fied the relevant operational knowledge and organized it on the intranet and through a help desk, staff trainings, on-the-job coaching, and team leader accreditation. The World Bank has no similar knowledge requirements for country and sector knowledge, apart from formal staff training organized under the “Open Learning Campus.” GP global units have knowledge man- dates but no guidance on how to best fulfill those. Units tend to follow their own knowledge management practices, led by managers with sectoral rather than knowledge management specialties. The World Bank’s approach to organizing knowledge has changed over time. At times, it has had action plans or strategies for knowledge with a central knowledge management team led by a director in charge, and at other times, it has left knowledge as an entirely decentralized responsibility. The here- tofore limited structure and support for knowledge ecosystems also stands in contrast to the World Bank’s structured approach to the Environmental and Social Framework, data, communications, procurement, and safeguards (table 3.1). Reforms in 2024 created a Knowledge and Learning Department and the World Bank Group Academy for client-facing offerings, thereby creat- ing welcome opportunities for central organization and senior management oversight of knowledge management. Other multilateral development banks also face challenges with embedding knowledge generation and use into core business processes. A benchmarking study by the European Bank for Reconstruction and Development Evaluation Department concluded that multilateral development banks (the sample included the World Bank and the International Finance Corporation) face the challenge of trying to embed knowledge and knowledge processes into a lending-oriented business model that has operated well in the perception of senior management without extensive knowledge management practices. The study found that knowledge management action plans and strategies need strong managerial endorsement and follow-through to be successful. Meanwhile, some large foundations have firmly embedded learning into Learning in World Bank Lending Chapter 3 core business processes and the DNA of their organizations. Three private sector firms (American Funds, Deloitte, and Royal Dutch Shell) selected as comparators based on their excellence in knowledge management have also been more successful in building a business case for knowledge management and learning, thereby gaining strong leadership support, creating a receptive culture for knowledge management and learning, and ensuring resourcing (EBRD 2021). 42  unctional Comparison Table 3.1. F Knowledge Management for Sector Management Environmental and Country of Operational and Social Development Functional Area Knowledge Knowledge Framework Communications Procurement Data Vision and strategy Yes Yes Yes Yes Yes Yes Dedicated unit(s) Newly created Yes Yes Yes Yes Yes responsible for the OPCS OPCS External and Development agenda corporate relations Data Group Dedicated staff Partially Yes Yes Yes Yes Yes with clear roles, responsibilities, and expertise Processes, No Yes Yes Yes Yes Yes standards, and 2018 ESF Framework and guidelines for policies agenda Source: Independent Evaluation Group. Note: ESF = Environmental and Social Framework; OPCS = Operations Policy and Country Services. Independent Evaluation Group World Bank Group    43 The GPs execute their knowledge mandates unevenly. In 2019, the Knowledge Flow and Collaboration Under the World Bank’s New Operating Model evalu- ation found that “each GP manages knowledge differently; some are more coherent and strategic than others.… The Global Solutions Groups lack mechanisms for channeling knowledge into country programs…. Differences in leadership attention and funding explain much of the variation” (World Bank 2019a, xiv, ix). This finding continues to ring true. As of June 2024, all 15 GPs have learning weeks and other internal knowledge sharing practices. However, only 8 GPs’ intranet pages have the learning week resources readily available with links that work. All 15 GPs have global leads, and all except 1 GP list the names of the global leads on the intranet, making these staff easy to find. Nine GPs list communities of practice or Global Solutions Groups (others may have communities of practice, but their intranet pages do not clearly list them). One GP, Water, has an integrated help desk (box 3.2); 3 GPs have a dedicated email helpline; and 4 GPs list the email of their knowledge and learning staff. In the remaining 7 GPs, it is unclear whom staff would turn to for help with accessing technical knowledge. Operational knowledge is the best organized, and country knowledge is the least organized, with sector knowledge in between (table 3.2). This can be explained by the fact that operational knowledge is nonconfidential, essential to the core lending business, and responsibility for organizing it rests with a central unit (Operations Learning and Engagement unit in the Operations Policy and Country Services vice presidency), along with “Country and Operations-mapped” staff affiliated with that unit. In contrast, Learning in World Bank Lending Chapter 3 responsibility for organizing sector and country knowledge rests with 22 GPs and theme groups and about 38 Country Management Units. 44  pproaches to Country, Sector, and Operational Knowledge Table 3.2. A Examples of Examples Approaches to Type of of Explicit Leverage Tacit Responsible Knowledge Approaches Knowledge Comments Departments Operational Policies, On-the-job learning The most systematic approaches to codifying and OPCS and Regions’ procedures, and requirements for TTL sharing knowledge are for operational knowledge. development guidance and staff accreditation, Academies effectiveness units training courses (IPF, DPF, PforR, CPF, Guarantees), and Operations help desk Sectoral ASA and other Webinars, learning GPs codify routinely sectoral knowledge in tools and GPs, cross-cutting theme reports, tools, and weeks, peer reviews, and ASA. Project-specific knowledge, in contrast, is at groups, Gender Innovation blogs GPs’ help desks best shared informally but without much method or Labs, Development support. Economics, and trust- funded partnerships Country Flagship ASA; Reliance on country The country engagement products are often not able CMUs and Regions country office staff and client to capture targeted outcomes, especially for outcomes engagement learning and workshops that are beyond project interventions or come from products, indirect development pathways, such as institutional including CPFs development, capacity building, knowledge transfer, and Completion demonstration effects, and market creation (World and Learning Bank 2020d). These products are little used, and Reviews no other system helps in for codifying, sharing, and learning from country knowledge. Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; CMU = Country Management Unit; CPF = Country Partnership Framework; DPF = development policy financing; GP = Global Practice; IPF = investment project financing; OPCS = Operations Policy and Country Services; PforR = Program-for-Results; TTL = task team leader. Independent Evaluation Group World Bank Group    45 Existing tools to capture and learn from country knowledge have shortcom- ings, making country office staff the primary repository of such knowledge. IEG’s evaluation of country programs’ outcome orientation found that how the Bank Group aims for outcomes at the country level is sound and, for the most part, country teams practice this model well, but that the Bank Group’s country evaluation product, the Completion and Learning Review, provides a partial picture of Bank Group contributions (World Bank 2020d). The picture is partial because of its overemphasis on those results that can be measured and on results from lending projects. The Completion and Learning Review rarely captures complementarities across instruments or institutions and so cannot establish whether the Bank Group’s contribution to country out- comes amounts to more than the sum of its parts. Because of their timing and limited content, the reviews are rarely used within the institution (World Bank 2020d), a finding echoed in this evaluation’s case studies. This is part of a larger pattern where the existing results architecture mechanisms serve upward reporting more than they support learning, adaptive management, or decentralized decision-making. There is ample evidence that staff engage with results architecture mechanisms with a “compliance mindset.” Some of the results tools and mechanisms tend to hinder rather than empower teams’ ability to engage in informed risk taking and adaptive management (World Bank 2016, 2020d). Apart from the country engagement products, the main responsibility of the Country Management Units is more to oversee the program than to provide knowledge, and IEG did not come across systematic efforts to foster the Learning in World Bank Lending Chapter 3 sharing of country knowledge except country team retreats. Admittedly, country knowledge is hard to organize because some of it is confidential and not easily codified. The World Bank therefore largely relies on tacit knowl- edge and relationships of staff located in country offices to understand a country’s context, political economy, governance issues, implementing agency capacity, and other country knowledge, as discussed in chapter 2. More systematic informal knowledge exchanges, including of country knowledge, could help improve learning conditions for lending teams. Yet knowledge and learning staff dedicate little time to these functions. Most of the World Bank’s knowledge management and learning practices focus 46 on disseminating explicit knowledge; as such, this is what knowledge and learning staff focus on as well. This is according to IEG’s interviews with managers and an electronic questionnaire of knowledge and learning pro- fessionals. Similarly, most of this explicit knowledge comes in the form of operational and sector knowledge rather than country knowledge. The transfer of tacit knowledge is deeply ingrained in the World Bank’s orga- nizational culture. Most staff rely on informal exchanges and transfer of tacit knowledge to gain a large part of the country and sectoral knowledge they need for their work. Previous IEG evaluations established the value of tacit knowledge (World Bank 2014, 2015, 2022a). For example, IEG reported that “interviews and the TTL survey show that informal interactions and idea exchanges among colleagues [were] one of staff’s most important sourc- es for global knowledge” (World Bank 2022a, 59). The sample case studies found that teams highly value tacit knowledge sources and less frequent- ly and prominently mentioned explicit and codified knowledge sources, such as templates, guidelines, document libraries, and training programs. In addition, staff often value opportunities to share knowledge: several of the teams involved in the sample cases described sharing project lessons at events, and several managers described how, for their staff, sharing lessons at events such as learning weeks provides recognition and incentives. Many of the managers whom IEG interviewed recognized the importance of infor- mal exchanges as opportunities to transfer tacit knowledge in World Bank culture and that staff often sought such opportunities instead of their units’ explicit knowledge sharing approaches. All 22 senior knowledge and learn- Independent Evaluation Group World Bank Group    47 ing professionals who responded to an IEG questionnaire acknowledged the importance of informal exchanges and tacit knowledge. Responding to the question, “In your experience, what sources of knowledge do staff consult to design and implement their projects?” most replied “peers” or “professional and personal networks.” Yet the World Bank has not optimized the transfer of tacit knowledge. Knowledge exchanges need not rely on random factors—they can be facilitat- ed and replicated and follow specific models and methods. Many units facilitate the unpacking of tacit knowledge through events, platforms, work- shops, learning weeks, communities of practice, and other people-focused approaches, but they do so with uneven attention to instructional design. Managers in GP global units explained that they lack knowledge on the most effective approaches to designing knowledge exchanges, have little profes- sional knowledge and learning support for this, and receive no corporate guidance on how to best organize these exchanges. Most World Bank knowl- edge sharing processes support internal knowledge sharing rather than sharing with external partners and clients and transfer codified knowledge rather than packaging and transferring tacit knowledge. These knowledge management and learning functions are defined by established knowledge management competencies, but the units apply these unevenly. In practice, knowledge and learning professionals support staff learning, content manage- ment, and communications—publishing or supporting newsletters, blogs, and briefing notes and event planning (figure 3.2). These can be helpful support functions. However, knowledge and learning staff are rarely involved in sup- porting knowledge inputs to the World Bank’s standard lending cycle or communities of practice and other occasions to unpack tacit knowledge, although with exceptions, such as in the Health, Nutrition, and Population GP.  rocesses That Knowledge and Learning Professionals Figure 3.2. P Support Processes supported by knowledge and learning professionals Planning internal events Content management Planning and delivering the staff learning program Managing communities of practice Producing blogs and newsletters Planning external events and high-level conferences Brokering technology and innovations Learning in World Bank Lending Chapter 3 Composing briefings, talking points, and knowledge notes Publishing reports Planning and delivering a data management program Planning and delivering the client learning programs 0 5 10 15 20 25 Share of professionals (%) Lead Support Do not handle n.a. Source: Electronic questionnaire of 22 knowledge and learning staff. Note: n.a. = not applicable. The absence of systematic mechanisms for exchanging knowledge hindered 48 the World Bank’s mobilization of new or existing knowledge. TTL interviews suggested that the absence of such mechanisms limited the integration of new or existing knowledge into projects during preparation and imple- mentation. One TTL cited an instance where Zambia’s government asked for Kenya’s and other countries’ experience in labor-based road works. The staff member was unaware of such experiences but later, when acting for the manager, discovered that a colleague’s aide-mémoire contained it. The example illustrates the lack of systematic knowledge sharing mechanisms even within units. The cases found that teams typically documented the major explicit knowledge inputs they used during the design but not the trust-funded studies and informal client learning engagements they used during the implementation, making such knowledge hard to reuse or build on. This is because the World Bank’s operational filing system focuses on tracking operations’ milestones, implementation challenges, disbursements, and safeguard compliance, and not their knowledge inputs and learn- ing. Conversely, IEG’s Mid-Term Review of the gender action plan found that when gender specialists acted as knowledge brokers, it enhanced the Regions’ and GPs’ uptake of gender knowledge (World Bank 2021). This ex- ample shows the value of brokering tacit knowledge. The World Bank appears to underuse some types of knowledge exchange. Many country teams do not appear to regularly exchange knowledge to enhance staff’s learning about countries’ context, political economy, and implementation environment. There may be room to expand exchanges Independent Evaluation Group World Bank Group    49 between knowledge producers in central units and regional staff, clients, and other intended users. Staff have few opportunities to share knowledge about mistakes and failures. Some business units have developed more systematic knowledge sharing approaches. Three examples that focused on sharing of technical knowledge stood out during IEG’s analysis: (i) the Africa Gender Innovation Lab has refined its approach to translating analytics into operations; (ii) the Water GP continues its long-running focus on brokering and systematizing tacit knowledge; and (iii) the Social, Urban, Rural, and Resilience leadership team cultivated an ecosystem centered on sharing tacit, experiential knowledge, although leadership changes and the COVID-19 pandemic have since re- duced the GP’s knowledge sharing efforts. The presence of leadership who championed knowledge and incentivized knowledge sharing was a common element in these three examples (box 3.2).  usiness Units with Systematic Approaches to Link Producers Box 3.2. B and Users of Knowledge The Africa Gender Innovation Lab. Initiated in 2008, the unit has refined its approach- es to evidence and uptake over time. The unit started by generating evidence but observed that World Bank operational teams want to know not only what works but how it works, how to adapt interventions to country and project contexts, and how cost-effective and scalable interventions will be. It also observed that many more actors than the World Bank are actively brokering gender evidence. As it began to understand the constraints to knowledge use, it changed to approaches that bet- ter influence operations and policies. For example, it fosters year-long collaborative engagements between its researchers and lending teams to promote the uptake of research findings. It focuses on impactful policy reforms, how to sequence poli- cies for impact, and how to expand or replicate projects. It has found much value in multicountry engagements that enable learning relationships across similar projects. Examples include engagements on a flagship initiative in the Sahel to help girls stay in school longer and delay marriage and childbearing and on a female entrepreneurship program, which reportedly influenced 20 projects across 15 countries. It hires staff with skills in translating knowledge into project and policy design and places some staff in countries where they can better provide advice and interact with governments and project implementation units. The Africa Gender Innovation Lab team found this oper- ational focus more impactful than producing and disseminating studies. The Water Global Practice. The Global Practice’s needs assessment found that Water staff prefer tacit knowledge from trusted colleagues. The unit values techni- Learning in World Bank Lending Chapter 3 cal, applied, and instrumental knowledge and tools that teams and clients can apply. It generates relatively few and focused reports, combined with brief notes, some of which document task team leaders’ tacit knowledge and project experiences. The Global Practice’s global unit uses webinars, learning weeks, and other events to promote staff’s understanding of these lessons. The unit uses its intranet page as a one-stop shop for knowledge resources and runs a service desk, AskWater, which about half of Water staff use in any given year. The service desk responds directly to queries or routes them to relevant experts and focuses on providing rapid answers and on codifying information. A typical request to AskWater might be for sample terms of reference or help with identifying an expert. (continued) 50  usiness Units with Systematic Approaches to Link Producers Box 3.2. B and Users of Knowledge (cont.) The Social, Urban, Rural, and Resilience Global Practice. After the Social, Urban, Rural, and Resilience Global Practice was created in the World Bank’s 2013 reorganization, its management invested in communities of practice to counter the Global Practice’s si- loed technical knowledge and as an integral element of how the Global Practice would carry out its business. The Social, Urban, Rural, and Resilience Global Practice manage- ment ensured that the communities of practice were useful by (i) relying on bottom-up processes to address the team’s needs; (ii) focusing on breaking knowledge silos across Regions and technical disciplines; (iii) creating “knowledge holders” (that is, one- stop shops or help desks for a particular business line); (iv) using effective knowledge management tools; and (v) targeting culture and behavior change to make strategies for knowledge and learning work. These managerial approaches are very different from the linear model’s specified entry points: “The GSURR [Social, Urban, Rural, and Resilience Global Practice] leadership team developed a unique label for the communities of practice— knowledge silo breakers [KSBs]—to signal the desire to break down traditional organizational barriers to knowledge flows and collaboration. But GSURR management did not stop at ideation…. [It] consistently and clearly communicated about the role and importance of KSBs. This approach provided an authorizing environment for the KSBs to pursue their ambitions and Independent Evaluation Group World Bank Group    51 innovations. It also gave an informal incentive to ambitious staff who saw KSBs as an opportunity to pursue business ideas, develop and strengthen leadership skills, and gain visibility and recognition from their peers and management.” (Ijjasz-Vasquez et al. 2024, 42) Source: Independent Evaluation Group. The World Bank’s few knowledge and learning staff are embedded in units and mostly without the remit to work across the organization. This is ac- cording to IEG’s electronic questionnaire of knowledge and learning staff and a light skills mapping. On the basis of a count of staff with “knowledge,” “learning,” or “knowledge management” in their business title, IEG esti- mates that the World Bank has about 102 professional-grade knowledge and learning staff in operational and corporate units—excluding Information Technology, Human Resources, and IEG—of which half are lower-level analysts or associates. This is a tiny fraction of the World Bank’s more than 10,000 staff. Most of these knowledge and learning staff are mapped to tech- nical units where they respond to the unit’s internal needs with limited line of sight to the World Bank–wide knowledge management needs and policies. Only one Vertical—Prosperity—pools its knowledge and learning staff at the vice presidential unit level, and some corporate knowledge and learning staff may work across. IEG could not identify a single knowledge and learning staff member who managed, connected, or disseminated knowledge streams across the institution or beyond. As such, these staff members could not contribute to breaking sectoral knowledge silos or recommend and imple- ment knowledge-enabling tools and processes across the organization, let alone with clients and partners. Learning in World Bank Lending Chapter 3 52 4 | Conclusions and Recommendations The World Bank has many strengths when embedding knowledge in its financing. It has a strong reputation as a provider of development knowl- edge that builds on its deep relationships with clients and cross-sectoral expertise. The World Bank is active in many countries’ policy processes and can back up its policy advice with financing. It is a Knowledge Bank with a long-term commitment to producing, disseminating, and applying knowl- edge through its established partnerships. It has teams and managers who go above and beyond World Bank requirements to pursue knowledge by creating their own learning spaces and knowledge ecosystems, sometimes opening these up for clients as well. The World Bank uses both a linear model and a multidimensional ecosystem to learn in and from its financing. Organizational learning is a multifaceted process that benefits from both the sharing of explicit knowledge and the transfer of tacit knowledge. Explicit lesson learning, such as that provided by ICRs, country programs’ learning reviews, IEG reports, and impact evalua- tions, offers a structured and scalable way to document and share knowledge from projects. The transfer of tacit knowledge—through job rotations, discussions, and other means—emphasizes interpersonal interactions and experiential insights. Both types of learning enrich the organization’s collec- tive knowledge base. Learning opportunities vary across lending instruments and project phases. In the linear model, World Bank financing instruments have formal poli- cies, procedures, and guidance, as well as budget norms, that create defined spaces, moments, and incentives for knowledge and learning in the design phase but lead to less attention during the implementation phase. PforR operations are somewhat of an exception in having explicit incentives and mechanisms for learning. The PforR’s annual DLI verification process creates     53 a formal, mandated, periodic, data-supported space for knowledge sharing and learning between the World Bank and clients. This learning opportu- nity is supported by data on PforR’s progress toward the DLIs, which tend to be aligned with important project outcomes. By contrast, DPF and IPF have fewer built-in formal learning opportunities, so they often rely on tacit knowledge and highly motivated TTLs to generate learning. ICRs provide formal learning opportunities, but these are underused because the culture and incentives treat the ICRs as a compliance function. In the World Bank’s broader knowledge ecosystem, business units run their own systems for learning outside of the linear, project-driven model. However, these systems do not have minimum standards or requirements and few formal mecha- nisms or incentives so vary in quality and scope. The World Bank’s reliance on tacit knowledge is valuable but risky. In line with previous evaluations, this evaluation finds that World Bank staff and managers are far more likely to rely on informal networks to share lessons. In part, this is because adult learners value social learning and tend to trust their peers to select the relevant advice among daunting volumes of codified knowledge. Learning from an exchange of tacit knowledge is simpler and requires only identifying the right person to talk to. However, despite their prevalence, the World Bank has not invested much in bringing methods and support for the transfer of tacit knowledge. Areas for Management Attention This evaluation shows that the World Bank’s current approach to learning in lending has imbalances, relies on tacit knowledge and informal learning, and rests on fragmented support (table 4.1): Learning in World Bank Lending Chapter 4 » First, the linear model leads to three major imbalances. The model creates a focus on producing reports but often neglects applying these reports’ find- ings to operations and country-level processes. Moreover, the project cycle steers management’s attention to the design phase, leading to less attention to knowledge and learning during the implementation phase. The model re- inforces the World Bank’s emphasis on generating global sectoral knowledge and the tendency to approach country knowledge through informal exchang- es of tacit knowledge. » Second, the World Bank’s lesson learning often relies on informal lessons and tacit experiences in prior or parallel operations. However, this approach is not 54 always reliable because it depends on chance discoveries and shortchanges smaller countries and new engagement areas that typically lack prior or par- allel operations to learn from. This is part of a larger issue of a culture that greatly values informal knowledge exchanges but organizes them with little methods and support. » Third, the World Bank’s knowledge ecosystems are fragmented, disconnect- ed, and underresourced. This lack of consistency and minimum standards leads to inefficiencies in the knowledge flows and challenges in knowledge reuse. For example, proper classification of knowledge with clear labels and contexts helps artificial intelligence models understand patterns and rela- tionships, leading to more effective learning and adaptation. The costs from these three characteristics are not measured. Little is therefore known about their contributions to inefficient work processes, inconsistent project quality, and missed outcomes. This evaluation does not attempt to assess such costs, but the evidence leaves little room for doubting that these imbalances and knowledge weaknesses adversely affect project performance.1 Independent Evaluation Group World Bank Group    55 56 Learning in World Bank Lending Chapter 4  nowledge and Learning Shortcomings Identified by the Evaluation Table 4.1. K Consequence of Suggestions to Address Shortcomings Shortcomings Reason for Shortcomings Shortcomings Focus on projects’ design Missed opportunities to course Linear model’s policies, Adjust policies and procedures to phase with less attention to correct during implementation, procedures, and guidance create learning moments knowledge and learning during leading to adverse effects on Foster adaptive management during implementation project performance implementation Focus on report production Costs of producing reports with The projectized approach that Strengthen the collaboration between over knowledge use and little use holds staff accountable for clients, partners, and World Bank dissemination report delivery more so than for authors of analytic work to elicit more influencing ownership of the intended audience and improve the reach of the work Focus on technical knowledge Projects with strong Few requirements for capturing or Systematize mechanisms to share, while leaving country knowledge technical design running into sharing country knowledge unpack, and accumulate country mostly dependent on learning implementation challenges that knowledge, notably political from informal exchanges require better contextual country economy, while protecting sensitive knowledge to prevent or resolve information Weak formal lesson learning Teams learn by unpacking tacit Behaviors and incentives around As part of the ongoing ICR reform, because ICRs are produced but knowledge from previous or the ICR and other self-evaluation improve the quality of ICR and ICRR not actively disseminated and ongoing operations, leading to systems lessons and promote use and learning little used issues with reliability and learning from them, including via enhanced Culture that discourages open in new areas of engagement perceptions of ICRs and ICRRs sharing of failures Provide managerial signals of openness to sharing of mistakes and failure (continued) Consequence of Suggestions to Address Shortcomings Shortcomings Reason for Shortcomings Shortcomings Fragmented and disconnected Lack of consistency and Approaches to knowledge Promote World Bank–wide knowledge ecosystems standards in knowledge management lacking governance knowledge management standards management and direction and processes and leverage professional knowledge management Reliance on chance discoveries Few World Bank knowledge staff to improve how the World Bank and learning staff to support its Inconsistent connection between supports knowledge capture, storage, operations knowledge production and sharing, and unpacking operations Invest in communities of practice and The most experienced staff technical help desks with heavy work programs who Give more emphasis to knowledge produce, use, and disseminate generation and knowledge impact in knowledge with little support staff awards Use the Academy programs to raise the bar for client and internal capacity development, knowledge management, and instructional design Source: Independent Evaluation Group. Note: ICR = Implementation Completion and Results Report; ICRR = Implementation Completion and Results Report Review. Independent Evaluation Group World Bank Group    57 Recommendations The procedures and guidance shape the incentives for integrating learning and knowledge generation into lending activities. The explicit expectations and incentives for learning from operations are limited to the lending pro- cess requirements, which staff generally follow, and managers pay attention to. However, many expectations and knowledge practices, such as how and when knowledge should be used and when and how clients are engaged, are implicit and implemented unevenly. Furthermore, the supervision budgets do not align with the mandates for knowledge, particularly in low-capacity countries and multicountry MPA programs. This indicates that adjusting the existing tools, procedures, and budgeting norms could encourage more con- sistent knowledge and learning in financing. Recommendation 1. Make better use of the learning opportunities that are already embedded in the lending processes. Operations Policy and Country Services should revise the procedures and guidance for lending to incen- tivize more consistent learning throughout the lending cycle. Specifically, Operations Policy and Country Services should set clear expectations to the type of knowledge and learning, the moments and processes for learning activities, and the level of client engagement required from project teams. Box 4.1 offers some examples of what such process tweaks could entail. At the same time, managers at all levels should create spaces for knowledge and learning, including via quality enhancement reviews and informal and early meetings with peer reviewers. Managers should also role model attention to knowledge and learning and openness to discussing failures. Learning in World Bank Lending Chapter 4 58  xamples of Lending Processes Where World Bank Box 4.1. E Management Could Infuse Learning Opportunities » The process around project Mid-Term Reviews could strengthen attention to learning—for example, by reengaging with the peer reviewers involved at approv- al. Documentation of Mid-Term Reviews can be broadened in scope to touch on knowledge, learning, and projects’ contribution to higher-level outcomes, as also suggested by the Corporate Scorecard’s “results narratives.” » Managers could recognize and reward task team leaders that go beyond in pursuing joint learning with clients, which would be easier if the Operations Portal, now called Workspace, captured learning with clients during implementation. » Country Management Units could incorporate learning agendas into Country Partnership Frameworks, as does the Asian Development Bank, with dedicat- ed budgets and objectives that can be monitored. They could also incorporate learning agendas into regular discussions with clients when planning activities to support Country Partnership Framework implementation. » Country Management Units could invite discussions of the reasons for dropped and canceled projects and of failures and challenges more broadly. Some of the existing tools such as the Program and Learning Reviews could be expanded to capture lessons from dropped and canceled projects and similar learning. » Managers could ensure that task team leaders use handover notes more inten- Independent Evaluation Group World Bank Group    59 tionally when they rotate. » Senior management could track the rate of task team leader turnovers by depart- ment to incentivize proactive management of it. » Preparation and review of Implementation Completion and Results Reports could pay more attention to the quality of lessons and their applicability to future proj- ects in the country and the sector more broadly. » The process for validating Implementation Completion and Results Reports could give more space for project teams and Independent Evaluation Group evaluators to discuss, identify, and capture lessons. (continued)  xamples of Lending Processes Where World Bank Box 4.1. E Management Could Infuse Learning Opportunities (cont.) » Responsibilities for learning from these lessons could be more clearly defined. Global Practices or Country Management Units could organize periodic sessions to reflect on lessons. » Implementation Status and Results Reports for Multiphase Programmatic Approach (MPA) operations could report on and revise the MPAs’ learning agendas. » As suggested in the Independent Evaluation Group’s MPA evaluation, “learning should also encompass institutional development over the program cycle. This learning may entail developing indicators that, for vertical MPAs, measure the ef- fectiveness of long-term institutional reforms and, for horizontal MPAs, incentivize and measure the effectiveness of collaboration among participants” (World Bank 2024, 41). » Practice managers overseeing MPAs could create intentional learning moments for staff beyond the immediate project team to reflect on lessons from imple- menting MPAs’ learning agendas. These could occur in the transition between MPA phases—for example, at the Concept Note meeting for the next phase. » Task budgeting norms could be adjusted to be commensurate with projects’ learning mandates. Source: Independent Evaluation Group. Learning in World Bank Lending Chapter 4 The knowledge ecosystems are fragmented and do not fully leverage learn- ing from informal exchanges of tacit knowledge. The World Bank currently lacks the ability to set central knowledge management standards regarding what types of knowledge should be codified, at what moments, using what methods and processes. It also lacks the ability and staff to bring methods and support to enhance knowledge exchanges. As a result, units run their own knowledge and learning practices with no central support or standards, few staff with professional credentials in knowledge management or adult learning, and uneven quality. For example, many units sponsor activities to share tacit and explicit country and technical knowledge, including quality 60 enhancement reviews, help desks, learning weeks, communities of practice, off-the-record exchanges with peer reviewers, and safe space clinics. These activities tend to provide valuable knowledge and networking opportunities and enable candid exchanges that complement the formal project milestone meetings, but they are too often ad hoc. Informal opportunities for knowl- edge sharing should be organized more systematically, with the help of more guidance and knowledge management support while not losing sight of the value of tacit knowledge and informal exchanges. Recommendation 2. The Knowledge and Learning Department should ensure core knowledge management capacity and set World Bank–wide standards and processes for knowledge capture, storage, sharing, and access. The World Bank should rationalize the core capacity to organize key learning events with instructional designers and knowledge management profes- sionals. The essential knowledge management capacities, standards, and processes should have oversight by senior management. Implementing this recommendation could entail: » Setting standards for how knowledge is tagged, classified, stored, and shared across units. » Supporting the GPs in adopting these standards, for example, by insisting that knowledge from key events such as Knowledge Weeks, client workshops, and long-term learning engagements is tagged, classified, stored, and shared Independent Evaluation Group World Bank Group    61 across units. » Professionalizing and enhancing the capacities of knowledge management staff. Supporting career management for knowledge management staff. » Working with the GPs and Regions to better leverage professional knowledge management staff and thereby relieve TTLs. » Working with the GPs and Regions to bring enhanced methods and support to knowledge sharing activities, including enhanced attention to instructional design. » Bringing methods and support to enhance informal knowledge exchanges. Investing also in communities of practice and technical help desks. » Periodically surveying staff’s need for, and satisfaction with, knowledge management. Progress on implementing these recommendations could be monitored in the following manner: » Recommendation 1—via monitoring of the number of new learning oppor- tunities created along the project life cycle, what mechanisms and incentives support these opportunities, and how teams and managers use them. » Recommendation 2—via staff surveys that periodically assess how staff perceive the World Bank’s learning environment and better evaluation of staff and client-facing learning events; by encouraging project documents and program documents to mention the ways in which communities of practice and other activities to unpack tacit knowledge helped the team solve problems, identify knowledge sources, or motivate design modifications; and via routine key performance indicators for the Knowledge and Learning Department. The evaluation identified several areas for future research. The relevance, quality, and influence of ASA are important for the World Bank’s ability to provide timely knowledge to clients but fell outside of the evaluation’s scope. Clients’ learning and capacity development outcomes were also out of scope. Both are important components of the Knowledge Compact. Furthermore, the evaluation provides only partial analysis of culture and in- centives around knowledge and learning, the weaknesses and effectiveness of using tacit knowledge, and ways to promote learning from mistakes and fail- Learning in World Bank Lending Chapter 4 ures. Future research and evaluations could consider covering these topics. 62 1 The project ratings system is not a good guide to the consequences of knowledge imbalances because it does not discern between delays and shortfalls stemming from weak knowledge and learning and other causes.  Independent Evaluation Group World Bank Group    63 Bibliography Ashton, Louise, Jed Friedman, Diana Goldemberg, et al. 2023. “A Puzzle with Missing Pieces: Explaining the Effectiveness of World Bank Development Projects.” The World Bank Research Observer 38 (1): 115–46. https://doi.org/10.1093/wbro/ lkac005. BRAC. 2024. Failure Report 2022–23. BRAC Social Innovation Lab. https://innovation. brac.net/_next/static/downloads/Failure-report-2022-23.pdf. Davenport, Thomas H., and Lawrence Prusak. 1998. Working Knowledge: How Organizations Manage What They Know. Harvard Business School Press. Denizer, Cevdet, Daniel Kaufmann, and Aart Kraay. 2013. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” Journal of Development Economics 105 (November): 288–302. https://doi. org/10.1016/j.jdeveco.2013.06.003. EBRD (European Bank for Reconstruction and Development). 2021. Learning and Knowledge Management at the EBRD: Experience of Other Organisations. EBRD. https://www.ebrd.com/evaluation-overview/special-studies-by-theme.html. Geli, Patricia, Aart Kraay, and Hoveida Nobakht. 2014. “Predicting World Bank Project Outcome Ratings.” Policy Research Working Paper 7001, World Bank. Goldhar, Joel D., Louis K. Bragaw, and Jules J. Schwartz. 1976. “Information Flows, Management Styles and Technological Innovation.” IEEE Transactions on Learning in World Bank Lending Bibliography Engineering Management EM-23 (1): 51–61. Honig, Dan. 2018. Navigation by Judgment: Why and When Top-Down Management of Foreign Aid Doesn’t Work. Oxford University Press. Honig, Dan. 2020. “Actually Navigating by Judgment: Towards a New Paradigm of Donor Accountability Where the Current System Doesn’t Work.” Policy Paper 169, Center for Global Development. Ijjasz-Vasquez, Ede, Philip Karp, and Monika Weber-Fahr, eds. 2024. Communities of Practice at the World Bank: Breaking Knowledge Silos to Catalyze Culture Change and Organizational Transformation. Routledge. https://doi. org/10.4324/9781003199083. 64 Knack, Stephen, Bradley C. Parks, Ani Harutyunyan, and Matthew DiLorenzo. 2020. “How Does the World Bank Influence the Development Policy Priorities of Low-Income and Lower-Middle Income Countries?” Policy Research Working Paper 9225, World Bank. Moll, Peter, Patricia Geli, and Pablo Saavedra. 2015. “Correlates of Success in World Bank Development Policy Lending.” Policy Research Working Paper 7181, World Bank. Orpen, Christopher. 1985. “The Effect of Managerial Distribution or Scientific and Technical Information on Company Performance.” R&D Management 15 (4): 305–308. https://doi.org/10.1111/j.1467-9310.1985.tb00041.x. Raimondo, Estelle. 2016. “What Difference Does Good Monitoring and Evaluation Make to World Bank Project Performance?” Policy Research Working Paper 7726, World Bank. Ravallion, Martin. 2016. “The World Bank: Why It Is Still Needed and Why It Still Disappoints.” Journal of Economic Perspectives 30 (1): 77–94. World Bank. 2010. “Transforming the Bank’s Knowledge Agenda: A Framework for Action.” Board Report 52989, World Bank. World Bank. 2014. Learning and Results in World Bank Operations: How the Bank Learns. Independent Evaluation Group. World Bank. Independent Evaluation Group World Bank Group    65 World Bank. 2015. Learning and Results in World Bank Operations: Toward a New Learning Strategy. Independent Evaluation Group. World Bank. World Bank. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Independent Evaluation Group. World Bank. World Bank. 2017a. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Independent Evaluation Group. World Bank. World Bank. 2017b. World Bank Group Engagement in Upper-Middle-Income Countries: Evidence from IEG Evaluations. Independent Evaluation Group. World Bank. World Bank. 2019a. Knowledge Flow and Collaboration Under the World Bank’s New Operating Model. Independent Evaluation Group. World Bank. World Bank. 2019b. Results and Performance of the World Bank Group 2018. Independent Evaluation Group. World Bank. https://ieg.worldbankgroup.org/ sites/default/files/Data/Evaluation/files/rap2018.pdf. World Bank. 2020a. Management Action Record Reform: IEG’s Validation Report. Independent Evaluation Group. World Bank. World Bank. 2020b. Results and Performance of the World Bank Group 2020. Independent Evaluation Group. World Bank. World Bank. 2020c. The World’s Bank: An Evaluation of the World Bank Group’s Global Convening. Independent Evaluation Group. World Bank. World Bank. 2020d. The World Bank Group Outcome Orientation at the Country Level. Independent Evaluation Group. World Bank. World Bank. 2021. World Bank Group Gender Strategy Mid-Term Review: An Assessment by the Independent Evaluation Group. Independent Evaluation Group. World Bank. World Bank. 2022a. Enhancing the Effectiveness of the World Bank’s Global Footprint. Independent Evaluation Group. World Bank. World Bank. 2022b. Results and Performance of the World Bank Group 2022. Independent Evaluation Group. World Bank. World Bank. 2023. Financial Inclusion: Lessons from World Bank Group Experience, Fiscal Years 2014–22. Independent Evaluation Group. World Bank. Learning in World Bank Lending Bibliography World Bank. 2024. Early-Stage Evaluation of the Multiphase Programmatic Approach. Independent Evaluation Group. World Bank. World Bank and Development Research Center of the State Council of China. 2022. Four Decades of Poverty Reduction in China: Drivers, Insights for the World, and the Way Ahead. World Bank. World Bank Group. 2018. “Sustainable Financing for Sustainable Development: World Bank Group Capital Package Proposal.” Paper prepared for the Development Committee Meeting, April 21, DC2018-0002/P, World Bank Group. World Bank Group. 2021. Realizing the World Bank Group’s Knowledge Potential for 66 Effective Development Solutions: A Strategic Framework. World Bank Group. World Bank Group. 2023. “Ending Poverty on a Livable Planet: Report to Governors on World Bank Evolution.” Development Committee, September 27, World Bank Group. World Bank Group. 2024. The Knowledge Compact for Action: Transforming Ideas into Development Impact—For a World Free of Poverty on a Livable Planet. World Bank Group. Independent Evaluation Group World Bank Group    67 APPENDIXES Independent Evaluation Group Learning in World Bank Lending Appendix A. Methods Approach for evaluation question 1: What types of knowledge inputs in- form the design and implementation of World Bank financing operations? For evaluation question 1, the evaluation used a mixed methods phased approach. In phase 1, the evaluation used a case-based, inductive approach to generate a set of hypotheses on how the World Bank learns in its lending operations. These hypotheses were then tested deductively through addi- tional quantitative and qualitative methods in subsequent phases. To understand what types of knowledge inform projects and how and when, the evaluation followed a simple conceptual framework as illustrated in fig- ure A.1. The evaluation team derived a framework from the commonly used Kipling method to problem-solving (5W1H). As shown in figure A.1, projects require sector, country, and operational inputs for successful design and im- plementation. Country knowledge inputs are likely to be far more critical in complex settings (for example, in countries affected by fragility, conflict, and violence and those undergoing political instability), and sectoral inputs are key to designing multisectoral approaches and experimenting with novel ap- proaches. Similarly, new client relationships and new sectoral issues present more challenges than clients and issues that are familiar to the World Bank. The injection of these knowledge inputs is closely tied to the project cycle. The World Bank has several structured entry points for learning during the project cycle, including quality at entry reviews, Concept Note discussions during appraisals, decision meetings for approval, Mid-Term Reviews at the time of implementation, regular Implementation Status and Results Reports, and lesson learning through the Implementation Completion and Results Report at closing. Staff use different knowledge inputs for various purposes in financing op- erations. Knowledge can be used for strategic purposes to build a business case for a project or to motivate a decision. Knowledge can also be used for instrumental purposes—for example, for solving a problem, such as how to design a specific project component. Instrumental use requires that task teams have access to the right knowledge inputs at the right time, often in 70 the design stage. Some knowledge use is more conceptual in nature, inform- ing thinking and promoting a shared understanding of an issue, and timing is less important. The process of using different types of knowledge inputs to drive action in lending operations is referred to as learning. These knowledge inputs can be explicit or tacit. Explicit knowledge is in tangible and written forms (reports, books, and databases). Conversely, tacit knowledge is gained through experience and engagement with peers relying on relationships to some extent.  onceptual Framework Figure A.1. C K N OW L E D G E Why? Uses of Knowledge Strategic Instrumental Conceptual How? Who? Form of Knowledge THE LINEAR MODEL Source of Knowledge Preparation Appraisal Implementation Completion and Approval Tacit Explicit Internal External What? Type of Knowledge Sectoral Country Operational Source: Independent Evaluation Group. Independent Evaluation Group World Bank Group    71 The evaluation uses a case-based approach to investigate 34 projects. It applies the conceptual framework to the projects that it selected using purposive and stratified random sampling techniques to optimize mean- ingful variation in the sample and thereby enhance the external validity of the findings. The evaluation team used stratified random sampling to select approved World Bank operations between 2014 and 2023 across differ- ent Regions, Global Practices, and country types (the International Bank for Reconstruction and Development and the International Development Association), ensuring a broad and meaningful variation in the sample across the World Bank’s broader organizational context. The team also used purposive sampling to ensure depth and inclusion of (i) various instrument types, such as Program-for-Results and development policy financing; (ii) the entire lending cycle for both closed and active operations; and (iii) specific Regions and Global Practices, including Social Protection and Jobs and West Africa, that have built a strong culture for knowledge and learning as a result of trust fund funding, among other reasons. The sample includ- ed 6 operations from Prosperity, 13 from Planet, 9 from People, and 6 from Infrastructure, across all 6 Regions. Table A.1 provides the list of projects included in the sample.  ampled Projects for Case-Based Approach Table A.1. S Project Global Practice Vertical Country Instrument Belize Agriculture and Planet Belize IPF Food Benin Multisectoral Health, Nutrition, People Benin IPF Food Health Nutrition and Population Project BR Amazona’s DPL Governance Prosperity Brazil DPF Burunga Wastewater Water Planet Panama IPF Management Project China Green Agriculture and Planet China PforR Agriculture and Rural Food Revitalization China Yunnan Honghe Transport Infrastructure China IPF Prefecture Diannan Center Urban Transport Learning in World Bank Lending Appendix A Climate and Disaster Urban, Disaster Infrastructure Türkiye IPF Resilient Cities Risk Management, Resilience, and Land Ecosystem Environment, Planet Sri Lanka IPF Conservation and Natural Resources, Management and Blue Economy Eswatini COVID-19 Health, Nutrition, People Eswatini IPF Emergency Response and Population Project FY17 Chad Emergency Social Protection People Chad DPF DPO and Jobs (continued) 72 Project Global Practice Vertical Country Instrument Greater Cairo Air Environment, Planet Egypt, Arab IPF Pollution Management Natural Resources, Rep. and Climate Change and Blue Economy Project India Water DPO Water Planet India DPF Iraq Emergency Social Social Protection People Iraq IPF Stabilization and and Jobs Resilience Project Jordan Inclusive, Governance Prosperity Jordan PforR Transparent and Climate Responsive Investments PforR Kenya Devolution Urban, Disaster Infrastructure Kenya PforR Project Risk Management, Resilience, and Land Madagascar Nutrition Health, Nutrition, People Madagascar IPF MPA and Population Mexico Environmental Environment, Planet Mexico DPF Sustainability and Natural Resources, Urban Resilience and Blue Economy Modernization and Governance Prosperity Serbia PforR Optimization PforR Morocco Green Growth Environment, Planet Morocco DPF DPL Natural Resources, Independent Evaluation Group World Bank Group    73 and Blue Economy Mozambique Agriculture and Planet Mozambique DPF Agriculture DPO Food National Immunization Health, Nutrition, People Pakistan IPF Support Project and Population Ocean Partnerships for Environment, Planet World IPF Sustainable Fisheries Natural Resources, and Biodiversity and Blue Economy Conservation Philippines First Finance, Prosperity Philippines DPF Financial Sector Competitiveness, Reform DPL and Innovation RMI Multisectoral Health, Nutrition, People Marshall IPF Early Childhood and Population Islands Development Project-II (continued) Project Global Practice Vertical Country Instrument Romania’s First Fiscal Macroeconomics, Prosperity Romania DPF Effectiveness and Trade, and Growth DPL Investment Senegal Safety Net Social Protection People Senegal IPF Project and Jobs Tanzania Sustainable Water Planet Tanzania PforR Rural Water Supply and Sanitation Tax Administration Governance Prosperity Moldova IPF Modernization Project Third Georgia Transport Infrastructure Georgia IPF Secondary and Local Roads Project Urban Resilience Urban, Disaster Infrastructure Côte d’Ivoire IPF and Solid Waste Risk Management, Management Resilience, and Land Urban Transport Transport Infrastructure Morocco PforR Vinh Phuc Flood Water Planet Viet Nam IPF Risk and Water Management Project West Africa Unique Social Protection People West Africa IPF Identification for and Jobs Regional Regional Integration and Inclusion (WURI) Program Learning in World Bank Lending Appendix A Zhuzhou Brownfield Environment, Planet China IPF Remediation Project Natural Resources, and Blue Economy Source: Independent Evaluation Group. Note: BR = Brazil; DPF = development policy financing; DPL = development policy loan; DPO = develop- ment policy operation; IPF = investment project financing; MPA = Multiphase Programmatic Approach; PforR = Program-for-Results; RMI = Marshall Islands. The evaluation’s case-based approach included document analysis, semi- structured interviews, and participant observations to thoroughly investigate knowledge use in the project cycle. Document analysis. The evaluation team collected and analyzed docu- 74 ments from the Operations Workspace throughout the project cycle to gain a foundational understanding of each project’s knowledge inputs. The team reviewed project Concept Note meeting minutes, peer reviewer comments and matrices, and Project Appraisal Documents during the preparation stage; Implementation Status and Results Reports, aide-mémoire (particu- larly for Mid-Term Reviews), and restructuring papers for implementation; and Implementation Completion and Results Reports, Implementation Completion and Results Report Reviews, Independent Evaluation Group re- views and Project Performance Assessment Reports, and notes for canceled projects for completion. In addition, the evaluation team analyzed trust fund reports and grant funding requests from external sources. Each case involved reviewing at least 10–14 documents. Semistructured interviews. The evaluation team conducted semistructured interviews with task team leaders (TTLs), co-TTLs, and managers. For each case study, the team interviewed one to three TTLs (both current and former ones, including co-TTLs) using a predefined interview guide (box A.1). Each interview lasted approximately 60 minutes and followed a protocol covering topics such as types of knowledge inputs, method of acquisition, specific use and timing, knowledge generated and channels of dissemination for projects, challenges encountered in knowledge flows, and perceived value and impact of different formal entry points created by the World Bank. The evaluation team took detailed notes for each interview. Observations. The evaluation team participated as observers in project Independent Evaluation Group World Bank Group    75 Concept Note and appraisal meetings to capture real-time knowledge use practices. The evaluation team took detailed field notes during observation for two lending operations. Within-case analysis. The evaluation team coded all information from all these sources for each case in a predefined protocol (as illustrated in figure A.2) to ensure systematic and consistent categorization of data. The evaluation team coded text excerpts in the documents, interview notes, and observa- tion field notes using the conceptual framework’s categories. Each case study offered patterns and granular insights into various knowledge inputs used instrumentally, strategically, and conceptually during the project cycle. This detailed examination of each case allowed the evaluation team to identify the extensive and in-depth knowledge used and produced within the World Bank’s lending operations.  ample Interview Guide Box A.1. S The purpose of this interview guide was to help the evaluation team cover a range of knowledge sources used to inform operations and capture change in subsequent ac- tion. This guide was tweaked based on the document review of the particular project: » What were the different knowledge sources used to inform the project, from appraisal to closure? » What do you think, in terms of knowledge or learning, has contributed to the success of the project? » Is there any specific example where a knowledge input enhanced your under- standing of and informed an action? » What were the challenges that hindered the progress of the project, and how did you use specific knowledge to address them? » Was that knowledge easily available and accessible? » In what way did the formal World Bank resources and processes, such as the proj- ect Concept Note meeting, Decision Note meeting, quality enhancement reviews, and Mid-Term Reviews, inform project design and implementation? » Were peer review processes being used as intended—that is, to provide mean- ingful insights into project design? » Did you refer to the Implementation Completion and Results Reports (same Learning in World Bank Lending Appendix A country and sector) before project design? » Did you reflect about the Implementation Status and Results Reports to under- stand what worked and what didn’t? Is that a knowledge input used in project implementation? » Have you used informal knowledge inputs and processes for this project? Informal inputs could include insights from global leads, collaboration with other task team leaders, and engagement with domain experts. (continued) 76  ample Interview Guide (cont.) Box A.1. S » How easy was it to access lessons learned from other projects, countries, and Regions? » Where do you go for learning for countries with limited World Bank experience? » Was there a cross–Global Practice collaboration for this project? » What processes were put in place from the World Bank side to ensure that knowledge flowed smoothly among the Global Practices? » Were there any sectoral networks you leveraged for knowledge and learning? These could include knowledge exchange between the World Bank staff and clients. » Were there any knowledge management processes set up for ongoing assessment? » Was there any special focus on knowledge management processes? » What inputs were used to inform the processes? » Did you rely on your and the team’s tacit knowledge to design and implement the project? » Some task team leaders have highlighted that they repeat activities across their projects without any reflection on the effectiveness of those activities. Do you Independent Evaluation Group World Bank Group    77 think that is the case? » How do you think the World Bank could have helped ensure improved learning and reflection for this project? » What kind of incentives can be leveraged to promote such learning? » What role did clients and other partners play in contributing to learning for this project? Source: Independent Evaluation Group.  rotocol for Capturing Case Studies Figure A.2. P Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; ISR = Implementation Status and Results Report; TTL = task team leader. Cross-case analysis. After completing the within-case analysis, the evalu- ation team conducted a cross-case analysis to compare findings across the Learning in World Bank Lending Appendix A 34 project cases. While single cases were deductively coded, the cross-case comparison was coded inductively. This comparative analysis helped identify commonalities and differences in knowledge production and use, as well as in World Bank processes and practices. It also highlighted the factors and enablers influencing the effectiveness of knowledge management practices. Based on within- and cross-case analyses, the evaluation team developed a set of hypotheses (as illustrated in this appendix), which were tested using mixed methods. Hypothesis: Projects during preparation are informed by strong analyt- ics, diagnostics, and academic research. The evaluation team conducted 78 an analysis of the references to reports, publications, databases, and other types of explicit knowledge as cited in financing projects’ Project Appraisal Documents and program documents. Specifically, the objective was to an- alyze explicit knowledge along four dimensions: (i) types of organizations generating the knowledge, (ii) proportion of knowledge generated by the World Bank, (iii) types of World Bank staff generating the knowledge, and (iv) types of World Bank knowledge. To do this efficiently for a large and representative sample from the population of projects, text mining and information extraction algorithms, along with generative artificial intelli- gence, were used. Administrative data related to World Bank staff were also leveraged. Over 3,100 references from over 1,000 projects were processed and analyzed. For further details, see appendix D. Hypothesis: World Bank operations leverage global sectoral knowledge and tend to incorporate country specificity, such as political economy, in financing operations. The evaluation team conducted an analysis of the responses of clients to the World Bank’s two-minute client satisfaction survey. Two (out of six) questions were relevant to the evaluation: one on the World Bank’s use of global expertise in financing projects and one on the World Bank’s tailoring of project design to country context. Survey response data were available for a random, representative sample of projects approved during the evaluation period, which covered 36 percent of the population of projects. For each question, the proportion of positive responses (that is, where the respondent agreed or strongly agreed with the question) was Independent Evaluation Group World Bank Group    79 calculated and aggregated by the project approval fiscal year, Practice Group and Global Practice leading projects, and Region. This analysis had the following key limitations: (i) survey responses were available for a random, unbiased sample of projects approved during the evaluation period because of which the satisfaction rates should be considered estimates of the true population satisfaction rates; (ii) survey responses could contain various types of response biases and therefore may not accurately capture true levels of client satisfaction; and (iii) a sizeable proportion (12 percent) of responses were neutral and therefore did not contain useful information. For further details, see appendix E. Hypothesis: Learning during implementation, especially client relation- ships, is disrupted by TTL turnover. The evaluation team conducted an analysis of the turnover of TTLs in financing projects approved during the evaluation period. This was done to understand to what extent there was possible disruption to knowledge continuity in projects and to what extent this was mitigated through co-TTL-ship arrangements. Necessary informa- tion on project TTL-ship was available for a random, representative sample of projects approved during the evaluation period, which covered 75 percent of the population of projects. These data were used to first calculate a simple TTL turnover rate, which identified and counted the instances of an individ- ual not being the TTL for a project through its entire life (Activity Initiation Summary sign-off to closing), and then, a co-TTL-ship–adjusted turnover rate, which included only instances where there was TTL turnover in the ab- sence of an overlap with a co-TTL. These two rates were examined over time, across Global Practices, lending instruments, and by fragility, conflict, and violence status of countries. This analysis had the following key limitations: (i) necessary information was available for a random sample of projects because of which the turnover rates should be considered estimates of the true population turnover rates, with a slight bias toward more recent years, and (ii) data on TTL-ship captured by the operations data systems were used, which might not accurately reflect realities of the project, such as informal handovers. However, because much of these informal transactions between TTLs cannot be accurately measured, system data were used as a proxy. For further details, see appendix F. Hypothesis: Peer reviewers, if selected strategically and engaged timely and consistently, can be great sources of knowledge for projects. The evaluation team conducted an analysis of the data on peer reviewers who Learning in World Bank Lending Appendix A provided advice to projects to understand (i) trends in the size of the set of peer reviewers over the evaluation period, (ii) consistency of peer reviewers providing advice within projects, and (iii) systematization of the process of peer reviewer selection. A sample of 3,972 observations (each observation was a system record of advice provided) from 1,011 projects was analyzed, representing 29 percent of the population of projects. Only those projects for which dates related to concept review, decision meeting, and quality enhancement review meetings were available were included in the sample because it was necessary to understand the patterns of use of peer reviewers across these three types of meetings. It was found that almost all develop- ment policy financing projects did not hold quality enhancement reviews; 80 therefore, all projects led by the Macroeconomics, Trade, and Investment Global Practice were excluded from the sample. Data on World Bank staff from its human resources data systems were also used to understand the third dimension. This analysis had the following key limitations: (i) a ran- dom sample was analyzed as the necessary information was not available for the rest of the population, because of which the patterns of use of peer reviewers should be considered estimates of the true population patterns, with a bias to exclude development policy financing and projects led by the Macroeconomics, Trade, and Investment Global Practice, and (ii) data on peer reviewers captured by the operations data systems were used, which might not accurately reflect realities of the project, such as external peer reviewers and those who did not enter the information in the system. For further details, see appendix G. Hypothesis: Multiphase Programmatic Approaches (MPAs) support sys- tematic learning. The evaluation conducted a desk review of all 34 learning agendas in 34 MPA projects approved from 2017 to 2024. These projects financed 24 MPA programs of which 17 were in their first phases and 7 were beyond the first phase; 5 MPAs were horizontal, or multicountry, and 19 were vertical, or single country. For further details, see appendix C. Approach for evaluation question 2: Have different operational units put effective knowledge enablers in place? Independent Evaluation Group World Bank Group    81 To identify good practices embedded by different operational units within the World Bank, the evaluation conducted interviews and rolled out a ques- tionnaire to the World Bank’s knowledge and learning professionals. World Bank staff interviews. After identifying the types of knowledge inputs that typically benefit World Bank projects, the team interviewed managers and some staff in nine Global Practices’ global units and three Regional Development Effectiveness units. A total of 22 interviews were conducted to identify practices put in place to facilitate knowledge exchange within and across teams,1 enabling factors and hinderances. These semistructured inter- views lasted 45–60 minutes each and were guided by a predefined protocol (box A.2). Detailed notes were taken for each interview.  rotocol for Interviews for Evaluation Question 2 Box A.2. P This protocol was meant to guide the evaluation team to conduct interviews with global units and regional teams: » What are the Global Practice’s or department’s knowledge and learning priorities? » What are the key knowledge enablers for operational staff? » Resources: Trust funds and past projects » Opportunities and capacities: processes, practices, global leads and Global Solutions Groups, events, ways of linking to operations, and staff motivations » Clients and partners » Does the Global Practice have major knowledge collaboration with external partners? » Does the Global Practice have major initiatives to learn with clients or build their capacity? Source: Independent Evaluation Group. The evaluation team categorized interview findings into three elements. These were extracted from the behavioral change wheel that was originally developed by Susan Michie and colleagues in 2011. The model defines three key elements shaping behavior change, including motivations, capabili- Learning in World Bank Lending Appendix A ties, and opportunity. In particular, the model helped the team understand how availability of financial resources and time to invest in “skill transfer” (capability or capacities), coupled with motivation and specific moments of opportunity for staff and counterparts, are critical to act on and use any knowledge identified. Given the importance of learning with clients in World Bank lending operations, the evaluation included this element into the mix. Questionnaire to the knowledge and learning community. The evaluation team rolled out a questionnaire to knowledge management professionals embedded in different operational units (covering all four Practice Groups, 10 Global Practices, two Global Themes, and two Regions) to understand their role in facilitating knowledge exchange. The evaluation team received 82 20 responses, which were then coded to identify patterns and best practices. Box A.3 summarizes the details of the questionnaire. In addition, the evaluation team discussed good practices for knowledge management across several multilateral development banks, including the Asian Development Bank, the Inter-American Development Bank, and the European Bank for Reconstruction and Development. The evaluation con- ducted semistructured interviews, which lasted about 60 minutes each, and reviewed a recently completed comprehensive benchmarking of knowledge management practices in multilateral development banks, large foundations, and large private sector firms done by the European Bank for Reconstruction and Development’s independent evaluation office.  nowledge and Learning Community Questionnaire Box A.3. K Knowledge and learning community questionnaire: » In your experience, what sources of knowledge do staff consult to design and implement their projects? » As a knowledge and learning professional, how are you helping staff acquire and generate knowledge for their operations? Please give one or more examples. » What processes are in place in your Region, Global Practice, or Global Theme to support the transfer of technical or sectoral knowledge (as opposed to operational Independent Evaluation Group World Bank Group    83 or contextual)? Please highlight up to three, if relevant. » What processes are in place in your Region, Global Practice, or Global Theme to support the transfer of operational knowledge (for example, knowledge of lending instruments, project life cycle, procurement policies, and so on)? » What processes are in place in your Region, Global Practice, or Global Theme to support the transfer of contextual and strategic knowledge (for example, Country Partnership Framework, client country political economy, and corporate priorities)? » Are you particularly proud of one particular product, process, innovation, or simple improvement you produced for your Global Practice, Region, or cross-cutting Global Theme? If so, please briefly share. (continued)  nowledge and Learning Community Questionnaire (cont.) Box A.3. K » Beyond the transfer of codified knowledge (for example, through the formal proj- ect review cycle, training, templates, reports, guidelines, and so on), do you feel that tacit knowledge and informal knowledge sharing are valued? » Does your Global Practice or Region have major knowledge collaboration with external partners? » Please list your most important partners and describe in one sentence the focus of your partnership with them. » Is your Region, Global Practice, or cross-cutting Global Theme engaged in client learning activities? » What type of client learning activity is your Region, Global Practice, or Global Theme engaged in? Briefly describe an example. » Does your Global Practice or Region have any rewards or recognition mechanisms to incentivize staff to generate and apply knowledge in operations? » What types of rewards or recognition mechanisms are provided? » What other activities, if any, are you leading or supporting?  » Some business units have modified the standard World Bank review and project cycle processes—for example, by holding internal premeetings, safe space meet- ings, clinics, or meetings around Mid-Term Review. What are examples of such Learning in World Bank Lending Appendix A modifications? In your opinion, are they worth scaling up?  » Please use this space to share any additional information, comments, and thoughts on the topic of learning in lending.  Source: Independent Evaluation Group. 84 Approach for evaluation question 3: What would it take to have more consistent learning across World Bank financing operations? The evaluation team triangulated evidence for evaluation questions 1 and 2 to identify potential areas for improvement within the formal linear process and best practices beyond it that could be replicated and scaled up. The team conducted several workshops with different groups of audiences to test if emerging findings and recommendations resonated with them. Each of these workshops was tailored for the target audience: (i) Knowledge Compact team, (ii) practice managers, (iii) TTLs, (iv) the knowledge and learning com- munity, and (v) advisers. Practice managers and TTLs were selected based on interactions during evaluation question 2 and evaluation question 1, re- spectively. In particular, those who went above and beyond the formal linear model within their project teams and units were included to understand what it would take to replicate and scale. Notes from these workshops were documented and helped the evaluation team to fine-tune the messages and ground the recommendations. Reference Michie, Susan, Maartje M. van Stralen, and Robert West. 2011. “The Behaviour Change Wheel: A New Method for Characterising and Designing Behaviour Change Interventions.” Implementation Science 6 (April): 42. Independent Evaluation Group World Bank Group    85 1 In some interviews, practice managers involved several of their team members.  Learning in World Bank Lending Appendix A 86 Appendix B. Past Independent Evaluation Group Evaluations and Research Past Independent Evaluation Group (IEG) evaluations have covered dif- ferent aspects of knowledge. IEG’s two previous evaluations of learning in World Bank operations found that for most staff, informal learning and tacit accumulation of knowledge predominate and are driven by incentives, mindsets, and aspects of organizational culture, such as group norms and diversity of teams (World Bank 2014, 2015). IEG’s Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group found limited use of evidence from the self-evaluation systems for learning and adaptive man- agement and stressed the role of staff values, motivations, and incentives regarding results measurement and using self-evaluation (World Bank 2016). IEG’s evaluation Knowledge Flow and Collaboration Under the World Bank’s New Operating Model examined the relationship between knowledge and the operating model, highlighting that although some Global Practices had co- herent and systematic approaches to managing and investing in knowledge, others did not, and that differences in managerial signals and incentives could explain much of the observed variation (World Bank 2019a). Enhancing the Effectiveness of the World Bank’s Global Footprint found that locating staff in country offices improves client relationships and brings other benefits but poses challenges to a global knowledge flow because many of the World Bank’s knowledge processes center on headquarters (World Bank 2022a). IEG’s evaluations Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity, World Bank Group Engagement in Upper- Middle-Income Countries: Evidence from IEG Evaluations, and The World’s Bank: An Evaluation of the World Bank Group’s Global Convening confirmed that data and knowledge are core sources of comparative advantage (World Bank 2017a, 2017b, 2020c). Several evaluations, including on disaster risk management, found that the World Bank is strong on producing strategic knowledge that motivates clients to reform.     87 Past IEG evaluations have often found that how World Bank pursues its knowledge work can be the most important determinant of results. The World’s Bank: An Evaluation of the World Bank Group’s Global Convening re- viewed knowledge-heavy convening activities across sectors and found that more purposeful selectivity and tighter management of the World Bank’s engagements improve results (World Bank 2020c). IEG’s Knowledge-Based Country Programs: An Evaluation of World Bank Group Experience found the following: In the sample of countries, the Bank Group was more effective when it worked on specific sectors rather than broad topics, designed tasks to address specific client concerns, customized international best practice to local conditions, generated data to support policy making, and formulated actionable recommendations that fit local adminis- trative and political economy constraints. The Bank Group was less effective when it did not address issues relevant to the client or was unable to follow up consistently with the client on the implementa- tion of advisory activities. Regardless of the level of government that operated as counterpart (central or local), client participation and good monitoring and evaluation systems were key to good results. (World Bank 2013, ix) Research and evaluations have found that projects informed by knowledge inputs—both explicit and tacit—tend to perform better. Econometric studies using IEG project ratings have linked the quality and stability of the project’s Learning in World Bank Lending Appendix B task team leader to project performance (Denizer et al. 2013; Geli et al. 2014; Moll et al. 2015). Ashton et al. (2023) reported that project design, including the estimated value added of design staff and the presence of prior analytic work, predicts project success. A key determinant of staff’s contribution is their experience with previous World Bank projects, indicating the contri- bution that knowledgeable staff make. Foundational knowledge work during project preparation matters for quality at entry because it not only enhances the World Bank’s understanding of local policy, capacity, and institutions but also allows it to build trusting relationships and fine-tune procurement arrangements (World Bank 2019b). Time pressures during preparation have a statistical association with projects’ quality at entry, presumably because of 88 less time to invest in knowledge work (World Bank 2020b). ndependent Evaluation Group Evaluations on Knowledge and Relevant Findings Table B.1. I Title Year Scope Relevant Findings Learning and Results in 2014 and These reports covered World Bank staff often rely on informal learning and gradual accumulation World Bank Operations: 2015 learning that takes place of tacit knowledge. Such learning and knowledge are based on observing How the Bank Learns through World Bank and copying the behavior of others in the group. They depend on mindsets, projects. group effects, and institutional incentives. Staff value mentoring and learning Learning and Results in from peers. Therefore, the World Bank should focus on making better use of World Bank Operations: informal learning and tacit knowledge, and an updated strategy for learning Toward a New Learning and knowledge sharing might be helpful in this regard. Strategy Behind the Mirror: A 2016 The report covered the The mandatory self-evaluation systems are seldom used for organizational Report on the Self- World Bank Group’s self- learning. ICRs are seen as not useful and provide only generic lessons. Staff Evaluation Systems of evaluation instruments, operational knowledge often comes from tacit sources (which is insufficient the World Bank Group including ICRs. because weaknesses in documenting lessons and overreliance on personal connections can lead to loss of important knowledge). Self-evaluation systems do not exploit dialogue and tacit knowledge formats to foster operational learning. Self-evaluation systems would benefit by being more flexible and geared toward socializing learning. World Bank Group 2017 This synthesis report The Bank Group’s analytic and advisory work has been key in supporting Engagement in covered the outcomes reforms in upper-middle-income countries and valued by country Upper-Middle-Income and lessons from the stakeholders. The analytic and advisory work also shapes the quality of Countries: Evidence from Bank Group’s work in the World Bank’s assistance, particularly in quality at entry of development IEG Evaluations upper-middle-income policy financing and during crises. However, there is little assessment of countries. the outcomes of knowledge services, and the potential for South-South knowledge exchange has been underused. (continued) Independent Evaluation Group World Bank Group    89 90 Learning in World Bank Lending Appendix B Title Year Scope Relevant Findings Knowledge Flow and 2019 The evaluation Some GPs have coherent and systematic approaches to managing and Collaboration Under covered how well the investing in knowledge, and others less so. Some GPs focus on learning by the World Bank’s New World Bank’s post- doing and tacit knowledge flow to support operations but do not emphasize Operating Model 2014 operating model generating or curating knowledge, innovation, client training, or global stimulates knowledge thought leadership. Differences in GPs’ attention to knowledge often reflect flow and enhances the availability of trust funds and leadership support. Contestability in quality collaboration to assurance is uneven. The mechanisms designed to pursue knowledge deliver multisector and excellence have been met with mixed results. Global leads have unclear roles multiservice tasks to and unfunded mandates. A few GPs made their Global Solutions Groups work clients. largely as intended, whereas others recast or disbanded the model. The World Bank should focus more on incentives, culture, and collaboration mechanisms than on structure. Incentives to enhance knowledge flow could include senior management support for knowledge excellence; metrics for knowledge uptake, impact, quality, and influence; more contestability in quality assurance; and nimbler budgeting arrangements. The World’s Bank: An 2020 The evaluation covered The Bank Group’s knowledge is key to its global role. The Bank Group’s Evaluation of the World which global issues the convening power as an independent generator and broker of global Bank Group’s Global Bank Group convenes on, knowledge allows it to inform policy makers and take a lead role in setting the Convening the factors that drive its agenda for global discussions on development. convening choices, and the determinants of its convening effectiveness. (continued) Title Year Scope Relevant Findings Enhancing the 2022 Among other things, this An expected benefit of decentralization is that it helps integrate local Effectiveness of the evaluation covered how knowledge into the World Bank’s global knowledge network and informs World Bank’s Global staff decentralization World Bank strategies and operation, but decentralization also poses Footprint affects the World Bank’s challenges to a global knowledge flow when staff members are away knowledge flow. from headquarters for extended periods. Knowledge management is often headquarters focused, and formal knowledge from the field is less appreciated and used globally. The evaluation recommended that the World Bank take measures to safeguard knowledge flow and the World Bank’s global nature. Results and 2022 Among other things, There is a good match between ASA topics and government policies, and the Performance of the the report presents a World Bank often uses ASA in its policy dialogue. However, there is limited World Bank Group 2022 qualitative analysis of evidence on governments’ ownership and use of ASA findings and on the the use of ASA based on use of ASA in World Bank programs and projects, and what evidence there is Completion and Learning shows mixed effectiveness (for example, because of issues with ASA’s timing Review Validationsa for 50 and dissemination). countries. Sources: Independent Evaluation Group; World Bank 2014, 2015, 2016, 2017b, 2019a, 2020c, 2022a, 2022b. Note: ASA = advisory services and analytics; GP = Global Practice; ICR = Implementation Completion and Results Report; IEG = Independent Evaluation Group. a. The Completion and Learning Review Validation was called Completion and Learning Review Review before May 1, 2023. No change was made to the methodology. Independent Evaluation Group World Bank Group    91 Past Independent Evaluation Group Recommendations on Knowledge Knowledge Flow and Collaboration (FY19) » The World Bank should strengthen its approach to knowledge in the Global Practices and Global Themes with clear goals, roles, and mechanisms; bud- gets commensurate with mandates; and metrics for knowledge uptake, quality, and influence. » It should improve budgeting systems to incentivize knowledge flow and collaboration. » It should link better the Global Practices and Regions to improve coordina- tion and enhance responsiveness to clients. » The World Bank should make better use of program leaders for cross-sectoral collaboration, integrated solutions, and complex client dialogue. » It should improve its quality assurance arrangements for both advisory and financing services. » The World Bank should ensure ongoing monitoring of its operating model to enhance its ability to attain its knowledge flow and collaboration goals. Report on Self-Evaluation Systems (FY16) » The World Bank should strengthen the quality of monitoring and evaluation Learning in World Bank Lending Appendix B (M&E) through revising the Implementation Completion and Results Report. » It should strengthen the quality of an intervention’s indicators. » It should strengthen the quality of M&E through better incentives and sig- naling, including for self-evaluation. » The World Bank should strengthen the quality of M&E across instruments, including for Program-for-Results and Country Partnership Frameworks. » It should expand voluntary evaluations that respond to learning needs of management and teams. 92 Learning and Results (FY15) » The World Bank should develop an updated strategy for learning and knowl- edge sharing, with clearly defined institutional accountabilities. » It should make optimal use of informal learning and tacit knowledge. » It should adjust institutional incentives to promote learning and develop- ment outcomes. » It should balance the focus on global and local knowledge. » It should promote adaptiveness in project design and implementation. Convening (FY20) » Scope engagements and contributions to major global convening initiatives more deliberatively. » Enhance how the World Bank’s and the International Finance Corporation’s internal systems and processes support managing major convening initia- tives over their life cycle. » Improve links between the World Bank’s global and country work. Global Footprint (FY22): Recommendation 2 Pertained to Knowledge » The World Bank should mitigate the risks to knowledge flow brought about Independent Evaluation Group World Bank Group    93 by decentralization and put in place safeguards to avoid developing country and regional silos. » The World Bank could tailor its knowledge management mechanisms better to field staff’s needs and ensure that knowledge produced in the field flows to other field locations and to headquarters. Improving the mechanisms for curating and sharing of knowledge produced in the field and investing in virtual and in-person channels for networking and knowledge sharing would facilitate this process. The headquarters-focused knowledge manage- ment approach might also need revisiting. » The World Bank should continue to promote staff mobility by rotating inter- nationally recruited staff between headquarters and the field and increasing cross-support opportunities for locally recruited staff. These efforts would enhance knowledge flow and ease the risk of the World Bank developing country and regional silos. Recommendation Follow-Up Strong follow-up has occurred for some past IEG evaluations’ findings and recommendations on knowledge. The Strategic Framework for Knowledge diagnostics built on IEG findings. Management has revised operational procedures to facilitate midcourse correction and restructuring, reformed aspects of the Implementation Completion and Results Report, introduced mandatory statements of operation’s theory of change, improved project M&E, and, in 2019 and 2020, realigned staff and adjusted reporting lines. Follow-up to past IEG evaluations’ findings and recommendations on cul- ture and incentives for knowledge has been more uneven. Recommendations on culture, incentives, M&E, and some other decentralized functions can be difficult for management to act on, in part because there is no clear locus for implementation and because changing organizational culture is genuine- ly difficult. As IEG has noted, “M&E quality and use, and the Bank Group’s broader outcome orientation, need continued attention. Recent and ongoing efforts in the World Bank…to renew outcome orientation and improve staff M&E tools and knowledge are useful but could be complemented with broad- Learning in World Bank Lending Appendix B er changes in capacity, approaches, and incentives” (World Bank 2020a, 20). It has proven hard to create useful performance measures of knowledge. The World Bank has a metric of knowledge flow based on staff time charged to cross-support, which usefully measures the flow of cross-support but does not get at the quality and influence of the knowledge itself. The citation metrics widely used in academia are not relevant for operational knowledge. IEG has recommended, and the Strategic Framework for Knowledge dis- cussed, improved measurement of the quality, relevance, and outcomes of knowledge (World Bank 2019a). The lack of metrics makes knowledge hard to manage. However, even if relevant measures could be constructed, the ques- tion of their usefulness remains, given the potential for misuse and displaced 94 incentives when indicators are used for oversight purposes. References Ashton, Louise, Jed Friedman, Diana Goldemberg, et al. 2023. “A Puzzle with Missing Pieces: Explaining the Effectiveness of World Bank Development Projects.” The World Bank Research Observer 38 (1): 115–46. https://doi.org/10.1093/wbro/lkac005. Denizer, Cevdet, Daniel Kaufmann, and Aart Kraay. 2013. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” Journal of Development Economics 105 (November): 288–302. https://doi. org/10.1016/j.jdeveco.2013.06.003. Geli, Patricia, Aart Kraay, and Hoveida Nobakht. 2014. “Predicting World Bank Project Outcome Ratings.” Policy Research Working Paper 7001, World Bank. Moll, Peter, Patricia Geli, and Pablo Saavedra. 2015. “Correlates of Success in World Bank Development Policy Lending.” Policy Research Working Paper 7181, World Bank. World Bank. 2013. Knowledge-Based Country Programs: An Evaluation of World Bank Group Experience. Independent Evaluation Group. World Bank. World Bank. 2014. Learning and Results in World Bank Operations: How the Bank Learns. Independent Evaluation Group. World Bank. World Bank. 2015. Learning and Results in World Bank Operations: Toward a New Learning Strategy. Independent Evaluation Group. World Bank. World Bank. 2016. Behind the Mirror: A Report on the Self-Evaluation Systems of the Independent Evaluation Group World Bank Group    95 World Bank Group. Independent Evaluation Group. World Bank. World Bank. 2017a. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Independent Evaluation Group. World Bank. World Bank. 2017b. World Bank Group Engagement in Upper-Middle-Income Countries: Evidence from IEG Evaluations. Independent Evaluation Group. World Bank. World Bank. 2019a. Knowledge Flow and Collaboration Under the World Bank’s New Operating Model. Independent Evaluation Group. World Bank. World Bank. 2019b. Results and Performance of the World Bank Group 2018. Independent Evaluation Group. World Bank. https://ieg.worldbankgroup.org/ sites/default/files/Data/Evaluation/files/rap2018.pdf. World Bank. 2020a. Management Action Record Reform: IEG’s Validation Report. Independent Evaluation Group. World Bank. World Bank. 2020b. Results and Performance of the World Bank Group 2020. Independent Evaluation Group. World Bank. World Bank. 2020c. The World’s Bank: An Evaluation of the World Bank Group’s Global Convening. Independent Evaluation Group. World Bank. World Bank. 2022a. Enhancing the Effectiveness of the World Bank’s Global Footprint. Independent Evaluation Group. World Bank. World Bank. 2022b. Results and Performance of the World Bank Group 2022. Independent Evaluation Group. World Bank. Learning in World Bank Lending Appendix B 96 Appendix C. Deep Dive into Multiphase Programmatic Approach The evaluation team conducted an in-depth review of a sample of Multiphase Programmatic Approaches (MPAs) and their learning agen- das to assess how and to what extent these agendas contribute to learning and knowledge within World Bank operations. MPAs embed an intentional learning agenda designed to foster a culture of continuous learning and im- provement in World Bank operations. The following outlines the steps used in the MPA analysis. Step 1: Design and Questions Using five criteria and guidance from Operations Policy and Country Services and two additional criteria, the evaluation team formulated seven research questions to evaluate the MPA’s learning agendas: » Knowledge gaps: Did the team identify knowledge gaps in the sector and related to the project before preparing the learning agenda? » Data sources: Did the team identify data sources for learning? Does it men- tion how data will be collected and used to support learning or measure learning progress? In this context, data are broadly defined as any informa- tion, including lessons learned from pilots, technical assistance activities, surveys, or monitoring and evaluation. » Capacity: Did the team identify capacity measures to support learning? This specifically refers to capacity building for learning, not just a capacity-building component. » Adaptive learning: Did the team use adaptive learning by systematically identifying and applying learning and information to progress through differ- ent phases and continuously improve? » Partners and feedback loops: Did the team identify partners, clients, or other stakeholders who would provide feedback loops or support the learning?    97 » Outcome indicators: Are there outcome indicators to measure learning? » Types of knowledge: Did the team identify the different types of knowledge that would be learned? Step 2: Sample, Data Collection, and Analysis The sample included 24 MPAs from 10 Global Practices, which included Transport; Health, Nutrition, and Population; Education; Water; Urban, Disaster Risk Management, Resilience, and Land; Agriculture and Food; Finance, Competitiveness, and Innovation; Energy and Extractives; Social Sustainability and Inclusion; and Social Protection and Jobs. The data collec- tion and analysis involved a systematic approach to ensure consistency and comprehensiveness. The steps involved are as follows. Before data collection and codebook development. On the basis of the research questions, the team conducted an initial review of the Project Appraisal Documents and developed a codebook. This codebook included seven codes for each research question, with broad definitions encompass- ing qualifiers and exceptions for each category. The purpose was to ensure consistency in data collection and analysis. For example, the code “data for learning agenda” included qualifiers for both quantitative and qualitative data, monitoring and evaluation, and data from assessments and pilot proj- ects. Conversely, the code “capacity” had specific exclusions and was defined not just in terms of components but also in terms of learning, including learning related to procurement, regulations, and additional capacity for regional organizations. Learning in World Bank Lending Appendix C » Data collection. Once the codebook was developed, the data were collected on the learning agendas contained in the Project Appraisal Documents, and the main text of the Project Appraisal Documents was analyzed along with the learning agendas. » Coding. The presence of each code in the learning agendas was recorded as “Y” (yes) and its absence as “N” (no). » Compilation and analysis. After compiling all “Y” and “N” entries from all the learning agenda, the team analyzed the data based on the number of “Y” entries. By counting the learning agendas with “Y,” the team determined which categories were used to assess learning in the project, such as whether 98 data were being used to assess learning or whether knowledge gaps were identified before developing the learning agenda. Step 3: Validation and Triangulation The third phase involved conducting interviews to validate the findings from the learning agenda. These interviews aimed to understand how the learning agenda was developed and to identify the learning challenges faced by the team. Independent Evaluation Group World Bank Group    99 Appendix D. Analysis of References in Project Design Documents This section describes in detail the methodology and findings of an analysis of references in footnotes of financing project design documents conducted as part of the evaluation’s quantitative analysis methods. Objective The analysis aimed to understand the patterns of authorship of explicit knowledge cited in project design documents. Various types of knowledge inputs from various sources are leveraged at various moments during the design and implementation of World Bank financing operations. While other analyses in this report have examined the use of tacit knowledge, it was also necessary to understand some characteristics of the explicit knowledge used to inform financing operations. Specifically, the objective was to analyze explicit knowledge along four dimensions: (i) types of organizations gener- ating the knowledge, (ii) proportion of knowledge generated by the World Bank, (iii) types of World Bank staff generating the knowledge, and (iv) types of World Bank knowledge. Methodology A record of the explicit knowledge used during the design of projects can be Learning in World Bank Lending Appendix D generated from the list of references cited in the footnotes of project design documents (Project Appraisal Documents and program documents). Such a record can be analyzed to understand the characteristics of the cited explicit knowledge along various dimensions. To that end, an analysis of a relatively large, unbiased sample of references was carried out using the following steps. Data Collection Collection of the necessary data involved the following steps: 1. All 2,280 financing projects approved during FY 2018–23 were identified.1 100 2. Project Appraisal Documents and program documents for the identified projects were downloaded in bulk using the application programming interface for the World Bank’s external document repository.2 3. A text extraction algorithm was developed and applied to extract the foot- notes from each document. 4. An initial data cleaning algorithm was developed and applied to remove observations that were unlikely to be references. The data collection steps resulted in the extraction of over 11,081 text fragments (that is, footnotes) from 1,672 projects.3 The number of footnotes extracted per document ranged from 1 to 75 and averaged 7,4 for the full set. The distribution of the projects mapped to the text fragments with respect to some key project attributes (financing instrument type, approval fiscal year, lead Global Practice, and Region) was not found to have any significant coverage gaps and was largely in line with the distribution of the overall portfolio for the period. Data Processing Processing of the collected data entailed the following steps: 1. The text fragments database developed in the data collection stage was processed via the OpenAI Chat Completions API with the GPT-4 Turbo Independent Evaluation Group World Bank Group    101 model to label each of the instances as either being a reference to a publi- cation or data source, or as not being a reference.5 2. In the cases where multiple footnotes were included in a single text frag- ment, the model was instructed to ignore the case. 3. For the text identified as being references to publications or data sources, GPT-4 Turbo was again used to extract the list of authors for each, sepa- rated by a semicolon. The data processing steps resulted in 3,102 text fragments from 1,020 proj- ects identified as being references to publications. This sample covered 44 percent of all 2,280 projects approved during the evaluation period. The distribution of this random sample by relevant project attributes is present- ed in table D.1. It should be noted that the unit for the counts is the number of references and not the number of unique projects due to the variation in the number of references per project. There was no systematic bias present in the sample compared with the population of projects (not shown).  istribution of Count of Observations in the Sample by Table D.1. D Different Project Attributes Project Attribute Observations (no.) Project approval fiscal year 2018  136 2019  309 2020  517 2021  594 2022  873 2023  673 Total  3,102 Lead Global Practice Agriculture and Food  145 Digital Development  72 Education  241 Energy and Extractives  178 Environment, Natural Resources, and Blue Economy  193 Finance, Competitiveness, and Innovation  255 Governance  126 Learning in World Bank Lending Appendix D Health, Nutrition, and Population  328 Macroeconomics, Trade, and Investment  551 Poverty and Equity  22 Social Protection and Jobs  219 Social Sustainability and Inclusion  98 Transport  120 Urban, Disaster Risk Management, Resilience, and Land  390 Water  164 Total  3,102 World Bank Region  Latin America and the Caribbean  644 102 (continued) Project Attribute Observations (no.) Eastern and Southern Africa  640 Western and Central Africa  484 East Asia and Pacific  405 Europe and Central Asia  376 South Asia  297 Middle East and North Africa  256 Total  3,102 Financing instrument type Investment project financing 1,710 Development policy financing  1,134 Program-for-Results  258 Total  3,102 Source: Independent Evaluation Group. Data Preparation Preparation of the processed data for the analysis entailed the following steps for each of the four aspects of the explicit knowledge being analyzed. Organizations as Authors Independent Evaluation Group World Bank Group    103 1. The list of authors from the database of 3,102 references was used to extract the names of individual authors and create a set of all unique author names. 2. Each author was coded as being either a person or an organization, and those which were organizations were extracted into a separate database. 3. This resulted in a database of 2,116 references (68 percent of total) with at least one of the authors as an organization. 4. A total of 2,341 organizations were identified as authors of this list of refer- ences (because one reference can have multiple organizations as authors). 5. A unique list of 852 organization names was extracted from the above list.6 6. Based on this list of unique organization names, a typology of organizations was developed.7 This typology is presented in table D.2. As can be seen, a separate category for the World Bank Group was retained in this typology. 7. Each organization name was manually coded to one of the organization types. While a single reference could contain multiple organizations and therefore multiple organization types, a single organization could be mapped to only one organization type. The World Bank as an Author or a Publisher 1. The list of all references with the Bank Group as the author was extracted from the manually coded list of organizations as authors. 2. This resulted in a list of 1,025 references (34 percent of total) with the Bank Group as an author. 3. A simple search taxonomy consisting of the names of the various Bank Group institutions was developed.8 4. A string search was conducted to identify which of the references con- tained a match to one or more terms from the search taxonomy. 5. This was done to identify all those references where the Bank Group was a publisher and not an author. 6. This resulted in a list of 245 references (8 percent of total) with the Bank Group as the publisher and not one of the authors. 7. Thus, 1,260 references (41 percent of total) with the Bank Group as an au- thor or a publisher were identified and extracted into a searchable database. Learning in World Bank Lending Appendix D World Bank Core and Extended Core Advisory Services and Analytics 1. A simple search taxonomy consisting of names of World Bank core and extended core advisory services and analytics knowledge products was developed.9 2. The database of all references authored or published by the Bank Group was used to search for the terms from the search taxonomy. 3. This resulted in a list of 66 references (5 percent of the total Bank Group references and 2 percent of the total references in the sample) mapped to 104 at least one of the terms. Of these, 55 and 11 were core and extended core advisory services and analytics, respectively. 4. This list was manually validated to confirm that the references actually contained references to the respective core or extended core advisory ser- vices and analytics document types. No false positives were identified. World Bank Staff as Authors 1. The list of authors from the database of 3,012 references was used to extract names of individual authors and create a set of all unique author names. 2. Each author was coded as being either a person or an organization, and those who were persons were extracted into a searchable database. 3. This resulted in a list of 994 references (32 percent of total) with at least one of the authors as an individual. 4. A unique list of 1,046 names was extracted from the above list. It should be noted that because of variation in the way names were written in the references, all these names were not in the same format, and it was not possible to convert them into the same one.10 5. A list of names of current technical staff (excluding various types of administrative, corporate, and fiduciary support roles and roles in grade levels GA–GD) in the Regional, Practice Group, and Development Independent Evaluation Group World Bank Group    105 Economics Research Vice Presidential Units was extracted from the World Bank’s human resources data systems. 6. The list of World Bank staff names was matched with each name in the list of author names, and exact matches were extracted. 7. To minimize the possibility of false positives, only those references with the Bank Group as an author or a publisher were retained in this analysis. 8. This resulted in a list of 109 references (16 percent of the Bank Group ref- erences) with at least one author as a current World Bank staff member. 9. A manual review of all identified names was conducted to ensure that World Bank staff were correctly being identified as authors. Results The results of the analysis are presented in this section. Organizations as Authors Table D.2 summarizes the distribution of references by author type.  istribution of References by Authorship Table D.2. D Author Type  References (no.) World Bank Group 1,025 Individual authors 994 Client government 396 United Nations agency 284 Multilateral institution 87 Private sector 79 International Monetary Fund 79 Nongovernmental organization 77 n.a. 76 Bilateral donor 52 Multilateral development bank 46 University 22 Research organization 20 Independent Evaluation Group 8 Learning in World Bank Lending Appendix D News organization 6 Total 3,102 Source: Independent Evaluation Group. Note: n.a. = no organization type could be assigned. The World Bank as an Author or a Publisher Table D.3 provides the breakdown by author type for the references authored or published by the Bank Group. 106  istribution of References Authored or Published by the Table D.3. D World Bank Group by Authorship Author Type  References (no.) World Bank Group 1,025 Individual authors 237 Client government 25 United Nations agency 20 Multilateral institution 13 Multilateral development bank 9 Independent Evaluation Group 8 International Monetary Fund 6 Private sector 6 Research organization 5 Bilateral donor 3 Nongovernmental organization 1 Total 1,271 Source: Independent Evaluation Group. World Bank Core and Extended Core Advisory Services and Analytics Independent Evaluation Group World Bank Group    107 Table D.4 provides the breakdown of references authored or published by the Bank Group by various core and extended core advisory services and analytics types.  istribution of References Authored or Published by the Table D.4. D World Bank Group by Core and Extended Core Advisory Services and Analytics ASA Type  References (no.) Core 55 Extended core 11 Total 66 Source: Independent Evaluation Group. Note: ASA = advisory services and analytics. World Bank Staff as Authors Table D.5 shows the distribution of references authored by World Bank staff by vice presidential unit type and vice presidential unit.  istribution of Observations in the Sample by Current World Table D.5. D Bank Technical Staff Vice Presidential Unit Type and Vice Presidential Unit VPU Type and VPU References (no.) Region 82 Practice Group 35 DEC 17 Grand total 109 Source: Independent Evaluation Group. Note: DEC = Development Economics Vice Presidency; VPU = vice presidential unit. Limitations The analysis was based on a random, representative sample that covered 44 percent of all projects approved during the evaluation period. The sample did not exhibit any systematic bias compared with the rest of the popula- tion along project attributes of interest, such as approval fiscal years, Global Practices, Regions, and financing instruments. Therefore, the authorship patterns should be considered unbiased estimates of the true population Learning in World Bank Lending Appendix D authorship patterns. Project Appraisal Documents sometimes do not cite all their sources for brevity, implying that the analysis may not be based on a complete inventory of all knowledge sources that project teams actually used. However, there is no reason to believe that Project Appraisal Documents are more or less likely to omit citations of core or noncore advisory services and analytics as compared with other types of references, implying that the ana- lyzed sample can be considered representative. Reference World Bank. 2024. “The World Bank Documents & Report API.” World Bank. https://documents.worldbank.org/en/publication/documents-reports/api. 108  Projects approved during FY 2014–17 were included initially but were dropped from the anal- 1  ysis after the data collection stage because of relatively low coverage of successfully extracted footnotes. The lower coverage was due to lower accuracy of the text extraction algorithm, as a result of changes in the format of the documents over time.  2  See documentation at World Bank (2024). 3  It should be noted that the initial list of 2,280 projects included those that did not have a Project Appraisal Document or a program document and therefore would be excluded from the denominator in calculating a coverage ratio, implying an actual coverage of more than 73 percent. 4  In many instances, however, sets of footnotes extracted from a single page could not be sepa- rated, and therefore the counts of footnotes are an underestimation. 5  The processing with GPT-4 Turbo was done in multiple iterations, with adequate human-in- the-loop quality assurance to ensure that the model was providing satisfactory outputs. 6  Because this list was not manually validated, the count mentioned here includes duplicates due to variations of names of the same organizations being used in different citations. 7  It contained the types shown in the first column of table D.2. 8  It contained the following search terms: ieg, ifc, independent evaluation group, international finance corporation, miga, multilateral investment guarantee agency, wb, wbg, world bank group, world bank. Independent Evaluation Group World Bank Group    109 9  It contained the following search terms: country climate and development, ccdr, Country Economic Memorandum, cem, country private sector diagnostic, cpsd, poverty assessment, pa, public expenditure review, per, agriculture sector review, asr, country environment assessment, cea, debt management performance assessment, dempa, fiduciary assessment, fa, method- ology for assessment of public procurement system, maps, public expenditure and financial accountability, pefa, financial sector assessment program, fsap, human capital review, hcr, infrastructure sector assessment program, infrasap, pandemic preparedness diagnostics, ppd, risk and resilience assessment, rra, sustainable cities review, scr. 10  While the full first name can be converted to an acronym, it is not possible to do the opposite. Appendix E. Analysis of Two-Minute Client Satisfaction Survey Data This section describes in detail the methodology and findings of an analysis of the World Bank Group’s two-minute client satisfaction survey data for financing projects conducted as part of the evaluation’s quantitative analysis methods. Objective Two-minute client satisfaction survey responses data were analyzed to understand the World Bank’s performance on questions relevant to learning in lending. A project-level client satisfaction survey (World Bank 2015) was started by the World Bank in FY 2015 after the changes to its operating model that sought to “improve global knowledge flow and technical collabo- ration to deliver more integrated, multisectoral solutions to clients” (World Bank 2019, ix). In this survey, clients are identified as “those who have the authority to approve or reject World Bank deliverables, as well as those who work with task teams on a day-to-day basis to implement activities or opera- tions. Stakeholders, partners, and beneficiaries may be engaged in an operation or activity but are not categorized as clients” (World Bank Group 2015). Specifically, the survey contained two questions related to the use of knowledge in the World Bank’s financing projects. Therefore, data on re- sponses to the survey were analyzed as part of the evaluation to understand the World Bank’s performance on these two relevant questions and to under- stand patterns in the performance based on internal administrative divisions. Methodology The survey response data were found to be representative of the underlying population of financing projects and provided a means to quantify perfor- mance as measured by client perceptions. Project-level anonymized survey response data for projects approved during FY14–23 were extracted from the World Bank’s internal corporate data systems and merged with project data. The survey is provided to clients at different stages of the financing project cycle, with two-thirds of the recorded responses provided at or before the 110 Board approval (68 percent), close to a fourth during implementation (23 percent), and a relatively small proportion at completion (8 percent). Out of 3,741 disclosed financing projects approved since FY14 (excluding addi- tional financing), 1,361 (36 percent) have at least one response to the surveys. Figures E.1 through E.3 illustrate the extent to which financing projects during the evaluation period are covered by the survey. As can be seen, there are no major temporal, geographic, or sectoral biases in terms of the coverage of the responses. The survey is composed of a set of six ques- tions to which the responses are on a five-point Likert scale. The proportions of positive, negative, and neutral responses presented in figure E.4 and tables E.1 through E.3 were calculated based on the number of positive, negative, and neutral responses, respectively, divided by the total number of responses.  emporal Coverage of Two-Minute Client Survey Responses  Figure E.1. T Sources: Independent Evaluation Group; World Bank (2023). Independent Evaluation Group World Bank Group    111 Note: Project data for FY23 are up to November 2022.  eographic Coverage of Two-Minute Client Survey Responses Figure E.2. G Sources: Independent Evaluation Group; World Bank (2023). Note: Project data for FY23 are up to November 2022.  ectoral Coverage of Two-Minute Client Survey Responses Figure E.3. S Sources: Independent Evaluation Group; World Bank (2023). Note: Project data for FY23 are up to November 2022. Results Survey results indicate that clients’ perceptions related to the World Bank’s knowledge in financing projects were the least favorable, although they were highly favorable overall. Table E.1 provides the proportion of positive, negative, and neutral responses to each of the questions for the set of 1,361 projects. The second and third questions are directly related to the scope of this evaluation. As can be seen from the table, clients’ perceptions regarding projects being tailored to country context were the least agreeable, followed Learning in World Bank Lending Appendix E by the question on timeliness and the deployment of global expertise. These are the two questions that are analyzed in more detail in this appendix. 112  wo-Minute Client Survey Responses for Projects Table E.1. T Projects with Positive Negative Neutral Responses Survey  (%) (%) (%) (no.) The World Bank worked well 94.9 1.6 3.5 1,381 with us during this activity  The World Bank brought global 88.6 1.6 9.8 1,378 expertise to support this activity  The World Bank’s expertise was 82.1 3.4 14.4 1,374 tailored to my country context  The World Bank provided 88.0 3.1 8.9 1,382 support at the right time  I would work with the World 94.7 1.3 4.0 1,376 Bank again for this type of activity  Overall, this project or activity 93.8 1.2 5.1 1,379 is likely to achieve its intended development outcomes  Sources: Independent Evaluation Group; World Bank (2023). Note: Project data for FY23 are up to November 2022. There was a decline in the proportion of positive perceptions related to the two relevant questions over time, with variation in the performance of various Practice Groups and Global Practices and Regions. Figure E.4 and tables E.2 Independent Evaluation Group World Bank Group    113 and E.3 provide the breakdowns over time, by Practice Group and Global Practice and by Region, respectively, for the two relevant survey questions.1 Figure E.4 shows that the proportion of positive responses to both relevant questions has declined since FY20.  ositive Responses for Relevant Questions Over Time  Figure E.4. P Sources: Independent Evaluation Group; World Bank (2023). Note: Project data for FY23 are up to November 2022. Dotted lines depict corresponding polynomial regression lines; FY23 is excluded due to small sample size of responses for each question. It can be seen from table E.2 that Health, Nutrition, and Population and Social Sustainability and Inclusion, respectively, had the lowest and second-lowest proportions of positive responses to both survey questions. Agriculture and Food and Poverty and Equity, respectively, had the highest and second-highest levels of positive responses to the question on global expertise, while Social Protection and Jobs and Agriculture and Food, respectively, had the highest and second-highest proportions for the question related to country contextualization. Learning in World Bank Lending Appendix E 114  ectoral Distribution of Two-Minute Client Survey Responses Table E.2. S Projects with Practice Group and Positive Negative Neutral Responses Survey Global Practice (%) (%) (%) (no.) The World Prosperity 87.0 1.4 11.6 307 Bank brought Finance, 85.9 2.2 12.0 101 global Competitiveness, and expertise to Innovation support this activity  Governance 85.9 0.0 14.1 87 Macroeconomics, 87.0 2.6 10.4 89 Trade, and Investment Poverty and Equity 94.5 0.0 5.5 28 Trade and n.a. n.a. n.a. 2 Competitiveness People 87.1 1.7 11.2 343 Education 90.3 1.3 8.4 96 Health, Nutrition, and 81.2 2.9 15.9 93 Population Social Protection and 91.1 0.6 8.3 101 Jobs Social Sustainability 82.4 2.7 14.9 53 and Inclusion Infrastructure 86.9 2.8 10.3 265 Independent Evaluation Group World Bank Group    115 Digital Development n.a. n.a. n.a. 28 Energy and 87.0 4.0 9.0 122 Extractives Transport 87.0 2.4 10.7 115 Planet 91.1 1.1 7.8 465 Agriculture and Food 94.7 0.0 5.3 116 Environment, Natural 89.6 2.6 7.8 107 Resources, and Blue Economy Urban, Disaster 89.9 1.4 8.7 148 Risk Management, Resilience, and Land Water 90.8 0.5 8.7 94 (continued) Projects with Practice Group and Positive Negative Neutral Responses Survey Global Practice (%) (%) (%) (no.) The World Prosperity 81.8 1.8 16.4 307 Bank’s Finance, 79.8 2.2 18.0 101 expertise was Competitiveness, and tailored to Innovation my country context  Governance 81.8 0.0 18.2 87 Macroeconomics, 83.8 3.4 12.8 89 Trade, and Investment Poverty and Equity 83.0 1.9 15.1 28 Trade and n.a. n.a. n.a. 2 Competitiveness People 82.7 3.9 13.3 343 Education 84.6 1.9 13.5 96 Health, Nutrition, and 77.8 8.9 13.3 93 Population Social Protection and 86.8 2.4 10.8 101 Jobs Social Sustainability 78.4 2.7 18.9 53 and Inclusion Infrastructure 79.5 5.6 14.9 265 Digital Development n.a. n.a. n.a. 28 Energy and 80.6 2.9 16.6 122 Extractives Learning in World Bank Lending Appendix E Transport 79.2 8.3 12.5 115 Planet 83.4 3.1 13.5 465 Agriculture and Food 85.5 0.5 14.0 116 Environment, Natural 81.8 5.2 13.0 107 Resources, and Blue Economy Urban, Disaster 84.2 2.8 13.0 148 Risk Management, Resilience, and Land Water 81.5 4.3 14.1 94 Sources: Independent Evaluation Group; World Bank (2023). 116 Note: Project data for FY23 are up to November 2022; n.a. = not applicable due to the sample size less than 30. In terms of the geographic distribution of survey responses, it can be seen from table E.3 that the World Bank’s South Asia Region had relatively low proportions of positive responses to both survey questions, and Eastern and Southern Africa had the lowest proportion of clients with positive perceptions of country contextualization of knowledge in World Bank financing projects. Western and Central Africa had the highest proportion of positive responses to the question on global expertise, and the Middle East and North Africa Region had the highest proportion for the question on country contextualization.  eographic Distribution of Two-Minute Client Survey Responses Table E.3. G Projects with Positive Negative Neutral Responses Survey Region (%) (%) (%) (no.) The World East Asia and Pacific 90.1 2.1 7.8 385 Bank brought Eastern and Southern 88.0 1.6 10.4 434 global Africa expertise to support this Europe and Central 89.3 2.4 8.3 253 activity  Asia Latin America and the 88.8 1.5 9.6 260 Caribbean Middle East and 85.1 1.2 13.7 168 North Africa South Asia 82.0 2.2 15.8 278 Independent Evaluation Group World Bank Group    117 Western and Central 92.9 0.7 6.4 451 Africa The World East Asia and Pacific 83.3 2.3 14.3 384 Bank’s Eastern and Southern 78.2 4.9 16.9 432 expertise was Africa tailored to my country Europe and Central 85.3 2.4 12.3 252 context  Asia Latin America and the 80.1 4.6 15.3 261 Caribbean Middle East and 86.4 2.4 11.2 169 North Africa South Asia 80.4 3.6 16.0 275 Western and Central 84.3 2.9 12.9 451 Africa Sources: Independent Evaluation Group; World Bank (2023). Note: Project data for FY23 are up to November 2022. Limitations The analysis had the following key limitations: » A random, representative sample and not the whole population of projects was analyzed. As mentioned, the survey responses were not available for all projects approved during the evaluation period, but rather for a random, unbiased sample that was 36 percent of the population. The sample did not exhibit any systematic bias compared with the rest of the population along project attributes of interest, such as approval fiscal years, Global Practices, Regions, and financing instruments. Therefore, the satisfaction rates should be considered unbiased estimates of the true population satisfaction rates. » The survey responses likely suffered from response biases. Similar to other satisfaction surveys, responses to the survey questions were likely to suffer from different forms of response bias and might not accurately reflect World Bank performance on the (or the clients’) satisfaction from it. Many neutral responses did not provide useful information, and the analysis therefore relied on the proportion of positive responses as a measure of the level of satisfaction rather than considering both the proportion of positive and negative responses. References World Bank. 2015. “World Bank Satisfaction Survey—Client Feedback on Our Performance.” World Bank. https://www.worldbank.org/en/news/feature/2015/05/11/ Learning in World Bank Lending Appendix E world-bank-satisfaction-survey. World Bank. 2019. Knowledge Flow and Collaboration Under the World Bank’s New Operating Model. Independent Evaluation Group. World Bank. World Bank Group. 2015. “Client Identification and Validation Process.” World Bank Group. 118 1 Country-level aggregates are not analyzed due to the small sample sizes (n < 30) for certain countries.  Independent Evaluation Group World Bank Group    119 Appendix F. Analysis of Project Task Team Leader Turnover This section describes in detail the methodology and findings of an analysis of task team leader (TTL) turnover in financing projects conducted as part of the evaluation’s quantitative analysis methods. Objective This analysis was conducted to measure the impact of turnover in TTL-ship and its mitigation by co-TTL-ship arrangements on the continuity of knowledge— especially tacit—in World Bank financing projects. To do so, it was necessary first to develop indicators to identify and quantify the instances of TTL turnover in projects. Because the focus of the inquiry was on the continuity of knowledge, it was also necessary to develop a measure for the degree to which knowledge handover took place to mitigate the effects of turnover. One observable proxy for knowledge handover is co-TTL-ship, so that outgo- ing and incoming or existing TTLs can have time to systematically exchange tacit and explicit knowledge. While there are many ways in which the trans- fer of knowledge from one TTL to another can take place, it was decided to focus on co-TTL-ship as an observable proxy for the same. This information is available as structured data on project team composition captured by the World Bank’s operations data systems. Furthermore, while there can be various factors exogenous to the direct management of financing projects— such as human resources policies or individual circumstances of staff—that affect TTL turnover, a key determinant of co-TTL-ship arrangements is expected to be the management “policies” at the Global Practice (GP) level. Therefore, co-TTL-ship arrangements are expected to be within operations management control even though TTL turnover might not be. Methodology To develop measures of TTL turnover and the use of co-TTL-ship to man- age the same, panel data on project team composition from the Operations Workspace were extracted for all projects approved during the evaluation 120 period. These data were used to first identify all the instances of TTL turn- over, defined as those where an individual who was the TTL for a project did not remain in that role from project initiation to completion.1 Then, for these instances of TTL turnover, it was noted whether the outgoing TTL had an overlap with another TTL (that is, whether there was co-TTL-ship) before their exit from the role. If there was an overlap, then that instance was not counted as turnover in the co-TTL-ship–adjusted TTL turnover indicator. Thus, a simple and an overlap-adjusted indicator of TTL turnover was devel- oped for each project in the portfolio. Because there can be multiple instances of turnover in a project, both indicators were normalized and aggregated at the project level (turnover per year) and then further aggregated by project attributes, such as project approval fiscal year, lead GP managing the project, among others. The details of the process are provided below. The work was done in Microsoft Excel. Data Collection Data containing information on project team composition were extracted from the World Bank’s data portal Data Explorer for all projects approved during the period July 1, 2013, to November 22, 2023. This panel data set included one ob- servation for each project–individual–role–duration combination.2 The data set included information on project identifier, unique personnel identifier, role type, and the start and end dates for each observation. Data Cleaning Independent Evaluation Group World Bank Group    121 The database was cleaned to retain only observations that met the following conditions: 1. The project was not additional financing. 2. The project had both Activity Initiation Summary (AIS) sign-off and closing dates available.3 3. The project was mapped to a lead GP. 4. The “role type” was “team lead.” 5. The duration between the start and end dates was at least five business days. 6. The start date of the earliest TTL for a project was not after the AIS date. 7. The start and end dates were not before the project AIS date. 8. The start date was before the project closing date.4 Coverage The data set contained 13,874 observations, or instances of TTL-ship, from 2,785 projects after the data cleaning steps. This sample represented 75 percent of the population of 3,702 nonadditional financing projects approved during the evaluation period. Figure F.1 shows the proportion of the population covered by the sample over time. It can be seen from the figure that the data coverage for FY 2014–16 was quite low and improved thereafter. This is in line with the expectation that the quality of the project team data captured by the World Bank Operations Workspace improved over time, starting from a relatively lower base.5  emporal Coverage of the Universe by the Subset Figure F.1. T Learning in World Bank Lending Appendix F Source: Independent Evaluation Group. This low coverage of the population for earlier fiscal years is also reflected in the data across most GPs, with coverage improving in later fiscal years, as shown in figure F.2. 122  overage of the Universe by the Subset for Global Practices Figure F.2. C Across Two Time Periods Source: Independent Evaluation Group. Note: GP = Global Practice. Thus, it can be concluded that while the sample underrepresents projects approved during FY14–17 overall and across GPs, with some variation therein, the subset represents well projects approved during FY18–23. Data Preparation After the data cleaning steps, the following variables were calculated to assist Independent Evaluation Group World Bank Group    123 in the computation of turnover indicators: 1. A binary indicator to identify contiguous observations with TTL-ship by the same individual in the same project that had been split into multiple observations in the system database.6 This indicator aided in the identifi- cation of all the instances where the same individual’s TTL-ship ended on a particular day and then was resumed the next day. Ideally, such obser- vations should not exist in the data, but because they did, they had to be accounted for so as not to overestimate TTL turnover. 2. A binary indicator to identify overlap between TTL-ship in the same project (that is, co-TTL-ship) based on the start and end dates for each observation. Figure F.3 illustrates the different possible ways in which such overlap can occur in projects. 3. A measure of project duration in years calculated as the number of days between project AIS sign-off date and project completion date divided by 365 days for closed projects, and the number of days between project AIS sign-off date and the portfolio snapshot date (November 22, 2023) divided by 365 days for active projects.  ifferent Possible Configurations for Task Team Leadership Figure F.3. D Overlap Source: Independent Evaluation Group. Note: TTL = task team leader. Estimation Based on the data prepared using the above steps, the following two TTL turnover indicators were computed. Learning in World Bank Lending Appendix F Simple Task Team Leader Turnover Rate This binary indicator identified all the instances where a noncontiguous TTL-ship of an individual started after the project’s AIS sign-off date and ended before the project’s completion date.7 Because this indicator was computed for each observation in the data set and each project had multiple observations, a project-level sum was computed to get the count of simple TTL turnover per project. Then, this value was divided by the project dura- tion in years to arrive at the number of turnovers per year for each project. For example, if a project had 4 TTL turnovers and lasted for eight years, then it had 0.5 TTL turnovers per year. 124 Co–Task Team Leadership–Adjusted Task Team Leader Turnover Rate This binary indicator identified all the instances of simple TTL turnover where the binary co-TTL-ship indicator computed above had a value of 0. That is, this indicator identified all those cases where an individual assumed project TTL-ship after the AIS sign-off date, exited the role before proj- ect closure, and did not have a co-TTL at any time during their TTL-ship.8 Because this indicator was computed for each observation in the data set and each project had multiple observations, a project-level sum was computed to get the count of co-TTL-ship–adjusted turnover per project. Then, this value was divided by the project duration in years to arrive at the number of co- TTL-ship–adjusted turnovers per year for each project. Figure F.4 illustrates two examples of these calculations. In the project with identifier P157809, the simple TTL turnover had the value of 0.4 per year, and the co-TTL-ship– adjusted TTL turnover was 0 because TTL2 was present during all turnovers. In the project with identifier P155480, the simple and co-TTL-ship–adjusted turnover both had the value of 1.5 per year. Independent Evaluation Group World Bank Group    125  ask Team Leadership Timelines for Two Sample Projects Figure F.4. T a. P157809 b. P155480 Source: Independent Evaluation Group. Note: AIS = Activity Initiation Summary; TTL = task team leader. Learning in World Bank Lending Appendix F Aggregation The project-level rates of simple and co-TTL-ship–adjusted TTL turnover were aggregated by different project attributes (approval fiscal year, lead GP, Region, among others) by taking their simple averages across projects. Results While the rotation of TTLs in the overall sample increased over time, increasing co-TTL-ship arrangements also increased, leading to a constant adjusted turn- over rate. The trends in the two measures (simple and co-TTL-ship–adjusted 126 TTL turnover rates) for the World Bank financing portfolio over time are shown in figure F.5. It can be seen that the two measures had divergent trends over time. The unadjusted measure of TTL turnover per project per year increased steadily over time, indicating increasing rotation of TTLs across projects. Conversely, the co-TTL-ship–adjusted measure of TTL turnover per project per year declined between FY14 and FY18 and remained approximately con- stant thereafter. This indicates that while the rotation of TTLs has increased over time, potentially as a result of policies exogenous to those within the control of individual GPs, increasing co-TTL-ship arrangements kept up with this increased rotation. This is expected to have been a desirable outcome from the perspective of both knowledge flows across projects and knowledge continuation within projects.  rends in Simple and Co–Task Team Leadership–Adjusted Figure F.5. T Task Team Leader Turnover Rates Independent Evaluation Group World Bank Group    127 Source: Independent Evaluation Group. Note: TTL = task team leader. There was variation in the adjusted turnover rate across GPs, indicating that some GPs managed TTL turnover better than others. Table F.1 shows that the adjusted turnover rate ranged from a minimum of 0.05 to a maxi- mum of almost three times as much, 0.14. It can be seen from table F.1 that the Macroeconomics, Trade, and Investment and the Environment, Natural Resources, and Blue Economy GPs had the highest rates of adjusted TTL turnover, followed by the Energy and Extractives and the Poverty and Equity GPs. Conversely, the Transport and the Water GPs had the lowest adjusted turnover rates, followed by the Urban, Disaster Risk Management, Resilience, and Land and the Agriculture and Food GPs.  imple and Co–Task Team Leadership–Adjusted Task Team Table F.1. S Leader Turnover Rates Across Global Practices Average TTL Average Co-TTL-ship– Turnover per Adjusted TTL Turnover Operations Year of Project per Year of Project Lead GP (no.) Duration Duration Agriculture and Food  177 1.11 0.08 Digital Development  49 0.81 0.09 Education  205 0.84 0.11 Energy and Extractives  286 0.80 0.13 Environment, Natural 206 0.86 0.14 Resources, and Blue Economy  Finance, 188 0.86 0.09 Competitiveness, and Innovation  Governance  190 0.85 0.11 Health, Nutrition, and 265 0.90 0.10 Population  Macroeconomics, 320 0.67 0.14 Trade, and Investment  Poverty and Equity  72 0.79 0.12 Social Protection and 165 0.84 0.11 Learning in World Bank Lending Appendix F Jobs  Social Sustainability 97 0.89 0.09 and Inclusion  Transport  162 0.87 0.05 Urban, Disaster 271 0.93 0.06 Risk Management, Resilience, and Land  Water  132 0.85 0.05 Total  2,785  0.85  0.10  Source: Independent Evaluation Group (2023). Note: GP = Global Practice; TTL = task team leader. 128 Development Policy Financing Management practices in the Macroeconomics, Trade, and Investment GP likely led to a higher adjusted turnover rate for development policy financ- ing, while the use of the instrument by other GPs was not associated with higher turnover compared with investment project financing. This can be seen in table F.2, which shows the rates of turnover across lending in- struments. This could indicate that although there was a low rate of TTL turnover in development policy financing, it was not being managed with co-TTL-ship arrangements as much compared with other instruments.  imple and Co–Task Team Leadership–Adjusted Task Team Table F.2. S Leader Turnover Rates Across Lending Instruments Average TTL Average Co-TTL-ship– Turnover per Adjusted TTL Turnover Operations Year of Project per Year of Project Lending Instrument  (no.) Duration Duration DPF  441 0.72 0.12 IPF  2,179 0.87 0.10 PforR  156 1.05 0.06 Total  2,776 0.85 0.10 Source: Independent Evaluation Group (2023). Note: Nine projects did not have a lending instrument field in the database and are excluded from this Independent Evaluation Group World Bank Group    129 table, thereby reducing the total number of projects to 2,785 − 9 = 2,776. DPF = development policy financing; IPF = investment project financing; PforR = Program-for-Results; TTL = task team leader. Two-thirds of the development policy financing projects approved during the evaluation period were led by the Macroeconomics, Trade, and Investment GP (table F.3). The table also presents the comparison of the two TTL turnover rates across investment project financing and development policy financing between the Macroeconomics, Trade, and Investment GP and all other GPs.  ask Team Leader Turnover Rates Across Lending Instruments Table F.3. T and Global Practices Average Co-TTL-ship– Average TTL Adjusted TTL Turnover Turnover per Year per Year of Project Projects (no.) of Project Duration Duration Lending Other Other Instrument  MTI GP GPs MTI GP GPs MTI GP Other GPs DPF 297 144 0.67 0.82 0.13 0.10 IPF  23 2,156 0.58 0.87 0.34 0.10 Total  320 2,300 0.67 0.87 0.14 0.10 Source: Independent Evaluation Group (2023). Note: Program-for-Results projects are excluded from the above calculations. DPF = development policy financing; GP = Global Practice; IPF = investment project financing; MTI = Macroeconomics, Trade, and Investment; TTL = task team leader. The following three observations based on table F.3 indicate that the higher adjusted turnover rate in the Macroeconomics, Trade, and Investment GP might not be due to the use of the development policy financing instrument, but instead due to the management practices in the unit: 1. For both development policy financing and investment project financ- ing, the Macroeconomics, Trade, and Investment GP had lower rates of unadjusted TTL turnover, but higher rates of co-TTL-ship–adjusted TTL Learning in World Bank Lending Appendix F turnover than other GPs, indicating that TTL turnovers were not ade- quately managed with co-TTL-ship arrangements by the Macroeconomics, Trade, and Investment GP for both development policy financing and investment project financing. 2. The co-TTL-ship–adjusted TTL turnover rates for both development policy financing and investment project financing were higher for the Macroeconomics, Trade, and Investment GP compared with other GPs, in- dicating that TTL turnover was managed with co-TTL-ship arrangements less intensively in the former compared with the latter. 130 3. If the Macroeconomics, Trade, and Investment GP is excluded, then the co-TTL-ship–adjusted TTL turnover rate is the same for development policy financing and investment project financing, indicating that the use of the development policy financing instrument itself might not lead to higher co-TTL-ship–adjusted turnover rates. Fragility, Conflict, and Violence Projects in countries affected by fragility, conflict, and violence (FCV) have taken adequate measures to ensure that the higher turnover of TTLs because of staffing policies is mitigated by overlaps with co-TTLs. Table F.4 pres- ents the turnover rates for projects based on the FCV status of the countries (World Bank 2024) in which they were (or are being) implemented.  imple and Co–Task Team Leadership–Adjusted Task Team Leader Table F.4. S Turnover Rates by the Fragility, Conflict, and Violence Status Average TTL Average Co-TTL-ship– Turnover per Adjusted TTL Turnover Country FCV Status Projects Year of Project per Year of Project (FY 2014–23)  (no.) Duration Duration FCV for at least five years  576 0.92 0.11 FCV for less than five 543 0.86 0.11 years  Never FCV  1,666 0.83 0.09 Independent Evaluation Group World Bank Group    131 Total  2,785 0.85 0.10 Source: Independent Evaluation Group (2023). Note: FCV = fragility, conflict, and violence; TTL = task team leader. Two key observations can be made based on table F.4: 1. The unadjusted TTL turnover rates were higher for countries that were on the World Bank FCV list for five or more years during the evaluation period. This is to be expected because there is a higher rotation of international staff in FCV countries. 2. There was not a large difference in the co-TTL-ship–adjusted TTL turnover rates for countries based on their FCV status. Limitations The analysis had the following key limitations: 1. A random, representative sample and not the whole population of projects was analyzed. As mentioned, the information required to conduct this analysis was not available for all projects approved during the evaluation period but for a random sample that was 75 percent of the population. The sample was slightly biased toward the more recent period; however, this does not affect the utility of the sample because more recent data reflect- ed more accurate current practices than older data. Therefore, the above turnover rates should be considered estimates of the true population turn- over rates, with a slight bias toward more recent years. 2. System data might not reflect the realities of projects. It is likely that the various entries made into the Operations Workspace by the project task teams do not reflect accurately the timing of key events, such as in the cases where a co-TTL might be brought on board informally. Another example might be cases in which knowledge transfers took place infor- mally between an outgoing and an incoming TTL without there being any overlap or co-TTL-ship between the two. However, since such informal “transactions” cannot be meaningfully estimated at the portfolio level, the system data are used as a proxy measure for the projects’ realities. Reference Learning in World Bank Lending Appendix F World Bank. 2024. “Classification of Fragile and Conflict-Affected Situations.” Brief, World Bank. https://www.worldbank.org/en/topic/fragilityconflictviolence/brief/ classification-of-fragile-and-conflict-affected-situations. 132 1 That is, turnover is defined as an individual not being in the TTL role for the entire duration of a project and is identified by the individual assuming TTL-ship after project initiation or exiting TTL-ship before project completion.  2 For example, if a particular individual was the TTL for a particular project from January 1, 2014, to May 1, 2014, then it was one observation in the database. If the same individual was a safeguard adviser for the same project from June 1, 2015, to August 1, 2016, it was another observation in the data set.  3  The AIS sign-off date is the date on which a country director approves an AIS (which is pre- ceded by concurrence by a practice manager) and therefore occurs after the date on which the AIS is actually created by a TTL. Thus, some work on a project’s design is possibly carried out even before this date. 4  As might be the case for Implementation Completion and Results Report TTLs because this report is produced after project closing. 5  There have also been major technical improvements in the Operations Portal since July 2013, which have enabled better capture of project-level data, such as on the project team. 6  A binary indicator is defined in this section as an indicator whose value is equal to 1 when it satisfies some condition and is equal to 0 when it does not. 7  Thus, turnover is defined as any deviation from an ideal scenario in which the same individ- ual(s) remains the TTL(s) of a project from its initiation to its completion. Independent Evaluation Group World Bank Group    133 8  Or, conversely, cases in which an individual assumed TTL-ship of a project after AIS sign-off or exited the role before project closure are not considered instances of co-TTL-ship–adjusted TTL turnover if there was a co-TTL present at any time during the TTL-ship. Appendix G. Analysis of Project Peer Reviewers This section describes in detail the methodology and findings of an analy- sis of project technical peer reviewers conducted as part of the evaluation’s quantitative analysis methods. Objective The objective of this analysis was to understand some relevant characteristics of the process of technical peer review during the design of World Bank fi- nancing projects. While a qualitative analysis of the technical1 peer reviewers’ inputs during project preparation was studied as part of the qualitative review of a sample of projects selected as cases, this analysis focused on extracting insights from a larger sample of projects and peer reviewers. Specifically, the analysis sought to understand the following three aspects, all of which were identified as being important by some of the interviewed staff: 1. Trends in the size of the set of technical peer reviewers. The count of unique individuals providing peer review to projects was used as a mea- sure to proxy the “diversity” of peer reviewers available to provide advice to projects, with more diversity being a positive for the incorporation of relevant and diverse technical knowledge in projects. 2. Consistency of peer reviewers across and within projects. A typical financing operation is expected to have two to three meetings (concept review, decision meeting, and quality enhancement review [QER], with the last being optional) in which peer reviewers would provide advice. Ideally, the same individual should provide advice in each of these meetings, which is referred to, in this appendix, as consistency of peer reviewers and therefore their advice. 3. Systematization of the process of peer reviewer selection. Recently, a peer reviewer database (PRDB) was introduced as a centralized database of individuals identified as being experts in specific topics by the relevant practice managers. This database was expected to make the process of 134 selection more systematic and transparent. It should be noted though that the same practice managers who are responsible for selecting peer reviewers for individual projects are responsible for selecting staff to be in the PRDB. Methodology The following steps were taken to collect, process, and analyze relevant data to understand the three characteristics of the project peer review process. Data Collection 1. All 5,016 disclosed financing projects approved during the period FY 2014–23 were identified from the World Bank’s external project data repository, and relevant project attributes were systematically extracted from internal operations data platforms using an application program- ming interface. Relevant project attributes included dates for key project design meetings: concept review, decision meeting, and QER. 2. The records of peer review advice provided to the identified projects were extracted from an internal operations database. This database includes unique personnel identifiers for the person providing the peer review ad- vice, as well as the project identification number for the project to which the advice was provided, and the date on which the advice was provided. 3. A copy of the internal PRDB was extracted from a data platform. This Independent Evaluation Group World Bank Group    135 database contained the unique personnel identifiers and technical areas of specialization of all staff members identified by the relevant practice managers as being suited to provide technical advice for design of projects in their areas of specialization.2 4. Finally, data on staff composition of various Practice Group vice presiden- tial units were extracted from an internal human resources database. This database contained latest information on the unique personnel identifiers, vice presidential unit mapping, and job titles of staff. Data Processing 1. For the population of projects approved during FY14–23, it was found that during FY14–17, the availability of system data on peer reviewers’ advice was below 50 percent (it had increased from 2 percent in FY14 to 34 percent in FY17); therefore, these years were excluded from the analysis. A total of 3,479 projects were in this population of projects approved during FY18–23. 2. Of the population of 3,479 projects extracted into the portfolio database from the preceding step, a random, unbiased sample of 1,011 projects (29 percent) with system data on meeting dates for all three review meet- ings and at least one record in the peer review advice data set was retained. The decision to include QER dates as a necessary data point was made even though it reduced the sample coverage by 14 percentage points to highlight the importance of QERs as providing a useful, less formal moment to incor- porate knowledge into design of operations. See the Data Coverage section for more information on the relationship between QERs and development policy financing (DPF). 3. From the database of records of peer review advice provided to the projects, only those 3,972 entries corresponding to the 1,011 projects were retained. Furthermore, this database contained entries for advice provided by the category of “other reviewers,” which were excluded from the analysis. 4. From the human resources database, only those staff currently mapped to the four Practice Groups were retained. Finally, current job titles of staff Learning in World Bank Lending Appendix G members were used to identify those belonging to the technical GH and GG cohorts, respectively. Data Coverage Table G.1 provides the proportion of the projects in the population covered by the sample based on certain project attributes of interest—namely, approval fiscal year, lead Global Practice, World Bank Region, and financing instrument type. One notable gap in coverage is observed: because almost all DPF in the population of projects did not have QERs, the QER dates were not available for these projects and therefore were excluded from the sample. 136  istribution of Count of Observations in the Sample by Table G.1. D Different Project Attributes Projects in Population in the Project Attributes Population (no.)  Sample (%) Project approval fiscal year 2018  448 33 2019  439 37 2020  531 31 2021  580 28 2022  534 40 2023  405 41 Total  2,937 34 Lead Global Practice  Agriculture and Food  202 54 Digital Development  49 69 Education  226 47 Energy and Extractives  253 38 Environment, Natural Resources, and 196 40 Blue Economy  Finance, Competitiveness, and 182 40 Innovation  Governance  166 38 Independent Evaluation Group World Bank Group    137 Health, Nutrition, and Population  419 16 Macroeconomics, Trade, and 262 0 Investment  Other  22 0 Poverty and Equity  44 20 Social Protection and Jobs  221 25 Social Sustainability and Inclusion  104 22 Transport  168 51 Urban, Disaster Risk Management, 274 42 Resilience, and Land  Water  149 62 Total  2,937 34 (continued) Projects in Population in the Project Attributes Population (no.)  Sample (%) World Bank Region  Africaa 5 0 East Asia and Pacific  391 33 Eastern and Southern Africa  583 35 Europe and Central Asia  362 31 Latin America and the Caribbean  376 34 Middle East and North Africa  227 25 Other  33 0 South Asia  361 45 Western and Central Africa  599 36 Total  2,937 34 Financing instrument type  Development policy financing 367 0 Investment project financing 2,366 36 Program-for-Results 204 73 Total  2,937 34 Source: Independent Evaluation Group. Note: The number of projects in the sample is 1,011. a. Africa, which stood for Sub-Saharan Africa, is no longer a World Bank Region, and most projects have been remapped to either Eastern and Southern Africa or Western and Central Africa, indicating that these projects might be outliers with incorrect Region tagging. Learning in World Bank Lending Appendix G Results The findings from the analysis using the above methodology were as follows. There was an increase in the diversity of peer reviewers providing advice to projects, albeit with some variation during the evaluation period. Figure G.1 shows the trend in the number of unique individuals providing peer review per project by approval fiscal year. As can be seen from the figure, this ratio increased from 2.49 individuals per project for FY18 to 2.81 in FY23, imply- ing an improvement in the set of peer reviewers. 138  nique Peer Reviewers Providing Advice to Projects Figure G.1. U Source: Independent Evaluation Group. Note: The number of projects in the sample is 1,011.  There was low consistency in the use of peer reviewers within projects. Figure G.2 shows the distribution of the number of times the same individual provided advice to a project. In a majority of cases (81 percent), an individual provided peer review advice to a project only once,3 implying a low level of consistency.  istribution of the Number of Times Advice Was Provided by Figure G.2. D the Same Individuals for the Same Projects Independent Evaluation Group World Bank Group    139 Source: Independent Evaluation Group. Note: The number of projects in the sample is 1,011. The PRDB is a useful tool to encourage systematization of peer reviewer selection. Tables G.2 and G.3 describe the demand and supply for peer reviewers by each Practice Group and Global Practice. It can be seen from table G.2 that a majority of peer reviewers who provided advice for projects approved during FY18–23 were listed in the PRDB. However, there was variation in this ratio across Practice Groups and Global Practices. At the Practice Group level, Planet had the lowest ratio at 58 percent and People the highest at 75 percent. At the Global Practice level, the lowest ratios were for Poverty and Equity and Macroeconomics, Trade, and Investment, whereas Education, Social Protection and Jobs, and Governance had the highest ratios.  roportion of Peer Reviewers from the Peer Reviewer Table G.2. P Database by Practice Group and Global Practice Advice Advice Advice Provided Provided Projects in Provided in by Persons by Persons the Sample the Sample from PRDB from PRDB PG and GP (no.) (no.) (no.) (%) Prosperity 146 556 353 63 Finance, 73 280 167 60 Competitiveness, and Innovation Governance 63 244 174 71 Macroeconomics, 1 5 2 40 Trade, and Investment Learning in World Bank Lending Appendix G Poverty and Equity 9 27 10 37 People 231 807 602 75 Education 106 399 322 81 Health, Nutrition, and 69 221 146 66 Population Social Protection and 56 187 134 72 Jobs Infrastructure 214 890 568 64 Digital Development 34 141 90 64 Energy and 95 360 234 65 Extractives Transport 85 389 244 63 140 (continued) Advice Advice Advice Provided Provided Projects in Provided in by Persons by Persons the Sample the Sample from PRDB from PRDB PG and GP (no.) (no.) (no.) (%) Planet 420 1,719 1,002 58 Agriculture and Food 110 451 281 62 Environment, Natural 79 344 224 65 Resources, and Blue Economy Social Sustainability 23 93 47 51 and Inclusion Urban, Disaster 115 441 231 52 Risk Management, Resilience, and Land Water 93 390 219 56 Total 1,011 3,972 2,525 64 Source: Independent Evaluation Group. Note: GP = Global Practice; PG = Practice Group; PRDB = peer reviewer database. It can be seen from table G.3 that in line with more use of peer reviewers from the PRDB, People also had more of its technical GH-level staff listed in the PRDB. The latter was, in fact, likely to have been an enabling factor for the former. Similarly, Prosperity and Planet had a lower supply of peer reviewers in the PRDB to match the low use of peer reviewers from the PRDB Independent Evaluation Group World Bank Group    141 as peer reviewers for the projects led by them.  roportion of Technical Staff in the Peer Reviewer Database by Table G.3. P Practice Group and Global Practice and Grade Level Proportion of Technical Proportion of Technical VPU  GH in PRDB (%) GG in PRDB (%) Prosperity 73 54 People, or Human Development 89 59 Infrastructure 82 56 Planet, or Sustainable 79 50 Development Total  79 54 Source: Independent Evaluation Group (2023). Note: The total number of technical GH-level staff is 106 and of technical GG-level staff is 591.  PRDB = peer reviewer database; VPU = vice presidential unit. Limitations The analysis had the following key limitations: 1. A random sample and not the whole population of projects was analyzed. As mentioned, the information required to conduct this analysis was not available for all projects approved during the evaluation period but for a random sample that covered 29 percent of the population. The sample did not exhibit any systematic bias compared with the rest of the popula- tion along project attributes of interest, such as approval fiscal years and Learning in World Bank Lending Appendix G Regions. However, the sample excluded DPF and therefore projects led by the Macroeconomics, Trade, and Investment Global Practice. Thus, the above patterns related to the use of peer reviewers should be considered unbiased estimates of peer reviewer patterns of the true population, ex- cluding DPF and Macroeconomics, Trade, and Investment. 2. System data might not reflect the realities of projects. It is likely that the various entries made into the Operations Workspace by the peer review- ers do not reflect accurately or completely the instances of peer review in projects. Furthermore, the system data might not capture external peer re- viewers because they would not have access to the Operations Workspace. 142 1 That is, excluding peer reviewers providing advice on fiduciary and compliance aspects, such as financial management, safeguards, and legal.  2  That is, the peer reviewer is identified as an expert in the relevant area and presumably has some space in their work programs to provide peer review for projects. 3  This measures only how many times an individual provided advice to a project. The same individual could have provided advice to multiple projects. Independent Evaluation Group World Bank Group    143 The World Bank 1818 H Street NW Washington, DC 20433