Knowledge Flow and Collaboration under the World Bank’s New Operating Model A N I N D E PE N D E N T E VA LUAT I O N © 2019 International Bank for This work is a product of the staff of The RIGHTS AND PERMISSIONS Reconstruction and Development / The World Bank with external contributions. The material in this work is subject World Bank The findings, interpretations, and to copyright. Because The World 1818 H Street NW conclusions expressed in this work Bank encourages dissemination Washington, DC 20433 do not necessarily reflect the views of of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they noncommercial purposes as long as full represent. attribution to this work is given. Attribution—Please cite the report as: World Bank. 2019. Knowledge Flow and The World Bank does not guarantee Any queries on rights and licenses, Collaboration under the World Bank’s the accuracy of the data included in including subsidiary rights, should be New Operating Model. Independent this work. The boundaries, colors, addressed to Evaluation Group. Washington, DC: denominations, and other information World Bank Publications, The World World Bank. shown on any map in this work do not Bank Group, 1818 H Street NW, imply any judgment on the part of The Washington, DC 20433, USA; fax: World Bank concerning the legal status 202-522-2625; e-mail: pubrights@ Cover Photo: faithie / Shutterstock of any territory or the endorsement or worldbank.org. acceptance of such boundaries. Knowledge Flow and Collaboration under the World Bank’s New Operating Model AN INDEPENDENT EVALUATION April 8, 2019 Careful observation and analysis of program data and the many issues impacting program efficacy reveal what works as well as what could work better. The knowledge gleaned is valuable to all who strive to ensure that World Bank goals are met and surpassed. contents abbreviations  v acknowledgments  vi glossaryvii overview  ix management response  xviii management action record  xxi 1. Developmental Context and Evaluation Approach  1 Introduction1 Methodology3 2. The Operating Model and How It Changed  5 Results and Performance of the The Change Diagnostics The Reform Processes 6 7 World Bank Group 2015 Overview of Changes to the Operating Model 10 Reform Implementation 13 AN INDEPENDENT EVALUATION 3. Global Knowledge Flow  16 The GPs’ Approaches to Knowledge 18 Global Leads 21 Global Themes 23 Quality Assurance 24 Other Mechanisms for Knowledge Flow 25 Drivers of and Barriers to Knowledge Flow 27 4. Collaboration for Integrated Solutions  31 Integrated Solutions in Country Programs 32 Patterns of Collaboration 35 Careful observation and Mechanisms that Make the Operating Model Function and Add Value 39 data analysis of program Collaboration Inefficiencies, Boundaries, Connectors, and Incentives and the many42 issues impacting program Incentives for Maximizing Finance for Development 45 what efficacy reveals Budget Arrangements and Incentives works as well46as what could work better. The knowledge gleaned is valuable to all who strive to ensure that World Bank goals are met and surpassed. Independent Evaluation Group | World Bank Group iii 5. Conclusions and Recommendations  48 Reform Design and Implementation 48 The Operating Model 49 Recommendations51 References  53 Boxes Major Change Process Milestones  Box 2.1.  9 L and Expertise  Box 3.1.  19 E xample of Operationally Relevant Knowledge Collaboration  Box 3.2.  20  he Hub Model of Knowledge Transfer: Malaysia  Box 3.3. T 26  xamples of Global Knowledge in Country Programs  Box 3.4. E 28  latforms for Multisector Engagements in Pakistan  Box 4.1. P 34 Figures Figure 2.1. Working Groups and Change Teams in Phase 1 of the Reform Process  8 Roles and Processes within a Country Team  Figure 2.2.  11  rganizational Charts  Figure 2.3. O 12  mployee Engagement Index (percent)  Figure 2.4. E 14 Figure 3.1. Global Leads’ Records of Publication as an Indicator of Their Research Contributions  22 Trust Funds as a Percentage of Total Global Engagement Expenditures  Figure 3.2.  29 Figure 4.1. Country Partnership Framework Objectives Requiring Collaboration (number)  33  taff Cross-Support Trends  Figure 4.2. S 36 Staff Cross Support, by Global Practice  Figure 4.3.  36 Multisectoral Projects in Each Sector  Figure 4.4.  37 Collaboration across Practice Groups in the Lending Portfolio  Figure 4.5.  38  ollaboration across Practice Groups in the Advisory Services and Analytics Portfolio  Figure 4.6. C 39 Leading and Supporting Projects, by Global Practice  Figure 4.7.  40 Operations with Co-Task Team Leaders  Figure 4.8.  41 Grade Composition of Task Team Leaders  Figure 4.9.  41 Tables Table 1.1. Theory of Change  2 Table 3.1. Advisory Services and Analytics Production and Spending  17 Table 4.1. Findings from Employee Engagement Survey (percent)  44 Appendixes Appendix A. Methodological Approach  57 Appendix B. The Previous Organogram 72 iv Knowledge Flow and Collaboration | Contents abbreviations ASA Advisory Services and Analytics CD Country Director CMU country management units CoPs communities of practice CPF Country Partnership Framework GP Global Practice GL Global Lead GSG Global Solutions Group GT Global Theme IBRD International Bank for Reconstruction and Development IFC International Finance Corporation MFD Maximizing Finance for Development PL Program Leader PM Practice Manager PPP public-private partnership SCD Systematic Country Diagnostic TTL task team leader VPU vice presidential unit All dollar amounts are U.S. dollars unless otherwise indicated. Independent Evaluation Group | World Bank Group v acknowledgments This evaluation was prepared by the Independent Evaluation Group (IEG). Rasmus Heltberg led the team under the overall guidance of Caroline Heider (director-general, Evaluation) and the supervision and direction of Emanuela Di Gropello (manager) and Auguste Tano Kouame (director). The team comprised Ecaterina Canter, Anis Dani, Katsumasa Hamaguchi, Malathi S. Jayawickrama, Chris Nelson, Maurya West Meiers, Estelle Raimondo, Anjana Sriram, and Disha Zaidi. The report’s editors were Maximilian Ashwill and Amanda O’Brien. Administrative support was provided by Carla Fabiola Coles. The approach paper for this evaluation was peer reviewed by Cheryl Gray (former director, Office of Evaluation and Oversight, Inter-American Development Bank); Ajay Narayanan (manager, Internal Audit, World Bank); and Manny Jimenez (executive director, 3ie). The draft evaluation report was peer reviewed by Anil Sood (principal, Centennial Group and former World Bank staff); Sharon Benjamin Bothwell (independent evaluation consultant); and Ajay Narayanan (manager, Internal Audit, World Bank). The team gratefully acknowledges the support of the Text and Data Analytics team, Global Themes Knowledge Management. The team also thanks the many Bank Group staff, in headquarters and country offices, who participated in interviews and focus groups and generously volunteered their time. vi Knowledge Flow and Collaboration | Acknowledgments glossary Abbreviation Term Explanation ASA Advisory Services and Analytics A knowledge task with (i) an external client, (ii) a clearly defined development objective, (iii) well- articulated deliverable(s), and (iv) a time limit. CCSAs Cross-Cutting Solutions Areas See Global Themes. CD Country Director Country Management Unit leads positioned at the country level. CMU Country Management Unit A unit responsible for dialogue with the country government and for preparing and implementing the Country Partnership Framework. Each of the six Regions has several CMUs. CoP Community of Practice A group of people who share a craft, profession, or interest. CPF Country Partnership Framework A strategy to align projects and programs in a country. In 2014, the CPF replaced the Country Assistance Strategy and Country Partnership Strategy. The CPF is a four- to six-year strategy that the World Bank Group develops for a country to guide its operational activities. All projects and programs financed by the Bank Group within the time frame of this strategy must be aligned with it. DEC Development Economics The vice presidential unit that houses most research and data staff. EFI Equitable Growth, Finance, and A Global Practice Group vice presidential Institutions unit. GP Global Practice A technical department. GPs have four broad tasks: (i) to define the strategic direction and the World Bank’s activity in their respective sector, (ii) develop and deploy expertise globally, (iii) deliver integrated solutions to client countries, and (iv) capture and leverage knowledge in their respective fields. GL Global Lead A role created to connect subsector knowledge and business lines across countries and Regions. The GLs also manage the new Global Solutions Groups. GSG Global Solutions Group Knowledge communities of staff expected to help pool and curate global and country knowledge and use it to serve client needs. Independent Evaluation Group | World Bank Group vii GT Global Theme A technical department. Initially called cross-cutting solution areas, GTs are responsible for coordinating on cross-cutting corporate priority themes: Climate Change; Fragility, Conflict, and Violence; Gender; Jobs; and Public-Private Partnerships, later renamed Infrastructure, PPPs, and Guarantees. For a while, the Knowledge Management department was managed as a GT. HD Human Development One of the Global Practice Group vice presidential units. MFD Maximizing Finance for An approach to drawing on private financing and Development private sector solutions; intended to reserve public financing for those areas where private sector engagement is not optimal or available. PL Program Leader A position created to connect Country Management Units and Global Practices, and reporting to Country Directors (and with a dotted line) to a director in the Practice Groups. Program Leaders help Country Directors integrate across sectors. PMs Practice Managers Managers in the Global Pracitces. PG Practice Group A vice presidential unit that oversees several Global Practices and Global Themes. Region regional Vice Presidencies Vice presidential units that manage the World Bank’s client relationships in specific regions: Africa, East Asia and Pacific, Europe and Central Asia, Latin America and the Caribbean, Middle East and North Africa, and South Asia. SCD Systematic Country Diagnostic A synthesis study intended to inform priorities for Bank Group country engagement, specifically goals and activities with high impact, aligned with the global goals of ending extreme poverty and boosting shared prosperity in a sustainable manner. VPU Vice Presidential Unit The main organizational unit of the World Bank. viii Knowledge Flow and Collaboration | Glossary highlights 1 Reforms to the World Bank’s operating model sought, among other things, to improve global knowledge flow and technical collaboration to deliver more Overview integrated, multisectoral solutions to clients. The reforms also sought to focus country programs more selectively on tackling critical issues, better delivering on global priorities, and deepening the World Bank’s ability to deliver expertise grounded in thought leadership and global knowledge curation. 2 Establishing Global Practices (GPs) improved knowledge flow and staff mobility across Regions, mobilized expertise for clients, and sometimes deepened expertise in operationally relevant areas. 3 A gap exists between the reform’s aspirations to deepen knowledge within GPs and the current reality. Some GPs have coherent and systematic approaches to managing and investing in knowledge, others less so. The Global Solutions Groups lack mechanisms for channeling knowledge into country programs. There is much variation in how GPs use and resource the ix Global Leads and in how they perform, and the role has been changing in many GPs. Differences in leadership attention and funding explain much of the variation. 4 Global Themes provide useful directions and coherence on major corporate priority topics. They have been a valuable component of the operating model even as the quality of their operational support is hard to assess. 5 The World Bank Group can provide integrated solutions addressing clients’ important development problems because of the leadership of Country Directors, who are supported by Program Leaders. Larger country programs attract global knowledge much more easily than smaller ones. 6 The model’s structure and some of its processes tend to inhibit collaboration and cause inefficiency, fragmentation, and internal competition. Mechanisms for working across GPs and among GPs and Country Management Units are inadequate. The GPs’ incentives favor own-managed lending and lead to internal competition. x x Knowledge Flow and Collaboration | Overview Knowledge Flow and Collaboration | Section T H I S E VA LUAT I O N ’ S P U R P O S E is to assess how well the World Bank’s current operating model stimulates knowledge flow and how well it enhances collaboration to deliver “integrated solutions”—or multisector and multiservice tasks and approaches—to clients. The operating model is defined to encompass organizational structures such as Practice Groups, Global Practices (GPs), and Global Themes (GTs); internal processes, such as budgeting, quality assurance, and strategic directions; and staff roles and reporting arrangements. The evaluation also looks at the incentives and behaviors the model inculcates. Specifically, it evaluates how well the model contributes to global knowledge flow—the generation and movement of customized knowledge among staff and across geographic regions and organizational structures—and to collaboration for integrated solutions. An effective operating model that enables collaboration and use of relevant knowledge is important for delivering on the World Bank’s ambitious agendas. Recently, these agendas have been defined in the contexts of the replenishment of the International Development Association, the capital increase for the International Bank for Reconstruction and Development and the International Finance Corporation, the Maximizing Finance for Development agenda, and the “Forward Look” with its ambitions of serving all client segments, leading on global issues, and enhancing institutional efficiency. This evaluation finds that certain aspects of the operating model are valuable, especially its enabling of global knowledge flow; these are the effect of setting up GPs that operate more globally than before and GT Groups that provide useful strategic directions and coherence to cross-cutting priorities. However, the model’s structure and processes tend to inhibit collaboration and cause inefficiency, fragmentation, and internal competition. Some GPs lack coherent and systematic approaches to managing and investing in knowledge. If left unaddressed, these issues can jeopardize the World Bank’s ability to deliver for clients. The evaluation’s methods and data collection focused on the effectiveness of the World Bank’s model for delivering client services: its operating model. Guided by a theory of change, the evaluation examined the reform process and the current functioning of the operating model—specifically, its ability to transfer technical knowledge and stimulate collaboration across organizational boundaries; customize global knowledge to country contexts; integrate GTs; and deliver multisector solutions, or “integrated solutions,” to clients. Data collection included interviews with staff and managers from all GPs and GTs and select Global Leads (GLs) and Global Solutions Groups (GSGs); in-country visits and interviews by phone or in person with 17 Country Management Units (CMUs); and other qualitative and quantitative data from employee engagement and other internal surveys, client surveys, and corporate data. The evaluation’s data sources and methodology had certain limitations which should be kept in mind when interpreting the findings. Independent Evaluation Group | World Bank Group xi Data analysis emphasized patterns and drivers of collaboration and knowledge flow, including how well roles, structures, incentives, processes, and management data enable desired behaviors. As an early-stage evaluation, the methodology did not include direct reviews of the quality or outcomes of World Bank services under the new operating model because they could not yet be observed. Direct comparison of the operating model before and after 2014 was not a central part of the scope of this evaluation; hence, data were benchmarked to the prereform period only in some instances. The Reforms Since 1997, the World Bank’s operating model has been a matrix system. That model had client- focused regional departments coexisting with technical sector departments, which were federated into networks. Under the former matrix, the World Bank operated as six regional banks, making it difficult to move staff and knowledge across Regions. It was also difficult for the entire World Bank Group to coherently manage its global corporate priorities. The World Bank’s client countries’ demands are evolving, and the former operating model did not enable the World Bank to serve these evolving demands as effectively as desired. Client countries increasingly demand sophisticated, multisectoral, knowledge-intensive reforms and investments. These require complex cross-sectoral interventions based on experiences from other regions of the world. To meet evolving client demands, a change process was initiated in 2012 to reform the World Bank operating model. The reform process fell into three distinct phases. First, an inclusive and comprehensive design phase, which was overseen by a change management team, occurred from fall 2012 to summer 2014. This phase included a diagnostic exercise that identified the Bank Group’s comparative advantages, diagnosed the former operating model’s problems, proposed internal reforms, and culminated in the 2013 World Bank Group Strategy. Second, during an initial implementation phase from summer 2014 to summer 2015, the change management team was dispersed and separate expenditure cuts went into effect. Consequently, several reform proposals intended to boost incentives to collaborate were never implemented, and implementation of reforms was marked by uncertainty, discontinuity, and plummeting staff morale. Third, during an adjustments phase ongoing since 2015, management has sought to resolve the new model’s weaknesses. Operating model reforms created a new set of organizational structures. The 2014 reforms retained the matrix structure but made some important changes. Structurally, the new matrix replaced the old anchors and networks with two different types of technical groups. First, GPs are responsible for delivering technical work within their respective areas—such as health, transport, agriculture, governance, and so on—to country clients. And second, five GT Groups are responsible for setting strategic directions internally; supporting operational mainstreaming; and engaging and convening external partners on major cross-cutting priority areas. Each of the 13 GPs and three of the GT Groups falls within one of four Practice Group vice presidential units (see figure O.1, which illustrates the organogram in place as of July 1, 2018). xii Knowledge Flow and Collaboration | Overview FIGURE O.1 Structure of the Operational Complex as of July 1, 2018 | CEO Operations Sustainable Equitable Development Policy and Human 6 Development Growth, Finance Infrastructure Economics Vice Country Development Regional Services (SD) and Institutions (HD) (INF) Presidency (EFI) VPUs (OPCS) (DEC) Climate Gender Infrastructure, Knowledge Fragility, Change Global Public-Private Management Conflict and Global Theme Partnerships and Violence Global Theme Global Theme Theme Guarantees Global Theme Agriculture Governance Education Environment Poverty and Health, Energy and and Natural Nutrition, and Resources Equity Population Extractives Social, Urban, Macroeconomics, Social Transport and Rural and Trade and Protection and Digital Resilience Investment Jobs Development Finance, Water Competitiveness Jobs and Innovation The reforms also introduced new internal processes and instruments. The GPs assumed greater quality assurance responsibilities. Budgeting processes were adjusted several times but continue to cause fragmentation and complicate collaboration. And, a new country engagement model, comprising the Systematic Country Diagnostics and Country Partnership Frameworks, was created to identify and address binding constraints to development and focus country programs more selectively on fewer priorities. The reforms also produced new staff positions and reporting arrangements. First, the Program Leader (PL) position was created to help Country Directors integrate GPs into their country programs. The PL role closely resembles the former Sector Leader position. Second, the reforms abolished the regional Sector Director position to have all technical staff report directly to senior GP directors. In the previous model, Sector Directors represented the sector networks in each Region, thereby connecting technical units to CMUs. Third, a Senior Regional Adviser position was created in regional vice presidents’ front offices to assume some of the former regional Sector Directors’ duties. This adviser position was later abandoned and some of the functions transferred in FY18 to the Practice Group Liaison Director position. Reforms also introduced GSGs to pool and curate global and country knowledge to serve clients’ needs. GSGs are formalized versions of the previous thematic groups, communities of practice, and Global Expert Teams. GSGs are knowledge communities of staff working within a defined topic that represents GPs’ current or emerging core business areas. Most GSGs are housed within Independent Evaluation Group | World Bank Group xiii individual GPs, though many have members from multiple GPs. GSGs are led by GLs who act as knowledge brokers to integrate issues, influence lending, support operations, and connect GP knowledge across Regions. When functioning well, GLs can be powerful conduits of knowledge across the World Bank. Findings on Knowledge Flow The new operating model encourages knowledge flow. Many GPs have structured processes to manage the substantial knowledge and learning embedded across their financing, analytics, and convening activities. Shifting technical work away from regional units into GPs and GTs, which are designed to be globally integrated, improved knowledge flow and staff mobility across Regions, made it simpler to mobilize expertise for client-facing activities, and sometimes deepened expertise in operationally relevant areas. Staff are positive about knowledge flows; for example, 78 percent of staff felt “encouraged to share knowledge with colleagues within [the Bank Group]” according to the 2017 employee engagement survey compared with 73 percent in 2016. Moreover, the World Bank maintains a robust Advisory Services and Analytics (ASA) production, spending over $600 million on 1,476 ASA activities completed in FY18. Each GP manages knowledge differently; some are more strategic than others. GPs with a coherent and strategic approach—for example, the Social Protection and Jobs and Social, Urban, Rural, and Resilience GPs—address knowledge gaps by creating and organizing knowledge and then transferring it to support operations. They use various mechanisms to do this, including staff cross support, staff rotations across Regions, staff and client trainings, online knowledge platforms, South- South knowledge exchanges, knowledge hubs in country offices, and knowledge communities like GSGs. GPs also use innovative knowledge sharing practices to support operational needs, including help desks, safe-space meetings, and other platforms. The GTs support coherent approaches to the World Bank’s cross-cutting priorities, and staff perceive them as collaborative because they don’t lead operational work and therefore need to sell their knowledge services in cross support agreements. Nevertheless, a gap exists between the reform’s aspirations to deepen knowledge curation and sharing within GPs and the current reality. Some GPs have strategic approaches to knowledge; others do not. No metrics measure how business units perform on knowledge. Larger country programs tend to attract global knowledge far more easily than smaller ones, based on evidence collected during country visits and CMU interviews. The mechanisms designed to promote knowledge excellence have mixed results. GSGs have performed unevenly, with few consistent mechanisms to channel knowledge into country programs. How GPs use and resource the GLs (who lead GSGs) and how GLs perform varies greatly. Not all GLs are considered thought leaders in their fields, as evidenced in publication records and interviews. The GLs are constrained by unclear roles and unfunded mandates and, in most cases, GSG knowledge collaboration is driven by GLs’ personal contacts. As such, the effectiveness of GLs xiv Knowledge Flow and Collaboration | Overview as knowledge brokers has varied widely across GPs and individuals. For these reasons, the GL role has been recast in many GPs. Concerns with the quality assurance process for operations and ASA products persist, though no hard data exist on the quality at entry. In the former operating model, Regions had greater responsibility for providing quality control, relying also on development effectiveness units. In the current operating model, this quality assurance moved to the GPs and the role of regional development effectiveness units was weakened. Interviews show concerns over this arrangement, namely: there is insufficient contestability, in part because managers do not always ensure an independent process with outside voices; it gives Country Directors insufficient oversight; and not all GPs have sufficiently filled the gap created by the weakening of the Regions’ development effectiveness units. Leadership and funding, especially access to trust funds, explain variation in ASA production and knowledge flow. Trust funds finance two-thirds of the Bank Group’s total ASA work and provide over 70 percent of the global engagement budgets. Trust funds are not always aligned with World Bank management’s priorities, and according to the internal documents and the Independent Evaluation Group’s 2011 evaluation on trust funds, they add to transaction costs. Besides ASA, knowledge flow also occurs in collaboration for operational activities. Findings on Collaboration for Integrated Solutions Integrated solutions that draw on global knowledge are increasingly in demand. Such solutions can involve multisector projects, programs, or platforms. Even so, single-sector interventions may be appropriate or even preferable in certain contexts. The Bank Group’s country engagement model is often used to define objectives that require integrated solutions, and a large and growing share of objectives will require cross-GP collaboration to deliver. Many positive examples show GPs collaborating on multisector activities designed to address clients’ critical challenges. Also, silos among Regions have decreased, as evidenced in increased interregional cross support. GP staff provided 6.6 percent of their time as cross support to other Regions in FY18 compared with 0.9 percent in FY10. At the same time, the operating model is heavy on competition and transaction costs and makes collaboration for integrated solutions unnecessarily complicated. Silos across sectors have worsened. Inter-GP cross support has declined. In FY17, GP staff spent 9.9 percent of their time supporting tasks mapped to other GPs compared with 15.6 percent in FY10. Staff are generally concerned with operational collaboration under the new operating model, according to surveys, interviews, and a “team effectiveness” analysis done by the human resources department, although the employee engagement survey shows that staff perceive that collaboration has improved since 2013. GPs collaborate more on analytical work than on lending operations and collaborate more within than across GPs, according to a social network analysis. Independent Evaluation Group | World Bank Group xv Many of the challenges of collaboration emerge from the current model’s structure and incentives. In the old operating model, the six Regions acted as silos, impeding knowledge flow. In the current operating model, 13 GPs and 5 GTs act as silos that complicate operational collaboration. This happens for two main reasons: First, in the old model, Regional Sector Directors played a major role in making the matrix function as intended. The elimination of the Sector Director position distanced GPs’ senior leadership from clients and made it harder to arbitrate and resolve tensions. Staff and managers on both sides of the matrix miss the Regional Sector Director role. In their absence, mechanisms for working across GPs and between GPs and CMUs are inadequate. GLs, PLs, and GT Groups are all tasked with integrating across boundaries but are without sufficient authority to do so. Second, the incentive for GP staff to increase own-managed lending volumes fosters competition among GPs for business, and, consequently, less collaboration. The reorganization did not change these long-standing incentives. Also, budget arrangements lead to fragmentation, even as they reflect the structure of the operational complex (see figure O.1). Strong leadership is necessary for collaboration to happen. In interviews and country visits, CMUs argued that collaboration often does not happen unless Country Directors strategically demand it. Apart from Country Director leadership, collaboration is strongest on initiatives with high visibility to senior management because prominent initiatives create an incentive structure that reduces infighting. This is partially why urgent situations, such as crises or disasters, bring out the best collaboration: they are often the most prominent and highly visible Bank Group engagements. Also, shared task team leadership, which has become common, improves collaboration, even as concerns arise about junior staff being given excessive responsibility, with more than 12 percent of all task team leads being at grade GF. Conclusions and Recommendations Despite the reforms’ many challenges, the operating model has shown value. GPs are thinking and operating more globally. They improve knowledge flow and staff mobility, mobilize expertise for client-facing activities, and sometimes deepen expertise in operationally relevant areas. GTs provide strategic direction and coherence to cross-cutting priorities. Although the objective of overcoming regional silos was sound, the new model created other barriers to knowledge flow and collaboration. Managers on both sides of the matrix incur high transaction costs from routine managerial functions. Further, key knowledge flow and collaboration mechanisms do not perform as intended. Quality assurance processes need enhancement. Budget incentives are not aligned with greater collaboration. Finally, more continuity in making course corrections for organizational effectiveness is needed. Every matrix system has pros and cons; the goal should be to mitigate the cons of the current one without disrupting the pros. The main challenge is to stimulate collaboration under the new operating model while enhancing the initial gains on knowledge flow. The World Bank aims to mainstream many issues into operations, including jobs, gender, climate change, maximizing finance, citizen xvi Knowledge Flow and Collaboration | Overview engagement, and fiduciary controls, placing further demands on managers’ ability to coordinate across boundaries. More robust mechanisms to work across the structure are needed while maintaining the global flow of staff and knowledge. To get there, the following evidence-based recommendations are offered: Recommendation 1. Strengthen the approach to knowledge in the GPs and GTs with clear goals, roles, and mechanisms, budgets commensurate with mandates, and metrics for knowledge uptake, quality, and influence. This will entail improving incentives for knowledge production, curation, management, and sharing, whether performed by GLs and GSGs or through other mechanisms. This could build on some of the good practices undertaken by some GPs. Recommendation 2. Improve budgeting systems to better incentivize knowledge flow and collaboration. This could be done, for example, through multiyear budgets or other mechanisms for enhanced year-on-year budget predictability; maintaining an adequate balance between budget allocations for ASA and other knowledge functions and lending; addressing time lags between cross support and the related settlement by recipient budget units; exploring additional options for stronger budget incentives for collaboration; and continuing reforms of World Bank–executed trust funds to improve their alignment with World Bank priorities. Recommendation 3. Better link the GPs and Regions to improve coordination and enhance responsiveness to clients. This could be done by enhancing the Region-facing responsibilities of GPs, providing Region-facing Practice Group Directors with adequate authority to arbitrate and link the two sides of the matrix; and, depending on the exact arrangement, by adjusting reporting arrangements for Practice Managers and PLs. Recommendation 4. Ensure a stronger and more consistent use and role of the PLs as a mechanism for cross-sector collaboration, integrated solutions, and complex client dialogue. This could be done, for example, by clarifying expectations on PLs’ role, authority, and accountability; achieving a better balance between technical and connector roles; ensuring more effective dual reporting to CMUs and GPs; and continuing to recruit PLs based on both leadership skills and technical proficiency. Recommendation 5. Review the existing quality assurance arrangements to improve the quality of knowledge embedded in advisory and financing services. The review should identify ways to enhance the contestability of quality assurance mechanisms across GPs, covering quality at entry for lending and ASAs, implementation, and reporting at completion. It should also assess the balance of oversight responsibilities between GPs and CMUs. Recommendation 6. Ensure there is ongoing monitoring of the operating model and more continuity in change management efforts to enhance the organization’s ability to attain its knowledge flow and collaboration goals. This could be achieved by continuously collecting and reviewing data on organizational effectiveness and learning from past reforms and from other organizations. Management should use these data to help it recognize and scale up successful innovations, and to design additional course corrections, as needed. Independent Evaluation Group | World Bank Group xvii management response World Bank management welcomes the report of the Independent Evaluation Group (IEG), Knowledge Flow and Collaboration Under the World Bank’s New Operating Model. Overall, the report provides management with useful findings about how knowledge flow and collaboration are working under the new operating model and considerations for course corrections. Many points in the report and in the Conclusions and Recommendations section resonate with management views. The World Bank Group is a learning organization, and management has adjusted the operating model and will continue to do so, drawing on lessons from this report and other inputs. IEG rightly notes the methodological limitations of the report. The report is focused on processes and perceived behaviors, and the analysis is based on perceptions largely derived from interviews and surveys. The report is useful for understanding staff perceptions of the operating model at a moment in time. Knowledge Flow Management appreciates IEG’s key findings on knowledge flows: that more knowledge is moving within Global Practices (GPs) at the global level, that staff mobility across Regions has increased, and that mobilizing expertise for client-facing activities has become easier. These findings are important, as these were the primary aims of the change process and the 2014 reorganization. The draft report cites several useful examples of good practice among the GPs in promoting global knowledge flows. Management will continue to look carefully at how knowledge is being generated, shared, and packaged for clients, and to explore options for improvements. The report puts due emphasis on the knowledge delivered through the World Bank’s Advisory Services and Analytics (ASA), but treats lightly the knowledge and learning embedded in financing operations. The bulk of the knowledge that we bring to our client engagements comes through the tacit knowledge our staff have accumulated and from the knowledge transfer and learning-by-doing that occur through our financing operations. There is close complementarity between ASA and financing in contributing to the knowledge flow. Management recognizes the pain points identified in the operational structure and is addressing them directly. Ensuring that Global Leads (GLs) can devote their global expertise to the benefit of country programs, ensuring that Program Leaders (PLs) have an effective connection with the Practice Group Directors, and providing Practice Group Directors with the managerial, human resources, and budget tools to connect Regions and Practice Groups (PGs) are all important challenges that management is working to resolve. Adjustments have been continually introduced as we learn from experience, including the integration of Global Themes groups (the former Cross-Cutting Solutions Areas) into the PGs (except for Fragility, Conflict, and Violence) and the creation of a new PG, Infrastructure. The Knowledge management Action Plan, now in its first year of implementation, includes action items to address the need for goals, clearer roles and responsibilities (including that of GLs and GSGs), incentives and knowledge-sharing mechanisms. xviii Knowledge Flow and Collaboration | Management Response Budget Management recognizes that the budget process can complement broader incentives for knowledge flow and collaboration. IEG’s findings will provide useful inputs as management continues to consider various options and measures to improve budget efficiency, effectiveness, and predictability, and explore ways in which budgets might help encourage greater knowledge flow and collaboration. Any changes to the budget system will of course require support from our authorizing environment and must be balanced against other priorities, including administrative simplification. At the same time, the budget process will not be the primary driver for improving knowledge sharing and collaboration. The budget process for operations plays a role, but it largely reflects the structure of the operational complex. The allocation of World Bank budget funds is grounded in the goals and priorities identified in the Work Program Agreement process and in each Country Partnership Framework (CPF), building on the development needs of our clients. Although each country program is now expected to produce a minimum ASA package, additional resources are allocated through the annual Work Program Agreement process to deliver against the CPF objectives. In recent years, this has resulted in a rising share of resources devoted to knowledge work, but future allocations will depend on economic circumstances, the specific knowledge gaps identified, and client demand. Collaboration The report does well to shed light on collaboration at the World Bank, a challenge faced by many large organizations. The findings are mixed. On the positive side, CPFs are directly tackling the multisectoral nature of client challenges. Nearly two-thirds of CPF objectives are cross-sectoral, requiring cross-GP collaboration to be achieved (compared with barely a quarter for the same countries in the previous CPF period). In addition, as noted in the report, staff perceptions of collaboration across units are rising (favorable responses in the staff surveys increased from 48 percent in 2015 to 67 percent in 2017). At the same time, there has been a decrease in the share of technical staff time working across practices, though the limits to gauging collaboration through the Time Recording System are well recognized. Collaboration at the World Bank extends far beyond formal collaboration through multisectoral operations and is extremely difficult to capture. While making changes to the operating model in 2014, the World Bank acknowledged the risk that working across sectors could become more challenging, and so it emphasized the continued importance of multisectoral solutions to help clients achieve their poverty and prosperity goals. However, multisectoral solutions are not limited to multisectoral projects or cross-sectoral teams working on individual ASAs or financing operations. They also include, for example, strategic-level coordination across GPs, across the Bank Group, and with partners. Management will continue to explore opportunities to enhance links between the Regions and Practice Groups and to learn from the emerging experience with Regional Liaison Directors, especially regarding their effect on the links between Practice Groups and Regions. PLs play Independent Evaluation Group | World Bank Group xix an important role in connecting across the matrix and coordinating activities between Country Management Units (CMUs) and PGs. Around a common set of responsibilities, PLs in different CMUs may have slightly different roles, depending on location, number of PLs in the CMU, number of countries in the CMU, and nature of the engagement and country program. Management will continue to proactively monitor and adapt the PL role and will carefully consider reporting lines as changes to the structure and operational systems are introduced. Quality Assurance Contestability is a critical component of quality assurance, and management shares the report’s conclusion that contestability needs to be strengthened. Recent changes in the accountability and decision-making framework to rebalance oversight responsibilities between the Regions and PGs are intended to enhance contestability and quality control and to rebalance the ownership of operations. Management will continue to seek opportunities to improve the quality assurance arrangements and will review and adjust the accountability and decision-making framework to reflect any adjustments in the operational model. Reform Implementation Management continues to see value in managing for organizational effectiveness while balancing change with stability. Senior management gives regular and ongoing attention to these matters, and over the last 18 months has rolled out multiple initiatives to improve our efficiency and working environment. Performance is monitored continuously and reported regularly, including through various corporate-level monitoring and reporting mechanisms and operations updates to the Board of Executive Directors. Management focuses its monitoring on efficiency, development effectiveness, and support to clients to achieve their poverty and prosperity goals. Moreover, management puts an additional spotlight on tracking key changes to improve our processes and efficiency, including through simplification, the Agile initiative, and serving clients in fragile situations. xx Knowledge Flow and Collaboration | Management Response management action record Incentivize Knowledge Production and Management IEG FINDINGS AND CONCLUSIONS Some Global Practices (GPs) have coherent and systematic approaches to managing and investing in knowledge, others less so. Global Solutions Groups (GSGs) have performed unevenly, with few consistent mechanisms to channel knowledge into country programs. Global Leads suffer from unclear roles and unfunded mandates, and the role has been recast in many GPs. Efforts to more closely integrate certain IFC and World Bank advisory services have met with mixed success. Knowledge performance has been stymied by weak incentives, including from budgeting arrangements and senior management signaling, and lack of metrics. IEG RECOMMENDATIONS Recommendation 1. Strengthen the approach to knowledge in the GPs and Global Themes with clear goals, roles, and mechanisms, budgets commensurate with mandates, and metrics for knowledge uptake, quality, and influence. This will entail improving incentives for knowledge production, curation, management, and sharing, whether performed by GLs and GSGs or through other mechanisms. This could build on some of the good practices undertaken by some GPs. ACCEPTANCE BY MANAGEMENT Partially agree. MANAGEMENT RESPONSE Management agrees that the World Bank can strengthen its approach to knowledge sharing. Transfer of knowledge and learning occur through multiple channels—not only through formal Advisory Services and Analytics (ASAs) and activities led by Global Leads and GSGs. The bulk of knowledge flows through our engagements in financing operations and through the tacit knowledge our staff have accumulated and bring to different types of engagements across the institution. Management will seek ways to enhance how knowledge is generated, shared, and customized to clients and will explore options for improvements, building on existing good practice examples and encouraging cross-GP, cross-PG, and cross-Regional learning on knowledge sharing. It will also continue to review its knowledge indicators, noting that clear and meaningful metrics, for example, for knowledge uptake and influence, remain elusive. Independent Evaluation Group | World Bank Group xxi Improve Budgeting for Collaboration and Knowledge Sharing IEG FINDINGS AND CONCLUSIONS Budget arrangements to some extent reflect the structure of the operational complex. Budget arrangements create incentives for the GPs to favor own- managed lending. The “collaboration premium” for collaborative tasks is not enough to offset these perverse incentives. Systems are fragmented. Trust funds complicate management’s efforts to align knowledge production with corporate priorities. IEG RECOMMENDATIONS Improve budgeting systems to better incentivize knowledge flow and collaboration. This could be done, for example, through multiyear budgets or other mechanisms for enhanced year-on-year budget predictability; maintaining an adequate balance between budget allocations for ASA and other knowledge functions and lending; addressing time lags between cross support and the related settlement by recipient budget units; exploring additional options for stronger budget incentives for collaboration; and continuing reforms of World Bank–executed trust funds to improve their alignment with World Bank priorities. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE Management recognizes that the budget process can be improved to reflect the evolution of the operational structure. The budget process can complement other incentives for knowledge flow and collaboration. Management will continue to explore options to improve budget predictability, flexibility, and integration of World Bank–executed trust funds into the planning process. A flatter structure in PGs, to be introduced July 1, 2019, is expected to help reduce silos between GPs and enhance substantive integration. xxii Knowledge Flow and Collaboration | Management Action Record Enhance Coordination and Engagement among Global Practices and Regions IEG FINDINGS AND CONCLUSIONS The operating model lacks robust connectors with authority to arbitrate between GPs and Country Management Units (CMUs). Too many boundaries cause fragmentation and inefficiencies and impede collaboration. This problem was exacerbated by eliminating Regional Sector Directors. In the previous model, Regional Sector Directors managed the sectors and linked them with CMUs and clients. They had the authority to arbitrate and require collaboration. Because of this absence integration must be driven by Country Directors, supported by Program Leaders (PLs) when empowered. The creation of Region-facing Liaison Directors is a recognition of this weakness but underestimates the need for more robust connector roles to link the two sides of the matrix. IEG RECOMMENDATIONS Better link the GPs and Regions to improve coordination and enhance responsiveness to clients. This could be done by enhancing the Region-facing responsibilities of GPs, providing Region-facing Practice Group Directors with adequate authority to arbitrate and link the two sides of the matrix; and, depending on the exact arrangement, by adjusting reporting arrangements for PMs and PLs. Agree. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE Management agrees that the links between the two sides of the operational matrix can be improved. Management has already introduced the formal role of Country Director in supervision, to enhance co-ownership of the portfolio. Management has also started to discuss the recommendations of this IEG report and is considering alternative ways to enhance the links across the matrix. Several options are under consideration, including IEG’s proposed creation of Region-facing Practice Group Directors, accountable for operations from a Regional perspective. Potential changes in resulting reporting arrangements for PMs and PLs would also be considered. Independent Evaluation Group | World Bank Group xxiii Clarify the Role of the Program Leader IEG FINDINGS AND CONCLUSIONS A common understanding of the PL’s role is lacking. The PL’s role varies across CMUs, as does the individual’s performance in that role. PLs can be effective in building collaboration if empowered through the authority of Country Directors. IEG RECOMMENDATIONS Ensure a stronger and more consistent use and role of the PLs as a mechanism for cross-sectoral collaboration, integrated solutions, and complex client dialogue. This could be done, for example, by clarifying expectations on PLs’ role, authority, and accountability; achieving a better balance between technical and connector roles; ensuring more effective dual reporting to CMUs and GPs; and continuing to recruit PLs based on both leadership skills and technical proficiency. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE PLs can and do help in connecting the two sides of the matrix. For PLs to do this effectively, a greater link between the two sides of the matrix is necessary at a level above PL. After building stronger cross-matrix linkages at the Director level, management will seek opportunities to clarify expectations for the PL role. Around a common set of responsibilities, PLs in different CMUs may have slightly different roles, depending on location, number of PLs by CMU, number of countries in the CMU, and nature of the engagement and country program. xxiv Knowledge Flow and Collaboration | Management Action Record Improve the Quality of Knowledge Embedded in Advisory and Financing Services IEG FINDINGS AND CONCLUSIONS Concerns persist about contestability in quality assurance arrangements and with the balance of oversight responsibility. IEG RECOMMENDATIONS Review the existing quality assurance arrangements to improve the quality of knowledge embedded in advisory and financing services. The review should identify ways to enhance the contestability of quality assurance mechanisms across GPs, covering quality at entry for lending and ASAs, implementation, and reporting at completion. It should also assess the balance of oversight responsibilities between GPs and CMUs. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE Contestability is an input into quality, both for operations and for ASA. Management has introduced changes in the accountability and decision-making framework for operations to enhance contestability and quality control, and to rebalance ownership of operations. Management will review the accountability and decision-making framework for quality assurance arrangements after adjustments in the operating model. Having noted this, management does not agree that there are concerns about quality assurance in operations: quality of World Bank operations, as validated by IEG, remains high. Independent Evaluation Group | World Bank Group xxv Continuously Collect and Review Organizational Effectiveness Data IEG FINDINGS AND CONCLUSIONS Discontinuity in reform implementation was caused by dispersing the central change management team in place during the early phases of reform. Having a central team to collect and use organizational performance data and oversee change management could have benefited the organization in recent years. Organizational effectiveness deserve a more permanent attention. IEG RECOMMENDATIONS Ensure an ongoing monitoring of the operating model and more continuity in change management efforts to enhance the organization’s ability to attain its knowledge flow and collaboration goals. This could be achieved by continuously collecting and reviewing data on organizational effectiveness and learning from past reforms and from other organizations. Management should use these data to help it recognize and scale up successful innovations, and to design additional course corrections, as needed. ACCEPTANCE BY MANAGEMENT Agree. MANAGEMENT RESPONSE Management is monitoring, assessing, and adjusting the impact of organizational and process changes. Monitoring of operational effectiveness is carried out at the Chief Executive Officer level, with inputs from other corporate units (for example, BPS, Human Resources, Operations Policy and Country Services), from IEG and the Internal Audit Department, and from client feedback. This IEG evaluation is a helpful input into this effort. xxvi Knowledge Flow and Collaboration | Management Action Record 1 I N R E C E N T Y E A R S , organizational reforms at the World Bank have created a new set of goals, structures, strategies, and processes to enhance its ability to deliver services to clients. Focusing on a subset of the larger reforms, this evaluation Introduction Developmental assesses how the World Bank’s current model for delivering Context and client services—in short, its operating model, including its Global Evaluation Practices (GPs) and Global Themes (GTs) influences knowledge Approach flow and enhances collaboration to deliver “integrated solutions”— or multisector and multiservice tasks and approaches (approaches refer to programs or platforms that combine interventions)—to clients. The evaluation examines how well the goals, structures, strategies, and processes of the World Bank’s operating model generate incentives and behaviors that work. In short, it is about how the operating model influences knowledge flow and collaboration. It does this by assessing the outputs, behavioral changes (direct outcomes), and intermediate outcomes of reforms, but for lack of data and the relative newness of the operating model, does not assess long-term outcomes. The evaluation also reviews the process of designing and implementing the reforms to understand how that process affected the model’s effectiveness. The International Finance Corporation (IFC) and the Multilateral Investment Guarantee Agency (MIGA)—the other parts of the World Bank Group—were outside the scope. The evaluation is based on a theory of change that was constructed through an examination of change process documents and key informant interviews (table 1.1). Following table 1.1, the evaluation examined the proposed reforms and the diagnostics that underpinned them. The evaluation reviewed the implemented reforms and whether they were necessary and sufficient to achieve the desired changes. The evaluation’s data collection and analysis focused largely on behaviors associated with operational, client-facing effectiveness— specifically, the operating model’s ability to transfer technical Independent Evaluation Group | World Bank Group 1 TABLE 1.1 | Theory of Change Reforms and Outputs Key Assumptions Proposed • Diagnostic studies identified reform needs and • Diagnostics were sound reforms directions • The proposed reforms are • Reform proposals made by change teams and sufficient for achieving desired working groups in the design phase behavior changes Implemented • New structures, including Global Practices • Reforms implemented reforms according to plan • Joint Practices (“One World Bank Group”) • Staff morale is not • New roles and mechanisms for knowledge flow (permanently) damaged and collaboration (Program Leaders, Global Leads, Global Solutions Groups) • New budgeting and other processes • New country engagement model Outputs • Country programs are more selective and • World Bank processes incorporate best global knowledge support selectivity • Country programs include multisector and • Clients need and demand multiservice solutions multisector solutions • Clear directions set for priority Global Themes Clients demand activities that •  incorporate select Global Themes Intended Behavior Changes and Outcomes Key Assumptions Expected Knowledge flows more freely across organizational • Reform proposals selected for behavior boundaries implementation are necessary changes and sufficient for achieving … and is customized to country context … desired behavior changes ... technical staff collaborate more across boundaries … • The desired staff behaviors … and integrate Global Themes in country programs … are supported by metrics, incentives, and processes … as a result of which teams become more effective at designing and delivering multisector and single-sector • Global mobility of staff services improves knowledge flow and does not impede customization to country context • Business units give adequate priority to knowledge, collaboration, quality assurance, staff development, and other intended behavior changes Intermediate • Improved quality of World Bank services due to • The implemented solutions outcomes stronger knowledge flow and collaboration were appropriate • Integrated, evidence-based solutions designed and • The World Bank serves all implemented client segments 2 Knowledge Flow and Collaboration | Chapter 1 knowledge across organizational boundaries or improve global knowledge flow; collaborate across organizational boundaries; customize global knowledge to country contexts; integrate GTs; and deliver “integrated solutions” to clients. The team’s analysis paid particular attention to how well roles, structures, incentives, processes, and management data enable desired behaviors. According to this theory of change, if the desired behaviors materialize, the World Bank is able to enhance its development effectiveness and impact by delivering high-quality services to its clients. In short, effective collaboration and knowledge flow enhance the World Bank’s effectiveness and development impact. This evaluation finds that certain aspects of the operating model are valuable, especially its enabling of global knowledge flow; however, evidence from the early years of implementing the model indicate that its structure and processes inhibit collaboration and cause inefficiency, fragmentation, and internal competition. Some GPs lack coherent and systematic approaches to managing and investing in knowledge. If left unaddressed, these issues pose risks to the World Bank’s ability to deliver for clients. Methodology The main question guiding the study was “do the operating model’s new features contribute to knowledge flow and improved collaboration for integrated solutions?” These new features include Global Leads (GLs), Program Leaders (PLs), GPs, Global Solutions Groups (GSGs), and GT Groups. The team reviewed how these features facilitate:   Knowledge  flow, especially across Regions and to clients. This includes mobilizing and customizing global expertise to country contexts; and   Internal  collaboration and integration of cross-sectoral perspectives in the delivery of services, especially integrated solutions, to clients. Data for this evaluation came from a comprehensive review of available qualitative and quantitative data combined with the team’s own data collection from interviews and desk reviews. As described in appendix A, data collection included semistructured interviews with staff and managers from all GPs and GTs, broader key informant interviews, a review of GLs and GSGs, social network analysis, reviews of staff and client surveys and relevant corporate data, analysis of budget arrangements, and analysis of Country Partnership Frameworks (CPFs). The evaluation collected data on knowledge flow and collaboration in country programs through a combination of in-country visits and interviews by phone or in person with Country Directors (CDs), Country Managers, PLs, and other members of Country Management Units (CMUs). It covered CMUs for Argentina, Azerbaijan, Botswana, Brazil, Bulgaria, the Caribbean, Chad, China, Central America, Côte d’Ivoire, the Arab Republic of Egypt, Malaysia, Mali, Myanmar, Pakistan, South Africa, Sri Lanka, Tunisia, Uganda, and Zambia. Client perspectives were obtained directly in five visited countries and indirectly from CMU staff in all the covered programs. Systematic triangulation was used to analyze the data for patterns and drivers Independent Evaluation Group | World Bank Group 3 of collaboration and knowledge flow. Key informant interviews and focus groups helped validate the findings. The evaluation’s data sources and methodology had certain limitations which should be kept in mind when interpreting the findings. These related to the following:   Timing. As an early-stage evaluation, the methodology did not include direct reviews of the quality of World Bank services or their outcomes, which could not yet be observed; instead, emphasis was more on processes and behaviors.   Scope. The evaluation pays closer attention to how the operating model varies across GPs than across Regions because GPs are new features of the operating model. The scope did not include broader changes in technology, human resources, and knowledge management that occurred in parallel with the reform of the operating model and may affect it. No counterfactual could be established, and data could be meaningfully benchmarked to the prereform period only in a few instances.   Data sources. Collaboration is measured using staff time records and corporate data on joint projects, but these are inaccurate proxies for collaboration. Perception data, collected through employee engagement surveys, change over time and may not reflect underlying realities. Also, there is no good corporate data on knowledge and how it flows into client services. This report is structured as follows: Chapter 2 explains the objectives and major components of the current operating model and how it came into being. It examines how reform implementation has enabled or hindered the functioning of the operating model. Chapter 3 reviews knowledge flows, focusing on the operating model’s values and inefficiencies. Chapter 4 examines collaboration and the model’s values and inefficiencies in delivering integrated solutions to clients. Chapter 5 concludes and offers recommendations 4 Knowledge Flow and Collaboration | Chapter 1 2 highlights 1 Internal reforms of the World Bank sought to improve knowledge flow across Regions, and collaboration to deliver more integrated, multisector solutions to clients. They also The sought to focus country programs more Operating selectively on tackling critical issues, better Model and How deliver global priorities, and deepen the World It Changed Bank’s ability to deliver expertise grounded in thought leadership and global knowledge curation. 2 Reforms resulted in unanticipated managerial challenges and the model has remained in flux. Dispersing the central change management team and undertaking simultaneous budget cuts caused lack of continuity in reform implementation. Some proposed measures on incentives and leadership never materialized. Independent Evaluation Group | World Bank Group 5 S I N C E 19 97, the World Bank’s operating model was a matrix system. In matrix organizations, client-oriented units exist side by side with technical units and have different but complementary objectives. Table A.2, appendix A, shows the version of the matrix that existed until 2014. That model had client-focused regional departments co-existing with technical sector departments, which were federated into networks. Matrix organizational frameworks create “creative tension” between focusing on clients and providing technical excellence. In an ideal scenario, the matrix allowed the Regions to focus on individual country clients and the then-networks to focus on sectoral expertise. However, matrix structures are complex and difficult to manage, and this matrix was no exception. The Change Diagnostics The World Bank’s client countries’ demands are evolving, which prompted rethinking of the former operating model. Client countries are increasingly prosperous and sophisticated, but their public institutions have not kept up with these economic advancements (World Bank 2017a). They increasingly demand multisectoral, knowledge-intensive support, which means, in practice, complex interventions based on experiences from other regions of the world. Their thirst for capital requires solutions that bring both public and private sector perspectives to bear. This is especially true for upper middle-income countries that no longer look to regional neighbors for learning but to wealthier countries in other regions that may have tackled similar challenges. Furthermore, working on global and regional issues and public goods has been another growing area for the World Bank and is likely to remain so, as the World Bank leverages its reputation and relevance as a major international organization with a broad development mandate. The former operating model did not enable the World Bank to serve these evolving demands as effectively as desired; this was the conclusion of IEG’s matrix evaluation and increasingly World Bank management had reached this view as well (World Bank 2013). Under the former matrix, the World Bank was operating as six regional banks, creating fragmentation by sector and region. It was difficult to move staff and knowledge across Regions. It was also hard for the World Bank and IFC, its private sector arm, to collaborate effectively, and for the entire Bank Group to coherently manage its global corporate priorities. A change process was started in 2012 that identified comparative advantages, diagnosed problems with the operating model, and proposed internal reforms. The process culminated in the 2013 Bank Group Strategy and a major reorganization in 2014. The diagnostics that underpinned reforms emphasize the World Bank’s comparative advantages in servicing these new client demands. The Bank Group’s close client relationships and deep technical and how-to implementation knowledge across sectors and Regions, positions it to offer solutions to complex and multidimensional problems. These solutions may be offered through investments, policy reforms, convening stakeholders, regulatory changes, and technical and knowledge assistance. 6 Knowledge Flow and Collaboration | Chapter 2 Solutions can span the public and private sectors. The expectation was that providing solutions to clients’ most complex problems would require the World Bank to draw on its deep expertise from different parts of the organization and different parts of the world. Compared with other providers of development finance and knowledge, the World Bank has a broad, global range of expertise suitable to serve countries seeking access to global experience, multisector solutions, or both. It can curate knowledge of solutions from upper- and middle-income countries and apply this knowledge to clients’ national contexts. The World Bank also has a value proposition around marrying its country presence with its work on global and regional issues and public goods (World Bank Group 2013). The change process formulated many different goals, which can be simplified as knowledge flow and collaboration for integrated solutions. The theory of change (table 1.1) reconstructs the logic steps and assumptions in the reform logic. The reform process sought to maintain a country-client focus while also giving priority to the most critical development problems. It sought to better deliver on global priorities such as jobs, gender, and climate change. And, it sought to deepen the World Bank’s ability to deliver global expertise grounded in thought leadership and curated from around the world. This evaluation focuses on a set of reform objectives related to knowledge flow and integrated solutions:   Knowledge  flow is the process of bringing the right global knowledge to the right clients at the right time. Less advanced client countries want to know how more advanced countries, such as Chile, Korea, Malaysia, or Singapore, handle technical reforms. This requires the free flow of people and knowledge across Regions and the Bank Group’s organizational boundaries. It also requires customizing knowledge to country contexts. And it requires strong knowledge production, curation, and management.    IntegratedSolutions address complex issues with broad, cross-sectoral, and multiservice programs involving diverse tools and knowledge. Clients’ development challenges often cut across sectors and require diverse technical expertise from different sectors, disciplines, and both public and private sectors. To provide integrated solutions staff must collaborate across sectoral boundaries on multisector approaches, programs, or projects. Integrated solutions are meant to complement, not replace, traditional single-sector projects and programs. The Reform Processes The reform processes fall into three distinct phases: an inclusive and comprehensive design phase from fall 2012 to June 30, 2014; an implementation phase from July 1, 2014, to June 30, 2015, marked by confusion and staff concerns; and an adjustment phase where management sought to resolve the new model’s weaknesses. Box 2.1 reconstructs the timeline. During the initial design phase in 2012 and 2013, the entire Bank Group was brought into the design process, which was overseen by a change management team. This team, initially headed by a senior vice president, was composed of five “change teams” that each looked at different aspects of the reorganization (figure 2.1). Twelve staff working groups, headed by experienced managers and involving representatives from most parts of the organization, developed reform proposals in areas Independent Evaluation Group | World Bank Group 7 such as risk, results, leadership, career development, incentives for collaboration and knowledge, and closer alignment of IFC, MIGA, and World Bank (“One World Bank Group”). Other staff not directly involved in the design process had numerous opportunities to engage through task forces and forums. The change management team also brought in external consultancy companies to provide inputs based on organizational change experiences elsewhere. The Board of Executive Directors and staff were kept informed as the process advanced. Different options for where to house technical staff were put forward in the reform discussions. Some senior managers believed in the primacy of a client-driven organization in which the Regions have large budgets and hold substantial power; such was the status until 2014. Others believed that this arrangement created regional silos that impeded knowledge flow across Regions; they pointed to large consultancy companies that organize globally integrated practices around business lines as a source of inspiration. Three options for how to organize technical staff were put forward in May 2013: (i) strengthen the matrix by adjusting incentives and accountabilities, improving management data, and removing barriers to knowledge sharing; (ii) create a global expert pool managed by the sectors’ former anchor units (central units not linked to any Region). These pools would be composed of technical experts and FIGURE 2.1 Working Groups and Change Teams in Phase 1 of the Reform | Process CHANGE TEAMS The five key areas of change. These priority themes were identified by staff through the Organizational Health Index. WORKING GROUPS The teams within each theme that were tasked to work with staff feedback to create recommendations for the World Bank Group by June 2013. Each team had an executive leader, one or more directors, a team manager, one or more communicators, a secondee, and representatives from various country and headquarters offices. WORKING GROUPS • Corporate Strategy • Budget World Bank Group Strategy • Operationalizing Goals • One World Bank Group • Finance CHANGE TEAMS Knowledge and Solutions • Knowledge and Solutions Client Impact & Results, • Risk • Results Accountability, and Risk Leadersip, People, • • Talent and People Leadership and Talent Global Footprint • Global Footprint Source: Management of the World Bank Group institutions. 8 Knowledge Flow and Collaboration | Chapter 2 Box 2.1 |  Major Change Process Milestones Phase 1: Design 2012, fall. Change process launched by the World Bank Group President, starting with a diagnostic phase that included creating a senior vice president for change management and extensive measures to survey, consult, engage, and inform staff through a variety of online and face-to-face communication mechanisms. 2013, April. The Bank Group formally announces its twin goals: fighting poverty and boosting shared prosperity 2013, July. Change Teams put forward 43 broad proposals in different areas, and senior management decided which ones would proceed. The accepted proposals included housing technical staff in integrated delivery practices, later called Global Practices; switching to a new Country Partnership Framework that would aim for more selectivity and include IFC; improving leadership and staff management capabilities; and adjusting incentives to focus more on client impacts and results. 2013, October. The Board formally approves the new Bank Group strategy. Phase 2: Initial implementation 2014, July 1. The new model becomes effective. GPs and cross-cutting solutions areas (CCSAs) start operations and are housed in one vice presidential unit (VPU) jointly overseen by two vice presidents. Phase 3: Adjustments 2015, July 1. The GPs and CCSAs are housed in three VPUs, which are given the name “Practice Groups” and overseen by three different vice presidents: Equitable Growth, Finance and Institutions: Governance; Finance and Markets; •  Macroeconomics and Fiscal Management; Poverty; Trade and Competitiveness. Human Development: Education; Health; Social Protection and Labor. •  Sustainable Development: Agriculture; Energy and Extractives; Environment and •  Natural Resources; Social, Urban, Rural and Resilience; Transport and Information Communication Technology; Water. 2017, July 1. The CCSAs are renamed Global Theme (GT) Groups and, except for the Jobs Group, are housed in a new Global Themes VPU together with the Knowledge Management Group. The Jobs CCSA previously merged as a unit with the Social Protection Global Practice (GP) on October 1, 2015. Figure 2.3, panel a, shows the resulting structure. (Box continues on the following page.) Independent Evaluation Group | World Bank Group 9 Box 2.1 | Major Change Process Milestones (continued) 2018, January 1. The Trade and Competitiveness GP was merged into two other GPs, which become the Finance, Competitiveness, and Innovation GP and Macroeconomics, Trade and Investment GP. With this change, there are now 13 GPs. 2018, July 1. The Sustainable Development Practice Group is split into two new Practice Group VPUs: Infrastructure: Energy and Extractives GP; the Transport and Information •  Communication and Technology GP; and the Infrastructure, Public-Private Partnerships, and Guarantees Global Theme Sustainable Development: Agriculture; Environment and Natural Resources; Social, •  Urban, Rural and Resilience; and Water GPs, plus the Climate Change Global Theme. Additionally, the Global Themes VPU is disbanded. With the exception of Fragility, Conflict, and Violence, the GTs are integrated into Practice Group VPUs, while the Knowledge Management Group is moved to DEC. Figure 2.3, panel b, shows the resulting structure. Source: World Bank documents. managed as a global resource that task teams could contact for cross support in high value-added activities; and (iii) integrated delivery practices that deploy staff to serve both global and country clients and invest in knowledge and staff career development. A large gathering of directors and higher-level management voted on these options, favoring the third. In interviews, key decision makers described this as a contentious decision to be made that caused divisions among management. Overview of Changes to the Operating Model The 2014 reforms retained the matrix structure but made some important changes. The new matrix replaced the old anchors and networks with two different types of technical groups: (i) GPs, which are responsible for delivering technical work within their respective areas, such as water, energy, education, and so on, to country clients; and (ii) GT, initially called cross-cutting solution areas, that are responsible for coordinating on cross-cutting themes, including jobs, gender, and climate change. These reforms were the most significant reorganization of the World Bank since the creation of the original matrix system in 1997. Before that, other significant restructuring took place in 1977 and 1987. 10 Knowledge Flow and Collaboration | Chapter 2 These new arrangements reduced the roles of Regions. As before, the Regions and CMUs remained responsible for client relationships and ensuring that technical work fits local contexts (figure 2.2). However, technical staff within the Regions would now report to GPs instead of to the networks’ Sector Directors in the Regions. In fact, as described later in this section, the Sector Director position was eliminated altogether despite being the fulcrum that connected the two sides of the matrix, ensuring integration across sectors and between networks and CMUs. New processes were introduced to support the new operating model. The new country engagement model, comprising the Systematic Country Diagnostics (SCDs) and the CPFs, was created to help identify and address binding constraints to development and focus country programs more selectively around corporate priorities. Oversight of technical quality shifted from Regions to GPs. The budgeting process was also adjusted several times. For example, in FY15, staff costs were fully paid with no settlements between units for cross support. These cross-support arrangements have since changed several more times. Many positions were changed, or new ones created, to make the model work:    The PL position was created to connect CMUs and GPs. The PL reports to CDs with a “dotted line” (meaning less important) to a Practice Group Director. PLs help CDs integrate across sectors. The PL role closely resembles the Sector Leader position which existed before.    The regional Sector Director position was abolished. In the previous model, Sector Directors represented the Networks in each Region, connecting the two sides of the matrix structure. Sector Directors were the main counterparts of the CDs and arbitrated among the operational managers. The reason for abolishing this position was to have all technical staff report directly to senior GP directors, regardless of where in the world that technical staff works. FIGURE 2.2 Roles and Processes within a Country Team | Products: *Lending *Advisory Services and Analytics Regions and Global Global CMUs Processes: Practices Themes *Systematic Country Diagnostic *Country Partnership Framework *Work Program Agreement and Budgeting Client Relationship Technical Work Source: Independent Evaluation Group based on World Bank documents. Independent Evaluation Group | World Bank Group 11    A Senior Regional Adviser position was created in Regional Vice Presidents’ front offices to assume some of the former Regional Sector Directors’ duties. This position was later abandoned and replaced in FY18 with a part-time Liaison Director role assigned to the pool of directors in the GPs.    GL positions were created to connect subsector knowledge and business lines across countries and Regions. The GLs also manage the new GSGs. The GSGs are formalized versions of the previous matrix’s thematic groups, communities of practice, Global Expert Teams, and other groups. GSGs are knowledge communities of staff working within a defined topic and are expected to help pool and curate global and country knowledge and use it to serve client needs. Figure 2.3 illustrates how the GPs and GTs were structured at the time of writing, June 2018. To make this model function, there were, as of June 2018, 18 senior directors, 30 directors, 175 Practice FIGURE 2.3 Organizational Charts | a. Structure of World Bank operational complex, January 1 to June 30, 2018 CEO Operations Policy Equitable Growth, Human and Country Finance and Sustainable 6 Regional Development Global Themes Services Institutions Development (SD) VPUs (OPCS) (EFI) (HD) Macroeconomics, Trade and Education Agriculture Climate Change Investment (MTI) (EDU) (AGR) Finance, Health, Nutrition, Energy and Fragility, Competitiveness and Innovation and Population Extractives Conflict, (FCI) (HNP) (EEX) and Violence Poverty and Social Transport and Equity Protection and Digital Gender Jobs Development (POV) (SPJ) (TDD) Infrastructure, Governance Jobs Water PPPs and (GOV) CCSA (WAT) Guarantees Social, Urban, Rural and Knowledge Resilience Management (SURR) Environment and Natural Resources (ENR) (Continues on the following page.) 12 Knowledge Flow and Collaboration | Chapter 2 FIGURE 2.3 Organizational Charts (continued) | b. Structure of World Bank operational complex as of July 1, 2018 CEO Operations Sustainable Equitable Development Policy and Human 6 Development Growth, Finance Infrastructure Economics Vice Country Development Regional Services (SD) and Institutions (HD) (INF) Presidency (EFI) VPUs (OPCS) (DEC) Climate Gender Infrastructure, Knowledge Fragility, Change Global Public-Private Management Conflict and Global Theme Partnerships and Violence Global Theme Global Theme Theme Guarantees Global Theme Agriculture Governance Education Environment Poverty and Health, Energy and and Natural Nutrition, and Resources Equity Population Extractives Social, Urban, Macroeconomics, Social Transport and Rural and Trade and Protection and Digital Resilience Investment Jobs Development Finance, Water Competitiveness Jobs and Innovation Source: Independent Evaluation Group. Note: PPP = public-private partnership; MFD = Maximizing Finance for Development; VPU = vice presidential unit. Appendix B shows the organogram from before 2014. Managers (PMs), 117 program leads, 4,863 nonmanagerial GP staff, and 334 nonmanagerial GT staff. This translates into an average headcount ratio of 30 GP or GT staff per PM. Reform Implementation The new matrix organization has remained in flux because of managerial challenges that later chapters discuss in full. During the first year of the reforms in FY15, two vice presidents (VPs) worked as one team to jointly manage the GPs and cross-cutting solutions areas. However, this structure did not work particularly well because it took too long to resolve many issues. The first follow- up restructuring organized GPs and cross-cutting solutions areas into three Practice Groups, an organizational construct that resembled the previous networks. Later, a GT vice presidential unit (VPU) was added. These VPUs were subsequently restructured, effective July 1, 2018. The reform’s design foresaw an eventual need to reorganize GPs based on evolving business needs, but no clear Independent Evaluation Group | World Bank Group 13 criterion for this need was put forward. So far, the largest GP reorganization was that the Trade and Competitiveness GP was merged into two other GPs in January 2018. The rationale for this was not well understood by staff. The joint World Bank-IFC units have been realigned. The central change management team was dispersed when the reform became effective in 2014, causing discontinuity in reform implementation. Some of the adjustments and reorganizations between 2014 and 2018 lacked a clear rationale or were not informed by performance data. There is no clear line of sight from reform plans laid out in 2013–14 and what has emerged today. Having a central team to collect and use organizational performance data and oversee initial reform implementation and later recommend course corrections could have benefited the organization. The timing of expenditure savings complicated reform implementation. Budget savings were not an original reform goal and did not form part of the consultative reform design process. In 2014 when the reforms were initially implemented, management carried out an “expenditure review” with the goal to safeguard the World Bank’s financial stability and increase its financial capacity by bringing expenditures in line with revenues. This review targeted budget cuts of $400 million in three phases over three years, implying also a need for staff reductions (although resources for operations were relatively protected). External management consultants warned that major budget cuts during organizational reforms was ill-advised. If budget cuts were a priority, the main goals of the reform— improved knowledge flow and collaboration for integrated solutions—could have been attempted FIGURE 2.4 Employee Engagement Index (percent) | 90 79 79 79 77 80 74 71 69 73 71 70 69 66 64 65 68 Percent favorable 70 65 64 60 50 40 30 20 10 0 All World Bank Group Equitable Finance and Human Development Sustainable Development Investment Practice Group 2009 2013 2015 2016 2017 Source: Employee engagement surveys. Note: In 2013, the Employee Engagement Index comprised an average of seven questions. In other years, the index was an average rating for four questions. All responses are percent favorable. For 2013, Equitable Growth, Finance, and Institutions was compared with Financial and Private Sector Development, Human Development Practice Group compared with Human Development Network, and Sustainable Development Practice Group to Sustainable Development Network. The index provides an average score of answers to questions relating to ”overall perceptions” of working at the World Bank, averaging four questions on “being proud of working at the World Bank,” ”the World Bank Group is the best place to work in development,” ”having a strong sense of belonging to the World Bank Group,” and ”World Bank Group inspiring employees to do their best work.” 14 Knowledge Flow and Collaboration | Chapter 2 without a major structural overhaul. Staff morale predictably suffered, with favorable responses in the employee engagement survey falling between 2013 and 2015 (figure 2.4). In a tight and uncertain budget environment, competition rather than collaboration can easily become the predominant behavior. Complementary measures to support the operating model have been implemented at an uneven pace. During the design phase, working groups envisioned changes to processes, incentives, behaviors, and support systems that would stimulate knowledge flow and collaboration and thereby improve results and client impacts. With no central change team, different units have implemented complementary measures at different speeds. This evaluation does not comprehensively list and assess all complementary measures, but some are notable. For example, there has been some progress on managing GP staff as one global pool. Information technology, including connectivity and mobile and internet tools, has noticeably improved. Data paucity and lack of leadership prioritization stalled progress on creating a results focus for staff, another area of proposed complementary measures. Operations Policy and Country Services created some useful data to assess the operating model’s results, such as the two- minute survey and client survey used later in this report. Overall, though, corporate data focus far more on disbursements and the lending pipeline than on results. Management has created corporate scorecards that “cascade” down to results agreements for business units and a Results Measurement and Evidence Community. But the World Bank has no comprehensive approach to improve project monitoring and evaluation, in part because it has no clear, broad-based leadership authority advocating for results and improved monitoring and evaluation functions, according to IEG’s evaluation of self-evaluation systems (World Bank Group 2016b) and a later follow-up review. There has been limited progress on creating incentives—through rewards, recognition, and accountability—to promote collaboration and knowledge capture. Virtually all respondents to this evaluation’s interviews and focus groups agreed that the World Bank incentivizes lending over knowledge, including via career advancements. One area of progress was the creation of joint team leaders, which incentivizes mentoring and collaboration in some instances (see chapter 4). Leadership accountability for greater collaboration has not materialized. Having a single management team across GPs proved impractical and was discontinued, and proposals to assess GP Senior Directors on their GP’s ability to collaborate were never implemented. In summary, there are multiple reasons for the reform’s uneven implementation. The working groups that designed reform proposals were dissolved once the reforms went into effect and, therefore, were not involved in implementation. The lack of a central change management team led to discontinuity. Implementing the expenditure review measures proved distracting. There were internal disagreements within senior management around reform directions. And, setting up the new GPs consumed much time: filling positions and putting new roles, processes, and reporting arrangements in place. Independent Evaluation Group | World Bank Group 15 3 highlights Global 1 Establishing GPs improved knowledge flow and Knowledge staff mobility across Regions, made it simpler to Flow mobilize expertise for client-facing activities, and, sometimes, deepened expertise in operationally relevant areas. Larger country programs tend to attract global knowledge more easily than smaller ones. 2 Some GPs have coherent and systematic approaches to managing and investing in knowledge, others less so. This often reflects the availability of trust funds and leadership support. Contestability in quality assurance is uneven. 3 GSGs have performed unevenly, with few consistent mechanisms to channel knowledge into country programs. Although views on GLs vary, GLs often suffer from unclear roles and unfunded mandates and the role has been recast or disbanded in some GPs. 4 The GTs support coherent approaches to the World Bank’s cross-cutting priorities, making them a valuable addition to the operating model. 16 x I N T H E A B S E N C E O F G O O D M E T R I C S of knowledge performance, this evaluation reviews the processes and mechanisms for knowledge flow, curation, and production. It finds several elements of value. The GP model has fostered greater sector identity and sector direction. GPs are often thinking and operating more globally. Some of the GPs create and curate global knowledge within well-defined technical niches and enable knowledge flow and staff mobility across Regions. GTs provide useful coherence on major corporate priorities. But the evaluation also finds that key knowledge flow mechanisms have not performed as intended and that some GPs lack coherent and systematic approaches to managing and investing in knowledge. Leadership, incentives, budget arrangements, and trust fund availability account for the World Bank’s uneven knowledge performance. The new operating model encourages knowledge flow. Many GPs have structured GP-wide processes to curate knowledge and identify and prioritize knowledge gaps. This includes managing the substantial knowledge and learning embedded across the GPs’ financing, analytics, and convening activities. GPs understand that their task is to move knowledge globally and have used various mechanisms to do so. These mechanisms, discussed more in the following, include: providing staff cross support (see data on this in chapter 4), rotating staff across Regions, appointing GLs, training staff and client, exchanging South-South knowledge, and creating knowledge communities such as GSGs, among others. A theme across most interviews was that it is easier than before for staff to access relevant technical knowledge in a timely fashion and channel it to clients. Also, the World Bank maintains a robust Advisory Services and Analytics (ASA) production, spending over $600 million on 1,476 ASA activities completed in FY18 (table 3.1), in addition to other knowledge related spending on research, knowledge management, and other areas. Spending on ASA has grown substantially since the reform. Management has recently introduced the concept of a minimum ASA package to inform the development of SCDs, CPFs, and enhance the World Bank’s ability to contribute to local and global debates. TABLE 3.1 | Advisory Services and Analytics Production and Spending Production or Spending FY11 FY12 FY13 FY14 FY15 FY16 FY17 FY18 Type Initiated ASA products (no.) 1,543  1,658  1,895  2,296  2,063  1,998  1,537  1,778  Completed ASA products 1,213  1,175  937  1,469  1,813  1,877  1,424  1,476  (no.) Cumulative expenditures on 327  346  249  402  659  677  497  607  completed ASAs ($, millions) Source: Business Intelligence and Standard Reports. Independent Evaluation Group | World Bank Group 17 Staff is increasingly positive about knowledge flows. From 2015 to 2017 the share of staff who said they could “access relevant knowledge” to do their jobs effectively increased from 74 percent to 79 percent in the annual engagement survey. From 2016 to 2017, the share of staff that felt “encouraged to share knowledge with World Bank Group colleagues” increased from 73 percent to 78 percent. Interviews show that GP staff believe knowledge flow has improved and collaboration inside GPs is easier because of enhanced staff mobility. They also perceive client services have improved at least some of the time. Many staff attribute improved knowledge flow to the operating model. They point to the GPs’ globally integrated nature and the various mechanisms that move staff and knowledge. A common theme in interviews was that GPs are getting better at applying lessons learned, not just from within Regions, but globally: “GPs have become less insular and more global.” CDs appreciate they can approach the GP Senior Directors whenever they want to access global knowledge. Some GPs have created specialized global teams—for example on topics such as taxation or land tenure—which has helped deepen expertise; support from those teams was appreciated in some of the covered country programs (see box 3.1). IEG’s forthcoming Country Program Evaluation of Mexico finds many positives in the World Bank’s knowledge support, rooted in its access to frontier cross-country knowledge and characterized by flexibility, responsiveness, and high quality. Conversely, some staff argue that improved knowledge flow and collaboration would have occurred just as easily under the previous operating model. Subsectors have become more integrated within many GPs. Subsectors are smaller sectors or issues that fall within a larger sector or individual GP. The Water GP mandated that teams working on water access collaborate with teams working on water resource management. GLs within the Transport and Digital Development GP have integrated urban mobility and road safety issues, resulting in a new approach to urban road safety. In the Social Protection and Jobs GP, staff with pension reform and modeling expertise from Eastern Europe developed a pension-related work program in Africa. The Social, Urban, Rural and Resilience GP encourages its four subsectors – land, social development, urban development, and disaster risk management – to collaborate on projects when it adds value for clients. An example is the multisector Crisis Recovery Project for North-Eastern Nigeria, which rehabilitates service delivery infrastructure, increases livelihood opportunities for conflict-affected and displaced communities, strengthens social cohesion, and builds emergency response capacity. The GPs’ Approaches to Knowledge Each GP manages knowledge differently; some are more coherent and strategic than others. Some GPs have a coherent approach to addressing knowledge gaps by creating and organizing knowledge then transferring it to support operations. For example, the SPJ GP systematically selects, researches, and publishes knowledge products that are aligned with the strategy and priorities set by its Leadership Team. Each year the GP releases one or two flagship publications on priority topics. For each publication, the GP (i) selects and researches the selected topic; (ii) curates the knowledge at a South-South Learning Forum with client counterparts; and (iii) synthesizes the accumulated 18 Knowledge Flow and Collaboration | Chapter 3 Box 3.1 |  Land Expertise Creating a dedicated Land team increased knowledge flow and improved the World Bank’s ability to address issues of land tenure and administration. Nearly all Land- related projects cut across sectors; 98 percent of projects with a land theme code are collaborative. Before 2014, staff working on land were housed in different sectors and the theme lacked visibility and a critical mass of staff. Staff working on land issues are now one global team and can easily learn from each other and from land interventions in all Regions. Clients in Côte d’Ivoire interviewed for this evaluation greatly appreciate World Bank support for the country’s complex land reform; this involves moving away from customary law by setting up and implementing the legal framework for modern land titling under a single, harmonized approach. World Bank support involved studies, capacity building, knowledge sharing, new technologies (for example, for land information and mapping land titles), and strong implementation support. The World Bank’s expertise is having transformational impact, according to clients. Source: Independent Evaluation Group. knowledge in a database that supports operations. The Environment and Natural Resources and Poverty GPs also have strategic approaches that identify knowledge gaps and create new knowledge. The Poverty GP houses the Secretariat for SCDs, which engages other GPs and CMUs to prepare SCDs. SCDs fill important knowledge gaps in many countries and help synthesize existing analytical work. GPs use a variety of knowledge sharing practices to support operational needs. Some GPs assign “quality assurance” staff to ensure that operational lessons and global knowledge are incorporated into operations; according to staff in the Regions, it would be useful if all GPs did this. Many GPs use help desks, safe-space meetings, and other platforms to connect task teams to experts and knowledge resources. These platforms identify knowledge gaps, create inquiry typologies, and log information. The goal is to create systematic long-term knowledge based on observed needs. For example, the Water GP’s well-used “Ask Water Help Desk” directs task teams to the knowledge resources they require, with a target response time of less than 48 hours. The Social, Urban, Rural, and Resilience GP’s leadership emphasizes knowledge and stimulates informal, bottom-up learning through knowledge communities called Knowledge Silo Breakers, nested under the GSGs. It supports these knowledge communities by hiring dedicated knowledge management specialists; empowering its GLs; and emphasizing that best practices from around the globe should inform operations. Some GPs like Water and Energy and Extractives use informal “safe-space” meetings to promote learning on difficult and sensitive matters and prepare newsletters on operational issues. The Health, Nutrition, and Population and Social Protection and Jobs GPs offer flagship Independent Evaluation Group | World Bank Group 19 courses, which are well attended by clients. The Health, Nutrition, and Population GP also piloted “Project Handover Checklists” and hosts a monthly “Knowledge Café,” similar to a townhall event, open to all staff. There are other examples of knowledge innovation within most GPs, too many to comprehensively list here. Knowledge work in some GPs is less strategic. Some GPs focus on learning-by-doing and tacit knowledge flow to support operations but do not emphasize generating or curating knowledge, innovation, client training, or global thought leadership. These GPs often deliver large lending volumes. The Transport and Digital Development GP, for example, has struggled to formulate its knowledge priorities, although more recently it has helped create new global partnerships on digital development and sustainable mobility. The Macroeconomics, Trade and Investment GP offers technical training courses to country economists and prepares SCDs but lacks the funds to finance major new analytical work for global audiences. The Energy and Extractives GP also emphasizes operations-focused learning. The new operating model established GSGs as the prime knowledge communities. GSGs pool and curate global and country-specific knowledge, and then channel it to country needs. At last count a few years ago, there were approximately 90 GSGs, led by 104 GLs, with about 15 to 25 active members in each, though some GSGs have upward of 250 members. Some GSGs were previously communities of practice (CoPs) under the old model and evolved into GSGs. GSGs represent GPs’ current or emerging core business areas and are meant to have operational relevance, being involved potentially across ASA, lending, and convening activities (box 3.2). Most GSGs are housed within individual GPs though many have members from multiple GPs. Example of Operationally Relevant Knowledge Collaboration Box 3.2 |  Natural disasters pose unique challenges that require integrative solutions. A Global Solutions Group on Resilience and Disaster Risk Management provides these solutions by brokering knowledge and initiatives. The GSG is composed of several smaller knowledge communities. One is focused on hydrological and meteorological hazards; another, called “Responding to Disasters Together,” brings staff together from the Social, Urban, Rural, and Resilience GP’s disaster risk management team, Finance, Competitiveness, and Innovation’s Disaster Risk Financing and Insurance program, and the Social Protection and Jobs GP, to promote comprehensive and collaborative disaster responses. The “Responding to Disasters Together” knowledge community links social protection systems with post-rapid-onset disaster assistance. The purpose is to use established systems to target people vulnerable to disasters for long-term disaster recovery. Source: Independent Evaluation Group. 20 Knowledge Flow and Collaboration | Chapter 3 GSGs have performed unevenly, with few consistent mechanisms to channel knowledge into country programs. Interviews suggest that, in most cases, GSG knowledge collaboration is executed through GLs’ personal contacts. Most CMU staff interviewed by IEG were either not aware of GSGs’ existence or questioned their relevance. In a survey of GSG members, 58 percent of respondents said they have reached out to individuals on different project teams. This is encouraging, but it also means that over a third of GSG members have not collaborated with other projects, let alone other GPs, on their particular GSG theme. The same survey shows that GSG members usually spend 5 percent or less of their time on GSGs, which seems low: GSGs were intended to be core mechanisms for knowledge collaboration. Locally recruited staff in country offices reported in focus groups that they use GSGs little. Global Leads There is much variation in how GPs use and resource the GLs and much variation in how they perform. The GLs are knowledge brokers that integrate issues, influence lending, support operations, and connect GP knowledge across Regions. GLs lead GSGs and were expected to guide the World Bank’s knowledge work in thematic areas or business lines, often in addition to other tasks not linked to the GL role. GPs have implemented the GL model differently, with much variation in mandates, budgets, and how much time these staff were given to perform the GL role. Time spent in this role varied from 10 percent to full-time, depending on how the GP funds the role. Some GPs treat their GLs as part of their management teams and expect them to help guide the GP’s knowledge creation and technical training of staff; others do not. Interviews with GLs, GPs, and CMUs show that when they function well, GLs can be powerful conduits of knowledge. Some GLs proactively identify knowledge gaps; some create new knowledge; some represent the World Bank at major international events. But their functioning and performance have been very uneven, and it is sometimes unclear how the GLs are supposed to channel global knowledge into country operations, apart from country visits by the GL in person. The GLs are constrained by unclear roles and unfunded mandates. Interviews reveal confusion as to whether a GL is an assigned role or a codified staff position. In a 2015 survey of GLs, half said they were unclear of their role, and nearly all agreed they needed more clarity on responsibilities. Some staff expect GLs to be mainly thought leaders; others emphasize their role in brokering, connecting, and leading knowledge communities. These are essential roles because tacit knowledge, embedded in staff’s experiences, is important for the World Bank’s effectiveness. But the knowledge broker role is a difficult one. Many GLs expressed this concern in interviews with IEG, saying they lacked time, sufficient budgets, and a clear work program to carry out their role effectively. GP managers agreed that a lot is expected from GLs, but GLs lack the budget or authority to be fully effective. Also, most GPs have only few knowledge management staff to support GLs. Perceptions of the effectiveness and usefulness of GLs as knowledge brokers varies widely. GP staff view GLs far more positively than CMUs. For example, one GP staff member said, “Some Global Leads are better than others and therefore it depends a lot on the individual, but the premise is Independent Evaluation Group | World Bank Group 21 a good one if the right person is in the role.” GP staff also said that different GPs expect different things from GLs and that different GLs fulfill the function in different ways. CMUs almost invariably expressed concerns or negative perceptions of the GL role. Some CDs said they did not know or notice that GLs existed. PLs were more familiar and appreciative, typically of a few specific GLs, but expressed concern that appointing technical experts as GLs makes them less available to clients. A quote illustrates CMUs’ concerns: “The Global Leads do not have visibility, and I would say, many are not really leaders in their field. Global Leads have not been an effective model for building expertise.” Not all individuals selected for the GL role were thought leaders in their fields (although having the skills to build knowledge communities is also important for this role). Some GLs are highly visible and seen as the “go-to” person within their technical field, but most are not. This is according to IEG’s interviews and observations, though metrics to assess GLs’ performance do not exist. GLs’ academic credentials are uneven. Although acknowledging that this is a narrow indicator of academic or technical expertise, in a random sample of 52 GLs, 19 (37 percent) had authored few or no scholarly publications, and 22 (42 percent) had authored 11 or more publications, according to Google Scholar’s broad publication listing. Figure 3.1 shows both that GLs’ publication record is very uneven, and that GLs in the Human Development GPs tend to have more evenly stronger academic credentials. However, the GL role requires a broader mix of skills than that of a researcher. Communication and integrator skills are critical to performing a connector role and successfully building knowledge communities, and, naturally, in practice these skills vary across individuals. FIGURE 3.1 Global Leads’ Records of Publication as an Indicator of Their | Research Contributions 40 Publications per Global Lead (no.) 35 30 25 20 15 10 5 0 1 5 10 15 20 Global Leads (no.) Human Development Equitable Growth, Finance, and Institutions Sustainable Development Source: Independent Evaluation Group based on data from Google Scholar. 22 Knowledge Flow and Collaboration | Chapter 3 Several GPs have revised their approaches to knowledge. The GL role has been recast or disbanded in some GPs because of low budgets and dissatisfaction with the model. Many vacant GL roles have been left unfilled. Some GPs, including the Governance GP, have reviewed and updated their approaches to knowledge flow. The Governance GP chose to disband the GSG and GL model completely, replacing it with more CoPs in a revamped Knowledge and Learning architecture. In contrast to the GL-led GSGs, the new CoPs have budgets, clear terms of reference, and support from the GP’s knowledge management team. PMs will oversee the CoPs, presumably to better align knowledge and management structures. The former Finance and Markets GP also replaced GL-led GSGs with manager-led CoPs, leveraging trust funds to support them. The Education GP divided the Basic Education GSG into four thematic groups to support knowledge exchange and delve deeper into subthemes. Meanwhile, GLs continue in their roles in other GPs. On the whole, GLs have not performed the intended role; this is hardly surprising given unclear and unrealistic expectations and few resources. It is positive that many GPs have rethought this role. In hindsight, the change team may have incorrectly assumed that thought leaders, like GLs, could be appointed. Thought leaders emerge from the ranks, sought out by peers for advice and mentoring. It is useful when management recognizes, empowers, and provides resources to these individuals, but merely creating a position does not ensure thought leadership. It is also unclear why so many GL positions were created in the first place also in light of the resource constraints. Global Themes The GTs provide coherent approaches to the World Bank’s cross-cutting priorities. As such, they have been valuable additions to the operating model. They align with major corporate priorities where the World Bank seeks external visibility and internal coherence. They have three broad functions: setting strategic directions internally; supporting operational mainstreaming; and engaging and convening external partners. All five GTs perform all three functions, but to different degrees. Interviews for this evaluation suggest that these functions are broadly useful and appreciated by staff. The GTs vary in size, from 27 staff in the Gender Group to 232 staff in the Climate Change Group. Interviews suggest it has at times been hard to recruit staff for GT positions, because staff perceive career advancement more difficult in GT positions than in GP positions. Setting strategic directions internally has been useful. The Bank Group has strategy documents and action plans guiding its work on gender; climate change; fragility, conflict, and violence; and Maximizing Finance for Development (MFD). All GTs offer training, knowledge resources, and analytical tools such as fragility assessments, jobs diagnostics, gender assessments, Infrastructure Sector Assessments, climate risk screening, and more. IEG encountered many examples where these tools and services informed country programs, albeit with some variations in uptake and visibility. The Gender Group, for example, has contributed to improved gender-disaggregated data and evidence. All GPs and Regions have action plans for integrating the World Bank Group Gender Strategy into their operations and dedicated gender focal points. Factors that contribute to this relative success include resource availability, strong senior management support, and a common Independent Evaluation Group | World Bank Group 23 vision enshrined in the World Bank Group Gender Strategy (FY16–23). The Climate Change team oversees the corporate requirement to track climate co-benefits and provides tools, advice, training, measurements, and monitoring to climate change projects. GTs support operational mainstreaming extensively, but the quality of this support is hard to assess because metrics are lacking. The GTs support operational mainstreaming by identifying strategic directions, conducting gap analyses, identifying entry points for operations, contributing to project preparation, connecting teams to global expertise, providing tools for measurement and monitoring, and helping meet corporate requirements. Staff perceive GTs as strong units for collaboration because they are barred from leading operational work and, therefore, need to sell their knowledge services in cross-support agreements. Interviewed staff appreciate when GTs provide thematic insights to country tasks and integrate strategic concepts into operations. However, demand far exceeds supply, and it can be challenging for managers to ensure that GT contributions are of adequate depth and customized to country and sector contexts. An example of inadequate depth is when GTs add a paragraph to a project document just to “include it,” but the theme may be completely left out of project implementation. Moreover, country offices sometimes find it difficult to locate the right expertise within a GT or identify the authority on a given topic. GTs also extensively engage and convene external partners, another useful role. All five GT Groups manage partnerships, trust funds, and external engagements. They serve as focal points for outside organizations to engage the World Bank on these themes. The Fragility, Conflict, and Violence Group, for example, partnered with UN agencies, helped double International Development Association (IDA) resources for fragile contexts, increased the GT’s focus on forced displacement, established a new Risk Mitigation Regime, identified a new approach to fragility, supported conflict prevention and famine response, and more. The Climate Change Group represents the World Bank in international climate negotiations and policy discussions, using flagship reports as part of its convening efforts. These convening functions are important from a corporate viewpoint as they contribute to the Bank Group’s visibility, influence, and fund raising but were not always visible to the operational staff interviewed for this evaluation. Quality Assurance There are persistent concerns with the quality assurance process for operations and for ASA services, though no hard data exist. In the former operating model, Regions had greater responsibility for providing quality control over operations, through development effectiveness units tasked with quality assurance. In the current operating model, this quality assurance moved to the GPs, specifically PMs. Now, there is a concern that the CD has insufficient oversight role or accountability during implementation and that the weakening of the Regions’ Development Effectiveness units left a gap in quality control, which GPs have not sufficiently filled. In terms of balancing the matrix’s GP and Regional sides, the pendulum may have swung too far to the GPs on quality assurance. However, there is no independent mechanism or data on operational quality. 24 Knowledge Flow and Collaboration | Chapter 3 Interviews show many staff believe contestability has decreased in the current operating model compared with before. In this context, contestability means airing and candidly debating diverse and conflicting viewpoints over the relevance and quality of products. GP review processes often do not include representation from GPs not leading the product under review. A representative quote from a CD illustrates the concerns: “There is no longer any contestability. In the old system the Sector Director would invite people from other Regions for technical review and would be the interlocutor. Now it is very difficult for GP staff to criticize others from the same GP since quality assurance resides within each GP.” Staff speculated that peer reviewers may not candidly challenge an approach if they know their Director favors it. Staff argued that more independent peer reviews would enhance contestability and improve the quality assurance process. Other Mechanisms for Knowledge Flow South-South knowledge exchanges, facilitated by GPs, are useful for transferring knowledge and experience among low and middle-income countries. The South Africa CMU brought Chile’s Minister of Finance to South Africa to discuss Chile’s approach to financial reforms and natural resource management. The World Bank has worked with Mexican officials on South-South exchanges in tax, gender, and green growth. When Morocco was looking to introduce a national identification (ID) system, a World Bank team arranged for specialists in India’s national ID system to visit Morocco, and Moroccan officials to visit India, to learn about the Indian system. Says one CMU staff member, “As a result, we feel we have state-of-the-art knowledge about having a unique ID system.” In another example, World Bank staff helped Chinese officials organize a visit to Costa Rica to study health insurance. Also, through a South-South exchange, Brazil and Costa Rica studied each other’s social protection systems. Clients consulted in Côte d’Ivoire said regional knowledge exchanges had been beneficial, for example on agriculture, nutrition, and adaptation of coastal areas to climate change. Advanced countries in East and Southeast Asia are seeking knowledge-based relationships with the World Bank and have asked for help exporting their development know-how to the world, including low-income countries. Knowledge hubs exist in World Bank country offices in Chile, Korea, Malaysia (which IEG visited for this evaluation), and Singapore (box 3.3). Online knowledge curation platforms are not always coherent or easily navigable. Reform proposals from 2013 to advance knowledge management and set out stronger expectations and accountabilities for staff’s and GP’s knowledge performance did not advance. There is still no World Bank-wide standard knowledge repository with effective tagging to organize knowledge. Different information technology platforms for internal knowledge have been installed but failed. Activating personal networks to inquire about experts and resources remains predominant. Many interviewed staff argued that this approach should be complemented by more systematic, platform- based knowledge curation. There has been recent progress on such platforms; for example, with automated “Pre-Design Knowledge Packages” that quickly curate operationally relevant knowledge to operational teams. The World Bank Knowledge Management Action Plan, discussed by the Board in October 2017, promises further advances on technology platforms. However, questions Independent Evaluation Group | World Bank Group 25 The Hub Model of Knowledge Transfer: Malaysia Box 3.3 |  The Malaysia Global Knowledge and Research Hub provides outbound knowledge (carries Malaysia knowledge and experience to other countries), inbound knowledge (provides advisory services to Malaysia, including through Reimbursable Advisory Services (RAS)), and supports research. The hub is unique because it is fully funded by the government and includes staff from the Development Economics unit, who are not normally located away from headquarters. Malaysian government counterparts appreciate how the hub accelerates outbound South-South knowledge transfer, including to several IDA countries. They value the hub’s ability to mobilize international expertise and ensure timely and continuous dialogue. World Bank in-country staff, including local and international experts from five GPs as well as Development Economics (DEC), translate inbound knowledge into local solutions. However, there are still opportunities to better contextualize the knowledge and make it more demand-driven. Overall, defining, measuring, and managing a successful hub model is complicated. Demand from the government tends to be sector and counterpart specific and collaboration within the World Bank does not systematically happen. There are silos on both ends, which may cause the hub to deliver below its full potential. There are also calls for the hub to serve a wider range of stakeholders, such as universities and subnational governments. Source: Independent Evaluation Group visit to Malaysia. remain, including which knowledge platforms to use, how to balance centralized and decentralized knowledge management, and how to produce knowledge more selectively. DEC and Regional Chief Economist units make notable contributions of global and regional knowledge. DEC and Chief Economist units are the authors of many of the most downloaded World Bank documents. World Bank data has earned a strong reputation and data-related web pages account for much of the traffic to World Bank web sites (World Bank 2017b). The relationships between DEC and the operational arms of the World Bank have long been discussed, with relatively weak linkages a persistent theme in internal discussions and external reviews of DEC (Besley and others 2015). The reforms abolished some mechanisms for dialogue between DEC and operational units such as the sector boards; apart from this, this evaluation did not find evidence that the interactions between DEC and operations have changed much. Senior Management is considering options for closer linkages. A limited number of interviews done for this and other IEG evaluations1 suggest that operational staff and managers tend to interact with relevant 26 Knowledge Flow and Collaboration | Chapter 3 DEC colleagues and to be familiar with data and research they produce. Direct cooperation occurs; DEC researchers are required to spend 30 percent of their time providing cross support, a higher share than any of the GPs and GTs. GLs share relevant DEC publications with GSG members, but, echoing earlier reviews, these staff tend to question the relevance of some DEC research (but not DEC data and flagship reports) to their work. There may be advantages to maintaining arms-length relationship between research and operations. The World Bank has a long-standing tradition of open, independent, credible, and high-quality research on a wide range of development issues, exemplified in DEC’s and Regional Chief Economist’s work which has been a driver of many intellectual agendas. This is made possible by having a research group that can recruit, retain, and nurture top research talent. Arguably, the independence of DEC and Regional Chief Economist research and thinking from operational pressures is a core strength of the institution: it retains accountability. For example, the approach behind the successful engagements in the Arab Republic of Egypt after the Arab Spring cited in this evaluation can be linked to a regional Chief Economist’s paper (Devarajan 2016) helping to position the Bank Group as supporting the Middle East and North Africa’s move toward a new social contract. The idea of supporting a new social contract became the dominant theme of the regional strategy and the CPFs, guiding a reorientation of the Region’s lending and analytics. The lesson for management when considering options for closer linkages is therefore to avoid jeopardizing core strengths—such as rigor, openness, independence, credibility—of DEC’s and Regional Chief Economists’ work. Drivers of and Barriers to Knowledge Flow Leadership drives knowledge flow and collaboration. This is true for both in-country and GP leadership. CDs can mandate collaboration through the CPFs design or when assigning projects to various GPs. Likewise, there is a direct link from GP leadership commitment to effective knowledge functions. Certain traits are consistent among GPs with effective knowledge functions; specifically, their leadership shows an active interest in knowledge generation and technical expertise, emphasizes sharing operational lessons, and encourages this behavior to staff. For example, the Social, Urban, Rural, and Resilience GP’s leadership strongly emphasizes knowledge and collaboration, and stimulates learning through a variety of mechanisms, as already mentioned. In other GPs, there is a link between effective GSGs and GLs and strong managerial support. In these GPs, GLs participate in management meetings, inform strategic discussions, receive dedicated funding, and help identify needs and technical skill gaps. One example is the Environment and Natural Resources GP, which funds its GLs on a full-time basis and expects them to contribute to strategy development. Larger country programs can attract global knowledge more easily than smaller ones. Interviews and country visits find a pattern where the Bank Group appears to serve knowledge needs better in countries with larger programs (such as Brazil, China, Egypt, or Pakistan) than in countries or regions with smaller ones (such as Azerbaijan, Botswana, Bulgaria, Central America, or Sri Lanka). Independent Evaluation Group | World Bank Group 27 Box 3.4 |  Examples of Global Knowledge in Country Programs The Arab Republic of Egypt, a country with a large program, has strong need and client demand for World Bank Group support that is geared to helping them address cross- sectoral issues; that involves both diagnostics and implementation support; and that is integrated across organizational boundaries in government and in the Bank Group. Addressing problems rather than promoting lending has helped the World Bank and IFC become trusted advisers. Large multisector work is difficult because it requires the client to work across governmental boundaries; but the Bank Group can have greater impact in sectors where the reforms enjoy strong client ownership. Examples where the Bank Group is addressing cross-sectoral issues include: fiscal decentralization; land allocation and land acquisition; urban transport; decentralization and local development; competitiveness; links between energy subsidies and macrofiscal management; and social housing. Clients appreciate the World Bank’s knowledge functions. Clients emphasize that Advisory Services and Analytics sometimes is more successful than finance and is of good quality. Some clients report that the World Bank has become faster and more responsive than before. They seek Bank Group involvement in designing policy and regulatory frameworks, and in implementing change. Clients recognize the strong quality of Cairo-based staff and do not perceive a difference between staff based in Cairo and staff elsewhere. In Côte d’Ivoire, clients appreciated World Bank support for nutrition, health reform, land reform, among others. World Bank support has been crucial for both technical and political aspects of health reform, according to clients: team leads had the right expertise and ability to connect to a wider pool of health financing expertise in the World Bank. Support involved advocacy, policy dialogue, and persuasion across multiple ministries and has led to a paradigm shift and complex reform, including also links with the social protection system to set up a common targeting system. Support also involves a nutrition project which embeds a community approach, informed by international experience that the World Bank is helping to adapt to the country. Most staff interviewed by IEG on this smaller country program said that access to GLs’ and GTs’ expertise was not easy and that they saw little use in GSGs. In Azerbaijan, IEG found client demand for specialized global knowledge but not for complex or multisectoral projects. As a smaller country program, it struggles to attract expertise, especially for ASA and small lending projects. The staff of the Country Management Unit reported relying on personal networks to mobilize expertise. Customizing knowledge relies on the ability of Country Management Unit staff to translate and localize ideas from elsewhere. Source: Independent Evaluation Group country visits. 28 Knowledge Flow and Collaboration | Chapter 3 For example, clients in Pakistan interviewed by IEG look to emulate other countries and see the World Bank as the prime organization that can bring them that international expertise. The GPs are responsive to client requests, often mediated by the CMU, according to IEG’s visit. Clients in other countries visited by IEG (see box 3.4) also greatly value the World Bank’s knowledge support, but, according to CMU staff, it is hard for countries with small programs to attract expertise. They explained that the incentives for GPs to create and manage knowledge largely come from how much lending demand the knowledge work stimulates. Access to trust fund resources enables global knowledge creation and knowledge management. CMUs tend to fund ASA products with residual resources, after first funding lending programs. By contrast, external trust funds finance much of the ASA and the larger knowledge agenda. Trust funds financed two-thirds of the Bank Group’s total ASA work. In fact, trust funds contributed to a 34 percent increase in ASA products from FY13 to FY17. For example, the Energy and Extractives, Health, Nutrition, and Population, and Education GPs use large trust funds to increase knowledge work, including work done in partnership with external partners. Trust funds provide over 70 percent of the global engagement budgets (the portion of the GP’s budget not tied to country operations) and finance many activities on data, outreach, partnerships, GLs and GSGs, and knowledge production, curation, and management (figure 3.2). Trust funds are often not aligned with World Bank priorities and add to transaction costs. This is according to IEG’s Trust Fund evaluation and internal World Bank management documents on trust fund reforms (World Bank 2011). It is hard for management to produce, curate, and apply knowledge FIGURE 3.2 Trust Funds as a Percentage of Total Global Engagement | Expenditures Percent World Bank total 71 Global Themes 34 Human Development 59 Sustainable Development 76 Equitable Growth, Finance, and Institutions 77 Source: Independent Evaluation Group, based on data from the Budget, Performance Review, and Strategic Planning Vice Presidential Unit. Independent Evaluation Group | World Bank Group 29 strategically when external finance is guided by different priorities; therefore, ongoing trust fund reforms aim for greater alignment. Trust funds may also have country eligibility restrictions that limit the ability to curate knowledge in middle-income countries. Many trust funds still focus on niche issues or programs that are no longer sector priorities. These “legacy funds” create supply-driven knowledge that deviates from sector or country priorities, making them hard to manage and causing fragmentation (World Bank 2011). Some CDs complain about having insufficient oversight to ensure that trust funds are used strategically. Some GPs and GTs have tried to make trust fund use more strategic. For example, the Water GP merged its trust funds, mainstreamed priority topics, and aligned knowledge creation with operational needs. The Health, Nutrition, and Population GP approach was to identify knowledge gaps, by country and subsector, and use trust funds to address these gaps. In Egypt, the United Kingdom set up a trust fund specifically to implement Egypt’s CPFs, an example of clear alignment with country programs. Management recognized trust fund misalignment as an issue and is pursuing reforms to encourage that their use is more strategic, efficient, and disciplined. 1 These other evaluations include Data for Development (World Bank 2016) and ongoing unpublished work. 30 Knowledge Flow and Collaboration | Chapter 3 4 highlights 1 There is demand for integrated solutions that draw on global knowledge. The World Bank can provide these solutions because of the leadership of CDs, who are supported by Collaboration Program Leaders. for Integrated Solutions 2 Silos across Regions have decreased, as evidenced in increased interregional cross support. 3 However, silos across GPs have increased, and lending has become less multisectoral. 4 The GPs’ incentives continue to favor own- managed lending, leading to competition and transaction costs. 5 Mechanisms for working across GPs and between GPs and CMUs exist but are not always strong enough. Groups and roles tasked with integrating and arbitrating across boundaries sometimes lack the authority to do so. Independent Evaluation Group | World Bank Group 31 T H I S E VA LUAT I O N finds many positive examples of World Bank collaboration across boundaries to deliver integrated multisector or multiservices solutions to clients, enabled by CDs’ leadership. But collaboration across boundaries is difficult and incurs high transaction costs. The model’s structure and incentives do not support collaboration. Integrated Solutions in Country Programs Integrated solutions are relevant to many of the development challenges faced by clients. Integrated solutions do not necessarily mean that individual projects need to be multisectoral or have multi-GP teams but also include multisector approaches (programs or platforms that combine interventions). Also, stakeholders were often keen to note that single-sector interventions are appropriate or even preferable for many other development issues. Of course, single-sector approaches need to take account of linkages or effects on other sectors. Demand for integrated solutions is either expressed by clients, particularly in middle-income countries, or generated through policy dialogues or World Bank analytical work, and then accepted by clients; this finding validates the assumptions about client demand behind the operating model (see theory of change in table 1.1.). Clients interviewed by IEG value the World Bank’s expertise and willingness to shepherd complex reforms across different parts of government. Clients said they value World Bank services that draw on global knowledge and convene government agencies and development partners. CMU staff said the same. For example, one CD said, “We have enormous demand from clients for cross-sectoral work.” Rwanda’s President asked for cross-sectoral support to reduce stunting. Also, CDs value the GPs that collaborate best with other units. The Bank Group’s country engagement model is often used to define objectives that require integrated solutions, and a large and growing share of objectives will require cross-GP collaboration or approaches to deliver. The country engagement model includes SCDs and the CPF, both of which are composed collaboratively by GPs, CMUs, and client counterparts. This makes the SCD and CPF useful mechanisms for finding common strategies among units. SCDs and CPFs are not required for integrated solutions; other documents can be used as well. For example, the Pakistan CMU used the Program and Learning Review, which is a CPF midterm review. IEG’s review of country programs shows that most country programs have several CPF objectives that plan integrated multisectoral approaches. Also, a large and growing share of CPF objectives will require cross-GP collaboration. Out of 151 CPF objectives in 18 country programs for which comparable data are available, 69 percent are objectives that, in IEG’s judgment, require cross-GP collaboration (figure 4.1). This is an increase from 28 percent in the previous strategy cycle for those same countries. Some of the subsectors that commonly have CPF objectives related to integrated solutions, include the following:   City planning and transportation systems;   Private sector competitiveness and fostering employment opportunities; 32 Knowledge Flow and Collaboration | Chapter 4   Human capital development combining interventions in education, health, jobs, and skills; and   Nutrition and Early Childhood Development. A smaller share of CPF objectives include integrated solutions that will require GP collaboration across Practice Groups. Out of a 151 CPF objectives in 18 reviewed CPFs, 20 percent frame objectives that, in IEG’s judgment, require collaboration or approaches that span across Practice Groups. This is an increase from 10 percent in the previous cycle for those countries. These types of objectives include:   Interventions  to improve public services for vulnerable or excluded groups that combine social services with access to finance, infrastructure, or productive opportunities    Interventions for lagging regions that improve access to markets, infrastructure, public services, financial services, rural livelihood assets, or employment opportunities. CMUs use different mechanisms to promote integrated solutions. For example, the Pakistan team set up broad engagement “platforms” to facilitate cross-GP collaboration on stunting, financial inclusion, and urban development in Karachi. These platforms, informed by analytical work, bring together multiple Bank Group teams and government counterparts to develop financing and knowledge services to address development problems (box 4.1). These platforms have been grafted onto local initiatives and have strong, high-level client support, either from provincial government or a federal agency, as IEG learned from a country visit. The Morocco team also uses cross-cutting platforms, or FIGURE 4.1 Country Partnership Framework Objectives Requiring Collaboration | (number) 178 All reviewed objectives (18 countries) 151 ..of which will require cross-GP 50 collaboration 98 ..of which will require collaboration 17 across Practice Groups 30 0 20 40 60 80 100 120 140 160 180 200 Number CPF (current period) CLR (previous period) Source: Independent Evaluation Group based on analysis of 18 CPF and Country Learning Review documents for which data could be compared. Independent Evaluation Group | World Bank Group 33 “task forces,” to bring GPs together around youth employment issues in lagging regions. These “task forces” receive additional budgets. Internal competitive pressures complicate CDs’ efforts toward selectivity within country programs. “Selectivity” in this context refers to the process of choosing or rejecting projects or programs depending on how coherent they are with the SCD. In interviews with CMUs and budget staff, it was a recurring theme that country programs tend to contain too many tasks for the budget and that CDs felt pressure to assign tasks to all GPs for the sake of maintaining cordial relationships. Some CDs Platforms for Multisector Engagements in Pakistan Box 4.1 |  The Stunting Platform builds on an earlier nutrition project and the Pakistan Poverty Diagnostics for Water Supply, Sanitation and Hygiene (WASH poverty diagnostic). This was jointly led by the Water and Poverty GPs. The diagnostic identified open defecation as the main cause of stunting among small children and has been instrumental to advancing the dialogue on nutrition and stunting. It also facilitated synergies among teams working on health, water, and agriculture. Under this platform, the World Bank financed two multisectoral nutrition projects in Sindh, the province with the worst stunting indicators. Regular dialogue between the CD and the Chief Minister of Sindh, and strong support from the Global Lead on Nutrition, are noteworthy elements of this intervention. The Karachi Platform builds on the Karachi Diagnostic Study, a nonlending technical assistance led by the urban team and comprising fiscal, inclusiveness, institutional, competitiveness, citizen participation, and service delivery assessments. The platform aims to make Karachi a more livable and sustainable city through institutional reforms, service delivery improvements, financial sustainability of local government agencies, and neighborhood development to make the city center pedestrian friendly. All three Practice Groups, IFC, and Multilateral Investment Guarantee Agency are involved. A Program Leader brought all participating GPs together for a two-day workshop with country stakeholders. The Financial Inclusion Platform sprang out of prior analytical work by the World Bank. It led to a national financial inclusion strategy, financed by the State Bank of Pakistan as a reimbursable advisory service (RAS). The RAS was led by a Program Leader with a team including IFC, Trade and Competitiveness, Financial Management, Finance and Markets, and Social Protection. From this platform, the World Bank and IFC provided support in strategy, investments, analytics, and technical assistance to improve poor people’s access to financial services. 34 Knowledge Flow and Collaboration | Chapter 4 said that to maintain the selectivity of lending programs, they assign analytical tasks more widely across GPs to make them feel included in country programs. Also, some PMs and Directors were concerned that competition makes PLs partisan arbiters, who may direct business to their “home GPs.” As a positive, though, IEG’s evaluation of the Bank Group’s country engagement framework (World Bank 2017c, 33) found “modest improvement” in the focus and quality of the current CPF compared with the previous strategy documents, and “clear indications of stronger selectivity,” but also room to enhance selectivity. Integrated solutions often require analytical, convening, and lending work. Compared with single- sector operations, integrated solutions involve multiple agencies from different sectors and often entail higher political and other risks to clients. To navigate these risks, integrated solutions often start with analytical work to identify the issues. These solutions cut across sectoral and organizational boundaries, requiring the World Bank to convene different agencies and stakeholders. Lending operations may follow, helping clients implement complex reforms. There are many examples across the World Bank combining knowledge, financing, and convening activities to deliver integrated solutions, far too many to list here. Patterns of Collaboration Silos across Regions have decreased, as evidenced in increased interregional cross support. GP staff in the key technical grades GF–GH in FY17 provided 6.6 percent of their time as cross support to Regions other than the one they are primarily working in. This is much more than before. According to IEG (World Bank 2011, 54 [figure 3.4]), staff mapped to Regional VPUs in FY10 spent 0.9 percent of their time providing cross support to other Regions (figure 4.2). Nevertheless, staff still provide more cross support to other GPs than to other Regions (figures 4.2 and 4.3). This was true in the previous model as well; establishing GPs has attenuated but not eliminated the focus on one Region for many staff. Practical considerations of client relationships and travel costs may explain this. But silos across sectors have increased as evidenced by decreased intersectoral cross support, among many other things. In FY17, GP technical staff in grades GF–GH spent 9.9 percent of their time supporting tasks mapped to other GPs. This is less than before. In FY10, staff in Regional Sector Management Units provided 15.6 percent of their time as cross support to other Sector Management Units (figure 4.3). In the 2017 employee engagement survey, 84 percent of staff agreed that “colleagues within the unit cooperate to get work done,” and 67 percent agreed that “colleagues collaborate effectively across different units.” Both are slight improvements from declines in 2015 and 2016, when staff were first feeling the effects of reforms (see table 4.1), pointing to an upward trend in staff perception of collaboration. Interviews with staff tended to dwell on collaboration challenges while also acknowledging some successes. World Bank lending may have become less cross-sectoral. This is suggested by analysis of the number of sector codes associated with all approved loans, although changes in sector codes and Independent Evaluation Group | World Bank Group 35 FIGURE 4.2 Staff Cross-Support Trends | 18 16 15.6 Technical staff (grades GF-GH), 14 12 12.2 time spent (%) 10 10 9.9 8 6.5 6.6 6 5.3 4 2 0.9 0 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 Cross-region Cross-sector Sources: FY10 data: IEG 2011 table 3.6 and figure 3.4. The data includes four Regions and excludes Middle East and North Africa and Europe and Central Asia for which comparable data were not available at the time. See also note for table 3.6. FY15–17 data: Standard Reports: Matrix VPs Memorandum of Understanding. FIGURE 4.3 Staff Cross Support, by Global Practice | 18 Staff (grades GG-GH) time spent (%) 16 14 12 10 8 6 4 2 0 FCI GOV MTI POV EDU HNP SPL AGR EEX ENV TDD SURR WAT EFI HD SD PG PG PG Global Practice Cross-region Cross-GP Source: Standard Reports: Matrix VPs Memorandum of Understanding. The underlying data are based on time recording by staff in professional nonmanagerial grades (GF–GH). Note: AGR = Agriculture; EDU = Education; EEX = Energy and Extractives; EFI PG = Equitable Finance and Investment Practice Group; ENV = Environment and Natural Resources; FCI = Finance, Competitiveness, and Innovation, and GOV = Governance; HD PG = Human Development Practice Group; HNP = Health, Nutrition, and Population; MTI = Macroeconomics, Trade, and Investment; POV = Poverty; SD PG = Sustainable Development Practice Group; SPL = Social Protection and Labor; SURR = Social, Urban, Rural, and Resilience; WAT = Water. 36 Knowledge Flow and Collaboration | Chapter 4 how they are assigned complicate comparisons over time. For example, lending with education codes that also contains codes from other sectors dropped from 63 percent in FY12–14 to 54 percent in FY15–17. The share of lending with energy sector codes that also contains codes from other sectors fell from 50 percent to 42 percent over the same period. This decline in multiple sector codes in projects is observed for all sectors, except for information and communication technology, where the ratio increased from 76 percent to 86 percent (figure 4.4). GPs collaborate more on analytical work than on lending operations and collaborate more internally than with other GPs. A social network analysis was used to measure collaboration. Data show that the GPs that tend to lead many joint projects, both lending and ASA projects, also tend to provide cross support to projects led by other GPs. The Social, Urban, Rural, and Resilience GP is a prolific leader and contributor to joint lending projects (figure 4.5), whereas the Governance GP supports the most lending projects. (This is true even though the data filter out routine safeguards and procurement support, activities typically carried out by these two GPs). Figure 4.6 shows that the ASA portfolio has more diverse collaboration patterns—143 connections between GPs out of 156 possible pairs—than the lending portfolio, 117 connections out of 156 possible pairs. The Equitable Growth, Finance, and Institutions GP frequently collaborates on ASA, showing 30 or more FIGURE 4.4 Multisectoral Projects in Each Sector | 100 90 80 Multisectoral projects (%) 70 60 50 40 30 20 10 0 l cia T n n h n s F n SW S ive tio tio alt io AF io IC IT n at ct He W ra rta ct na uc e st tra ot po Fi Ed i in Pr Ex ns m ial a Ad d Tr an c So ic gy bl Pu er Sector En Before reorganization (FY12, FY13, and FY14) After reorganization (FY15, FY16, and FY17) Source: World Bank projects data analyzed by the Text and Data Analytics team, Global Themes Knowledge Management. Note: The data show the share of lending that contains codes from other sectors. AFF = Agriculture, Fishing, and Forestry; ICT = Information and Communication Technology; ITS = Industry, Trade, and Services; WSW = Water, Sanitation, and Waste Management. Independent Evaluation Group | World Bank Group 37 FIGURE 4.5 Collaboration across Practice Groups in the Lending Portfolio | Note: Node size represents number of collaborative projects a GP leads. Node color represents Practice Groups. Edge size represents number of collaborative projects between GPs, those with 10 or more highlighted in red. AGR = Agriculture; EDU = Education; EEX = Energy and Extractives; ENV = Environment and Natural Resources; FCI = Finance, Competitiveness, and Innovation, and GOV = Governance; HNP = Health, Nutrition, and Population; MTI = Macroeconomics, Trade, and Investment; POV = Poverty; SPJ = Social Protection and Jobs; SURR = Social, Urban, Rural, and Resilience; TDD = Transport and Digital Development; WAT = Water. connections with Human Development and with Sustainable Development, but the latter two do not collaborate frequently. Equitable Growth, Finance and Institutions is the most active Practice Group in leading and contributing to joint ASAs, and Macroeconomics, Trade, and Investment, Social, Urban, Rural, and Resilience, and Governance are the most active GPs (figure 4.7). The social network analysis data are not normalized, so larger GPs tend to show more collaboration than smaller ones. Collaborative behaviors vary across GPs. According to the two-minute survey, the Social Protection and Jobs, Macroeconomics, Trade, and Investment, and the Poverty and Equity GPs had the most positive views about working as coordinated teams. Interviews reinforced this: The Social Protection and Jobs GP was seen as open to a joint GP model, and the Macroeconomics, Trade, and Investment and Poverty and Equity GPs were seen as leaders on cross-sectoral tasks, such as policy operations and diagnostic studies, including Public Expenditure Reviews. Cross support varies much across GPs. The groups that spent the most time supporting tasks led by others were Climate Change; Macroeconomics, Trade, and Investment; Agriculture; Poverty; and Finance, Competitiveness, and Innovation. The GPs that spent the least time supporting tasks led by others were Energy and Extractives and Education (figure 4.3). The two-minute survey shows that the Water, Governance, and Environment and Natural Resources GPs were perceived as the least effective at working in coordinated teams, though the reasons for this are unclear. 38 Knowledge Flow and Collaboration | Chapter 4 FIGURE 4.6 Collaboration across Practice Groups in the Advisory Services and | Analytics Portfolio Note: Node size represents number of collaborated projects a GP leads. Node color represents Practice Groups. Edge size represents number of collaborated projects between GPs, those with 30 or more highlighted in red. AGR = Agriculture; EDU = Education; EEX = Energy and Extractives; ENV = Environment and Natural Resources; FCI = Finance, Competitiveness, and Innovation, and GOV = Governance; HNP = Health, Nutrition, and Population; MTI = Macroeconomics, Trade, and Investment; POV = Poverty; SPJ = Social Protection and Jobs; SURR = Social, Urban, Rural, and Resilience; TDD = Transport and Digital Development; WAT = Water. Mechanisms that Make the Operating Model Function and Add Value CD leadership drives collaboration for integrated solutions. Interviews and country visits repeatedly show that collaboration across GPs depends on CDs, and that lending operations tend to include adequate cross-GP collaboration only when CDs and PLs insisted on it. Some CMUs have set up platforms for cross-sector collaboration. CMU staff insist uniformly that the operating model incentivizes GPs to embrace single-sector lending support more than multisector approaches; they ascribe this behavior to the competition among GPs to lead on lending. As a result, collaboration for integrated approaches must be driven by country programs, CDs specifically, who uniformly complained about high transaction costs. In interviews, CDs often described the GPs as seeking to lead on lending operations and unenthusiastic about cross-GP collaboration. Collaboration is strongest on initiatives with high visibility for senior management. This is according to interviews. There are several reasons for this. First, in prominent projects, GPs will receive visibility if they are involved even if in a supporting role. Second, prominent challenges with clear goals and important consequences reduce organizational infighting. Third, senior management is more likely to pay attention to these initiatives and require collaborative integrated solutions. To put simply, prominent initiatives seem to realign the incentive structure toward collaboration. This is partially why Independent Evaluation Group | World Bank Group 39 FIGURE 4.7 Leading and Supporting Projects, by Global Practice | a. Lending Portfolio 140 URS 120 100 Leading (no.) 80 SD_Ave MTI EFI_Ave AGR 60 FO SPL [ENV] 40 GOV HNP EEX WAT 20 POV HD_Ave EDU TDD 0 0 10 20 30 40 50 60 70 80 Contributing (no.) b. ASA Portfolio 350 MTI 300 250 SPL URS Leading (no.) SD_Ave 200 ENV FCI EFI_Ave GOV 150 HD_Ave WAT 100 POV EEX TDD HNP AGR 50 0 EDU 0 50 100 150 200 250 300 Contributing (no.) Note: AGR = Agriculture; EDU = Education; EEX = Energy and Extractives; EFI_Ave = Equitable Growth, Finance and Investment average; ENV = Environment and Natural Resources; FCI = Finance, Competitiveness, and Innovation, and GOV = Governance; HNP = Health, Nutrition, and Population; MTI = Macroeconomics, Trade, and Investment; POV = Poverty; SD_Ave = Sustainable Development average; SPJ = Social Protection and Jobs; SURR = Social, Urban, Rural, and Resilience; WAT = Water. situations of urgency, such as crises or disasters, bring out the best in the Bank Group: they are often the most prominent and visible engagements. Shared task team leadership has become common and is one area that improves collaboration. In FY17, 55 percent of projects had co-task team leaders (co-TTLs), up from 21 percent in FY12 (figure 4.8). Co-TTL arrangements are used to recognize, and sometimes compel, cross-GP collaboration. This can be illustrated with a representative quote from a CD who said, “We have a project on irrigation with the Water GP, but water affects the Agriculture GP. So, we need to ensure that they work together. We have a TTL from Water and another from Agriculture.” Co-TTLs often 40 Knowledge Flow and Collaboration | Chapter 4 FIGURE 4.8 Operations with Co-Task Team Leaders | 600 60% Projects with co-TTL (%) Lending projects (no.) 400 40% 200 20% 0 0% 2012 2013 2014 2015 2016 2017 Lending with Co-TTL All Lending Ratio of Lending with Co-TTL Source: World Bank projects data analyzed by the Text and Data Analytics team, Global Themes Knowledge Management. Note: TTL = task team leader. share task management between international and country-based staff, which clients say increases the teams’ response times. The co-TTL system promotes mentoring by pairing experienced TTLs with newer staff members, but there are concerns about junior staff becoming TTLs prematurely. Data show more junior TTLs, suggesting a lower threshold for new staff to become TTLs, though pairing them with more senior co-TTLs protects average TTL seniority (figure 4.9). Interviews showed concern, especially from CDs of smaller countries, that inexperienced TTLs give poor advice to clients and are unfamiliar with internal policies and procedures. Additionally, some locally recruited staff felt they were inadequately recognized as mentors in their training of younger international staff. FIGURE 4.9 Grade Composition of Task Team Leaders | 1,000 14 12 800 10 Number Percent 600 8 400 6 4 200 2 0 0 2012 2013 2014 2015 2016 2017 GF GG Other GF Ratio Source: World Bank projects data analyzed by the Text and Data Analytics team, Global Themes Knowledge Management. Note: TTL = task team leader. Independent Evaluation Group | World Bank Group 41 Several examples show how co-TTL arrangements support project collaboration. In Côte d’Ivoire, there was strong teamwork among three GPs, including Transport, Social, Urban, Rural, and Resilience, and Trade and Competitiveness, on three urban development projects. These projects aimed to transform two secondary cities into primary economic hubs and foster development in less advanced areas of the country. The collaboration was driven by the need for solutions in several themes, including land tenure, infrastructure, job creation, the private sector, and planning and regulation. Staff from the three GPs were co-located in the country office and shared the task leadership. Because there were three projects, each GP could lead one while contributing to the others. The PMs from the different GPs allowed contributing staff to put as much time and effort as needed into the operation, independent of whether the staff was the TTL or a team member. In Egypt, three co-TTLs could quickly prepare a series of complex development policy operations that involved eight GPs. This successful co-TTL arrangement was made possible by having common incentives, clear and open internal communication, clearly defined roles within the team, and strong leadership and support from the CD. Collaboration Inefficiencies, Boundaries, Connectors, and Incentives The operating model has new boundaries that hinder collaboration. The overall perception among staff interviewed for this evaluation remains that GPs operate as sector silos that have replaced the regional silos that existed in the previous model. This led to the view that collaboration is harder now than before the reorganization because of increased fragmentation and lack of adequate mechanisms to work across GPs and between GPs and CMUs. Additionally, there are incentive issues stemming from competition among the GPs and the importance of own-managed lending. Mechanisms for Working Across GPs and Between GPs and CMUs Eliminating Regional Sector Directors caused collaborative inefficiencies and distance between GPs and clients. In the previous model, Regional Sector Directors managed the sectors, helping bind them together and connecting them with CMUs and clients. Regional Sector Directors had budget, human resources, and substantive roles that afforded them the authority to require collaboration and arbitrate across sectors. The new operating model eliminated this role. Because of the absence of Regional Sector Directors integration must be driven by CDs, supported by PLs when empowered. Further, conflicts among GPs escalate to the VP level rather than being resolved at the Director level, but the VPs have little time to resolve operational disagreements. Also, the GPs’ leadership is perceived as less visible to the clients. These views were frequently reinforced in interviews. One CD said, “The operating model does need change. The loss of the Sector Director has weakened the ability of the sectors to engage in the Regions with client countries. Most of the Senior Directors and Directors are too far removed from the front line to be able to serve our clients effectively. … There is no real connection among the sectors any more at the regional level. … We have lost the ability to work as an integrated team.” Another CD said, “Previously, when we had Sector Directors in the Regions, they could help. We are not seeing this kind of support under this current setup. When there is a problem, such as with safeguards 42 Knowledge Flow and Collaboration | Chapter 4 or procurement, there is less ability to solve problems. Many of the PMs are unable to rise to the occasion.” A third CD said, “The operating model is not responsive…the role of CD has turned into a Sector Director.” One other person said, “the leadership of GPs is focused on identifying and building the management team— not focused on teams and the client —the whole reform has distracted people from the core business.” Several also highlighted loss of the ability to easily arbitrate across GPs. CDs complained about high transaction costs and resistance toward cross-GP collaboration. CDs and PLs say they must push back against GPs that want to lead more lending operations, often without clear client demand or when the situation does not merit a loan. Clients were generally more positive, suggesting that not all the internal issues noted here affect service delivery. Staff is generally concerned with operational collaboration under the new operating model, although employee engagement surveys point to improvements over time. This emerges from surveys, interviews, and a “team effectiveness” analysis done by the human resources department. The team effectiveness analysis was based on nearly 200 hours of interviews with staff and managers and reported widespread concern over collaboration. At the same time, the annual employee engagement survey shows that staff perceives collaboration across units, between headquarters and country offices, and between different Bank Group institutions as having improved steadily between 2013 and 2017. Satisfaction with collaboration between different units has recovered to 2009 levels. The 2013 employee engagement survey (only that year), asked staff if the Bank Group was effective in delivering multisectoral solutions, but only 49 percent of respondents agreed that it was (table 4.1). Survey responses also show staff is concerned that that institutional decisions are not timely and that processes for client delivery are inadequate. The two-minute surveys, which are short surveys on task delivery, show that satisfaction with collaboration and cross support is lower than for any other aspect of task delivery. In interviews with staff from the Equitable Growth, Finance, and Innovation Practice Group, most perceived that collaboration was easier before the reforms, when macroeconomists, poverty economists, and public-sector specialists were housed in the same network, with a lead economist to coordinate on each country; the reorganization increased competition among the various specialists and eliminated the lead economist’s coordinating function. The Human Development Practice Group, in contrast, has had far more continuity, because its roles and boundaries did not change with the reorganization. Transaction costs are high for managers on both sides of the matrix. CDs, PLs, and Country Managers spend much of their work time coordinating with GPs. They negotiate compromises with GPs to ensure funding and sustain work programs. These interactions incur costs in time, effort, and energy. PLs say tensions arise most frequently with PMs around budgets and about which GP should lead work programs and which GPs should contribute. Both CDs and PLs spend time arbitrating among GPs that are competing to lead tasks, especially lending tasks. PMs say the high collaboration transaction costs ensue from administrative requirements, the multiplication of structures and interaction points, the complexity of budget procedures, and the delays in cross-support settlements. Senior Management recently created a “Regional Liaison Director” role, which is unlikely to fill the gap left by Regional Sector Directors. This role is occupied by a GP Director from each Practice Group who attends regional management meetings as a representative for all the GPs. This role is not a new Independent Evaluation Group | World Bank Group 43 TABLE 4.1 | Findings from Employee Engagement Survey (percent) Survey Question 2009 2013 2015 2016 2017 Colleagues within my unit cooperate to get work done. 79 78 78 79 84 Colleagues collaborate effectively across different units. 66 55 48 51 67 Colleagues in country offices and Washington collaborate effectively. 49 50 51 60 66 Colleagues across the World Bank Group (IBRD and IDA, IFC, MIGA, n.a. 23 27 34 40 ICSID) work together effectively. Internal processes and procedures enable me to deliver high-quality 42 42 31 38 45 services to our clients. I am encouraged to share knowledge with colleagues within Bank n.a. n.a. n.a. 73 78 Group. Bank Group makes institutional decisions in a timely manner. n.a. 25 22 24 33 I have access to relevant knowledge to do my job effectively. n.a. n.a. 74 76 79 The World Bank Group is effective in delivering multisector solutions. n.a. 49 n.a. n.a. n.a. I can easily access the relevant knowledge I need to serve my clients. n.a. 70 n.a. n.a. n.a. Source: Employee engagement surveys. Note: The survey covers all staff in the World Bank Group. IBRD = International Bank for Reconstruction and Development; ICSID = International Centre for Settlement of Investment Disputes; IDA = International Development Association; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; n.a. = not applicable. position but is simply added to the duties of the person assigned it. These Regional Liaison Directors can assist with coordination and communication but do not have managerial authority over staff, budgets, or operations outside their individual GPs. The PL role is useful. Country visits and interviews show repeated examples where PLs led client dialogues, cross-GP collaboration, and major multisector initiatives. PLs can be effective in building collaboration if empowered through the authority of CDs. In such cases, PLs have a lot of influence to connect and integrate GPs, CMUs, and clients. Lacking formal authority, PLs often must rely on persuasion and personal networks to build collaboration. PLs often tap into the GPs’ global expertise, including the GLs, and ensure that GP knowledge is adapted to country contexts. Some illustrative quotes from CDs: “A good and experienced PL…protects me and is able to manage relationships with the PMs because of his previous networks. With a junior PL this will be a problem. We need to make sure the PL has the right leadership skills if we want to make this work;” “the PL is the way that cross-sectoral work happens.” In practice, the PL’s role varies. The variation seems to reflect both how CDs choose to use and empower the PL’s and individuals’ performance in the role. When PLs are not empowered by CDs 44 Knowledge Flow and Collaboration | Chapter 4 to drive integration, they may be given less substantive administrative or quality assurance tasks. GP staff and managers noted the variability in PL performance across CMUs, for example one interviewee said, “PLs are not sufficiently informed about the strategic direction and the way they are expected to function.” Recent appointments of PLs at lower grade levels raises concerns about eroding the clout that this role necessarily must have. Recruiting the right profile for the PL role is essential. A PL needs to be able to influence staff without budgets or formal authority. They advise CDs, to whom they report, but must also maintain good relationships with PMs, GLs, and other experts. PLs must understand the reality of operations and provide tangible support to teams when needed. PLs, thus, need a broad skillset that includes technical, leadership, operational, communication, and client interface skills. In GPs, from where PLs are recruited, hiring and promotion criteria emphasize technical skills instead of the many other skills that PLs need. As in any organization, certain issues overlap among business units and create tensions: taxation, urbanization, and job creation, for example, do not fall neatly within the mandate of a single GP. This has sometimes resulted in tensions and mixed signals to clients as when many groups work on job creation issues, without a clear division of labor with the Jobs GT. In another example, drawing the line between environment and climate change required moving staff and changing the reporting structure. The reorganization redrew the agricultural sector’s boundaries, placing land tenure, irrigation, rural finance, rural development, and community-driven development in groups other than the Agriculture GP. According to interviews, these divisions in agriculture contributed to further competition and fragmentation. Collaboration across hierarchical boundaries also suffers because many of the groups and individuals tasked with this role are not given authority. GLs, PLs, and Global Theme Groups are all tasked with integrating across boundaries but are without authority to do so. Instead, they must appeal to intrinsic motivations to collaborate, and they frequently blame small budgets and lack of authority for their agendas not gaining traction. For example, several GP managers questioned the capacity or mandate of the Jobs Group to promote job creation, instead saying that expertise already exists within the GP. Likewise, some managers disputed the Infrastructure, PPPs, and Guarantees GTs’ right or ability to lead on MFD. In contrast, the gender and climate change GTs have greater visibility and buy-in from GPs because GP managers more clearly understand the mandate of these two groups and the need to mainstream these themes into operations. Incentives for Maximizing Finance for Development MFD will require a shift in mindsets and behaviors that will be hard to engineer. The purpose of MFD is to help countries’ access private capital. For many years, success and staff promotion were based on getting projects approved, more specifically approving lending projects for the World Bank and investment projects for IFC. By contrast, the MFD agenda calls for seeing success not in terms of Independent Evaluation Group | World Bank Group 45 approvals but in terms of leveraging the Bank Group’s influence and interventions to unlock private sector investment. For example, instead of lending for roads, the Bank Group might support a multimodal transport strategy that removes barriers to private investment in bus, rail, and toll roads. MFD is more advanced in some countries than in others. In Egypt, the MFD agenda is well advanced. There, the World Bank is working with the government to carry out energy policy reforms that attract private investments in energy. This, in turn, creates opportunities for IFC to invest and advise on renewable energy, the largest portion of IFC’s Egypt portfolio. Elsewhere, the mentality shift required for MFD is only starting to emerge. Many GPs, for example, have yet to formulate clear positions on the private sector’s role in their sectors, slowing adoption of the MFD approach. Also, interviews suggest that units have been slow to adopt MFD approaches. To encourage the adoption of MFD, the Infrastructure, PPPs, and Guarantees GT, which oversees MFD’s implementation, offers guidance and staff trainings. Efforts to achieve closer integration of IFC and World Bank services have taken different forms. Calls for IFC-World Bank collaboration go back many years. For a while, Joint Implementation Plans were the agreed platform. Later, joint CPFs had some success. Different approaches to structurally integrate IFC and World Bank units have been attempted. The former Trade and Competitiveness GP worked hard to streamline procedures and incentives; this particular approach stopped when management reorganized Equitable Growth, Finance, and Institutions on January 1, 2018 (see chapter 2). Joint practices were realigned with integration now at the director level. Despite improvements over time, staff perceptions of IFC-World Bank collaboration remain poor: In 2017, 40 percent of staff felt that Bank Group colleagues work together effectively across the Bank Group (table 4.1). The problem at its core is that IFC and the World Bank have different clients, incentives, and organizational cultures. This is also bound to complicate efforts to make IFC and World Bank work in lockstep on the MFD approach. Budget Arrangements and Incentives GPs are incentivized to attain the greatest volume of own-managed lending. The reforms did not change this, in part because some of the complementary measures were not implemented (see chapter 2). More own-managed lending gives GPs recognition, budget certainty for the project’s duration, and access to clients for policy dialogues. PMs and Directors who add lending volume are successful and can hire more staff with a guaranteed multiyear budget for supervision based on agreed coefficients: a major positive incentive. PMs who fail to secure sufficient lending volumes face budgetary anxieties: a negative incentive. A CD says, “The incentives are still for the PMs to give jobs to their staff…. Feed your own goat before you fatten somebody else’s goat.” Units with large lending volumes feel safe, whereas units with strong collaboration, large ASA programs, but small lending portfolios might be at risk of disappearing. Therefore, the tendency among GPs to compete for own- managed lending and to lobby CDs for work programs is entirely rational but causes tensions. 46 Knowledge Flow and Collaboration | Chapter 4 Budget arrangements create incentives for own-managed work programs, negatively affecting collaboration for integrated solutions. This is in addition to the structural barriers to collaboration already mentioned. A theme emerged in interviews that tied collaboration difficulties to PMs’ anxiety over budgets and spending targets. This anxiety results in risk aversion, which creates a preference for more own-managed work programs and less collaboration because collaboration has consequences that are hard to foresee when doing budget planning. Many interviewed staff said PMs are unwilling to have staff contribute to projects led by other GPs because it may not lead to additional lending work. These perverse incentives reduce the demand to carry out knowledge work or work with nonborrowing client countries. Management introduced a “collaboration premium” that provides modestly higher budgets for collaborative tasks. But, the budget system remains too complicated and fragmented to fully align budget incentives with greater collaboration. Budget processes often came up when talking to managers about ways to improve knowledge flow and collaboration. In interviews and focus groups, managers expressed challenges with some of the unit-level budget processes, characterizing them as fragmented and time consuming to manage. These managers tended to perceive unit-level budget processes as a contributing factor rather than a major impediment to knowledge flow and collaboration. Management is aware of these pain points and has repeatedly explored opportunities for improvement in budget processes to enhance budget predictability and improve collaboration. Why are these reforms apparently not working as intended? The budget system is aligned with the structure of the organization; the GPs receive annual budgets. The GPs’ senior directors delegate spending authority to PMs, who manage spending targets and are accountable for controlling spending. In practice, PMs spend much time managing different sources of funds – including trust funds, World Bank budget, and reimbursables – and different spending categories – including country engagement budgets, global engagement budgets, sustaining costs, and cross-support settlements. PMs also wished for greater budget predictability. At the “budget holder” senior director level, cross support is prepaid based on collaboration in the previous year with settlement at the end of the year, resulting in relative budget stability. But for the units that collaborate, there can be time lags in budget settlements, causing uncertainty and anxiety for the PMs accountable for managing spending targets. Timing differences between cross support provided in one year and budget reimbursement in the next could hinder collaboration. 1  The numbers of leading and contributing projects were used as metrics to measure collaboration among Global Practices (GPs). The data source is the collaboration data from the corporate Standard Monitoring Report published by Information and Technology Solutions and was accessed and downloaded on February 26, 2018. In the collaboration data, a contributing GP is defined as: Co-task team leadership from a GP other than the lead GP; contributions of over five weeks from another GP (could be across multiple staff); or a qualitative decision (based on a discussion between the involved GPs and relevant Country Management Unit). 2 A strength of the PL role is that PLs maintain connection to the GPs, their technical stream. However, the previous  version of the operating model had Sector Leaders who dually reported to Country and Sector Directors. This kept them even more embedded in the technical streams than does the current Program Leader reporting structure. Nearly all those interviewed who were intimately familiar with both setups found the previous Sector Leader’s dual reporting more helpful than the current setup. Independent Evaluation Group | World Bank Group 47 5 T H I S E VA LUAT I O N finds that certain aspects of the operating model are valuable, especially its improvements to global knowledge flow. However, the early evidence examined in this report indicates that the model’s structure and processes Conclusions and tend to inhibit collaboration and cause inefficiency, fragmentation, Recommendations and internal competition. Some GPs lack coherent and systematic approaches to managing and investing in knowledge. If these issues are left unaddressed, they pose risks to the World Bank’s ability to deliver for clients. This chapter elaborates on this conclusion and proposes a set of recommendations to address the weaknesses of the current operating model while preserving its strengths. Reform Design and Implementation Prereform diagnostics argued that the Bank Group’s comparative advantage to clients is its ability to deliver knowledge and finance in a complementary way. The goal of the operating model reforms was to deepen the World Bank’s development impacts by improving knowledge flow and collaboration for integrated solutions. This requires strong collaboration among the Bank Group’s many business units and instruments. Improved knowledge flow also requires thought leadership, internal coherence, and knowledge mainstreaming. In hindsight, these diagnostics seem sound. Later strategy statements, such as the Forward Look and the Capital Increase package, reiterated these diagnostics and the strategic directions they imply (World Bank 2016, 2018). Reforms to the operating model were designed to respond to the diagnostics’ recommendations. A new country engagement approach of CPFs and SCDs would make Bank Group support more selective and strategic. GPs would lead operations, deepen technical knowledge, and ensure global knowledge flow by removing barriers to staff and knowledge mobility. New GT units 48 x would not lead their own operations but would ensure coherence on cross-cutting priorities and provide knowledge, influence, and leadership across the World Bank (and in some cases also IFC) on major corporate themes. New staff roles were created to make the model function as intended: PLs would connect CMUs with GPs. GLs would serve as thought leaders and guide knowledge generation, curation, and sharing. Reform implementation did not go as the designs intended and caused significant turmoil. The central change management team in charge of designing the reforms was dispersed during implementation and not replaced. This meant that there was no leadership that could focus on organizational effectiveness, ensure continuity in change management, and help business units navigate the complex changes under way. Reform proposals to deepen support, metrics, incentives, and leadership for knowledge and collaboration were not implemented. Moreover, several of the new mechanisms and incentive structures that were introduced have faltered since the reforms took effect. Implementing expenditure review measures simultaneous with the reforms caused anxiety among staff, even as these measures eventually may have helped improve the World Bank’s finances. Structures— the organogram—have and still receive much attention. In addition, sustained attention to organizational effectiveness and change management could have enabled timely resolution of many of the operating model’s downsides and helped realize more of the reforms’ goals. In hindsight, reform design and implementation succeeded in reducing regional silos as barriers to knowledge flow but did not succeed in enhancing collaboration or change incentives, behaviors, and organizational culture. Replacing four sector networks with 13 GPs and 5 GTs has resulted in greater fragmentation and silos because of the GPs’ competitive behaviors, which are driven by the size of their own-managed lending portfolios. A simpler approach to the reorganizations may have been to retain existing groups and structures, perhaps with some adjustments among sector boards, but realign incentives and reporting arrangements. This may have minimized additional fragmentation, confusion over the reforms, and damage to staff morale. More could also have been done to create metrics for organizational effectiveness and use them to inform course corrections. Better data on the quality of services to clients might also help managers focus on results. In future reforms, careful attention, supported by relevant data, should be paid to behavior change, knowledge leadership, quality of outputs and results, and the balance of roles and responsibilities between the two sides of the matrix. The Operating Model The operating model has shown value. GPs are thinking and operating more globally than ever before. GPs improve knowledge flow and staff mobility across Regions, mobilize expertise for client- facing activities, and, sometimes, deepen expertise in operationally relevant areas. GTs provide strategic direction and coherence to cross-cutting priorities. Data and many examples show some GPs, GTs, and CMUs making efforts to integrate issues and collaborate across boundaries to deliver integrated solutions to clients. Management should strive to identify, preserve, and enhance the features that are working well, and institute mechanisms to address the inefficiencies and weaknesses of the operating model. Independent Evaluation Group | World Bank Group 49 Collaboration across boundaries is difficult. The operating model works as intended when CDs and PLs show leadership and when structural incentives are aligned with the reforms’ goals. Managers on both sides of the matrix incur high transaction costs from routine managerial functions across the GP structure. Although the objective of overcoming regional silos was sound, it proved difficult to effectively coordinate between 44 CMUs, 13 GPs, and 5 GTs. CMUs consistently cite GP collaboration as a key pain point. The creation of Region-facing Liaison Directors is a recognition of this structural weakness but underestimates the need for more robust connector roles to link the two sides of the matrix. Also, there are no clear rewards for those groups and individuals that collaborate; structural changes have sent a message that GPs without strong lending may not be safe in a reorganization. Incentives for GP staff continue to favor own-managed lending, complicating collaboration. PMs without much own-managed lending navigate fragmented and uncertain budget arrangements in which they are responsible for managing many separate expenditure targets. The operational agenda seems overcrowded. The World Bank mainstreams many issues into operations, issues such as jobs, gender, climate change, maximizing finance, citizen engagement, fiduciary controls, and others. Some issues are universal mandates, others are optional. Mechanisms for mainstreaming are in place, but they tend to increase the time and costs of preparation and implementation, depending also on client country demand and capacity. Mainstreaming requires collaboration across boundaries and implementing this crowded agenda in the current structure and resource environment is managerially challenging. Many of the roles and units tasked with integrating, coordinating, and connecting across boundaries—GTs, GLs, and PLs—are not given authority or adequate budgets to carry out their responsibilities. Senior management should seek to lighten the corporate demands placed on operations, for example by ensuring decisionmakers are empowered to make flexible judgments based on the costs, benefits, and country demand for mainstreaming. There is a large gap between the reform’s aspirations to deepen knowledge and the current reality in many GPs. Some GPs have strategic approaches to knowledge; others do not. This often reflects differences in the availability of trust funds and leadership support. The mechanisms designed to promote knowledge excellence have met with mixed results. A few GPs made the GSG model work largely as intended, whereas others recast or even disbanded the model completely. GLs, who lead GSGs, suffer from unclear roles and unfunded mandates. There are concerns with quality assurance processes, specifically around contestability and the balance of oversight responsibility. Efforts to integrate IFC and World Bank have met with mixed success. Despite these challenges, there have also been bright spots, such as the creation of country knowledge hubs, better IT services, and innovative knowledge sharing mechanisms within certain GPs. Management has signaled its intention to propose a new strategic framework for knowledge premised on deploying knowledge and finance in complementary ways. The success of this framework will require continued investments in knowledge management, as proposed by management in the Knowledge Management Action Plan combined with a revamped incentive structure. A revamped structure should involve: (i) strong senior management signaling in support of knowledge excellence; (ii) creating metrics for knowledge uptake, impact, quality, and influence; (iii) reforming internal budgeting arrangements; and (iv) continuing trust fund reforms. 50 Knowledge Flow and Collaboration | Chapter 5 Every matrix system has pros and cons; the goal should be to mitigate the cons of the current system without disrupting the pros. The overall approach to knowledge needs improvement. The two sides of the matrix need robust, authoritative connectors. The existing pool of directors in the GPs is large enough to constitute a new cadre charged with connecting and arbitrating between GPs and CMUs in each Region. PLs could be used more effectively. Incentives, including those coming from budget arrangements, need to better promote collaboration. There should be more contestability in quality assurance. Organizational effectiveness deserves continuous attention. Recommendations IEG makes the following evidence-based recommendations: Recommendation 1. Strengthen the approach to knowledge in the GPs and GTs with clear goals, roles, and mechanisms, budgets commensurate with mandates, and metrics for knowledge uptake, quality, and influence. This will entail improving incentives for knowledge production, curation, management, and sharing, whether performed by GLs and GSGs or through other mechanisms. This could build on some of the good practices undertaken by some GPs. Recommendation 2. Improve budgeting systems to better incentivize knowledge flow and collaboration. This could be done, for example, through multiyear budgets or other mechanisms for enhanced year-on-year budget predictability; addressing time lags between cross support and the related settlement by recipient budget units; maintaining an adequate balance between budget allocations for ASA and other knowledge functions and lending; exploring additional options for stronger budget incentives for collaboration; and continuing reforms of World Bank–executed trust funds to improve their alignment with World Bank priorities. Recommendation 3. Better link the GPs and Regions to improve coordination and enhance responsiveness to clients. This could be done by enhancing the Region-facing responsibilities of GPs, providing Region-facing Practice Group Directors with adequate authority to arbitrate and link the two sides of the matrix; and, depending on the exact arrangement, by adjusting reporting arrangements for PMs and PLs. Recommendation 4. Ensure a stronger and more consistent use and role of the PLs as a mechanism for cross-sectoral collaboration, integrated solutions, and complex client dialogue. This could be done, for example, by clarifying expectations on PLs’ role, authority, and accountability; achieving a better balance between technical and connector roles; ensuring more effective dual reporting to CMUs and GPs; and continuing to recruit PLs based on both leadership skills and technical proficiency. Recommendation 5. Review the existing quality assurance arrangements to improve the quality of knowledge embedded in advisory and financing services. The review should identify ways to enhance the contestability of quality assurance mechanisms across GPs, covering quality at entry for Independent Evaluation Group | World Bank Group 51 lending and ASAs, implementation, and reporting at completion. It should also assess the balance of oversight responsibilities between GPs and CMUs. Recommendation 6. Ensure there is ongoing monitoring of the operating model and more continuity in change management efforts to enhance the organization’s ability to attain its knowledge flow and collaboration goals. This could be achieved by continuously collecting and reviewing data on organizational effectiveness and learning from past reforms and from other organizations. Management should use these data to help it recognize and scale up successful innovations, and to design additional course corrections, as needed. 52 Knowledge Flow and Collaboration | Chapter 5 references Besley, T., Paxson, C., Peter, H., and C. Udry. 2015. “Evaluation Panel Review of DEC. A Report to the Chief Economist and Senior Vice President.” Washington, DC: World Bank Group. Devarajan, S. 2016. “An Exposition of the New Strategy,’Promoting Peace and Stability in the Middle East and North Africa”. openknowledge.worldbank.org World Bank. 2011. Trust Fund Support for Development. Washington, DC: World Bank. ———. 2013. The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness. Wash- ington, DC: World Bank. ———. 2015. Learning and Results in World Bank Operations: Toward a New Learning Strategy. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016b. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017a. World Bank Group Engagement in Upper-Middle-Income Countries: Evidence from IEG Evaluations. Independent Evaluation Group, Synthesis Report. Washington, DC: World Bank ———. 2017b. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Inde- pendent Evaluation Group. Washington, DC: World Bank. ———. 2017c. World Bank Group Country Engagement: An Early-Stage Assessment of the Systematic Country Di- agnostic and Country Partnership Framework Process and Implementation. Independent Evaluation Group. Washington, DC: World Bank ———. 2018. “Sustainable Financing for Sustainable Development: World Bank Group Capital Package Proposal.” Paper prepared for the Development Committee Meeting of the World Bank Group, Washington, DC, April 21, 2018. World Bank Group. 2017. “Forward Look: A Vision for the World Bank Group in 2030—Progress and Challenges.” Washington, DC: World Bank Group. Independent Evaluation Group | World Bank Group 53 A PPE NDI X E S Knowledge Flow and Collaboration under the World Bank’s New Operating Model AN INDEPENDENT EVALUATION Appendix A. Methodological Approach Evaluation Questions The evaluation’s purpose was to provide early information about how the World Bank’s current operating model was working and identify areas that could be strengthened to improve knowledge flow and collaboration for integrated solutions. This served three objectives: ƒƒ Providing the Board of Executive Directors and management with early information about the functioning of select aspects of the new operating model and the extent to which the short- comings related to knowledge flow and collaboration in the earlier operating model (pre-2014) have been addressed; ƒƒ Identifying good practice examples and lessons learned thus far; and ƒƒ Enabling changes to improve aspects of the operating model via possible midcourse correction or deepening of the implementation of the model. The evaluation’s purpose inspired four main evaluation questions that guided the collection and analysis of data and the framing of findings and recommendations (box A.1). Box A.1. Four Evaluation Questions • What has been the experience so far with select new features of the operating model such as globally integrated Practices, Program Leaders, Global Leads, Global Solutions Groups, and cross-cutting solutions areas, especially regarding their contribution to collaboration and knowledge flow? • Have these features of the new operating model facilitated internal collaboration and integration of cross-sectoral perspectives in the delivery of client solutions? Why? • Have these features of the new operating model facilitated knowledge flow? Why? • Have they facilitated the mobilization of global expertise and its customization to country context? Why? Overarching Principles Three central principles underpinned the evaluation design, data collection, and analysis: theory- based evaluation, mixed methods, and participatory evaluation processes. The evaluation was grounded in a set of nested theories of change. At the broadest level, the theory of change of the evaluation represents a reconstruction of how changes to the operating model were intended to improve knowledge flow, collaboration, and thereby team effectiveness and, ultimately, the relevance and quality of World Bank Group services and its development effectiveness. It is based on the reconstruction of management intent and refined based on key informant interviews. The theory of change seeks to trace a causal link from reform proposals, to implemented Independent Evaluation Group | World Bank Group 57 changes, desired behavior changes among staff, intermediate outcomes, and ending with long- term outcomes. Table A.4 contains a detailed description of the theory of change that guided this evaluation. Guided by the principle of mixed methods, the evaluation combined a large amount of existing secondary qualitative and quantitative data with the team’s own data collection. The evaluation relied on semistructured interviews, focus group discussions, portfolio review analysis, desk review, social network analysis, and survey analysis. The evaluation applied systematic triangulation to ensure the robustness of the findings. The data collection and analysis covered both sides of the Bank Group’s matrix—the Global Practices and Global Themes on the one side and the Regions and CMUs on the other side. In support of the formative, learning-oriented purposes of this evaluation, participatory engagement of stakeholders in all parts of the evaluation process was another major principle. Participation was used in data collection, validation, and triangulation of findings, as well as in the quality control (through a REACT workshop). The IEG team engaged closely with counterpart teams working on related tasks (typically teams collecting or analyzing data on knowledge, collaboration, team effectiveness, or aspects of client delivery). A series of small-group meetings and focus group discussions with counterparts were used to identify good practice cases and elements that work well, probe how processes and incentives enable or hinder the desired behaviors, assist in triangulation, and promote learning and stakeholder engagement during the evaluation process. Participation of critical thinkers was actively sought in this process. Evaluation Components Table A.1 lists the evaluation components, and figure A.1 shows their articulation within the overall evaluation design. The next two sections provide more details on each component. Table A.1. Evaluation Components Evaluation Component Description Desk review Review to collect data and evidence on collaboration and knowledge flow from Bank Group corporate data and existing studies Country Partnership Framework Structured review of CPF and Country Learning (CPF) Analysis Reviews for 21 countries. These are countries which have CPFs that could be found in the World Bank database. Reconstruction of a Theory of Reconstruction of how changes to the operating Change model were intended to improve knowledge flow, collaboration, and thereby team effectiveness and, ultimately, the Bank Group services based on change process documents and key informant interviews. 58 Table A.1, continued Evaluation Component Description Global Practice (GP) Analysis Analysis of the extent to which the new operating model facilitated knowledge flow and collabora- tion through the 13 GPs. It primarily relied on 65 semistructured interviews with staff and manage- ment. Other data used for this analysis included the cross support and budget data, sector codes, social network analysis, MoUs, staff and client surveys, and human resources team effectiveness study. Global Solutions Groups and Analysis of GSG/GL’s effectiveness regarding the Global Leads (GSG/GL) Analysis knowledge flow and collaboration. It relied on nine semistructured interviews with GLs, the GSG model survey from 2015, and interviews with other staff Global Themes (GT) Analysis Analysis of the effectiveness of GTs to promote corporate priorities and contribute to knowledge flow and collaboration. It relied on six interviews with GT staff and management, review of the memoranda of understanding used to assess GTs performance, and dedicated GT related questions in semistructured interviews with GP staff Budgeting Arrangements Analysis of budget arrangements related to knowl- Analysis edge flow and collaboration Staff Survey Analysis Analysis of perspectives of staff and management on the quality of collaboration within and across Practice Groups based on existing survey data from the CMU and GP two-minute surveys, and the employee engagement surveys Country Program and Client Analysis of country programs including CMU’s views Perspective Analysis on knowledge flow and collaboration. It was covered through a combination of in-country visits and inter- views by phone or in person in 20 CMUs Figure A.1. Methodological Design Independent Evaluation Group | World Bank Group 59 Table A.2 lays out the evaluation design matrix, focusing on the links among the evaluation questions, aspects that were considered to answer the questions, and the sources of data used. Table A.2. Evaluation Design Matrix Evaluation Questions Aspects to Consider Sources of Data What has been the experience so • GSG/GL model: functioning • Change process and other internal far with select new features of the and contribution to larger documents operating model such as globally objectives • Validation interviews on the theory of integrated Practices, Program • Program Leaders as inte- change Leaders, Global Leads, Global grators Solutions Groups, and cross-cut- • Corporate data ting solutions areas, especially • Mainstreaming of Global • Interviews regarding their contribution to Themes collaboration and knowledge flow? • GSG survey; GSG internal documents • GPs as globally integrated providers of technical work • Assess key aspects of the model against what was proposed • Incentives, processes, and efficiency of the model • Budget data Have these features of the new • Extent, quality, and efficiency • Employee engagement and two-min- operating model facilitated internal of collaboration ute survey results collaboration and integration of • Enablers and obstacles • Interviews with staff, GP manage- cross-sectoral perspectives in the to collaboration, “positive ment, CMUs, and clients delivery of client solutions? outliers” • Documents • Client perspectives • Social network analysis • Budget Arrangements Analysis • Scan of IEG evidence on collaboration Have these features of the new • Global mobility of staff • Review of task documents (PADs, operating model facilitated knowl- ASA reports, and so on) • Role of GSGs and GLs edge flow? Have they facilitated • Interviews with team members, CMU, the mobilization of global expertise • Types, sources, relevance, and clients and its customization to country quality of knowledge; cus- context? tomization to country context • Two-minute surveys Country surveys Ensuring the Validity of Findings The team ensured validity by systematically triangulating findings within and across all evaluation components. First, the team applied triangulation by crosschecking evidence sources within a given methodological component. For example, within GP analysis component, the team compared and contrasted evidence from interviews with staff in different positions, that is, Regional Practice Managers, Directors, and task team leaders, and quality assurance staff on the same topic. Second, the team ensured validity by systematically triangulating findings across all evaluation components, relying on all available sources of data, and in dialogue with the refined theories of change. For example, the team cross-validated findings from interviews with findings from surveys, corporate data, portfolio analysis, and focus group discussions. The evaluation team also applied external validation mechanisms at various intervals during the evaluation process. Three peer reviewers provided feedback at the beginning of the evaluation 60 process, and four experts peer reviewed the draft final report. A methodology workshop was used to fine-tune the evaluation design. The team convened a “REACT” workshop with counterparts around the draft final report. Finally, the team organized workshops with a panel of key stakeholders at the end to ensure the relevance and feasibility of the evaluation recommendations. Limitations Notwithstanding these steps, the team documented several limitations to the evaluation design that broadly fell into three categories: timing; conscious choices about scope; and other methodological and data availability reasons. Limitations related to timing include ƒƒ As an early-stage evaluation, the methodology did not include direct reviews of the quality of World Bank services or their outcomes, which could not yet be observed; instead, emphasis was more on processes and behaviors. ƒƒ Several reorganizations took place during the data collection for this evaluation. Limitations in scope include the following: ƒƒ Execution of the change process itself was not evaluated directly. Neither an audit of how man- agement carried out the organizational change process nor a detailed documentation of each phase of the change management process was part of the scope of this study. ƒƒ The evaluation didn’t include a direct review of the quality of Bank Group services or client outcomes. As an early-stage evaluation, a limitation of the evaluation design is that findings on staff behaviors and team effectiveness were hard to link to development effectiveness and development outcomes because this type of data was not yet available. The team’s detailed theories of change were used to link findings on mechanisms, processes, behaviors, and team effectiveness to the likelihood of achieving intermediate and longer-term outcomes. ƒƒ It was not possible to account fully for how factors other than the organizational reforms have affected performance. For example, improvements in IT likely facilitated knowledge flows and collaboration between headquarters staff and country office staff independently of other reforms. ƒƒ The evaluation pays more attention to how the operating model varies across GPs than to how it varies across Regions because of its focus on the new features of the operating model; the Regions were therefore not a focus area. ƒƒ The focus of the evaluation was to capture the patterns and experiences of the Bank Group as an organization providing services to the client rather than those of client governments. Al- though the evaluation analyzed the client perspectives on the delivery of World Bank services, the interaction with clients was somewhat limited and included only representatives of borrow- ers, sector ministries, and Project Management Units in five countries (that is, Azerbaijan, Côte d’Ivoire, the Arab Republic of Egypt, Malaysia, and Pakistan). ƒƒ The evaluation focused more on the patterns of knowledge flow and collaboration at the level of Independent Evaluation Group | World Bank Group 61 the entire operational complex rather than at patterns within specific business units. ƒƒ The focus of the evaluation was on the World Bank. Those parts of IFC that were embedded in the GPs and Global Themes formed part of the scope. Other parts of IFC, as well as Multilateral Investment Guarantee Agency, were not covered. Development Economics and other non- operations parts of the World Bank were also not covered in the original scope. IEG added a limited amount of material on Development Economics at the very final stage at the request of management. ƒƒ The limitations of this evaluation also related to the sampling of interviewees. Because of time and resource constraints, the evaluation team set limitations on the number of interviews in each GP and to the types of positions to be chosen for interviews. The role of the Global Themes Groups in fostering integration of select corporate priority themes (such as jobs; gen- der; climate change; fragility, conflict, and violence; and public-private partnerships) was cov- ered, while their more external-facing roles (such as raising the global profile of the Bank Group within their themes) was not covered. These roles may be covered in IEG’s planned evaluation of convening power, scheduled for FY19. ƒƒ The creation of the GPs and GTs was part of a larger change process, occurring simultaneous- ly with several other ancillary reforms that are outside the scope but needed to be considered as part of the context influencing the performance of the operating model: ƒƒThe Expenditure Review ƒƒNew country engagement process (Systematic Country Diagnostic, Country Partnership Framework) ƒƒIT Investments ƒƒSimplification; “Agile Bank” ƒƒ“One World Bank Group” initiatives aiming at bringing the World Bank, IFC, and Multilat- eral Investment Guarantee Agency closer, apart from the joint GPs ƒƒDetailed functioning of the framework for accountability and decision making ƒƒDisbanding the Learning, Leadership, and Innovation Group. Limitations due to data availability included: ƒƒ The corporate data on joint projects and activities, partially derived from staff time recording, were inaccurate proxies of the extent of collaboration because staff have not been consistent at recording their time spent working on other projects. ƒƒ There is no good data on how knowledge flows into the products delivered to clients. ƒƒ The employee engagement surveys may be subject to swings over time in perceptions that 62 need not reflect underlying realities. Change fatigue and the emotional imprint of the FY15 reforms might interfere with staff perceptions related to knowledge flow and collaboration. ƒƒ There are no directly comparable baseline and ex post data on knowledge flow and collabora- tion before and after FY15 when reforms were launched. ƒƒ The aggregate budget data by budget buckets was available but not specific or itemized for knowledge flow and collaboration activities.  ƒƒ The analysis of GL publications did not distinguish between different types of journals and publications. Description of Evaluation Components Four methodological components completed early in the evaluation process informed further methodological choices, selection strategy, and the substance of the evaluation findings. They included a reconstruction of the theory of change, a systematic Country Partnership Framework Analysis, initial key informant interviews for scoping the analysis, and a desk review. Theory of Change The evaluation drew on a theory-based evaluation approach, systematically and iteratively reconstructing and recalibrating the theory of change. The overarching theory of change focuses on reconstructing the causal chains and processes from reform intentions to implemented changes, targeted staff behaviors, intermediate results and long-term results. Table 1.1 in chapter 1 contains a reconstruction of how changes to the operating model were intended to improve knowledge flow, collaboration, and thereby team effectiveness and ultimately the relevance and quality of Bank Group services and its development effectiveness. The theory of change also informed the design of data collection methods. IEG reconstructed the theory of change by integrating insights from the key informant interviews. Nested within this, specific theories of change were developed for some of the features of the operating model selected for in-depth treatment such as GSGs and GLs model and for describing how factors that promote or hinder knowledge flow and collaboration can impact World Bank delivery to clients. Particularly, the nested theory of change on GSGs/GLs (figure A.2) helped to assess the extent to which GSG/GL model had standardization or flexibility around GL roles, number of GSGs per GP (inputs) as well as how effective the GSG/GL model is related to (i) desired outputs; (ii) outcomes around pooling and curating relevant country and global knowledge; and (iii) creating a strong technical stream. The nested theory of change on knowledge flow (figure A.3) provides a conceptual framework for analyzing various components of this evaluation related to knowledge Independent Evaluation Group | World Bank Group 63 creation, curation, and transfer. These detailed theories of change were developed from relevant literature and documents combined with key informant interviews.1 Figure A.2. Nested Global Solutions Group/Global Lead Theory of Change 64 Figure A.3. Nested Knowledge Flow Theory of Change Country Partnership Framework Analysis The CPF analysis involved a systematic review of CPFs for 21 countries, 18 of which could be benchmarked against the earlier cycle as documented in the Country Learning Reviews. The 21 CPFs were the clear majority of all the 28 potentially available at the time of this work. Most of the seven CPFs not reviewed could not be readily retrieved from, or were unavailable in, the database. The main goal of the CPF analysis was to see whether country teams have CPF objectives identifying and planning for integrative multisectoral interventions. The Independent Evaluation Group (IEG) compared the framing of the objectives, the analysis of the issues, and the intent to work across sectors as laid out in the CPF documents with the same variables as in the previous Completion and Learning Review. IEG looked at (i) whether there was a change in the number of CPF objectives that, in IEG’s judgment, would require cross-GP collaboration and (ii) areas of collaboration within the Practice Group Vice Presidential Unit (VPU) and across the Practice Group VPUs. The secondary goal of the CPF Analysis was to inform the selection of country visits and country program analysis. First, IEG identified the country CPFs that represented the positive and negative outliers in the number of CPF objectives that are framed in a way that would require cross-GP collaboration. The second step was to review key project documents of the country portfolio of projects and ASA to inform the country visits and country program analysis. Additionally, the team compiled data on the extent of co-TTL-ship to provide background information on collaboration patterns. Key Informant Interviews The scoping of the evaluation was informed by 27 broad semistructured interviews with key informants. Most of these informants were World Bank employees at manager or higher levels; a few were former staff. Their selection emphasized people that were involved in the change process, via working groups or in other ways, complemented by staff in different roles or with unique insights. Additional scoping and methods design interviews (on top of the 27) and focus groups involved IEG Independent Evaluation Group | World Bank Group 65 staff and an external evaluation specialist. These early-stage interviews informed the evaluation’s scope as laid out in the Approach Paper. Interviews were also used for other evaluation components as described in the next few sections, in particular box A.1. Global Practice Analysis The GP analysis assessed the extent to which the new operating model of the World Bank has facilitated knowledge flow and collaboration through the 13 Global Practices. The goal of this component was to answer how GPs are organized to facilitate knowledge flow and collaboration internally and externally. Specific questions in which this evaluation component focused on staff incentives and organizational and budgeting arrangements to do the following: ƒƒ Identify knowledge gaps that require a multi-GP collaboration ƒƒ Create, curate, store, and make accessible knowledge within and beyond GP boundaries (re- view of GPs’ knowledge management systems) ƒƒ Collaborate within the GP (across Regions) or outside the GP on integrated solutions projects ƒƒ Collaborate with the Global Themes The evaluation team relied on qualitative data collected during semistructured interviews and focus group discussions with GP staff and managers. In each of the 13 GPs, the evaluation team conducted about five interviews with Practice Managers or directors (often the Practice Manager for Strategy and Operations, and a Regional Practice Manager), a TTL, quality assurance or development effectiveness staff, and knowledge management staff, for 65 interviews. Within a specific job description, there was a random selection from among the staff. For example, if there were six regional PMs in one GP, one of these was selected on a random basis to maintain neutrality across Regions. The evaluation team took detailed, written notes for each interview, and systematically analyzed those to derive themes and key messages from the interviews that could be triangulated with each other and with other information sources. The objective of social network analysis (SNA) was to understand the patterns of collaboration among GPs. It mapped out patterns of cross-GP collaboration, relying primarily on the corporate collaboration indicator. The analysis was done at the level of Practice Groups and GPs and distinguished between collaboration on lending and ASA. The team used Cytoscape as SNA software to calculate the network metrics and to generate the network maps. The layout was purposefully adapted by the team to illustrate the cross-practice group collaboration. Numbers of leading and contributing projects were used as metrics to measure collaboration among GPs. The team used collaboration data from Standard Monitoring Report published by Information and Technology Services. The data were accessed and downloaded on February 26, 2018. The team 66 used two sets of data: number of active lending projects and number of active ASA tasks.” In the collaboration data, a contributing GP is defined as: ƒƒ Co-TTL-ship from a GP other than the lead GP ƒƒ Contributions of over five weeks from another GP (could be distributed across multiple staff members) ƒƒ A qualitative decision (based on a discussion between the involved GPs and relevant CMU). Within this component, IEG conducted a thematic analysis of the World Bank’s Land-related projects. The team conducted a portfolio review of the projects containing a Land theme code in the Bank Group’s database and a desk review of information about Land-related projects. After the reorganization, the Land professional staff joined the Social, Urban, Rural, and Resilience GP within a dedicated Land unit; hence, the goal of the analysis was to identify potential quantitative and/or qualitative portfolio changes in the portfolio about knowledge flow and collaboration. IEG looked at 955 projects containing a Land theme code in the World Bank system. The evaluation team also analyzed the perceptions of interviewees on how the World Bank leverages trust funds to enhance knowledge flow and collaboration. The team relied on an analysis of all the notes of the semistructured interviews conducted by the team within the GP analysis component. Other quantitative data this component used include the following: the employee engagement surveys, the two-minute surveys, client surveys, corporate data on cross support via the Time Recording System, budget, sector codes, data provided by the knowledge management team in Chennai, Memoranda of Understanding, GSG/GL surveys, and the human resources team effectiveness study. Global Themes Analysis The goal of the Global Theme Analysis was to answer specific questions related to (i) how Global Themes were organized to mainstream corporate priorities, and (ii) the effectiveness of the support they provide to GPs and CMUs. The data collection and analysis relied on six semistructured interviews with staff and manager/director of Global Themes which covered the entire Global Themes universe. The interviewees were selected from people who have enough experience in respective Global Theme. GSG/GL Analysis The goal of this component was to assess the effectiveness of the GSG/GL model to contribute to desired outputs and outcomes around pooling and curating relevant country and global knowledge, Independent Evaluation Group | World Bank Group 67 as well as creating a strong technical stream. The team developed a nested theory of change establishing a causal link between the GSG/GL role, new operating model and intended outcomes (figure A.2). Through focus on GSGs and GLs, this evaluation component assessed the following: ƒƒ How GSGs and GLs were organized to facilitate knowledge flow and collaboration across GPs, GTs, Regions, and CMUs ƒƒ How GSG/GL model was performing ƒƒ Good practices and challenges in the GSG/GL model, and ƒƒ Alignment of GSG/GLs with the new operating model The data collection and analysis were organized in two phases. The first phase systematically gathered data on GSGs through GSG Spark or Intranet pages, organized GSGs by GP, analyzed GSG model survey results, met with GSG model survey staff, and conducted key informant interviews with GLs. The second phase focused on conducting a deeper dive into specific components of the GSG model, based on findings from phase 1. For example, the team conducted stakeholder interviews with GP staff and CMU staff on the GSG/GL model through country case studies. The GLs who were interviewed were randomly selected. IEG also tracked Global Leads’ record of publications in Google Scholar. The goal of this research exercise was to look at the prominence of the World Bank’s Global Leads in their field of expertise, as reflected by the number of their publications captured by the Google Scholar database. The research relied on a sample of 52 GLs (50 percent) out of 104 GLs; every other GL listed in the database was selected. Overall, the distribution of GLs researched across Practice Groups was as follows: Equitable Growth, Finance, and Innovation: 20 GLs (out of 41 GLs operating in Equitable Growth, Finance, and Investment); Human Development: 12 (out of 23 GLs in Human Development); and Sustainable Development: 20 (out of 40 GLs in Sustainable Development). Almost all references to GLs’ names were related to the recognition of co-authorship of publications; individual authorship of publications was very rare. Budgeting Arrangements Analysis Analysis of budgeting arrangements added insights on whether, how, and why budget issues and systems drove findings, via their influence on incentives. The team reviewed budget data across all sources of funds from FY15–18. These data were kindly provided by the Budget, Performance Review, and Strategic Planning VPU. The goal was to track trends and see if there were major discrepancies in allocation that would lead to constraints in the flow of knowledge and collaboration. The team discussed emerging findings in-depth with key informants. 68 Staff Surveys Analysis The perspectives of staff and various layers of management, as captured by surveys, were a major component in the evaluation. Analysis of the CMU and GP two-minute surveys and the employee engagement surveys from various years provided data on staff perceptions of the quality of collaboration and knowledge flow within and across groups. The employee engagement survey collects information on workforce perspectives related to employee engagement, including employee performance, career opportunities, managerial effectiveness, and provides information on various dimensions of knowledge flow and collaboration between and within units. For this evaluation, the employee engagement survey was analyzed to assess knowledge flow and collaboration trends before and after the implementation of the new operating model introduced in 2014. Specifically, the survey analysis focused on collaboration and knowledge flow across and within Bank Group units, between country offices and Washington, knowledge sharing across units, and internal processes that facilitate delivery of services to clients. The two-minute survey captures information on whether the new operating model is functioning as planned in the context of specific products and activities and focuses on the World Bank’s operational and analytical work (and excludes global and regional products). The two-minute survey consists of three separate surveys with different sets of questions, and survey respondents are specific staff identified when the survey is initiated. The three surveys are as follows: ƒƒ CMU Survey: respondents are Country Directors and Country Managers ƒƒ GP Survey: respondents are task team leaders and Practice Managers ƒƒ External/ Client/ World Bank Satisfaction Survey: respondents are clients in decision making or borrower roles, and implementing agencies For this evaluation, the two-minute survey analysis was conducted using the CMU, GP, and client surveys. Trends relating to knowledge flow and collaboration were analyzed based on aggregated quarterly data collection from December 2014 to October 2017. Country Program and Client Perspective Analysis IEG performed selected country programs and CMU analysis. The objective of this component was to look at knowledge flow and collaboration through the lens of country programs and to capture the views of CMU staff and GP staff based in field offices on the effectiveness of knowledge flow and collaboration. The country program analysis was covered through a combination of in-country visits and interviews by phone or in person with country management teams (mostly Country Directors, Country Managers, and Program Leaders) for Argentina, Azerbaijan, Botswana, Brazil, Bulgaria, the Caribbean, Chad, China, Central America, Côte d’Ivoire, the Arab Republic of Egypt, Malaysia, Mali, Independent Evaluation Group | World Bank Group 69 Myanmar, Pakistan, South Africa, Sri Lanka, Tunisia, Uganda, and Zambia. Additional criteria for country selection include regional diversity. The client perspectives on the delivery of World Bank services were assessed based on interviews with representatives of borrowers, sector ministries, and Project Management Units in five countries (that is, Egypt, Côte d’Ivoire, Azerbaijan, Pakistan, and Malaysia). Triangulating with document reviews and team interviews, this assessment answered questions related to whether (i) integrated solutions were truly called for in each task; (ii) collaboration led to better quality of the service, and (iii) knowledge was customized to the country context. Box A.1. Interviews and Focus Group Discussions As described in this appendix interviews were an important data source for many of the evaluation’s components. The initial scoping of the evaluation was informed by 27 broad semistructured interviews with key informants. Informant selection emphasized (i) involvement in the change process, (ii) a variety of roles, and (iii) individuals with unique insights. Semistructured interviews provided evidence for the GP, GT, GL, and CMU analysis. Staff and managers in each of the 13 GPs and 5 GTs were covered, with 71 interviews. GP interviews covered, for each GP, Practice Managers or directors, a TTL, quality assurance or development effectiveness staff, and knowledge management staff. Within a specific job description, there was a random selection from among the staff. For example, if there were six regional PMs in one GP, one of these was selected on a random basis to maintain neutrality across Regions. GT interviews covered managers or directors with enough experience in the respective unit. Nine GLs were interviewed, selected via random stratification across GPs. The clear majority of GP, GT, and GL interviews were done face to face. CMU interviews covered 21 CMUs from all six Regions and including large and small programs. In many CMUs the CD and some PLs were interviewed. In one instance, the CD, a country manager, and all three PLs were present, and the interview was really a focus group. In other instances, a country manager and a PL were interviewed. CMU interviews were done via a combination of phone or video calls and face to face, either during country visits or at headquarters. The CMU perspectives were complemented with interviews with client and country office staff in five visited countries. The interviews were semistructured, guided by templates. These templates varied depending on the purpose of the interview. Templates were tested and adapted based on experience. The CMU interview template covered knowledge flow (finding and bringing global expertise to clients, customization to country context, support from GTs and GSGs); collaboration (what works well, what doesn’t, PLs’ role, working with IFC, rewards and recognition); clients and results (service quality, demand); and the operating model overall. 70 Box A.1, continued Country office staff interviews covered the same topics. The GP interview template covered how the GP operates (including its knowledge production, curation, and management); working across Regions; working with other Groups (enablers and barriers for cross-GP support and collaboration, working with GTs); and influencing on cross-cutting priorities. The GL interview template was specific to that role. Interview findings were triangulated across types of interviews and with other evidence sources. For example, the evaluation findings on GLs and GSGs came from triangulating GL, GP, CMU, and country office staff interviews; the nested GSG/GL theory of change (figure A.2) which in turn was based on management documents; and a survey. The evaluation findings on PLs came from triangulating CMU and GP interviews with insights from country visits, including from client interviews. A small number of validation interviews and focus groups with key informants were also conducted. These tended to focus on deepening findings on specific topics and processes such as budget arrangements or links across the matrix. Semistructured focus groups were used to complement the interviews. Some went into depth on specific topics such as enablers and barriers to collaboration or budget processes (with participation by PMs and BPS staff). Focus groups with country office staff covered the same general topics as the GP and CMU interviews. Source: Independent Evaluation Group. 1 Sources for the Global Solutions Group/Global Lead Theory of Change include the Global Leads Terms of Reference, the Global Solutions Group model survey results, and semistructured interviews with Global Leads. Independent Evaluation Group | World Bank Group 71 Appendix B. The Previous Organogram Figure B.1. Organogram for the World Bank Operational Complex prior to the 2014 reforms 72 Knowledge Flow and Collaboration | Appendix B The World Bank 1818 H Street NW Washington, DC 20433