Approach Paper Evaluation of the World Bank’s Support to Procurement and its Principles March 21, 2023 1. Background and Context 1.1 In 2015, the World Bank approved a new procurement framework with the aim of reforming its approach to procurement. The overall objective was to enhance the World Bank’s development effectiveness, continue to exercise leadership in procurement for development, and remain at the forefront of public procurement as a setter of international standards (World Bank 2012a). The vision of the procurement reform was for investment project financing (IPF) procurement to help clients achieve value for money with integrity in delivering sustainable development (World Bank 2015b). Making procurement of works, goods, and services less of a barrier is critical to project success, especially in countries with lower institutional capacity to implement procurement. 1.2 The World Bank’s intention was to reform its procurement systems and the way procurement is supported in client countries. Although only applicable to IPF, the World Bank may support procurement capacity building at the level of a project or as part of the country dialogue, using a range of measures depending on the specific context of the country, sector, and agency. The reform also introduced procurement monitoring and support systems and influenced the reform efforts of other multilateral development banks to improve the development effectiveness of donor procurement in countries and of governments to promote internationally recognized good practices. To take forward the procurement reform, from fiscal year (FY)17 to FY20, responsibility for policy making and oversight lay with the Operations Policy and Country Services vice presidency of the World Bank, and responsibility for implementation lay with the Governance Global Practice in Equitable Growth, Finance, and Institutions (World Bank 2021b, 2022a). 1.3 The procurement reform emphasized seven core principles: value for money for decision-making, efficiency, economy, integrity, fairness, transparency, and fit for purpose. Value for money for decision-making is at the center of the procurement framework—which requires evaluating relevant costs and benefits, along with assessing risks, and encompasses the contextual application of the other core procurement principles to successfully use resources as appropriate. Efficiency is to be achieved by streamlining procurement information systems and focusing review on higher-risk and higher-value activities. Economy requires procurement to address cost and noncost 1 dimensions, including competitiveness, quality, life cycle costs and benefits, and sustainability (economic, environmental, and social). Fit for purpose is to be obtained by finding the “right solution” to clients’ procurement challenges given the context. Integrity, transparency, and fairness are to be promoted by protecting procurement from fraud and corruption, ensuring openness at all stages of procurement, and providing equal opportunities to bidders on contracts. 1.4 Anchoring the reform in a set of principles was intended to promote greater flexibility and more effective operational procurement. Although the previous guidelines recognized most of the principles, they were organized in relation to rules to be followed, which often led to risk aversion and limited innovation within operations to achieve the principles. The reform was expected to support a culture change to strategically and flexibly plan procurement to help clients implement good practices and support development processes (World Bank 2015b, 2017a). The reform also introduced new or enhanced areas to help clients implement procurement, summarized in box 1.1 (World Bank 2015b). Box 1.1. New or Enhanced Areas to Support Client Procurement in 2016 Reform The World Bank’s new procurement framework became effective in 2016. New or enhanced areas included the following. • Making the project procurement strategy for development (PPSD) a critical component in delivering procurement. The client can use the PPSD to identify the most appropriate procurement methods in a country context for achieving the procurement principles. The optimum procurement approach for each operation is intended to be based on an analysis of the project needs, market, risks, and other influencing factors identified through a PPSD. The PPSD can also be used to identify the project-specific technical assistance needed within the country’s broader governance environment. • Expanded use of innovative approaches and methods for procurement that support good practice application of the principles. Examples include expanded methods to support sustainable procurement, transparent electronic processes, and quality assessments in procurement decisions. • Increased client capacity building and institutional strengthening. Procurement staff have the dual role of supporting fiduciary insurance and capacity building. This includes technical assistance or training for clients on the procurement approaches introduced by the reform to support good practices. Capacity building activities may be identified in countries based on government ownership, the track record of existing programs, evident capacity gaps, and the need for additional resources. • Options for hands-on expanded implementation support to facilitate procurement where a need is identified, especially in low-capacity country situations. The World Bank may use this support at any given stage of procurement if it is useful to help the client achieve the development objectives and outcomes of an investment project financing operation. 2 • Development of an electronic system—Systematic Tracking of Exchanges in Procurement—to help carry out transparent and timely procurement and provide end- to-end transaction monitoring data. Systematic Tracking of Exchanges in Procurement is intended to help clients and World Bank teams plan and manage procurement and provide public information on procurement timelines by sector, type, agency, and country. It also provides monitoring procurement to inform decisions. It is a key instrument for improving efficiency, transparency, and accountability. • Use of alternative procurement arrangements by clients for projects. The procurement arrangements of other development banks, agencies, and organizations, including client implementing agencies, can be used for procurement if they meet the World Bank’s assessed standard. This was important to better align the World Bank’s procurement with that of other development banks and partners to make project implementation easier for country clients and support development effectiveness. • Enhanced approach to procurement-related complaints from suppliers, consultants, and contractors. Actions include an updated system to report complaints and a new standstill period of 10 calendar days, so a bidder has enough time to raise procurement-related complaints before a contract is formalized. • Training program to develop the skills of World Bank procurement staff. Training furthers the professionalization of procurement specialists supporting World Bank projects. • Streamlined requirements for the use of a country’s own national procurement system in World Bank projects. The framework supports a significantly larger use of a country’s own system for lower-value or lower-risk procurement below an agreed threshold. • Optimized prior review activity to free staff time for other support to client countries. The framework focuses on prior review of high-risk and high-value procurements, reducing reviews of lower-value and lower-risk procurements and freeing resources. • Increased support to contract management in the most significant procurements. This support includes procurement subject to review by the Operations Procurement Review Committee or identified for increased contract management support, including using key performance indicators. Sources: World Bank 2015b, 2021b. Continuous Change Management Process 1.5 The procurement reform is a continuous change management process. The history leading up to the reform is summarized in appendix B. The reform became effective for all projects approved starting July 1, 2016, with some projects being able, on an exceptional basis, to continue to apply the previous procurement and consultant guidelines (World Bank 2015b). Moreover, the approaches adopted by the reform, such as for more flexible procurement, likely have had a spill-over effect on other projects in the portfolio. Since the early consultations and the approval of the reform, ongoing reporting to track progress and gradual implementation of actions has taken place, including (i) training of staff and clients; (ii) development of guidance for operational 3 procurement in projects, such as for the project procurement strategy for development (PPSD), post review of procurement processes and alternative procurement arrangements (APAs); (iii) rollout of the World Bank’s information system—Systematic Tracking of Exchanges in Procurement (STEP); and (iv) revision of the project risk assessment and management system (table 1.1). There have also been resources for country capacity building, including the e-procurement tool kit, the global procurement partnership, the redevelopment of the methodology for assessing procurement systems, and reports on procurement data analytics. More recent efforts in the early stages of development include guidance on sustainable procurement for projects. Table 1.1. Timeline of World Bank Procurement Reform Launch and Tracking of Reform Dates Ongoing Implementation Board discussion on procurement reform. 2012 Board paper on procurement reform plan. Consultations with external stakeholders. Propose new procurement framework; other 2013 MDBs initiate similar reforms; IEG evaluation of procurement. Board endorsement of final procurement 2015 framework. New procurement policy, regulations, 2016 directives, and procedures effective. Includes launch of STEP, guidance on VfM, sustainable procurement, APAs, PPSD, e- procurement tool kit, and help desk. Board seminar on launching framework. Training of procurement staff and e-modules for task teams; client training. EFI annual Board updates, with no updates 2017 Updates of policy; contract management during COVID-19. guidance; standard bidding documents. 2018 Ongoing launch of new features; complaints; beneficiary ownership and direct payment pilots; assessment of PPSD; redevelopment of MAPS. Dashboard for procurement monitoring. 2019 Introduced emergency procurement guidance: World Bank–facilitated procurement for COVID- 19; launched the Procurement Risk Assessment and Management System integrated with SORT; Global Public Procurement Database. 2020 Introduced guidance for civil works, supporting ESF; launched post review system. EFI procurement progress report. 2021 Updated directive; introduced Global Procurement Partnership; World Bank report on green procurement. 4 Launch and Tracking of Reform Dates Ongoing Implementation Five-year procurement retrospective, 2022 Updated procedures; World Bank report on using informing next steps for strengthening data analysis in procurement; new supply chain procurement. management guidance; analysis of hands-on expanded implementation support. 2023 Forthcoming guidance for sustainable procurement; quality-based selection for VfM decisions. Sources: OECD 2018; World Bank 2012a, 2012b, 2013a, 2013b, 2014, 2015a, 2015b, 2016a, 2016b, 2016d, 2017a, 2017b, 2017c, 2018a, 2018b, 2018c, 2018d, 2019a, 2019b, 2020, 2021a, 2021b, 2021c, 2021e, 2022a, 2022b, 2022c, 2022d, 2022f. Note: APA = alternative procurement arrangement; EFI = Equitable Growth, Finance, and Institutions; ESF = Environmental and Social Framework; IEG = Independent Evaluation Group; MAPS = methodology for assessing procurement systems; MDB = multilateral development bank; PPSD = project procurement strategy for development; SORT = Systematic Operations Risk-Rating Tool; STEP = Systematic Tracking of Exchanges in Procurement; VfM = value for money. 2. Strategic Positioning of the Evaluation and Conceptual Underpinning 2.1 This evaluation by the Independent Evaluation Group (IEG) builds on a range of efforts to assess the World Bank’s procurement performance since the July 2016 procurement reform. The 2014 IEG evaluation of World Bank procurement was conducted to help guide decisions on the World Bank’s procurement reform (World Bank 2014). Since the launch of the World Bank’s reform and its procurement framework, Equitable Growth, Finance, and Institutions and Operations Policy and Country Services have led several internal reports to review the implementation of the reforms, including a five-year retrospective analysis (World Bank 2021c, 2022b, 2022d). IEG’s early evaluation addressing the health and social response to COVID-19 (World Bank 2022e) and its evaluation on mobilizing technology for development (World Bank 2021d) also looked at aspects of procurement. Conceptual Underpinning 2.2 The conceptual underpinning of the evaluation is the results logic, in terms of the breakdown of the results chain, of the World Bank’s procurement framework and its core principles (value for money, efficiency, economy, integrity, fairness, transparency, and fit for purpose) to support development outcomes (figure 2.1). The evaluation takes a systems approach to look at how procurement (since the 2016 reform) has supported and measured results toward these interlinked principles. The World Bank’s current approach suggests that procurement is successful when these principles are applied in combination in a country context. The immediate procurement principle to measure success is value for money for decision-making. Value for money for decision-making and other procurement principles may be estimated at any stage in a project, but the final outcomes are at the end of the project’s lifetime. Achieving value for money requires addressing other underlying principles (efficiency, economy, or noncost 5 considerations, such as quality and sustainability aspects, integrity, fairness, and transparency). The successful application of the procurement principles in a project requires fit for purpose approaches that are flexible and proportional to the risk of each process. 2.3 For IPF projects, the World Bank’s procurement framework is intended to provide additional resources to help clients achieve successful procurement through applying the principles. Since the 2016 reform, the World Bank has added more resources to help clients strategically and flexibly plan, implement, and monitor IPF procurement, such as the PPSD, STEP, renewed borrower assessments, and standard bidding documents. These resources are intended to help country clients achieve more efficient, economical, integral, fair, and transparent operational procurement that brings value for money in terms of supporting a project to achieve its development objectives. The intention is also to place a greater focus on contract management and consideration of environmental and social aspects and the life cycle cost of the project to facilitate better-quality goods, works, and services. 2.4 Client engagement for capacity building is an essential part of the World Bank’s procurement reform. Projects, especially in fragile and conflict-affected and lower- capacity countries, may receive assistance to help clients undertake effective operational procurement. This assistance may take the form of dedicated staff support, training, hands-on expanded implementation support, and APAs. Moreover, in some countries, IPF procurement support may be synergized with broader country or sector support to build procurement-related institutional capacities. For example, this support might consist of reforming procurement laws to better address sustainability considerations or developing e-procurement systems for greater efficiency. Capacity building is intended to develop knowledge, know-how, and systems to contribute to institutional capacities for applying procurement principles. Procurement capacity building may take the form of a dedicated project, activities embedded in projects, or advisory services and analytics (ASA). Whereas resources to support IPF procurement are often delivered by World Bank–accredited procurement specialists, capacity building for broader country reforms may be delivered by governance specialists or by task teams in various sectors and in coordination with partners. 2.5 The country-specific situation and procurement risk are at the base of the procurement framework and determine the context in which to strategize IPF procurement and additional capacity-building support. Factors of fragility, emergency, complexity, and capacity are among those that can create different country scenarios to apply procurement methods and approaches, balance trade-offs between speed and quality, and focus assistance for capacity building. Task teams and procurement specialists are expected to iteratively assess procurement arrangements and risks in 6 World Bank projects to update PPSDs and procurement plans and prepare actions to improve procurement. In some contexts, World Bank Country Management Units may also engage in dialogue about procurement capacity building challenges and plan targeted support with partners. 7 Figure 2.1. The Logic of the World Bank’s Procurement Reform Followed by the Evaluation Sources: World Bank 2015b, 2017a. Note: APA = alternative procurement arrangement; EFI = Equitable Growth, Finance, and Institutions; MAPS = methodology for assessing procurement systems; MDTF = multidonor trust fund; OPCS = Operations Policy and Country Services; OPRC = Operations Procurement Review Committee; SME = small and medium enterprise; STEP = Systematic Tracking of Exchanges in Procurement. 8 Added Value of the Evaluation 2.6 Consultations informed areas in which evaluation findings can contribute to learning to inform future actions for improving the World Bank’s procurement performance. The evaluation team consulted with procurement specialists, staff, and managers from Operations Policy and Country Services and Equitable Growth, Finance, and Institutions; country teams; task teams; the World Bank Board of Executive Directors; clients; and external experts. The following areas identified in these consultations frame the design of the evaluation. • Reviewing operational procurement outcomes. What is the success of the new procurement framework in supporting procurement outcomes through applying the seven core principles? It is important to highlight what is and is not working to ensure procurement is not a constraint on project development achievements, which was the key reason for the 2016 reform. The results assessment should also include a review of the metrics used by the World Bank to monitor, report, and evaluate procurement achievements. • Understanding the perspective of countries. The evaluation can shed light on how the World Bank’s procurement reform has changed the flexibility of procurement in countries with different institutional environments, including those affected by fragility and conflict and lower capacities to conduct procurement. Moreover, it is important to assess how the enhanced emphasis on capacity building since the 2016 reform has supported World Bank projects to successfully carry out smooth procurement processes with fewer obstacles and countries to improve institutional capacities for successful procurement. • Identifying the successes and challenges of implementation. The evaluation can help identify the successes of, and hindrances to, the reform process to date. 3. Objectives and Audience 3.1 The objective of the evaluation is to assess the results, successes, and challenges of the World Bank’s procurement since the reforms made in 2016 and thus help inform its continuation. The main audiences for the evaluation are the World Bank Board of Executive Directors and managers and staff supporting procurement in countries and using the new procurement framework. The secondary audiences include clients using the World Bank’s procurement framework in countries and development partners and practitioners who engage with the World Bank in countries to support procurement capacity-building activities. 9 4. Evaluation Questions and Scope Evaluation Questions 4.1 Motivating the evaluation is the following overarching question: To what extent has the World Bank’s reform supported successful procurement to help contribute to development outcomes? Underlying this question are two main lines of inquiry. 1. To what extent has the new procurement framework supported successful operational procurement in World Bank projects (that is, the achievement of fit for purpose, efficiency, economy, integrity, fairness, transparency, and value for money), and what are the success factors and challenges? 2. To what extent has the World Bank improved its procurement capacity building in countries since its procurement reform, including areas of support, tailoring to country needs, evidence of results, and success factors and challenges? Evaluation Scope 4.2 Four parameters determine the scope of this evaluation: portfolio coverage, time coverage, country perspectives, and conceptual boundaries. • Conceptual boundaries: The scope of the evaluation is defined by the results logic outlined for the World Bank’s reform (figure 2.1), including its principles and intended lines of support. The evaluation will focus on the World Bank’s operational procurement and any synergistic capacity building support in client countries. Emphasis will be placed on assessing how results toward the procurement principles are being achieved in practice, from early outputs to outcomes. We will focus on intermediate outcomes because the portfolio of IPF projects using the World Bank’s new procurement framework has few closed projects to look at procurement outcomes after the lifetime of the project (about 4 percent are closed). However, for projects of at least three years’ duration, we will use feedback from case studies as a proxy to estimate procurement outcomes. Corporate procurement and an internal audit of procurement functions are outside the evaluation’s scope. The World Bank Group’s internal audit already assessed World Bank–facilitated procurement and the World Bank’s Procurement Risk Assessment and Management System, which will not be reviewed in the current evaluation. The higher-level influences of procurement support on country development outcomes (the orange box in figure 2.1) are also outside the scope of the evaluation. 10 • Country perspectives: The focus will be on understanding the application and benefits of the World Bank’s 2016 procurement reforms in different types of country contexts. The reforms are intended to provide countries with more flexible and fit for purpose solutions to ease the constraint on procurement in World Bank projects and tailored support to help fragile and conflict-affected countries and lower-capacity countries implement procurement. • Portfolio coverage: We will look at two main portfolios: (i) IPF using the World Bank’s new procurement framework and associated systems reforms (such as STEP) and (ii) support to capacity building, which may consist of assistance within lending projects or ASA support to countries for procurement reforms. Data for these portfolios will come from STEP and the World Bank’s operational portal (appendix C). The portfolio of IPF using the new procurement framework is focused on operations financed by the International Development Association (IDA) and the International Bank for Reconstruction and Development (IBRD), given these projects are the main users of the World Bank’s operational procurement. The portfolio will focus on Global Practices where IPF is among the main lending instruments, including Governance; Finance, Competitiveness, and Innovation; Education; Social Protection and Jobs; Health, Nutrition, and Population; Agriculture and Food; Water; Social Sustainability and Inclusion; Urban, Disaster Risk Management, Resilience, and Land; Energy and Extractives; Transport; and Digital Development. The review of procurement assistance in IPF projects will focus on the use of hands-on expanded implementation support, APAs, and the allocation of procurement staff support to help clients across countries. The review of procurement capacity building will look at ASA and lending support (including IPF, recipient-executed trust funds, Program-for- Results financing, development policy operations, and institutional development funds) to country procurement reforms. The trust fund support will include the support of the Global Procurement Partnership multidonor trust fund. • Time coverage: The portfolio for IPF operations using the new procurement framework will focus on projects with a Concept Note approved on or after July 1, 2016. The IPF portfolio includes projects that are either closed or have been active for at least one year as of December 8, 2022, to ensure the evaluation focuses on projects with some implementation experience to carry out procurement activities. The portfolio for capacity building support focuses on July 1, 2016, to October 25, 2022. 4.3 Where possible, based on the sufficiency of available data, the evaluation will compare the procurement performance of IPF operations with benchmarks from past World Bank procurement. We will analyze STEP data on current IPF procurement over 11 time to see if progressive improvements have taken place over the six years of implementing the framework since the World Bank’s procurement reform was launched in 2016. Comparison data on IPF procurement will draw on the previous IEG evaluation, where possible, because it includes some qualitative and quantitative benchmarks assessing procurement before the 2016 reform, including perceptions and data on efficiency (World Bank 2014). However, limited data on procurement are available from before the introduction of STEP. Procurement capacity building support will be compared with the portfolio before the 2016 World Bank reform (approved from July 1, 2010, to June 30, 2016). 4.4 The preliminary portfolio includes IPF operations across Global Practices and a mix of procurement capacity building activities. The preliminary IPF portfolio includes 713 projects (54 percent IDA, 40 percent IBRD or IDA and IBRD blend, and 6 percent regional projects) using the new procurement framework. The preliminary portfolio covers projects across Practice Groups: 33 percent in Human Development, 32 percent in Sustainable Development, 21 percent in Infrastructure, and 14 percent in Equitable Growth, Finance, and Institutions. The estimated portfolio supporting procurement capacity building estimates 106 lending projects, 80 development policy loans, and 349 ASA activities. These projects and ASA range from those that provide dedicated support to procurement capacity building, such as for e-procurement systems, or public financial management assessments that cover procurement, to those with a few procurement support activities embedded in their components. 5. Evaluation Design and Components Evaluation Design 5.1 The following approaches anchor the evaluation design. • Multilevel analysis and triangulation: The evaluation will bring together results from different levels (country, project, and procurement activity) and types of analyses. For example, evidence from the portfolio, country situation, case study, and perception analyses will be combined to shed light on the extent to which the World Bank’s procurement reform has supported successful procurement to help contribute to development outcomes. We will use a mixed methods approach that combines quantitative and qualitative evidence. The emphasis will be on procurement support within projects and countries to ensure that the evaluation’s learning informs efforts to strengthen the effectiveness of procurement in the World Bank’s country programs. 12 • Theory-based design: The evaluation follows a theory-based design, built on the results logic of the World Bank’s procurement reform (figure 2.1). The results logic will guide the protocols used for data collection and analyses to collate evidence from the evaluation components and answer the evaluation questions. We will also develop more detailed, nested theories of change to map the results chain, from early outputs to outcomes, to advance procurement principles and final procurement outcomes. The theories of change will help further guide data collection and interpretation of findings. Insights into how the procurement principles are applied in countries will be used to refine the results logic and theories of change outlined by the World Bank’s procurement reform. Evaluation Components 5.2 The evaluation components at each level are outlined in the next section (IPF operational procurement portfolio analysis, procurement capacity building portfolio analysis, country situation analysis, case analysis, and analysis of perceptions). See appendix A for the evaluation design matrix. Country Level 5.3 IPF operational procurement portfolio analysis: The portfolio analysis of IPF procurement will assess results, successes, and challenges. The focus will be on disaggregated analysis to understand the procurement achievements of projects in different country contexts (for example, fragile and conflict-affected situations, IDA, and IBRD) and how achievements differ for different types of procurement (for example, by the level of complexity, support to COVID-19 or other crises, risk, volume, and amount). The IPF portfolio analysis covers the following areas. • STEP data: STEP data on IPF operational procurement transactions (contracts, timelines, suppliers, bids, procurement plans, and complaints) will be used to assess achievements toward procurement principles, from early outputs to outcomes.1 To inform the indicators to measure the principles, the evaluation will review existing indicators used by the World Bank and proposed in the literature. Where limited data are available, we will identify proxy indicators and outputs that can inform progress. Indicators may include the speed of procurement, contracts awarded to local firms, balance of prior and post review, use of market analysis, correlation between procurement and project ratings, uptake of innovative procurement methods, and use of quality criteria in procurement methods. The review of measurement information will also help the evaluation analyze what is being tracked in procurement progress reports 13 and dashboards by the World Bank. It will help respond to the question of whether it is providing the results information needed to support clients. • Project documents and reports: Data from Implementation Status and Results Reports and the PPSD repository will be coded to look further at successes and challenges. The analysis will be used to understand strategic support for procurement achievements and procurement issues reported in Implementation Status and Results Reports. Content extracted from documents will be analyzed using natural language processing techniques as much as possible. • Modeling: The evaluation will explore econometric analysis or another relevant modeling analysis of the IPF procurement portfolios to understand what has supported good procurement ratings and good achievement of principles for different types of procurement and countries. 5.4 Procurement capacity building portfolio analysis: The portfolio analysis of capacity building will use a consistent protocol, including text analytics, to assess procurement assistance within country portfolios and support to procurement reforms in countries. We will review the intermediate outcomes achieved and the types of activities supported, emphasizing how this support has benefited different types of countries (for example, fragile and conflict-affected situations and countries with different levels of procurement risk and existing capacity). • Procurement assistance within country portfolios: We will look at the benefits of activities, such as hands-on expanded implementation support, APAs, and time dedication of procurement staff, in countries. This will be done by analyzing STEP and data on staff support across countries. Moreover, procurement assistance will be one of the topics asked about in the case study interviews. • Support to procurement reforms in countries: We will look at projects and ASA, which include capacity building activities, such as methodology for assessing procurement systems and support to e-governance, to reform country procurement. The results of capacity building will be captured by assessing available project indicators, and for closed projects in countries extracting evidence on achievements from documents. Country and sector studies assessing procurement institutional capacities will be reviewed to develop a checklist that can help categorize portfolio information by capacity building activities, challenges, and types of intermediate outcomes. 5.5 Country situation analysis: A heat map will be constructed to look at how institutional capacities in areas of the procurement principles may affect operational procurement achievements. The literature will be reviewed to help identify the best 14 available indicators for the heat map from available public data sources (such as the Global Public Procurement Database). The focus will be on countries that use more IPF procurement to understand how the institutional capacity characteristics of a country may facilitate or hinder World Bank operational procurement. We will look at the alignment between the successful application of procurement principles by IPF projects and the procurement institutional capacities of the country. For example, we will try to understand whether World Bank operational procurement is better in countries with procurement systems that have greater efficiency and transparency. Another question is whether these institutional capacities have helped countries make better use of the innovative procurement methods in the World Bank’s new procurement framework. We will also look at the alignment among the procurement capacities of countries; the provision of World Bank procurement assistance, such as hands-on expanded implementation support and capacity building support; and procurement problems reported by World Bank projects. Clustering analysis of the heat map data will be explored to understand whether countries can be grouped in terms of differences in their procurement capacities; the types of procurement they carry out, in terms of complexity, procurement volume, and use of innovative methods; and their operational procurement achievements. Project and Procurement Activity Level 5.6 Case analysis: The aim of the case study analysis is to understand the factors that make operational procurement more or less successful. A case analysis protocol will be designed to examine how projects successfully or unsuccessfully operationalized strategic procurement approaches and fit for purpose methods to apply procurement principles in countries and trade-offs that may have been made (for example, between speed and quality procurement processes). The case analysis will examine a random sample of about 60 projects, including projects with high procurement ratings (highly satisfactory or satisfactory) and low ratings (moderately unsatisfactory or unsatisfactory). The project will have at least three years of implementation to allow for cases to estimate procurement outcomes and be clustered to cover IDA, IBRD, and fragile and conflict-affected countries in different regions and sectors. The data collection will encompass interviews with task teams, procurement staff, suppliers, and clients and a review of project procurement documents, including the PPSD and selected procurement activities. We will draw insights into the project’s procurement outcomes and how and why the project has applied or not applied procurement methods and arrangements in a fit for purpose way in the country context. 5.7 Analysis of perceptions: The short survey will include questions to understand what has been working and not working in the World Bank procurement system since 2016, successes and challenges in the flexible application of the procurement principles, 15 and satisfaction with their achievement in a project context. The questions will include closed and open-ended formats to facilitate the synthesis of responses and will also allow accounts from experience. The survey will be sent to a selective sample of task teams and procurement staff who work on projects that have used the World Bank’s new procurement framework for at least four years to ensure feedback on the procurement system benefits since the 2016 reform. An adaptive approach will be taken to follow up with staff to try to obtain representation across projects with both satisfactory and unsatisfactory procurement ratings in IDA, IBRD, and fragile and conflict-affected countries. Limitations 5.8 The main limitations of the evaluation components are as follows: • The focus of the evaluation is on a mainly active IPF portfolio, limiting the focus to completed procurement activities within these projects and assessing intermediate outcomes. A later assessment could look at the final procurement outcomes at the end of a project’s lifetime. • The evaluation will use available data to estimate achievements for procurement principles from STEP data (for example, speed of procurement, use of quality criteria, the correlation between procurement ratings and outcomes, and use of competitive bidding). These indicators will be compared across the portfolio for similar projects. The availability and quality of STEP data in the World Bank’s information systems will limit the calculation of evidence on the procurement principles and their achievement. Components of STEP, such as the database for post review and the complaints module, have been rolled out at different times. Moreover, the transition period to help clients use STEP suggests that data will likely be more complete for recent years. Furthermore, indicators for procurement principles can be estimated at different stages in a project’s procurement life cycle using a range of approaches. In particular, value for money is a soft concept that can be calculated in different ways and cannot be fully evaluated until the end of the life of the asset. The definition of indicators will draw on expert guidance, the literature, and available World Bank guidance. • Regarding procurement capacity building, the focus will be on intermediate outcomes because capacity achievements can, in some cases, take many years to advance. Moreover, we will follow up with selected project and ASA task teams, where documents lack details on their achievements to help ensure a good capture of results. 16 6. Quality Assurance Process 6.1 The evaluation will go through IEG’s quality assurance processes to ensure the quality and usefulness of evaluation findings. This includes interview process requirements and independent external review by five peer reviewers: Charles Kenny, senior fellow at the Center for Global Development and former World Bank Governance staff; Jeff Taylor, deputy director general of procurement, and Dharmesh Mahendra Dawda, senior procurement specialist, at the Asian Development Bank; Mihály Fazekas, professor at Central European University, data scientist, and procurement and governance expert; and Sope Williams-Elegbe, professor at Stellenbosch University and deputy director of the African Procurement Law Unit. Peer reviewers will be consulted regularly during the evaluation, as well as at key decision points. 6.2 The evaluation will complement IEG’s regular quality assurance through informal engagements with advisers and an advisory panel. Frannie Léautier (Senior Partner at SouthBridge Group and former vice president at the World Bank and African Development Bank) will serve as a senior expert adviser to the evaluation. Kirsten Ejlskov Jensen serves as a technical adviser to the evaluation (former United Nations procurement expert). Some of the peer reviewers will also advise the team on specific issues within their areas of expertise. We will also hold virtual workshops with a small advisory panel of experts from the private sector, international associations, and client regions to gather feedback and help interpret key lessons and messages from the preliminary findings of the evaluation. 7. Expected Outputs and Outreach 7.1 The evaluation timeline is FY23–24, with the expected review of the evaluation report by IEG and World Bank management and the Committee on Development Effectiveness discussions in FY24. The final output will be a report of up to 20,000 words including appendixes. 7.2 Throughout the evaluation, IEG will continue its consultative engagements with World Bank staff and external experts to shape the evaluation methods and interpret findings from the evaluation. Participants in consultative engagements will vary by topic. For example, technical consultations on case analyses may be held with procurement specialists with country experience. 8. Resources 8.1 The evaluation will be prepared with an estimated budget of US$785,000. The task team leaders of the IEG evaluation are Jenny Gold and Elisabeth Goller. Core team members for the evaluation are Santiago Ramirez Rodriguez, Deryck Brown, Ariya 17 Hagh, Michael Premson, Kirsten Roster, Isabelle Adams, and Gaby Loibl. The work will be conducted under the guidance of Estelle Raimondo (methods adviser), Galina Sotirova (manager), Carmen Nonay (acting director, Human Development and Economic Management), and Oscar Calvo-Gonzalez (acting Director-General, Evaluation). 1 Systematic Tracking of Exchanges in Procurement (STEP) is the World Bank’s procurement information system that streamlines and automates key stages of the procurement process and enables monitoring and reporting. It helps the World Bank and clients to plan, record, and track the procurement processes of investment project financing. The STEP system includes data on transactions for procurement activities and contracts, timelines, methods of procurement activities in investment project financing, procurement complaints, procurement notices, bids, suppliers, no objections, post review, project procurement risk assessments, and procurement accreditation. STEP data will be used to estimate indicators on procurement for the principles. For example, indicators estimating efficiency may include the time to process contracts, use of framework agreements, the number of failed transactions, and cost overruns. Indicators to estimate economy may include the use of qualitative methods and use of local suppliers. Integrity, fairness, and transparency may be estimated by looking at complaint handling, bid publication, competitive bidding, and the number of new suppliers engaged. Fit for purpose may be estimated by looking at data on the use of market analysis, amount of the procurement, complexity, flexibility to apply new methods, and procurement risk ratings. Value for money may be estimated by looking at the composite estimate of indicators for procurement principles for a project and the correlation between procurement ratings and outcomes. Case analyses, survey data, capacity-building data, and data from the country situation analysis will be triangulated with STEP data to understand perceptions more deeply and provide additional data on procurement achievements and variations across countries. 18 Bibliography GDELT Project. 2022. GDELT 1.0 Event Database, The GDELT Project (accessed November 1, 2022). https://www.gdeltproject.org. GTI (Government Transparency Institute). 2017. “World Bank Public Procurement Data for Fiscal Years 1998–2013.” Data Publication, GTI, Budapest. https://www.govtransparency.eu/data-publication-world-bank-public-procurement- data-for-fiscal-years-1998-2013. OECD (Organisation for Economic Co-operation and Development). 2018. Methodology for Assessing Procurement Systems. Paris: OECD. https://www.mapsinitiative.org. World Bank. 2012a. The World Bank’s Procurement Policies and Procedures: Policy Review . Approach Paper. Washington, DC: World Bank. World Bank. 2012b. The World Bank’s Procurement Policies and Procedures: Policy Review. Initiating Discussion Paper. Washington, DC: World Bank. World Bank. 2013a. Procurement in World Bank Investment Project Finance: Phase I: A Proposed New Framework. Washington, DC: World Bank. World Bank. 2013b. Procurement Policy Review: Feedback from Consultations with External Stakeholders. Washington, DC: World Bank. World Bank. 2014. The World Bank Group and Public Procurement—An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2015a. Procurement Framework—Implementation Strategy and Plan. Washington, DC: World Bank. World Bank. 2015b. Procurement in World Bank Investment Project Financing: Phase II: The New Procurement Framework. Washington, DC: World Bank. World Bank. 2016a. Methodology to Assess Alternative Procurement Arrangements in Borrower Implementing Agencies for Procurements Financed under IPF. Washington, DC: World Bank. World Bank. 2016b. “New Procurement Framework: Update on the Status of Implementation .” Internal Seminar, World Bank, Washington, DC. World Bank. 2016c. Procurement Post Review, Independent Procurement Review, and Integrated Fiduciary Reviews. Washington, DC: World Bank. World Bank. 2016d. Project Procurement Strategy for Development: Long Form Detailed Guidance. Washington, DC: World Bank. World Bank. 2017a. Bank Policy: Procurement in IPF and Other Operational Procurement Matters. Washington, DC: World Bank. 19 World Bank. 2017b. Procurement Guidance: Contract Management General Principles. Washington, DC: World Bank. World Bank. 2017c. Procurement in World Bank Investment Project Financing: Procurement Framework—Implementation Update, July 1, 2016, to June 30, 2017 . Washington, DC: World Bank. World Bank. 2018a. “2016 Procurement Framework: Mandatory Direct Payment and Beneficial Ownership.” Private Sector Seminar, World Bank, Washington, DC. World Bank. 2018b. Procurement in World Bank Investment Project Financing Procurement Framework—Implementation Update, July 1, 2017, to June 30, 2018. Washington, DC: World Bank. World Bank. 2018c. Procurement Risk Global AI Tool (PRAIT): Project Scoping Report. Washington, DC: World Bank. World Bank. 2018d. Project Procurement Strategy for Development (PPSD) Review: Detailed Analysis Report. Washington, DC: World Bank. World Bank. 2019a. New Complaints Module in Systematic Tracking of Exchanges in Procurement (STEP) System. Washington, DC: World Bank. World Bank. 2019b. Procurement Risk Assessment Management System (PRAMS): User Guide. Washington, DC: World Bank. World Bank. 2020. The World Bank: Procurement Regulations for IPF Borrowers. Washington, DC: World Bank. World Bank. 2021a. A Global Procurement Partnership for Sustainable Development: An International Stocktaking of Developments in Public Procurement. Washington, DC: World Bank. World Bank. 2021b. Bank Directive: Procurement in IPF and Other Operational Procurement Matters . Washington, DC: World Bank. World Bank. 2021c. Influencing Transformative Public Procurement: A World Bank Report. Washington, DC: World Bank. World Bank. 2021d. Mobilizing Technology for Development: An Assessment of World Bank Group Preparedness. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2021e. Procurement Sexual Exploitation and Abuse and Sexual Harassment (SEA/SH) Prevention and Response Measures: Contractor/Sub-contractor Disqualification for Breach of SEA/SH Contractual Obligations: Questions and Answers for Borrowers and Contractors. Washington, DC: World Bank. World Bank. 2022a. Bank Procedure: Procurement in IPF and Other Operational Procurement Matters. Washington, DC: World Bank. 20 World Bank. 2022b. “Development of Public Procurement: Key Initiatives and Future Outlook.” Procurement Report, World Bank, Washington, DC. World Bank. 2022c. Hands-on Expanded Implementation Support (HEIS)—Latin America and the Caribbean (LAC) Region. Washington, DC: World Bank. World Bank. 2022d. Procurement in World Bank Investment Project Financing: Review of Results after Five Years of Implementation of the Procurement Framework, FY17 to FY21. Washington, DC: World Bank. World Bank. 2022e. The World Bank’s Early Support to Addressing COVID-19: Health and Social Response to COVID-19: An Early-Stage Evaluation. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2022f. Using Data Analytics in Public Procurement: Operational Options and a Guiding Framework. Washington, DC: World Bank. 21 Evaluation Design Matrix Table A.1 indicates the key questions, sources, data collection and analysis methods, and their associated limitations. The overall question is as follows: To what extent has the World Bank’s reform supported successful procurement to help contribute to development outcomes? Question 1. To what extent has the new procurement framework supported successful operational procurement in World Bank projects (that is, the achievement of fit for purpose, efficiency, economy, integrity, fairness, transparency, and value for money), and what are the success factors and challenges? Table 0.1. Evaluation Design Matrix for Question 1 Information Required Information Sources Data Collection and Analysis Methods Limitations Results for procurement World Bank information system IPF operational procurement portfolio: Indicators assessed will data on IPF project and data principles since the reform • Procurement results: STEP data on IPF operational procurement will be used to be limited by available for different types of from STEP. assess indicators for procurement principles that can point to early outputs and data and data quality countries, and information Nested theories of change to outcomes. The analysis will be disaggregated by country (for example, fragile issues of STEP. for the counterfactual. help outline the results chain and conflict-affected situations, IDA, and IBRD) and different types of We will focus on Measurement information and literature and World Bank procurement, such as the complexity and sector, to understand achievements intermediate outcomes on principles. documents on possible in different contexts. To inform the analysis, we will review existing indicators because most projects Procurement successes, indicators to measure for used by the World Bank and in the literature to measure the procurement in the portfolio are challenges, and procurement principles. principles and will draw on the logic of the framework and nested theories of active. innovations. PPSD repository and ISRs. change. The review of measurement information will also look at the adequacy of what is currently being tracked in World Bank reports and dashboards. • Procurement innovations and challenges: As part of the IPF portfolio analysis, content from a sample of PPSDs will be coded to understand their support for procurement achievements, and procurement issues in ISRs will be coded to identify the successes and challenges. Content extracted from documents will be analyzed using natural language processing techniques as much as possible. • Econometric analysis: This analysis will explore modeling to understand what has supported good procurement ratings and good achievement of principles for different types of procurement and different types of countries. Procurement successes Data collection will follow a Case analysis: The case analysis will examine a random sample of projects that fall Cases will be limited to and challenges of projects protocol to select sample cases in the top and bottom quartiles of the World Bank portfolio for their procurement about 60 projects across 22 Information Required Information Sources Data Collection and Analysis Methods Limitations in countries and estimated and codify information from rating to learn where the new procurement framework has been successful or has IDA, IBRD, and fragile outcomes. interviews with task teams, faced challenges. We will focus on projects with at least three years of and conflict-affected procurement staff, suppliers, implementation using the World Bank’s procurement framework, clustered across countries, with low and and clients and document IDA, IBRD, and fragile and conflict-affected countries. The case analysis protocol high procurement review for inference across will be designed to consistently capture information on procurement successes ratings (highly cases. and challenges related to operationalization of procurement principles in a project satisfactory or and to estimate final procurement outcomes. In addition to interviews, we will look satisfactory compared at project documents, such as the PPSDs. with moderately unsatisfactory or unsatisfactory). Perceptions of operational Survey of task teams and Analysis of perceptions: A short staff survey will include questions related to what The survey could have a procurement. procurement specialists. is working and not working in the World Bank procurement system, successes and low response rate. challenges in the flexible application of the procurement principles, and satisfaction with their achievement in a project context. The survey will focus on a selective sample of task teams and procurement staff who have at least four years of experience implementing a project using the new procurement framework to ensure feedback on recent procurement experiences. Source: Independent Evaluation Group. Note: IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IPF = investment project financing; ISR = Implementation Status and Results Report; PPSD = project procurement strategy for development; STEP = Systematic Tracking of Exchanges in Procurement. Question 2. To what extent has the World Bank improved its procurement capacity building in countries since its procurement reform, including areas of support, tailoring to country needs, evidence of results, and success factors and challenges? Table 0.2. Evaluation Design Matrix for Question 2 Information Required Information Sources Data Collection and Analysis Methods Limitations Types of learning activities Data on HEIS, training, and staff Portfolio supporting capacity building: Consistent protocol, including text analytics, Projects may not have supporting procurement in time in IPF projects; project will be used to identify, categorize, and analyze client assistance and capacity good documentation of projects and their benefits. reports. building support for operational procurement and broader country reforms: capacity building Types of activities and Data on projects and advisory • Procurement assistance to IPF portfolio in countries: Activities such as HEIS and activities and evidence of results services and analytics procurement staff support will be analyzed in the portfolio for different types achievements. supporting procurement supporting procurement of countries. Evidence will also draw on projects selected for case analysis. capacity building and reforms and comparison data • Support to procurement reforms in countries: Capacity building activities will be reforms in countries. reviewed in the portfolio using text analytics and assessment of project 23 Information Required Information Sources Data Collection and Analysis Methods Limitations on capacity building support indicators. Evidence of achievements will be extracted from project documents from the previous portfolio. for closed projects. Studies assessing capacity building to identify results areas from the literature. Information on differences The heat map will draw on Country situation analysis: A heat map will be constructed to look at how the Data on institutional in procurement successes portfolio data and secondary institutional capacities in countries related to procurement principles may affect capacities related to and challenges across data sources with information World Bank project procurement. The literature will be reviewed to identify the procurement will be countries with different on institutional capacity factors best available indicators for the heat map. The focus will be on countries with estimated based on institutional contexts. related to procurement greater use of IPF procurement. We will look at the alignment between the success available indicators. principles, such as from the of operational procurement in projects and the institutional capacities of the Global Public Procurement country. For example, we will assess whether the country has policies supporting Database and Procurement sustainable procurement to understand if these institutional capacities helped Anticorruption countries make better use of the innovations and flexibility in the World Bank’s and Transparency platform. new procurement framework. We will also look at the alignment between the provision of procurement capacity building support and procurement challenges in countries. Clustering analysis of the heat map data will be done to understand whether countries can be grouped in terms of differences in their procurement capacities and characteristics (such as procurement ratings, complexity, or use of innovative methods). Source: Independent Evaluation Group. Note: HEIS = hands-on expanded implementation support; IPF = investment project financing; MAPS = methodology for assessing procurement systems. 24 Figure 0.1. Diagram Providing an Overview of How the Evaluation Methods Will Inform the Questions Source: Independent Evaluation Group. Note: ASA = advisory services and analytics; IPF = investment project financing; ISR = Implementation Status and Results Report; STEP = Systematic Tracking of Exchanges in Procurement. 25 Evolution of the World Bank’s New Procurement Framework Several reasons justified the World Bank’s 2016 procurement reform. By the early 2000s, the World Bank’s procurement and consultant guidelines were complex and risk-averse and were not providing the needed flexibility to support different country situations. Although the guidelines were updated at several points historically, a full procurement reform had not taken place since their introduction. Moreover, many international agenda commitments had been made between the 1990s and 2011 that shaped the need for World Bank procurement reform, notably the 2005 Paris Declaration on Aid Effectiveness, its companion piece on governance and anti-corruption, and the international commitments made in 2011 in Busan on aid effectiveness. The procurement reform also built on learning from the World Bank’s use of country systems pilot pre- 2012. Procurement was seen as a major bottleneck to project performance, especially in fragile and conflict-affected and low-capacity countries. In addition, the guidelines were not well tuned to the wide range of procurement needs across sectors that had expanded over the years with World Bank operations. For example, some sectors have many small procurement activities. Moreover, rapid globalization changed markets in countries and the availability of suppliers, goods, and services, and there was a need for greater alignment of procurement approaches across development partners (World Bank 2012a, 2012b, 2014). It was increasingly important to shift the culture of procurement to apply good practices for realizing principles such as value for money to ensure efficiency, economy, integrity, fairness, and transparency in a fit for purpose way that fit the country context. World Bank stakeholders gave feedback that these principles should be emphasized more systematically in the World Bank’s procurement and the use of methods and approaches that could help projects and countries address them in a flexible, relevant way should be expanded. For example, such methods and approaches can help address quality and noncost considerations, such as social, gender, and environmental factors in procurement decisions. Moreover, modern information technologies had opened new opportunities for transparency and efficiency in procurement management and oversight. As part of strengthening oversight, it was important to shift the focus from the up-front procurement task to the full procurement cycle, from strategy and planning to contract management and the end-of-life benefits of the procurement. This shift would help address failures as a result of procurement being inadequately carried out after bidding is complete (World Bank 2012a, 2012b, 2014). The World Bank and other development partners increasingly recognized the need to strengthen procurement systems in client countries. Clients needed support to 26 strengthen their procurement systems, develop their own markets, and align with international good practices. This included support in reforming procurement laws, e- procurement, sustainable procurement, and development of competition in local markets. Some capacity building had occurred previously, but more strategic attention was needed to help countries address capacity constraints on procurement outcomes (World Bank 2012a, 2012b, 2014). The World Bank’s previous procurement and consultant guidelines needed modernization. They often led to suboptimal results, and too much staff time was spent reviewing low-value and low-risk transactions. World Bank support needed to be more proportional to risk, so that staff could free time previously spent on small processes to better support clients. Moreover, routine procurement information was needed to inform decisions and respond to procurement bottlenecks that could impede project success. Procurement data were sparse, with disjointed information technology systems that made monitoring procurement performance difficult. In 2012, these concerns prompted World Bank management to initiate a procurement reform process (World Bank 2012a, 2012b, 2022). To design the procurement reform, the World Bank held extensive consultations with client countries, development partners, civil society organizations, academia, and the business community. The following guiding themes emerged: (i) the need for a robust set of principles to guide the World Bank’s operational procurement, reflecting internationally recognized principles; (ii) use of procurement methods that are context specific, proportional, and fit for purpose and that create room for innovation to address quality, which requires consideration beyond issues of lowest cost; (iii) use of country systems with due caution and management of risks in a country context; (iv) use of e- procurement and other technologies; (v) special attention to the needs of fragile and conflict-affected countries, along with small economies, which faced unique procurement challenges; (vi) increased support for capacity building, especially for lower-income countries; (vii) more up-front market analysis and strategic thinking about the shape of markets, competition, and impacts, especially on local industry; (viii) coverage of the full procurement cycle through contract execution, with more attention on performance; (ix) expansion of governance and integrity aspects, including transparency, access to information, monitoring by civil society, internal controls such as more effective audits, and dispute resolution mechanisms; (x) renewed “push” on advancing the global context for public procurement (harmonization, international standards, and capacity assessments); and (xi) review of the World Bank’s internal organization to support procurement, roles and responsibilities, staffing, efficiency, dispute handling, internal collaboration, and global partnerships (World Bank 2013a, 2013b, 2015). 27 References World Bank. 2012a. The World Bank’s Procurement Policies and Procedures: Policy Review. Approach Paper. Washington, DC: World Bank. World Bank. 2012b. The World Bank’s Procurement Policies and Procedures: Policy Review . Initiating Discussion Paper. Washington, DC: World Bank. World Bank. 2013a. Procurement in World Bank Investment Project Finance: Phase I: A Proposed New Framework. Washington, DC: World Bank. World Bank. 2013b. Procurement Policy Review: Feedback from Consultations with External Stakeholders. Washington, DC: World Bank. World Bank. 2014. The World Bank Group and Public Procurement—An Independent Evaluation. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2015. Procurement in World Bank Investment Project Financing: Phase II: The New Procurement Framework. Washington, DC: World Bank. World Bank. 2022. Procurement in World Bank Investment Project Financing: Review of Results after Five Years of Implementation of the Procurement Framework, FY17 to FY21 . Washington, DC: World Bank. 28