Measuring Public Procurement Rules and Practices Benchmarking a Recurrent Infrastructure Contract

The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.


Introduction
The process by which governments purchase goods, services, and works from the private sector has significant implications for the business environment.Governments are the sole or principal buyer in many markets critical for economic development such as education, health, and infrastructure.The sheer size of these "government markets" makes procurement an important policy area to harness for economic development.A recent study estimates the size of the global procurement market at US$ 13 trillion, 1 noting at the same time that only 2.8% of the market's total value is accounted for in publicly available data.Thus, data availability and government transparency continue to be an obstacle for research on government spending and efficiency.
Government markets can range from a simple purchase of office furniture to a complex infrastructure project.Despite the vast range of government markets, and subsequently of optimal purchasing strategies, the foundation of all procurement contracts is based on regulations and their implementation.Markets themselves exist because governments decide that the private sector will be more efficient in providing the required good, service or works than themselves.Tenders and contracts are governed by a combination of binding rules (laws, regulations, tender documents and contracts).Thus, market conditions are heavily influenced by government regulations, rather than by the behavior of market participants. 2e literature on public procurement has identified regulatory principles that are associated with better outcomes.Transparency, competition, accountability, efficiency and innovation are most commonly noted as enablers to achieve best value for money (see Section 2 for an overview of relevant papers).Sound public procurement practices are an important policy tool to support poverty reduction, while inefficient public procurement can be extremely costly.Inefficiency compromises market access and competition, and raises the price paid by public entities for goods and services, directly impacting public expenditures and, therefore taxpayers' resources.
Despite the significance of public procurement, large-scale, frequently updated and comparable data on such processes are scarce.While high-income economies have been collecting more and more data on their own procurement systems, little work has been done in middle-and low-income economies.The result is that most of the economic research on public procurement has relied on country-level databases from highincome OECD economies.
To address this knowledge gap, in 2016 the World Bank started to design a new indicator to measure public procurement regulations that would have a global scope and could be updated yearly. 3In the following years, the questionnaire was refined in consultation with internal and external experts, the assumptions of the indicator were laid out and data were collected through an expanded pool of respondents and in an increasing number of economies.As a result, in 2019 the World Bank published a brief chapter on the contracting with the government indicator, including some descriptive findings from the data collected across years. 4The data set was subsequently published in 2020 on the Doing Business website. 5e goal of this paper is to provide the full technical background of the methodology foundations of the indicator and describe additional findings.It uses the latest iteration of cross-country data, obtained for 2020. 6This indicator should easily inform researchers and policy makers on areas that can be improved to promote better governance and better private sector development in government markets.
From a policy perspective, this research exercise aims to: (i) encourage regulatory reforms aimed at developing domestic private sector participation in government markets through streamlined and efficient procurement regulations; and (ii) incentivize governments to innovate and improve transparency to attract more local companies to participate in public procurement.
Designing comparative indicators on public procurement is relevant especially now, when the COVID-19 crisis and subsequent disruptions to global supply chains have increasingly strained governments' ability to achieve the best value for money and to ensure timely and efficient delivery of public goods and services.

Literature review a) Economic literature
A big part of the existing literature focuses on identifying the pitfalls of current public procurement systems.By identifying undesirable outcomes, policy makers can learn how to improve public procurement frameworks.For instance, Porter and Zona (1993) examine bidding in auctions for US state highway construction contracts, in order to determine whether bid rigging occurred, finding that collusion was prevalent, but did not take the form of a bid rotation scheme.Instead, several ring members coordinated bids on the same jobs.Relatedly, a World Bank Group (WBG) study estimated that the costs of collusion and cartels in the road sector were between 8% and 60% of the contract value (World Bank 2011).Bandiera et al. (2009) analyze purchases of standardized goods by Italian public bodies and find that some public bodies pay systematically more than others for equivalent goods, and differences are correlated with the governance structure.Several papers focus on common pitfalls in infrastructure procurement.Most notably, Flyvbjerg et al. (2003) find that average cost overruns for roads are about 20% for projects in Europe and North America.The paper shows that cost estimates used in decision making are inaccurate, and that substantial cost escalation is the rule rather than the exception.Along the same line, Alexeeva et  al. (2008, 2011a, 2011b) estimate cost overruns in the road sector in developing economies using the Roads Cost Knowledge System (ROCKS) database. 7other branch of the literature tries to assess whether some features of the public procurement framework enhance desired outcomes.In this area, transparency plays a leading role.De Silva and others (2008)  examine the impact of a policy change by the Oklahoma Department of Transportation that led to the release of the state's internal estimate of the costs to complete highway construction projects, and compare it with bidding in Texas, a state that had a uniform policy of revealing the same information during the same period.The results show that, in comparison to Texas auctions, the average bid in Oklahoma fell after the change in engineers' cost estimate (ECE) policy.Coviello and Gagliarducci (2014) find that increased publicity requirements induce more entry and higher winning rebates using a large database of Italian procurement auctions.This leads to a reduction of costs and rationalizes public spending.Increased publicity also selects different winners: it increases the likelihood that the winner hails from outside the region of the public administration and that the winner is a large company.Such companies tend to win repeated auctions gaining market share.The study further finds that publicity seems to have no adverse effect on the ex-post renegotiations of the works.
In a similar fashion, Lewis-Faupel et al. (2016) examine the relationship between transparency and competition using databases in India and Indonesia, focusing on a specific characteristic of the system: eprocurement platforms. 8The paper finds that e-procurement leads to quality improvements.In India, where the authors are able to observe quality directly, e-procurement improves road quality, and in Indonesia, eprocurement reduces delays in road works.Winning bidders in regions that use e-procurement are also more likely to come from outside the region.Overall, the results suggest that e-procurement facilitates entry from higher quality contractors.
Finally, another subset of research focuses on procurement methods and evaluation criteria, and their impact on desired outcomes.Lewis and Bajari (2011) use an extensive data set of highway projects awarded by the California Department of Transportation between 2003 and 2008, to assess whether implementing a scoring system to evaluate bids leads to welfare gains for commuters.Using a structural model that endogenizes participation and bidding, they estimate that the counterfactual welfare gains from switching all contracts from the standard design to a scoring design are nearly 22% of the total contract value.Szucs (2017) studies the role of discretion in the context of a Hungarian reform which removed the obligation to use an open auction for contracts under a certain value.The paper finds that discretion increases the price of contracts.The actual threshold redistributes about 2 percent of the total contract value from taxpayers to firms and decreases the average productivity of contractors by approximately 1.6 percent.
In summary, the literature suggests that a good public procurement framework should promote transparency, integrity, innovation and efficiency in the management of public resources.These factors increase firms' confidence in the process and attract the best firms to participate in public procurement (Bovens et al. 2007).
In what follows, the principles identified above will guide the choice of components to be included in the indicator.

b) Existing comparative public procurement policy instruments and frameworks
Data sets on public procurement are scarce.Some economies with sophisticated e-procurement systems (such as KONEPS in the Republic of Korea, CONSIP in Italy and ChileCompra in Chile) collect data on the domestic public procurement market.The ability to produce disaggregated data on public procurement covering the bidding and contract management phases varies significantly across economies.Only few economies can produce data sets of disaggregated data that might be used for research, and most of them are in the high-income group.Very few of these existing databases can provide detailed information on the process to award a public contract, and even less data is available on contract implementation.Progress has been made to build more comprehensive platforms at the supranational level, such as the EU opentender 9 and the World Bank Step platform. 10Other initiatives, such as the open contracting partnership,11 provide standards for governments to publish data in an open manner.In any event, there is no up-to-date worldwide data set on public procurement that benchmarks the law and the implementation of public procurement systems in practice.
However, since the early 1990s there has been a push to assess, identify and promote good practices in public procurement across the globe.The World Bank has played a crucial role in this area, from the preparation of Country Specific Procurement Assessments to the latest iteration of the Methodology for Assessing Procurement Systems (MAPS, see Annex 1 for more details).MAPS methodology and assessments have been fundamental references while developing the methodology presented in this paper, and for selecting the good practices that are benchmarked by the indicator.The good practices that are awarded higher indicator scores are aligned with key recommendations emanating from MAPS.Given its in-depth scope, the number of MAPS assessments carried out each year is limited, thus, they are not ideal to build data sets or indicators that can be regularly updated over time.This paper aims to complement MAPS by creating indicators that are aligned with its purpose.
The design of the indicator was also heavily influenced by the efforts of the United Nations Commission on International Trade Law (UNCITRAL) to harmonize public procurement regulations, and to promote good public procurement governance in developing countries in particular.The indicator also builds on the provisions of the 2011 UNCITRAL Model Law on Public Procurement, which inspired multiple public procurement reforms in developing economies.
In the same spirit, the indicator also takes into account the provisions included in the standard bidding documents and general contract clauses published by the International Federation of Consulting Engineers (FIDIC12 ), and the adoption of Performance Based Contracting to identify good practices in contract management in road maintenance. 13Although these sources are not economy-specific assessments or methodologies, they were prepared based on internationally recognized good practices in contract management, particularly as related to contract changes and streamlining the process to certify the completion of works.
To sum up, the instruments described in this section are and will continue to be extremely relevant, as they comprehensively lay out internationally recognized good practices in public procurement.The main contribution of this paper is to introduce comparable and quantitative indicators that can be collected for a global set of economies and updated on a regular frequency.We believe that such data will have a high added value for future research on public procurement and, from the policy-making perspective, it can help spur discussions on best practices and promote reforms.

Methodology assumptions
The ultimate goal of this indicator is to provide worldwide annually updated and comparable data on public procurement regulations and practices.In order to be relevant for research, the indicator needs to provide quantitative data to measure implementation gaps in public procurement systems.Thus, the design prioritizes comparability and replicability over time.
To ensure comparability across economies, the data is collected on the basis of a hypothetical case study with strict assumptions.The use of a case study poses a tradeoff between comparability and representativeness, since public procurement is very vast, and the performance in implementing public procurement might vary significantly across sectors within an economy and across different types of transactions.Thus, the indicator will not be able to capture all strengths and weaknesses of public procurement systems given the specific case study used.However, it is worth noting that many steps and processes are designed to be applied similarly across different contracts, and that some of the legal framework dimensions benchmarked by the indicator will also be applicable to other types of contracts.This section describes the assumptions made on the (i) procurement sector; (ii) subject matter of the contract; (iii) contract value; (iv) location of the contract; (v) method of procurement; (vi) source of funds; and (vii) profile of companies participating in the tender.

a) Procurement sector
The first aspect to consider for the case study assumptions is the sector which the indicator should focus on, as procurement rules and practices might differ across sectors.Industries with specific safety or national security regulations were excluded from consideration to achieve the highest levels of comparability.For example, defense and health procurement represent a large share of government spending, usually up to a quarter of all public spending.However, heavily regulated products and services are not suitable for a global benchmarking exercise, as their regulations often supersede the application of general legal procurement rules.Such procurement is confidential or avoided altogether through the use of state-owned entities who serve as the buyer.Thus, sectors that have specific regulations are not included in the case study.
It is generally established that public procurement rules can be divided into three main categories depending on the type of purchases: goods, services (consulting and non-consulting services) and works. 14In each category, there is a high likelihood of different procurement strategies, procurement procedures, different implementing agencies, and different profile of companies that participate in these markets among other traits.
Three factors have been considered to select within these categories: (i) comparability; (ii) relevance for development; (iii) relevance for benchmarking rules and practices.Comparability poses a challenge no matter which sector is selected, however, works contracts tend to be the ones in which countries use similar procurement procedures and strategies.For instance, goods contracts might be procured through centralized purchasing bodies to maximize returns of scale, while services might be more susceptible to variability in public procurement procedures depending on the technical requirements needed.
Important from the development perspective, works contracts are mostly used to execute infrastructure projects, which play a crucial role on the productivity and social inclusiveness of an economy.There is a robust correlation between income inequality and infrastructure quantity and quality (Calderon et al. 2004).Moreover, corruption and collusion have marred infrastructure procurement.It is estimated that between 20% and 30% of the value of public construction projects is lost through corruption.Works contracts also allow to benchmark more areas that are relevant to identify implementation gaps and bottlenecks.For instance, they tend to require more planning and budgeting, and also have more complex and lengthier implementation.Thus, an indicator that benchmarks public procurement practices in works contracts can enhance the governance of such projects and is relevant for economic development.

b) Subject matter of the contract
Within works contracts, there is a broad variety of projects that could be selected for the case study.The goal for this indicator is to be replicable over time: the subject matter to be procured should happen regularly in all countries so that the indicator can monitor improvements over time.
The road infrastructure sector has been chosen as the focus of the indicator's case study due to the central role it plays in the development of an economy.Government investment in road transport alone can account for between 2% and 3.5% of GDP (Collier et al. 2016).In order to analyze the relevance and representativeness of this type of contract for the benchmarking exercise, an analysis was conducted using a database of road-related projects completed by the WBG over the last twenty years.The ROCKS database was initially developed by the transport department of the WBG in 1999 and was updated by the research department in 2018 (World Bank, 2006a; 2018b). 15 addition, data were extracted on projects implemented between 2014 and 2021 for 9 economies where such data are made publicly available online. 16Out of the more than 150,000 works contracts published by these economies, the proportion of road works contracts was calculated, together with the proportional value of such contracts in the total contracts.Not surprisingly, high income economies tend to have lower shares of both the number of road works contracts and the value (Table 1).This could be explained by a greater role of other types of infrastructure other than roads associated with the level of development (for instance, rail transportation).Still, in the period analyzed, high income economies spent between 8% and 23% of the total value of works contracts in the road sector.The numbers can of course be impacted, to some extent, by the time period and country characteristics.The analysis supports the decision to focus on a hypothetical road maintenance contract.While road construction contracts represent a higher share of expenditure, road maintenance contracts are more likely to occur regularly over time.For comparability purposes, further assumptions had to be made about this contract to be able to identify the applicable legal framework.The main assumption on the road contract itself -drawn for comparability purposes from the ROCKS projects' analysis -is that the works happen in a two-lane flat road and at least there should be a road resurfacing component of 20 km.The scope and value of the contract were determined by analyzing the projects related to work maintenance in the ROCKS database, taking into account the average length of each section and the unit cost.

c) Contract value
The value of the contract is commonly used as a threshold for different public procurement rules and practices.For instance, higher value contracts tend to have stricter regulations aiming to guarantee transparency, accountability, and best value for money.Most countries in the world divide the catalog of public procurement procedures as a function of the contract value.Thus, the contract value of the case study needs to be similar across countries to ensure that the rules and practices are comparable.For instance, using a range of potential contract values would be detrimental for comparability (even if it would be more realistic as the unit cost of a road maintenance contract can change even within a country), as public procurement thresholds defined in a country's regulation could fall within it, and thus multiple regulations could apply within the same range.
Given that this indicator aims to benchmark public procurement regulations as applied to domestic public procurement and taking into consideration the data extracted from the ROCKS database, the case study assumes a fixed value of USD 2.5 million. 17To assess the adequacy of this estimate, the thresholds defined in the Agreement on Government Procurement (GPA) were considered as upper limits that the case study should not surpass.The median threshold in the GPA for works contracts is 5,000,000 IMF Special Drawing Rights (SDRs), which depends on the exchange rates reported by the signatory parties.As an example, the current threshold for works contracts in European Union economies is EUR 5,350,000, while in the United States the threshold is USD 7,008,000.Contract values above this parameter are generally considered to have international interest, and thus might be subject to additional rules to attract foreign competition in tenders.

d) Location of the contract
Procurement rules and practices can also differ according to the location of the works contract.Indeed, the constitutional division of powers and competences might grant the authority to conduct public contracts on roads to different procuring entities.These entities, in turn, might be allowed to enact specific procurement rules or implement common rules with a different set of practices.Central governments, independent authorities, state-owned companies, regional governments, metropolitan authorities and municipalities can procure the case study contract if it pertains to roads under their authority.Unfortunately, given that data are scarce, it is not feasible at this stage to obtain estimates for all authorities that could procure such contracts.One authority had to be selected that would be at the same time broadly representative for an economy, and comparable across all countries.
As a consequence, the indicator assumes that the location of the contract is in a road that connects the largest business city of a country with another city.In federal countries, or countries that have devolved authority to subcentral authorities, the road must connect to another city within the same state, province or region.Thus, streets or roads that do not connect cities are not relevant for selecting the authority and corresponding applicable rules and practices.

e) Method of procurement
The contract described above can be procured with different procedures, which have different rules and practices.To maximize relevance and comparability, the indicator assumes that the method of procurement used by the procuring authority is open tender or equivalent in that country.Not only open tender is an internationally recognized good practice to promote competition and transparency, but also tends to be the most common method of procurement used in practice.
An analysis was conducted of the Pillar 24 (Procurement) of the last Public Expenditure and Financial Accountability (PEFA 18 ) reports published since 2015.These reports usually provide an estimate of the number of tenders carried out in open procurement as a percentage of total tenders (available for 15 economies), and also an estimate of the contract value of open procurement tenders as a percentage of all procurement (available for 50 economies).Regarding the number of projects, the median share of procurement projects procured in an open tender procedure is 71%, with some notable outliers such as The Gambia, Madagascar and Mali (with shares below 20%).On the other hand, the median share of the value of all contracts procured in an open tender procedure across countries is 75%. 19 17 The value was determined by selecting all the sections of projects in ROCKs that had a road rehabilitation component and used the unit costs per kilometer of each of them to estimate a representative value per economy.The median was used as the most neutral estimate, to avoid being influenced by projects that were more volatile, or larger projects.Then, the team averaged the median unit cost per kilometer of all the economies with road rehabilitation sections in ROCKS, to obtain the final unit cost per kilometer estimate.Then the unit cost per kilometer was multiplied by the length of the assumed project (20 kilometers).The actual average was rounded down from USD 2.7 to USD 2.5 million to facilitate communication with questionnaire respondents. 18For more information, see Annex 2. PEFA information can be retrieved at: https://www.pefa.org/ 19Data retrieved from PI-24 section in PEFA reports published since 2015 at: https://www.pefa.org/

f) Source of funds
Procurement regulations and practices can also be different depending on the source of funds for the project.
For instance, projects might be funded by International Financial Institutions, bilaterally through development aid agencies, or by regional international organizations, which might require additional procurement rules and practices as part of their fiduciary duties.While in some low-income countries a relevant share of projects is funded by external parties, the focus of this indicator is to benchmark regulations applicable to projects managed entirely by governments and directed towards domestic companies.Therefore, the case study assumes that the project is funded entirely by the procuring authority.
This assumption also has an impact in countries where there is co-funding between different authorities (e.g.federal authorities partially fund a state project), which also can lead to apply a specific set of rules for this project.

g) Companies that participate in the procurement process
One final assumption needs to be made on the companies that participate in the tender for the case study contract.Since this exercise focuses on benchmarking domestic public procurement as a tool to promote more private sector participation in government markets, the case study assumes that the companies that participate are locally incorporated and do not have foreign ownership.Thus, the case study focuses on regulations and practices that would apply to a hypothetical domestic firm.Company characteristics, including the size, turnover, legal form, are often requirements laid out in tender document for companies to even participate in the tender.However, the indicator does not assume any further characteristic as these are more project-specific and subject to the discretion of procuring authorities.Table 2 summarizes the components of the case study and its assumptions.
Table 2: Case study assumptions The Procuring Entity − Is the agency in charge of procuring construction works for the authority that owns the road described below.− Is the sole funder of the works, has budget for the works and is solvent.The business (BidCo) − Is one of multiple companies participating in the call for tender for the contract.− Is a limited liability company (or its legal equivalent).If there is more than one type of limited liability company in the economy, the limited liability form most common among domestic firms is chosen.− Operates in the economy's largest business city.For 11 economies the data are also collected for the second largest business city.− Is 100% domestically owned.− Is owned and controlled by private individuals or entities.− Is up to date with all regulations and is in good standing with all relevant authorities, including those related to taxes.− Has all licenses and permits needed to operate in this technical area.− Has already responded to a public call for tender and is already registered with the procuring entity.− Meets all solvency, technical and administrative requirements to compete.The road − Connects the largest business city to another city within the economy (and within the same state, region or province as the largest business city, if applicable).− Is not a highway.− Is not under concession.

The contract
− Entails resurfacing 20 km of a flat two-lane road with an asphalt overlay of 40 to 59 mm (or its most common equivalent used in the economy).− Is valued at USD 2.5 million.− Does not include any other work (such as site clearance, subsoil drainage, bridgework, or further routine maintenance).

The procurement process
− Is an open, unrestricted, and competitive public call for tender for resurfacing a road like the one described above.− Is completed without complaints, challenges, or protests from interested parties, unless explicitly provided.− Ends with the awarding of the contract to BidCo, whose bid satisfied all technical and administrative criteria and offered the best value for money.

h) How the data are collected
Under these case study assumptions, the indicator obtains estimates through a questionnaire distributed to around 3,500 private sector experts in public procurement regulations, and companies that participate in similar tenders, across 191 economies.The questionnaire clearly indicates the case study assumptions to ensure comparability across economies and over time.
Data are collected in a standardized way.Texts of the relevant laws and regulations are collected, and answers checked for accuracy.The methodology offers several advantages.It is transparent, using factual information about laws and regulations and allowing multiple interactions with local respondents to clarify potential misinterpretations of questions.The methodology is easily replicable, so data can be collected in a large sample of economies on an annual basis.Because standard assumptions are used in the data collection, comparisons and benchmarks are valid across economies.Finally, the data not only highlight the extent of specific regulatory obstacles to business, but also identify their source and point to what might be improved (for more details about the data collection process, see Annex 4).
One of the important novelties of the indicator is that data collection does not depend on the ability of economies to keep records or statistics on public procurement -although good procurement practice naturally encourages them to do so.Quantitative data are collected by reviewing the applicable laws and regulations, and gathering information from carefully selected private sector firms and experts based on whether they have direct or indirect experience in procurement processes of road works contracts (see Sections 3c and 4d).Thus, data are not collected through a statistical survey.

i) Limits to what is measured
The methodology has four limitations that should be considered when interpreting the data.First, the collected data refer to a works contract implemented by a procuring entity in the largest business city of an economy and may not be representative of regulation in other parts of the economy.Second, the data focus on regulation that applies to a specific business form-a domestic limited liability company (or its legal equivalent)-and may not be representative of the regulation on other businesses (for example, foreign companies).Third, transactions described in the case scenario refer to a specific set of issues and may not represent the entire landscape of public procurement processes and strategies within an economy.Fourth, the measures of time involve an element of judgment by the expert respondents.When sources indicate different estimates, the time indicators reported represent the median values of responses provided by several respondents, given the assumptions of the standardized case (the average number of respondents per question in each economy is 5).

Dimensions used as estimates of efficiency, transparency, innovation, and integrity
The traditional focus of the international literature on public procurement has been on the bidding phase, which also tends to be the most regulated and transparent phase of the procurement process in most economies.Other phases -in particular the planning and contract management phases (from needs assessment to contract management and payment) as well as the exceptions to competitive steps (from urgency to contract fractioning) -are less frequently subject to transparency requirements, and therefore potentially more vulnerable to abuse and corruption.
As pointed out in the literature described above, all public procurement steps are intertwined.Poor planning can lead to significant cost overruns during contract management, and it reduces incentives for capable companies to enter government markets.Delays in the tender process can lead to contract changes during execution if initial circumstances changed.Poor contract supervision might disrupt the process to complete works, and it might even lead to require further tender procedures to ensure the project is completed.These few examples show that data are needed on all procurement phases to evaluate the implementation of public procurement regulations in a project.
The same principle should apply when benchmarking regulations.Thus, this indicator should be comprehensive, and examine the quality, transparency, integrity and efficiency of public procurement systems throughout all phases of the procurement life-cycle, from needs assessment and budgeting to invoicing and payment, as applicable to the case study.For the purposes of this indicator, we have classified the procurement life-cycle in three phases: (i) planning, (ii) tendering, and (iii) contract management phase (for more details see Annex 3).
In each phase of the procurement life cycle, the indicator measures four factors: i.
The number of steps needed to complete each phase.A step is an interaction between the contractor and the relevant public agency (the procuring entity, any governmental office issuing permits, auditing department, etc.).ii.
The time associated with completing the steps described above.The indicator measures how long those interactions take in calendar days.iii.
The transparency and functionality of the e-procurement platform used by the procuring entity.
Here, the indicator measures the number of features available in the e-procurement platform.iv.
Whether the legal framework regulates a minimum set of internationally recognized good practices that promote competition, integrity and transparency.

a) Steps
The indicator records all steps officially required, or commonly carried out in practice, for a company to participate and win a works contract through an open and competitive method of public procurement, until the contract is successfully executed, the company obtains the last payment and no longer has pending financial restraints related to these works with the procuring entity.
A step is defined as any interaction of the company with external parties that is either required by law or is carried out in more than 25% of similar work contract cases in practice.Interactions between company employees are not counted as a procurement step.The indicator assumes that steps cannot be completed simultaneously.Each electronic procurement step is counted as a separate procurement step.
Only steps required for the tendering and contract management phase of the specific works contract assumed by the indicator are included.For example, procurement steps to comply with environmental regulations are included only when they apply to any company that would bid for the contract and execute it if awarded.Steps carried out when designing the procurement project are also not measured.
Some steps are recorded as mandatory for each economy, as those are necessary to ensure that public procurement is carried out under the principles of transparency, competition, integrity and best value for money (for more details, see Annex 4).To calculate the procurement steps score, the number of steps is normalized using the linear transformation (worst -y)/(worst -best)*100.The sum of mandatory steps (12 steps) is identified as the best practice, while the worst practice is the sum of all potential steps (22 steps).As a result of this transformation, the steps score ranges on a scale from 0 to 100, where 0 represents the worst and 100 represents the best performance.

b) Time
Time is recorded in calendar days.The measure captures the median duration that public procurement lawyers or construction companies indicate is necessary in practice to complete a procurement step with minimum follow-ups with government agencies, no unofficial payments and no collusion.The bidding process is considered complete once BidCo is able to commence the works officially.The contract management phase is considered complete once BidCo receives the last payment, but it does not include the time for the works to be carried out.Instead, it includes the time for contract amendments, if any, for inspections and payment.It is assumed that the company is aware of all participation requirements and their sequence from the beginning, but it has had no prior contact with any of the officials involved in awarding the contract described above.
To calculate the time score, the component is normalized using the same linear transformation explained above.The shortest time represents the best performance on the component across all economies.To mitigate the effects of extreme outliers, the 95th percentile is used to determine the worst value of time.As a result of this transformation, the time score ranges on a scale from 0 to 100, where 0 represents the worst and 100 represents the best performance.
c) E-procurement index E-procurement systems have several advantages compared to traditional paper-based procurement procedures.First, an e-procurement system creates a single online portal for stakeholders to access information on procurement opportunities, learn about the procurement process, and obtain documents including technical specifications, user friendly templates, and the terms and conditions for all types of public contracts.The open availability of information promotes equal access for all types of businesses, including small and medium enterprises, by reducing the possibility of large or well-connected firms gaining an advantage because of information asymmetries, and potentially increases competition for government contracts.In addition, e-procurement facilitates quick and easy decision making.Government officials can easily see detailed information on bids through the online system, rather than having to sift through paperwork. 20e e-procurement index measures whether economies have implemented an electronic platform to carry out the public procurement process of the works contract assumed by the indicator's case study.The eprocurement index assigns 1 point for each of 10 features that promote transparency, competition, integrity and efficiency in the bidding and contract management phases (for more details see Annex 4).The indicator only recognizes an e-procurement feature as available if it can be fully completed electronically and no hard copies are required.Publication of annual procurement plans (0-1) Publication of tender notices (0-1) Publication of tender documents (0-1) Asking and receiving clarifications on tender documents and procurement process (0-1) Bid submission (0-1) Bid opening (0-1) Notification of decisions (0-1) Publication of award decision (0-1) Contract signing (0-1) Submitting invoices and receiving electronic payment order/receipt (0-1) The e-procurement score is reflected on a scale from 0 to 10 points, where 0 represents the lowest and 10 represents the best performance.Then the e-procurement component is normalized to a scale of 0 to 100, so that all components have a common unit.

d) Legal framework index
A robust legal framework is an important cornerstone for public procurement systems.These legal frameworks can be comprehensive and complex.Since the focus of this benchmarking exercise is to compare regulations enabling companies to participate in government markets, we have constrained the selection of internationally recognized good practice to provisions that promote competition, transparency, integrity, and best value for money.
In order to identify a minimum common denominator in international regulatory best practices, we consulted the existing international regulations (UNCITRAL, WBG Procurement Framework) and methodologies (MAPS and PEFA), and identified 16 best practices (Annex 3 contains a table comparing each of the instruments with the legal framework index we have constructed).
The legal framework index measures whether economies regulate certain good practices that promote transparency, competition, integrity of the evaluation of the bids and integrity of the contract as applicable to the case study measured by the indicator (for more details, see Annex 4).What does the legal framework index measure?(0-16) Transparency (0-4) Publication of tender notices (0-1) Publication of tender documents (0-1) Publication of award decisions (0-1) Publication of contract amendments (0-1) Competition (0-4) Open tendering as default method of procurement (0-1) Prohibition of dividing contracts to circumvent open tendering thresholds (0-1) Minimum deadlines for bid submission (0-1) Standstill between award and contract signing (0-1) Integrity of Evaluation (0-4) Minimum content of tender notice and tender documents (0-1) Best value for money criteria (0-1) Definition of abnormally low bids (0-1) Notification of award decision to bidders (0-1) Integrity of Contract (0-4) Regulation of contract changes (0-1) Established timeframe for payment (0-1) Bidder entitled to interest on late payments (0-1) Protocols and guidelines for inspections (0-1) The legal framework index ranges on a scale from 0 to 16 points, where 0 represents the lowest and 16 represents the best performance.Then the component is normalized to a scale of 0 to 100, so that all components have a common unit.
The overall score for the indicator is obtained as the simple average of the scores in each of the above four dimensions: steps, time, e-procurement index, and the legal framework index.

Data: Summary statistics 21
The data collected using the methodology described above can shed some light on the differences in public procurement practices across economies and identify patterns across income groups or regions.In what follows, we provide summary statistics for the four dimensions measured by the indicator, based on the data collected as of May 1, 2020 in 191 economies (for summary statistics by income group, see Annex 7). 22

a) Steps & Time
The indicator measures the efficiency with which economies can complete public procurement steps relative to a specific road resurfacing hypothetical case.In the Republic of Korea, for example, it takes 270 days to complete all the procurement steps that are required or that are common in practice during the bidding and contract management phases of a road resurfacing contract.It is common in Korea to require bidders to participate in a prequalification step for such contracts even though the applicable framework recommends it for complex or high value works contracts.The Korean Public Procurement Service (KPPS) takes 20 days to evaluate and select the contractors that meet the pre-qualification criteria.Then, these bidders have 30 days to submit their proposals.From the moment of submission, including bid opening and evaluation, alongside with a bid security.KPPS takes 60 days to notify the award decision to BidCo, from the moment of bid submission (this includes both bid opening and evaluation).The step to draft and sign the works contract is very efficient, taking only 7 days, including the time it takes BidCo to obtain the required performance guarantee.The bidding phase ends when BidCo can commence the works, and it takes 30 days since the moment the contract was signed.The process is quite streamlined as BidCo does not need to obtain additional authorizations or permits to commence the works.During the implementation phase, contract amendments and new contracts awarding additional works to BidCo are common.However, the time required to sign the contract amendment or the new contract is very short (21 days).The contract management phase ends with the process of certifying the completion of works, which takes 60 days from the moment BidCo notified that the works were completed until the certificate of completion is issued.Finally, Korea is also very efficient in paying BidCo, taking only 7 days from the moment of submission of the invoice.Norway; Canada; Singapore; Hong Kong SAR, China; Finland; Lithuania; Sweden; Latvia and Australia are among the most efficient economies in terms of timeliness of procurement processes.In each of these nine economies, it takes a year or less to complete the public procurement steps described above.
However, some economies do not manage to complete the procurement life-cycle measured by Doing Business as efficiently.In São Tomé and Príncipe, Guinea-Bissau, Timor-Leste, the Islamic Republic of Iran, Brunei Darussalam, Angola, Lesotho and Cameroon it takes more than 3 years to complete the same steps.
In addition, there are 13 economies where private sector experts reported that there has not been any open procurement for a works contract over the last 5 years.Some of these economies are affected by conflict which impedes the government's ability to purchase goods, services, or works, such as Libya, the Syrian Arab Republic or the Republic of Yemen.Some other countries, such as Equatorial Guinea or the República Bolivariana de Venezuela, have not introduced or implemented open procurement practices for our case study.Finally, there are some countries where, due to budget constraints, authorities are not purchasing road works contracts with their own budgets, relying on international financing institutions or donor funds to implement such contracts, and thus their practice is not comparable to the other countries.These economies are not scored by the indicator, 23 as there are no open government markets for the private sector. 24e global average in completing all the relevant procurement steps identified above is 723 days.However, the standard deviation is large, at 242 days.For instance, countries in the most efficient group, High income: OECD, complete the procurement steps in 554 days on average, while in the least efficient region, Sub-Saharan Africa, it takes on average 836 days (Table 6).High-income economies 25 are more efficient in completing all the administrative processes related to the bidding and implementation of the case study contract, with a median time shorter by a considerable margin of 137 days, relative to the next income group ("upper middle income").In comparison, low income and lower-middle income economies are completing this process in similar times.
Furthermore, the data also reveals some differences regarding the number of steps required to complete procurement processes as captured by the case study (Table 7).In practice, no economy attains the best possible performance in terms of steps.Austria is the best performer with 13 steps, followed by a group of economies with 14 steps, such as Japan, Ireland or Germany.At the other end of the spectrum, Oman is the country with most procurement steps ( 22), followed by a group of five economies, including Honduras, Hungary, Iran, Islamic Rep., Myanmar and Tonga, which recorded 21 steps to complete a similar procurement process.
23 However, we compute their score for the e-procurement and legal framework indices as their components can be measured and compared independently of the economy's ability to procure in an open and competitive manner. 24The 13 economies are: the Central African Republic, Equatorial Guinea, Eritrea, Haiti, Kiribati, the Lao People's Democratic Republic, Libya, Palau, Somalia, South Sudan, the Syrian Arab Republic, the República Bolivariana de Venezuela and the Republic of Yemen. 25According to WDI income classification.A median high-income economy requires only one less step than those in other income groups.While this component shows some differences in the implementation of a procurement procedure across countries, the small differences in the average and median number of steps across income groups shows that the procurement process for the case study is quite standardized between economies.
The steps and time scores (on a 0-100 scale) are then compared with other publicly available indicators.
The time score is correlated with the standards and procedures on internal audits, as measured by PEFA. 26he score on the time component is associated with indicators 26.2 and 26.4 of PEFA (Figure 1), which measure the nature of audits and standards applied and the response to internal audits, respectively.Although the PEFA sample mostly includes lower-middle-income and low-income economies, this simple correlation would suggest that there could be an association between the efficiency in implementing procurement processes and good practices in internal auditing standards.Or, in other words, economies in which internal audits meet professional standards, and where management takes action to respond to audit reports seem to be able to implement more efficient procurement procedures.Accountability can thus be an enabling tool to improve efficiency.Note: The correlations between the PEFA indicators and the indicator's Time score are significant at the 1% level for indicator 26.4 and 5% for indicator 26.2 after controlling for income per capita.

b) E-Procurement index
By benchmarking e-procurement features, the indicator helps to assess at what stage economies are in the transition to e-procurement.Rwanda and Bangladesh provide good examples on recent reforms to transform their public procurement processes and move towards e-procurement. 28In both economies, the eprocurement platform has managed to implement successfully the features of publication of (i) annual plans; (ii) tender notices; (iii) tender documents; (iv) and award decisions of the procuring entity, and it has also successfully developed the following interactive features: (v) submit and receive clarifications on the tender documents and procurement process; (vi) submission of electronic bids; and (vii) electronic notification of decisions.Both economies are among the top scoring in e-procurement in the respective regions.
On the other side of the spectrum, some economies do not have e-procurement in place.Economies such as Algeria have just recently launched a website providing some basic information on the legal framework and on tender notices, although it is not an e-procurement system yet.Therefore, in these economies, the procuring entity measured by the case study must carry out the whole procurement process on paper. 29hese economies would receive the lowest score in the e-procurement index.
The global average of e-procurement features available to the procuring entity benchmarked by the data is 3.8 out of 10.The broad message from these data is that, on average, countries are publishing public procurement information online (although not all the key documents), but systems have not yet transitioned to a transactional e-procurement management system.The average number of available features is above 6 only in high-income OECD economies and in Europe & Central Asia (Table 8).In sum, the message is clear: there is plenty of room to improve in e-procurement.2), when considering the 45 countries assessed under the PEFA 2016 framework.These indicators focus on procurement monitoring and public access to procurement information, respectively.Electronic platforms to carry out public procurement processes play a relevant role in storing data on the procured contracts as well as in improving access to procurement information.

c) Legal framework index
When comparing the legal index across regions, the best performing are economies in Europe & Central Asia, High Income OECD and Sub-Saharan Africa (Table 9).The very good performance of the latter might be due to two factors.First, since economies in Sub-Saharan Africa did not have public procurement laws or regulations until the 21 st century, they were able to incorporate the most recent internationally recognized good practices from the start.Second, the WBG has had a crucial role in shaping public procurement policy in Africa, and thus the laws and regulations in this region are more closely aligned with the standards recommended in MAPS or in the WBG Procurement Framework.If the analysis is carried out by income levels, the best performer is the group of low-income economies.A potential explanation of this rather surprising result is that it might reflect differences in the size and complexity of procurement contracts between countries in different income groups that are not entirely captured by the case study assumptions.For example, high income economies might prioritize implementation of best practices in more complex infrastructure contracts (that are designed to attract international companies).

d) Overall score
As explained in section 4, the overall score for the indicator is computed by transforming scores in the four measured dimensions to a 0-100 scale, assigning each an equal weight, and taking a weighted average.
The global average score is 45.4 (with a standard deviation of 15.1).Scores across regions follow similar patterns as for each of the four dimensions.However, broadly speaking, High income OECD and Europe & Central Asia economies score quite above the global average, at around 58 points each, implying a large performance gap between these regions and the rest of the world.
Figure 3 below shows the distribution of scores in each of the dimensions captured by the indicator, and the overall score density function for comparison.More economies are performing well in the legal index than in other indices.This is no surprise, as the first policy response to address inefficiencies, corruption and introduce innovation is by adoption of internationally recognized good practices, which has often been incentivized by international financing institutions, donors and international regional organizations.
The graphs also show that there is considerable room for improvement in the e-procurement dimension on average, as the left tail of the distribution covers a significant proportion of economies.
Figure 3: E-procurement is the dimension that has most room for improvement, while regulatory good practices have been more widely implemented in public procurement frameworks Overall, economies with higher income tend to have better scores on time, procurement steps, and especially on the e-procurement index.

e) Correlations with other related indicators
The analysis of the data collected reveals significant correlations between the indicator and other (external) relevant variables.For this exercise, we use the log of GDP per capita from the World Development Indicators, the sixth criteria of the WBG Country Policy and Institutional Assessment (CPIA), Transparency International's Corruption Perceptions Index (2020), the Government Effectiveness indicator of the Worldwide Governance Indicators (WGI) in 2019 and World Economic Forum's quality of roads indicator in 2018.In Table 10, we summarize the results of simple regressions of the indicator (and its components) on these variables.In addition, we show results that take income per capita into account.Notes: Coefficients reported in the section "Unconditional regressions with external indicators" are coefficients from univariate regressions of the indicator scores on external indicators, considered one at a time.Coefficients reported in the section "Regressions with external indicators controlling for log GDP per capita" are coefficients on external factors when additionally controlling for log GDP per capita.t statistics in parentheses; * p < 0.1, ** p < 0.05, *** p < 0.01.
Higher overall scores on the indicator are associated with higher income levels (Figure 4)-in line with expectations that developed economies usually have in place more efficient and transparent procurement processes.The new indicator provides data on the business regulatory environment as applied to a specific type of government contract.Thus, another interesting exercise is to compare the indicator scores with the CPIA's sixth criterion31 since it measures the business regulatory environment in an economy.As shown in Table 10, there is a positive association between the indicator's overall score and the assessment carried out in CPIA.The steps component, though, shows a somewhat less statistically significant link with the CPIA's sixth dimension.
The indicator also positively correlates with Transparency International's CPI index.A more efficient, regulated and transparent procurement process, as measured by the indicator, is thus associated with lower levels of corruption (Table 10).The legal framework score shows the weakest link with the CPI.On one hand, it is possible that the regulatory framework component is not capturing all regulations relevant for fighting corruption.On the other hand, the cross-section correlations do not capture potential positive relations between legal reforms and reduction of corruption over time within the same economy.The same reasoning can explain the weak link between the CPI and the e-procurement component, as the relationship becomes not significant when controlling for income.Since e-procurement reforms are more recent, and as pointed out above, mostly introduced in high-income economies, the relationship with corruption in a crosssection analysis is unclear.
Next we analyze the indicator scores in relation to the Government Effectiveness indicator of the WGI.The Government Effectiveness indicator focuses on the quality of public services, policy formulation and implementation.WGI scores are correlated with the indicator scores, suggesting that higher scores in the proposed indicator are associated with more effective governments (Table 10).The relationships are statistically significant for all of the indicator's components also when controlling for income.
As noted above, PEFA indicators are also useful for this comparative exercise.Pillar 24 (Procurement) of the PEFA 2016 framework, which examines key aspects of procurement management, shows a strong alignment with the indicator scores on the time and e-procurement components (Table 10).
Finally, considering the focus of the indicator on roadworks, its relationship with the quality of the roads is also worth noting.An efficient procurement process, where all the relevant information is publicly available, should attract higher quality bids and eventually be reflected in the final quality of the works contracted.Indeed, higher scores in the indicator are significantly associated with better quality roads.The components that have a stronger relationship with the quality of roads are the ones that focus on the efficiency of the procurement process in practice, that is the time and steps components.Following the same logic, the implementation gaps in the legal frameworks might explain the lack of a statistically significant relationship between the road quality index and the legal component.

Conclusion
This paper presents the full technical description of the methodology underlying a new indicator designed to benchmark public procurement regulations and their implementation in practice, in a comparable manner across 191 economies and across time.The indicator should help inform researchers and governments on the regulatory and practical obstacles the private sector faces when participating in the market for public contracts, and help governments optimize public spending-two elements crucial to supporting the WBG's focus on ending extreme poverty and boosting shared prosperity.
The resulting data set should inform governments on global and regional best practices to encourage national and subnational reforms, and it should allow them to track progress over time through yearly data updates.Moreover, the indicator can contribute to measuring progress in the Sustainable Development Goals, particularly targets 16.5 (Substantially reduce corruption and bribery in all their forms) and 16.6 (Develop effective, accountable and transparent institutions at all levels). 32Dozens of economies, either through self-assessment or through multilateral or donor institutions, have carried out MAPS assessments with the old methodology.The WBG is currently undertaking around 25 MAPS assessments across different regions with the 2019 revised methodology.However, there is no publicly available dataset summarizing the findings of these assessments, and no comparable data on public procurement systems. 36 Annex 2 -PEFA/PIMA In addition, the development of the indicator has been informed by other assessments related to public finance management practices, as they indirectly affect many steps of the public procurement life-cycle.
Most notably, the indicator has been inspired by the Public Expenditure and Financial Accountability (PEFA 37 ) methodology, which is applied to assess the status of public financial management, and to a lesser extent by the Public Investment Management Assessment (PIMA 38 ) carried out by the IMF to assess infrastructure governance.
PEFA was first published in 2005 with the aim of harmonizing various Public Finance Management (PFM) systems diagnostic tools.The PEFA framework provides the foundation for evidence-based measurement of PFM systems.A PEFA assessment measures the extent to which PFM systems, processes and institutions contribute to the achievement of desirable budget outcomes: aggregate fiscal discipline, strategic allocation of resources, and efficient service delivery. 39PEFA has become the most widely used assessment of PFM performance in low-and middle-income countries.Since its launch in 2005, nearly 600 formal assessments (national and subnational) in 150 countries and territories have been undertaken and verified by the PEFA Secretariat. 40Since its inception, PEFA has a specific pillar measuring good practices in public procurement systems from a PFM perspective.In 2016, PEFA was upgraded with new benchmarks to reflect better new PFM reforms and the evolution of good practices over the last decade.As regards to public procurement, a new subcomponent was added to measure procurement monitoring, and the scoring criteria to assess the use of competitive procurement methods were revised.
In The IMF's Public Investment Management Assessment (PIMA) Framework was also used as a reference since the case study of the indicator focuses on the infrastructure sector.PIMA provides a comprehensive assessment of the public investment decision-making process at three key stages: (i) planning sustainable levels of investment across the public sector; (ii) allocating investment to the right sectors and projects; and (iii) implementing projects on time and on budget.The PIMA measures elements related to macro-fiscal frameworks, integration of investment planning in medium-term budgeting, coordination of public investment across levels of government, and private sector participation in the provision of public infrastructure.43 Annex 3 -Procurement life-cycle captured by the indicator a) Planning phase The needs assessment and budgeting phase analyzes the tools that procuring entities should use to identify procurement needs and secure necessary funds.When procuring entities do not begin the procurement cycle with an adequate needs assessment, the process is unlikely to have a successful outcome.When budgeting is not done properly and funds are not specifically allocated to each procurement opportunity, the companies' ability to receive agreed payments -and their potential to stay in business -are compromised.
Unrealistically optimistic budgets from faulty needs assessments at the onset of the procurement cycle often result in projects delivered with cost overruns and delays (Kahneman et al., 2003; Flyvbjerg, 2017).
Improper needs assessments may also lead to unnecessary purchases, waste of public funds and excessive renegotiations, in the context of both legitimate and illegitimate reasons (Decarolis et al., 2015).
Transparency and accountability in these steps allow citizens to hold the public administration accountable for intentional acts of corruption (Beth, 2005).Good practices increasing transparency, efficiency and accountability during this phase include preparing multi-year procurement plans, publishing annual procurement plans, estimating the contract value of upcoming procurement opportunities on the basis of technical studies and data from previous projects, and securing the required public funds prior to advertising the procurement opportunity.

b) Tendering phase
The tendering and bid collection phase benchmarks how the procurement method is chosen, how the information related to the tender is made publicly available, how bids are collected from the private sector, the time for bidders to submit proposals, the processes to clarify the content of the tender documents and to modify the initial tender documents.The indicator also measures innovative features of e-procurement systems in this phase, such as the submission of digital tenders online.
Implementation of the pillars of an effective public procurement system during the tendering and bid collection phases is key to reaching optimal economic outcomes (Kinsey, 2004).Information that is not provided consistently risks affecting the level playing field for bidders, as technical specifications may be vague or tailored to one company, award criteria may be unclear, and records of the step may be lacking.Good practices during the tendering and bid collection phases include ensuring that a wide number of qualified bidders can participate, and that rules on open procurement are not circumvented through the overuse of emergencies (UNCITRAL, 2011).If applicable, the system must trace who is responsible for and who has access to the tenders.Tender documents should be free, publicly available, and easy to navigate, and they should set all selection and award criteria.Any clarifications on those criteria should be made available to all participating bidders.Any form of required bid guarantee should not impose unnecessary burdens on businesses or preclude the participation of small companies.
This phase also includes bid opening, evaluation and contract signing, which covers all the steps required from the moment companies submit their proposals, until the procuring authority has signed the contract with the winner and the commencement of works.From a regulatory perspective, the indicator measures, among other elements, the criteria used to evaluate bids and award the contract, the definitions of abnormally low bids and non-substantial errors in the bids, and standstill provisions.Several features of eprocurement systems are also measured in this phase, such as the possibility to sign the contract electronically.
These steps are particularly vulnerable to abuse when transparency, efficiency and accountability are not prized.Conflicts of interest can take the shape of excessive familiarity between agents of the procuring entity and bidders.Poorly defined evaluation criteria may be used to award the contract to a bidder that did not offer best value for money or deter smaller companies from participating.Also, poor record keeping and lack of clear communication on award decisions compromise transparency and erode accountability.
Good practices during evaluation and contract award include: opening all tenders at the time, place and manner specified in the solicitation documents; using a well-defined and quantifiable award criterion; providing a standstill period to allow bidders to file a complaint with the procuring entity; enforcing a code of ethics and conduct for employees of the procuring entity; communicating award and exclusion decisions promptly to all bidders; and limiting informal interactions between bidders and the procuring entity (OECD,  2016; UNCITRAL, 2011).

c) Contract management phase
This phase benchmarks the steps carried out from the time the contract is signed to when the works are completed, including requirements on guarantees and contract renegotiations.Failure to properly monitor the contractor's performance may result in the works not meeting the contract's specifications, theft of materials, improper use of subcontractors and excessive renegotiations.The delivery of sub-standard works -or lack of delivery altogether -are also important risks during this phase.Private companies may be required to perform unfeasible contracts or be bound by one-sided contract terms.
The contract management phase, which starts after the contract is signed and works can commence and ends when the works are completed and the contractor is paid, poses a set of challenges of its own.Failure to properly monitor the contractor's performance -i.e. a general lack of supervision over the quality, cost and timing of the process -may result in works not meeting contracts specifications, theft of assets, improper use of subcontractors and excessive use of renegotiations.Changes in contract terms and values are among the basic indications of the likelihood of corruption in public procurement (Dudkin, et al., 2006).When the work is complete, low quality goods can be used to defraud procuring entities and create aboveequilibrium profits for corrupt businesses.The delivery of sub-standard works, or the overall lack of delivery represent the biggest risks of this phase.Good practices during contract management include tight rules on renegotiations and inspections, and control mechanisms.
During this phase, the indicator also measures the practices around the timeliness and frequency of payments.Lack of transparency and accountability may lead to cost misallocation, late payments and misuse of public funds.Delayed or missing payments impose severe obstacles on private sector growth, and may have serious consequences for companies, especially the smaller ones, whose ability to remain in business may depend on a handful of government contracts.

Annex 4 -Data notes for the scoring methodology a) Procurement steps
The detailed list of procurement steps that are assumed to be mandatory in each economy are: • Bidding phase: o Advertisement of the procurement opportunity: Initiating step whereby the procuring entity invites all potential bidders to participate in an open and competitive method of procurement.o Bid submission: Bidders obtain the tender documents, prepare their bids/proposals and submit them within the timeframe specified in the advertisement of the procurement opportunity.o Bid opening: The procuring entity opens the bids/proposals submitted by potential bidders after the deadline to submit bids.o Award decision: The step of evaluating the bids/proposals, from the moment bids are opened until the final decision of award the contract to BidCo has been approved.o Contract signing: The procuring entity and BidCo formalize their bilateral agreement to execute the procurement opportunity, taking into account all the processes needed between the moment of the final award decision until the formalization of the agreement through a contract or letter of acceptance where applicable.o Commencement of works: The procuring entity and BidCo conduct all the necessary requirements to initiate the execution of the agreement, including all processes between the moment of formalization until BidCo can actually start the execution of the contract.

• Contract management phase:
o Notice of completion of works: The BidCo notifies the procuring entity that the works have been completed according to the tender specifications and the conditions of the agreement.o Certificate of completion: the procuring entity verifies that BidCo executed the agreement according to the tender specifications and conditions of the agreement, including all the processes from the moment of reception of the notice of completion of works until the certificate of completion is issued.o Payment: The BidCo gathers all the necessary documents to prepare the invoice to request the final payment, until the payment is actually disbursed.

b) E-procurement index
The e-procurement index measures the following features: • Whether the e-procurement platform publishes annual procurement plans of the procuring entity measured by the indicator.A score of 1 is assigned if the feature is available; or 0 if not.• Whether the e-procurement platform publishes tender notices of works contracts of the procuring entity measured by the indicator.A score of 1 is assigned if the feature is available; or 0 if not.• Whether the e-procurement platform publishes all tender documents of works contracts of the procuring entity measured by the indicator in an electronic format.A score of 1 is assigned if the feature is available; or 0 if not.• Whether the e-procurement platform allows potential bidders to ask clarifications on tender documents and receive the response from the procuring entity.A score of 1 is assigned if the feature is available; or 0 if not.
• Whether the e-procurement platform allows the submission of electronic bids in works contracts of the procuring entity measured by the indicator.A score of 1 is assigned if the feature is available; or 0 if not or hard copies are needed to complete the bid submission.• Whether the e-procurement platform allows the procuring entity to open the bids electronically.A score of 1 is assigned if the feature is available; or 0 if not.• Whether the e-procurement platform allows to notify public procurement decisions or communications entirely though the electronic system.A score of 1 is assigned if the feature is available; or 0 if not.• Whether the e-procurement platform publishes award decisions.A score if 1 is assigned if the feature is available; or 0 if not or if the information published is insufficient. 44• Whether the e-procurement platform allows to sign the works contract of the procuring entity measured by the case study electronically.A score of 1 is assigned if the feature is available; 0 if not.
• Whether the e-procurement platform allows contractors to submit invoices and all additional documents electronically and receive the electronic payment order/receipt.A score of 1 is assigned if the feature is available; 0 if not.

c) Legal framework index
The transparency component of the legal framework index measures whether the applicable legal framework mandates: • The publication of tender notices.A score of 1 is assigned if yes; or 0 if no.
• The publication of tender documents.A score of 1 is assigned if yes; or 0 if no.
• The publication of award decisions.A score of 1 is assigned if yes; or 0 if no.
• The publication of contract amendments.A score of 1 is assigned if yes; or 0 if no.
The competition component of the legal framework index measures: • Whether the applicable legal framework regulates that open method of procurement should be the default method.A score of 1 is assigned if yes; or 0 if other methods of procurement can be used without restrictions, including prequalification steps, or if the selection of the method of procurement is not regulated.• Whether there is a specific provision in the applicable legal framework prohibiting the division of contracts to circumvent the regulations or thresholds to determine the method of procurement.A score of 1 is assigned if yes; or 0 if no. • Whether there is a specific provision in the applicable legal framework establishing a minimum deadline to submit bids.A score of 1 is assigned if yes; or 0 if no. • Whether the applicable legal framework regulates a standstill period after the award decision. 45A score of 1 is assigned if yes; or 0 if no.
The integrity of evaluations component of the legal framework index measures: • Whether the applicable legal framework regulates the minimum content of tender notices and tender documents.A score of 1 is assigned if both are regulated; or 0 if only one is regulated, or the legal framework is silent. 44Bidder should be able to identify the best evaluated offer, the grounds of evaluation and the grounds of exclusion, if any. 45 Standstill is defined as a period of time in which the public procurement process is suspended to allow bidders to submit challenges on the decision, if any.
• Whether the applicable legal framework regulates best value for money criteria.A score of 1 is assigned if the legal framework regulates the criteria and evaluation method to determine the Most Advantageous Bid, including systems to include rated criteria; or 0 if no. • Whether there is a specific provision regulating abnormally low bids.A score of 1 is assigned if there is a definition and the process to identify is regulated; or 0 if no. • Whether the applicable legal framework regulates the process to notify the award decision to bidders.A score of 1 is assigned if the full decision is notified to all relevant bidders; or 0 if bidders have to request access to the full decision, or if there is no notification.
The integrity of contract component of the legal framework index measures: • Whether the applicable legal framework regulates contract changes.A score of 1 is assigned if substantial changes (renegotiations), non-substantial changes (variation/changing orders) and new additional contracts (additional works) are all regulated; or 0 if no. • Whether there is a specific provision in the applicable legal framework establishing a maximum deadline to pay.A score of 1 is assigned if yes; or 0 if no. • Whether there is a specific provision in the applicable legal framework establishing compensation in case of late payment.A score of 1 is assigned if yes; or 0 if no. • Whether the applicable legal framework provides for protocols and guidelines for inspections.A score of 1 is assigned if yes; or 0 if no.

Annex 5 -Comparison between the indicator and other relevant products
The indicator has been inspired by several methodologies, indicators, and standards to design the questionnaire and identify good practices.At the same time, during the consultation process for the development of the indicator, several concerns have been raised on whether the recommendations of the indicator would be different than other WBG products and frameworks, and whether the assessments would be different as well.In order to provide more guidance on this matter, table 11 shows how the e-procurement and the legal framework subcomponents compare with MAPS (2016), WBG Procurement Framework (2016), and UNCITRAL (2011).

Annex 6 -Data collection
Questionnaires are administered to more than 4,000 local experts, including lawyers, construction companies, procuring entities,46 and government officials.Respondents include local lawyers with expertise in public procurement and infrastructure development, and construction companies specializing in the road sector.These experts have several rounds of interaction with the data collection team, involving conference calls, written correspondence, and visits by the team.In 2019, team members visited 36 economies to verify data and recruit respondents.The data from questionnaires are subjected to numerous rounds of verification.
Lawyers with experience in administrative and procurement law are pivotal in directing the team to the right laws and regulations, in providing a clear understanding of administrative powers, as well as in providing insights on their practical interactions with the procuring entity and the contractors.Their contribution is also helpful to understand the context of each economy.
Construction/engineering firms are key in helping the team understand how regulations are applied in practice, especially in some phases of the project in which lawyers might be less involved (such as quality controls and payments).
The above contributions are complemented, where relevant and appropriate, by insights provided directly by the procuring entity, for example on budgeting or internal needs assessment processes.
Data collection and analysis draws on the team's expertise in examining large datasets and conducting the necessary checks to ensure accurate and unbiased data.Information gleaned from the respondents, combined with the team's research into procurement laws and steps worldwide, are used to directly assess procurement systems.The data are reviewed several times by different members of the team, and also by WBG colleagues during a Bank-Wide Review process. Steps

Figure 4 :
Figure 4: Correlation between the score of the indicator, and its components, and GDP per capita, as measured by the World Development Indicators (WDI)

Table 1 :
Road sector works contracts are more prominent in developing

Table 3 :
Steps measured by the indicator Tendering phase Submission of performance guarantee in the form of a financial instrument Renegotiations: Substantial changes to the contract, usually requiring a contract addendum or amendment Variation Orders: Non-substantial changes that are often established in the contract or in the legal framework, requiring the issuance of a variation, changing or work order Additional Works: Change in the scope of the initial project that requires a new contract, often

Table 4 :
List of e-procurement features measured by the indicator What

Table 5 :
List of subcomponents measured by the legal framework index of the indicator

Table 6 :
Summary statistics for the time dimension (days)

Table 7 :
Summary statistics for the steps dimension (number of steps)

Table 9 :
Summary statistics for the legal index dimension (0-16)

Table 10 .
Regressions of the indicator scores on external variables.
External audit).Poor planning and budget oversight might cause delays during the process to award and sign contracts, as well as higher frequency of contract changes that delay the contract management phase.Furthermore, payments can also be delayed if economies do not implement good PFM practices.45 new economy-level assessments have already been completed with the new PEFA 2016 framework, 42 and more projects are currently underway.

Table 11 :
Summarized comparative table across relevant public procurement indicators and standards The table compares the components included in the tentative scoring methodology of the indicator constructed in this paper with other indicators and standards.N/A is displayed when there is no explicit corresponding guidance available in MAPS, WBG Procurement Framework, or UNCITRAL.