GOVERNANCE GOVERNANCE EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT Governance Risk Assessment System (GRAS) Advanced Data Analytics for Detecting Fraud, Corruption, and Collusion in Public Expenditures © 2023 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: “World Bank. 2023. The Governance Risk Assessment System(GRAS) Advanced Data Analytics for Detecting Fraud, Corruption, and Collusion in Public Expenditures”. © World Bank.” All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202- 522-2625; e-mail: pubrights@worldbank.org. >>> Contents Acknowledgements 4 Executive Summary 5 1. Introduction 6 2. The Governance Risk Assessment System: 7 Conceptual Framework and Structure Innovative and Operationally Relevant Risk Detection 7 Methodology GRAS Red Flag Framework 9 1. Risk Group: Procurement Cycle 10 2. Risk Group: Collusion 12 3. Risk Group: Supplier Characteristics 16 4. Risk Group: Political Connections 19 3. GRAS in Practice: Implementation and Results in 23 Brazil 4. A Roadmap for GRAS Implementation 31 5. Conclusions and Potential for Extensions 37 Further Enhancing GRAS 38 References 40 Appendix I. GRAS Data Field Requirements 43 Appendix II. Feasibility Assessment at the Indicator 48 Level Appendix III. GRAS Architecture 58 >>> Acknowledgments This report was prepared by a team led by Daniel Ortega Nieto (Senior Public Sector Management Specialist, World Bank) and formed by Mihály Fazekas (Government Transparency Institute and Central European University), Bianca Vaz Mondo (Government Transparency Institute), Bence Tóth (Government Transparency Institute and University College London), and Rafael Braem Velasco (Aethos Brazil). Ruben Interiam, and Igor Carpanese provided valuable research assistance. The team worked under the overall guidance of Adrian Fozzard (Practice Manager, Governance Public Sector Institutions, Latin America and the Caribbean, World Bank) and Robert Taliercio (Director, Equitable Growth, Finance and Institutions, for Latin America and the Caribbean, World Bank). Other World Bank and international experts consulted during development of the Governance Risk Assessment System include: Sinue Aliram, Alexandre Borges, Francesca Recanatini, James Anderson, Gavin Ugale, Alexandra Habershon, Rafael Munoz, and Gustavo Bozzetti. The development of the Governance Risk Assessment System (GRAS) was made possible by financing from the Spanish Fund for Latin America (SFLAC) Trust Fund. >>> Executive Summary Corruption poses a significant threat to development and has a disproportionate impact on the poor and most vulnerable. Government agencies struggle to identify fraud and corruption in public expenditures. Risk assessments usually rely on manual analysis and follow-up on specific complaints or anecdotes which requires substantial resources. Assessments are often limited in scope and ineffective, failing to generate the evidence needed to build strong cases. The World Bank developed the Governance Risk Assessment System (GRAS), a tool that uses advanced data analytics to improve the detection of risks of fraud, corruption, and collusion in govern- ment contracting. GRAS increases the efficiency and effectiveness of audits and investigations by identifying a wide range of risk patterns. GRAS makes use of public data and is based on a robust and comprehensive conceptual framework which draws on insights from experienced practitioners and sound academic research. This report presents GRAS’s main features, examples of GRAS implementation, and outlines the steps government agencies can take in applying GRAS in their countries. GRAS was de- veloped in Brazil, where it has been piloted in four subnational governments and has helped to investigate fraud, corruption and collusion in public procurement. Concrete results include the identification of over 850 suppliers with strong indication of collusive behavior, 450 suppliers likely registered under strawmen, 500 cases of conflict of interests involving suppliers owned by public servants, and about 4500 companies with connections to political campaigns, among other examples. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 5 1. >>> Introduction Public procurement is highly vulnerable to corruption, given the complexity of procurement pro- cesses, the high degree of official discretion, and the close interaction between the public and private sectors (OECD, 2016). The costs of procurement corruption are enormous. Conservative estimates from the empirical literature suggest that corruption can amount to about 8 percent of the value of procurement contracts worldwide, reaching some US$ 880 billion lost yearly.1 Interventions to prevent corruption in public procurement have focused on procedural standard- ization, strengthened transparency, reduced scope for discretion, and digitization in the procure- ment process. Evidence on the impact of these approaches on corruption is mixed (Bajpai and Myers, 2020, Fazekas and Blum, 2021). Methods for detecting and investigating corruption are inherently limited compared to the scale of the problem. Corruption risk assessments usually rely on manual analysis and follow-up on specific complaints or anecdotes. This is time-con- suming and inefficient, requiring the use of vast human and financial resources. Assessments are often limited in scope and ineffective, failing to generate the evidence needed to build strong cases to identify potential risks. Improvements in data collection, digitization, and public sector transparency have unlocked op- portunities to better address corruption, providing for the development of data-driven approach- es. The World Bank developed and implemented the Governance Risk Assessment System (GRAS) in Brazil to exploit the opportunities offered by a data rich environment. GRAS broadens the scope of risk assessments in public procurement, covering multiple risk patterns linked not only to corruption, but also to fraudulent or collusive practices. It improves the accuracy of cor- ruption detection in government contracting; consequently increasing the efficiency and effec- tiveness of audits and investigations. GRAS is based on a robust and comprehensive concep- tual framework covering 60 red flags, linked to 23 broad risk patterns along 4 dimensions. GRAS uses large volumes of contract-level and company data from public datasets: electoral registers; social program beneficiaries; public sector payroll; and blacklisted firms. Cross-referencing these datasets and leveraging algorithms, GRAS screens relationships among stakeholders, indicat- ing risks associated with collusive practices, supplier characteristics and political connections. As a result, GRAS’ interface can provide users with comprehensive aggregated risk reports that can have multiple use cases; for example, pre-screening firms before being awarded public con- tracts; anti-corruption investigations by internal and external control agencies; conflict of interest reviews of public or elected officials by monitoring bodies, among others. GRAS can also be used to identify potential tax fraud and collusive networks, as well as atypical spending patterns in strategic sectors. GRAS has been used in Brazil at the State and Municipal levels, leading to the identification of potential fraud, corruption, and collusion worth millions of US dollars. 1. https://blogs.worldbank.org/developmenttalk/reducing-corruption-public-procurement EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 6 2. >>> The Governance Risk Assessment System: Conceptual Framework and Structure Innovative and operationally relevant risk detection methodology The Governance Risk Assessment System (GRAS) systematically analyzes large public pro- curement and linked administrative datasets in order to generate actionable risk reports. If used as an integral part of regular investigations and audits by well-trained users, it can make a profound contribution to effective anti-corruption. Integrating GRAS into operational and inves- tigative processes has to be supported by staff training, a user manual and tailored operational processes. Data-driven risk assessment tools can rarely point at actual cases of fraud, corruption or collu- sion, rather they identify transactions or actors of high risk (Fazekas et al, 2019). The risks identi- fied indicate a higher likelihood of wrongdoing for transactions, entities (such as specific suppli- ers or procuring agencies) or individuals based on some generally validated features which we call risk indicators or red flags. Any risk-based approach inevitably will flag some transactions or entities as high risk even though they are compliant with the rules (i.e. false positives).2 How- ever, on average high risk transactions or entities are expected to be more susceptible to fraud, corruption and collusion than those that are assessed as low risk. The red flags used by GRAS are indicators of behaviors that are associated with fraud, corrup- tion or collusion based on lessons learned by auditors, investigators, and academics (Velasco, 2019). Datasets and data science approaches allow the derivation of robust and valid risk in- dicators across a wide variety of markets and countries even though case evidence is sparse. Risk indicators can be identified because fraud, corruption, and collusion involve specific forms of economic behavior that consistently leave traces, such as inexplicably successful govern- ment contractors, low bid participation rates, and abnormal cost overruns (Fazekas et al, 2018). These signals may appear as anomalies or outliers in the data while others may represent the norm that is average market behavior. In the latter case, corruption and collusion is systemic in the market leaving integrity as the outlier. 2. Similarly, such a system will inevitably generate “false negatives” too, that is, real malfeasance cases that cannot be detected. In the case of GRAS, this is minimized by the sheer comprehensiveness and breadth of the risk assessment framework. Nevertheless, some situations can be poorly identified based on big data alone, and corruption, collusion and fraudulent “technologies” in contracting tend to evolve with time, thus requiring permanent revisions and adjustments to the framework to better reflect new or adapted risk patterns. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 7 GRAS’ comprehensive and detailed risk assessment The accuracy and usefulness of red flags depend on the framework allows for lowering false positive rates. GRAS can availability and combination of the specific data points which are draw on a wide range of related risk indicators allowing for needed for their calculation. Oftentimes, the relevant data are triangulation, parametrization, and validation of predictions. located in different datasets. The core dataset indispensable This lowers the frequency of imprecise signals. Many risk for the application of GRAS is micro-level public procurement indicators are unreliable individually: the presence of a data (detailed data on procurement processes, contracts and single red flag may lead to a high rate of false positives. The purchased items). Such data are required for the calculation accuracy of risk assessment can be increased by combining of all red flags in the risk framework and depending on their or collating multiple risk indicators characterizing the same comprehensiveness, they may be sufficient to make one third underlying risky behavior. In this respect, GRAS offers a of the framework operational where complementary datasets high level of flexibility to knowledgeable users with filtering are not available. functions that enable them to explore different types of high- risk profiles based on criteria most relevant to their scope Additional datasets enrich GRAS and its indicators. Firm of action. Auditors, for instance, may define the parameters level data is particularly useful. Data on incorporated legal for prioritizing among red-flagged suppliers based on entities include: business registry information, shareholder their agency’s strategies, or on their empirical experience and management data, information on employees, and about different risk patterns that are commonly observed information on firms’ financial activities. Other datasets that in combination in their jurisdictions. Therefore, instead of can be used in GRAS include: electoral data; company using pre-defined risk-ranking parameters or intransparent blacklists or debarment lists; public sector payroll data; asset methodologies, GRAS provides for a more context-tailored and interest declarations of politicians and bureaucrats; social definition of complex risk profiles by users themselves, based registries (e.g. lists of welfare recipients that can help identify on transparently defined and documented indicators. While strawmen); and criminal records of individuals. this offers a high degree of flexibility to users, it also requires sophistication and data analytic proficiency from them. As GRAS is designed to allow for a robust and scalable risk many fundamental analytical decisions are made by the assessment process, incorporating large volumes of data user rather than decided by the system, inadequately trained across a whole country or jurisdiction. The framework can staff can quickly reach seemingly attractive, but incorrect be flexibly adapted to different contexts with varying data conclusions. Hence, offering in-depth technical training along readiness. with the introduction of GRAS is a must. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 8 GRAS Red Flag Framework GRAS indicators comprehensively assess a range of corrupt bidder (e.g. tailoring specifications to a particular company) practices in public expenditures. The precise definitions (World Bank, 2009). Such corrupt behaviors may manifest of these behaviors may vary from country to country or themselves in public tendering processes or outcomes, while even over time within the same country, still, a few high- they may also involve suppliers and linked individuals with level considerations are important to delineate the scope of risky features (Fazekas et al, 2018). behaviors assessed. Collusion is a distinct phenomenon from corruption, as it does not Procurement fraud, for example, may involve any act or require the participation of a public actor, only the coordination omission by an individual actor with the intent of deceiving or among supposedly competing companies. Collusion in public misleading other involved parties in order to obtain a (financial) procurement entails coordination of companies’ decisions advantage (World Bank, 2009). It may also occur independently regarding price, quantity, quality, or geographical presence to of corruption (e.g. bribery) or fraud (e.g. a bidder presents a eliminate competition in public procurement processes and forged certificate in its qualification documentation), but it is earn a mark-up above competitive conditions. This strategy often observed in connection to corrupt or collusive practices can only be sustained if (a) companies can coordinate, (b) (e.g. a shell company with hidden connections to a politician is internally sustainable (credible punishment system and bids in a tender, or operates in collusion with other bidders). effective detection of cheating), (c) it is externally sustainable (ability to exclude market entrants), and (d) the scheme can Corruption in public procurement refers to the allocation and go undetected and circumvent sanctions (Fazekas and Tóth, performance of public contracts by distorting principles of open 2016). and fair government contracting in order to benefit some to the detriment of all others (Fazekas & Kocsis, 2020). The aim of GRAS rests on 4 risk groups with 3 of them targeting fraud and corruption is to steer the contract to the favored bidder without corruption and 1 dedicated to inter-bidder collusion. These 4 detection in an institutionalized and recurrent fashion, by groups gather the 60 core red flags of GRAS as classified avoiding or biasing competition (e.g. unjustified sole sourcing under 23 broader risk pattern categories (Figure 1). or direct contract awards) in order to favor a certain, connected EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 9 > > > F I G U R E 1 - Overview of GRAS Risk Groups and Risk Areas Covered 2.2. Fixed differ- 2.3. Bid vari- 2.1. Top loser ence bids ance biases 1.1. Non-competi- 1.2. Non-competi- 1.3. Contract imple- 2.4. Unusual con- 2.6. Common tive processes 2.5. High price registration data tive tender results mentation biases tract value 1. Procurement 2.7. Common 2.8. Common cycle 2. Collusion shareholder employee 3. Supplier 4. Political characteristics connections 4.2. Personal 4.3. Personal con- 3.2. Unusual 3.3. Broad scope 3.1. Unusual size 4.1. Political finance connections to nections to public profitability of activities politicians official 3.4. Young supplier 3.5. Non-registered 3.6. Sanctions supplier 3.7. Shareholder 3.8. Shareholder 3.9. Tax haven with low socio-eco- with criminal record registration nomic status The first risk group includes those indicators which are closely Risk Group 1: Procurement Cycle tied to the different phases of the procurement cycle (OECD, 2016), namely tendering processes such as non- The risk group for the procurement cycle comprises indicators advertisement of call for tenders; tendering results such as of corrupt and fraudulent behaviors in public procurement single bid submitted; and contract implementation such as processes. These indicators capture risky behaviors in large cost overruns. The second group consists of indicators the three main phases of public procurement: tendering, approximating inter-bidder collusion, such as indicators of award, and contract implementation. They are indicative coordination opportunities among presumable competitors of deliberate manipulation of public procurement aiming to (e.g. common shareholder) and indicators of likely coordinated favor a particular supplier. While these indicators are highly bidding behavior (e.g. unusual bid price variance) (Adam et relevant on their own, they are especially useful as they al, 2022). The third group captures fraud and corruption risks further support and strengthen indicators from risk groups 3 centered on supplier characteristics. These risk factors (supplier characteristics) and 4 (political connections). Table 1 might relate to the company’s registry information such as enumerates 8 red flags related to each of the 3 risk patterns. registration in a tax haven jurisdiction; company financial records such as unusual profitability; multiple economic activities; or the company’s shareholders such as the criminal record of a company owner. The fourth group of risk factors is relational, capturing risks associated with political connections of a supplier. Connections can be established through personal connections to politicians or public official, or through companies’ political finance activities, that is, donating to an electoral campaign or political party (OECD, 2019). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 10 > > > T A B L E 1 - Individual Risk Indicators in the Procurement Cycle Risk Group (higher values indicate higher risk) Risk pattern Red flag/ indicator Description 1.1. Non-competitive 1.1.1. Contract share through Percentage of contracts won in high-risk, non-competitive processes non-competitive procedures procedure types (direct awards, invitation procedures, etc.) compared to all contracts won in a given time period (based on number of contracts or contract value) 1.1.2. Contract share after Percentage of contracts won in tenders without prior call for call for tenders absent tenders published compared to all contracts won in a given time period (based on number of contracts or contract value) 1.1.3. Contract share after Percentage of contracts won in tenders where advertisement shortened advertisement period (time between tender publication and submission period deadline) is too short compared to all contracts won in a given time period (based on number of contracts or contract value) 1.2. Non-competitive 1.2.1. Contract share as sin- Percentage of contracts won as single bidder compared to tender results gle bidder all contracts won in a given time period (based on number of contracts or contract value) 1.2.2. High winning rate Percentage of winning bids compared to all bids presented in a given time period (based on number of bids or bid value) 1.2.3. Contract share in Percentage of contracts won compared to the buyer’s total buyer's portfolio annual procurement (based on number of contracts or total value spent) 1.3. Contract imple- 1.3.1. Contract share with Percentage of contracts won with cost overrun above a given mentation biases sizeable cost overruns threshold (e.g. 5% more expensive than planned) compared to all contracts won in a given time period (based on number of contracts or contract value) 1.3.2. Contract share with Percentage of contracts won with delivery delay above a gi- sizeable delivery delays ven threshold (e.g. 5% longer than planned project) compared to all contracts won in a given time period (based on number of contracts or contract value) Among the red flags in the non-competitive processes sub- award the contract to a favored and/or connected company. group, the non-publication of call for tenders is one of This pattern is especially indicative of risks if it happens the most widely used (Fazekas et al, 2016). This indicator repeatedly with the same company. Naturally, buyers rather is initially defined for each tender where a call for tender than bidders decide whether or not to publish. However, publication is either present (indicator value=1) or absent corruption is typically well-organized and based on a network (indicator value=0). Then it can be aggregated to the level of of corrupt individuals across the public and private sectors. a supplier based on all the contracts won by the company in a Hence, a company repeatedly receiving information about period, resulting in the share of contracts without prior call for non-advertised bidding opportunities is more likely to have tender publication. This red flag points at potential corruption connections and be favored. Countries differ in the degree because not publishing the call for tenders makes it less likely to which non-advertisement is allowed, depending on the that eligible bidders notice the bidding opportunity, weakening procedure type, with a few countries where non-advertisement competition and allowing the contracting body to more easily is virtually non-existent.3 3. For a systematic mapping of regulatory requirements for European countries see: http://europam.eu/ and country overview statistics for selected countries: https://www. procurementintegrity.org/countries (integrity indicators panel). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 11 > > > F I G U R E 2 - Call for Tenders Advertisement and the Likelihood of Single Bidding in Mexico, Federal Procurement, 2017-2018 60% 50% 40% 30% 20% 10% 0% YES NO Was the tender advertised at the national procurement portal? Source: Government Transparency Institute calculation Among red flags in the non-competitive tender results category, to robust and fine-grained threshold definition (Fazekas and single bidding is by far the most widely used (e.g. European Kocsis, 2020). 20 percent cost overrun threshold is considered Commission, 2022, p. 227). Single bidding occurs when only high risk for World Bank funded projects in Fazekas and Márk one bid is submitted in an otherwise competitive market. (2017). This indicator is first calculated for each contract and While competition can be limited for a range of non-corrupt then can be aggregated to the level of supplier in order to reasons, corrupt deals almost invariably require some form of characterize organizational behavior. limited competition in order to award contracts to connected firms (Fazekas & Kocsis, 2020). The association between non-competitive tendering processes such as non-publication Risk Group 2: Collusion of call for tenders and single bidding supports the validity of these red flags (Figure 2). This indicator is initially calculated The risk group for collusion comprises indicators which signal on the level of lot or contract and then can be aggregated to collusive behavior among bidders such as cartels and bid- the level of the supplier in order to characterize the supplier’s rigging practices. GRAS collusion indicators capture collusive bidding behavior. outputs, such as coordinated bid prices or persistent losers in tenders, and the means by which companies may coordinate Corruption can also take place during contract implementation. bidding, such as common shareholders or employees across Sizeable cost overruns are an important red flag. While there supposedly competing firms. The 24 collusion risk indicators can be justifiable reasons for increasing contract value during are grouped under 8 broader risk patterns: top loser, fixed implementation, contract modifications can be used to extract difference bids, bid variance biases, unusual contract value, unwarranted profits, cover the costs of bribes spent to secure high price, common registration data, common shareholder, a contract, or cover expenses if the favored company could and common employee (Table 2). only win the contract by offering a competitive or even below- market price (Collier et al, 2016; Alexeeva et al, 2008). A crucial Collusive behaviors involving private actors, i.e. bidders, may challenge with this indicator, as with many other indicators in take place without the participation of public sector actors GRAS, is the definition of sizeable: the threshold above which (e.g. officials in a buying organization). However, GRAS can cost overruns may be considered as higher risk. There is no identify if corruption and collusion take place together by universally agreed threshold for risky behaviors, but data simultaneously applying collusion and corruption-related red analytics exploiting correlations among red flags can lead EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 12 flags. Under the collusion risk group, indicators are calculated require the definition of a market and the characterization at the tender, organization, or market levels before they can be of bidding behavior in relation to behaviors of other market related to individual suppliers. This is because some indicators participants (Fazekas and Tóth, 2016). > > > T A B L E 2 - Individual Risk Indicators in the Collusion Risk Group (higher values indicate higher risk, unless otherwise specified) Risk pattern Red flag/ indicator Description 2.1. Top loser 2.1.1. Low winning rate Percentage of winning bids compared to all bids presented in a given time period (based on number or value; lower values = higher risk) 2.1.2. Number of competitors Number of companies against which the Top Loser lost in a given time period (lower values = higher risk) 2.1.3. Number of wins against Number of bids won against Top Losers in a given time period Top Losers 2.1.4. Winning rate against Percentage of bids won against Top Losers compared to all Top Losers bids won in a given time period (based on number or value) 2.1.5. Number of Top Loser Number of Top Losers defeated by the bidder competitors 2.2. Fixed difference 2.2.1. Number of colluding Number of companies with which the pattern of fixed differen- bids partners with fixed difference ce bids is present, i.e. the company in question and another bids bidder repeatedly present a pair of bids with the same absolu- te or percentage difference over different tenders 2.2.2. Number of bids with Number of individual tenders/bids (items or lots) in which the fixed difference bids company bid in a fixed difference pattern, i.e. the company in question and another bidder repeatedly present a pair of bids with the same absolute or percentage difference over different tenders 2.2.3. Frequency of fixed Percentage of bids with fixed difference pattern compared to difference bids all bids presented in a given time period (based on number or value), i.e. the company in question and another bidder repeatedly present a pair of bids with the same absolute or percentage difference over different tenders 2.3. Bid variance biases 2.3.1. Bid share in low varian- Percentage of bids submitted on tenders with the Coefficient ce tenders of Variation (standard deviation divided by the mean of bids) very low, i.e. close to 1. 2.3.2. Bid share in high relati- Percentage of bids submitted on tenders with the relative ve bid distance tenders distance between the lowest and second lowest bid (distan- ce between the lowest and second lowest bid divided by the value of the lowest bid) very high 2.4. Unusual contract 2.4.1. Contract share with Percentage of contracts won in a given time period whose value contract value violating Ben- first digits of contract prices violate Benford’s Law (based on ford’s Law number of contracts or contract value) 2.5. High price 2.5.1. Contract share with Percentage of contracts won with a relative contract value very high relative contract (winning bid price divided by estimated value) above a given value threshold (e.g. 0.98) in a given time period (based on number of contracts or contract value) (also an indicator of potential corruption) CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 13 Risk pattern Red flag/ indicator Description 2.6. Common registra- 2.6.1. Number of competitors Number of competing bidders with which the company shares tion data sharing registration data the same registration information (e.g. same phone number, same postal address, same website, same legal representati- ve, same accountant) 2.6.2. Number of tenders with Number of individual tenders in which a competitor shares the bidders sharing registration same registration information data 2.6.3. Share of contracts won Percentage of contracts won against bidders with the same against competitors sharing registration information compared to all contracts won in a registration data given time period (based on number of contracts or contract value) 2.7. Common sharehol- 2.7.1. Number of competitors Number of competing bidders with which the company shares der with common shareholder a common shareholder 2.7.2. Number of tenders with Number of individual tenders in which a competitor shares the competitors sharing a sha- same shareholder reholder 2.7.3. Share of contracts won Percentage of contracts won against bidders sharing the against competitors with com- same shareholder in a given time period (based on number of mon shareholder contracts or contract value) 2.7.4. Number of competitors Number of competing bidders which belong to the same cor- in the same corporate group porate group 2.7.5. Number of tenders Number of individual tenders in which a competitor belongs to with competitors in the same the same corporate group corporate group 2.7.6. Share of contracts won Percentage of contracts won against bidders belonging to the against competitors in the same corporate group in a given time period (based on num- same corporate group ber of contracts or contract value) 2.8. Common employee 2.8.1. Number of competitors Number of competing bidders employing someone associated with common employee with the company (e.g. employee, shareholder) 2.8.2. Number of tenders Number of individual tenders in which a competitor employs with competitors sharing an someone associated with the company employee 2.8.3. Share of contracts won Percentage of contracts won against bidders employing against competitors with com- someone associated with the company in a given time period mon employee (based on number of contracts or contract value) One of the price-based collusion indicators is Benford’s law. first digit of contract prices of Swedish construction contracts Benford’s law is a statistical rule commonly used in forensic (Fazekas and Toth, 2016). Panel A shows contracts from the accounting, election monitoring, and in the study of economic Stockholm region, which shows that the actual distribution crime including collusion and corruption (Berger and Hill, 2015). is almost identical to our theoretical expectation. However, It posits that the first digit of most naturally occurring sets of number 3 was overrepresented, while numbers 5 to 8 were numerical data follows a specific distribution.4 Competition as underrepresented compared to the theoretical expectation such can be regarded as a natural process, hence contract in Jönköping County. These distribution differences are also prices in public procurement markets, assuming that prices statistically significant, suggesting that contract prices are are distributed across multiple magnitudes, should follow likely not a result of a competitive process and triangulating Benford’s law.5 As an example, we show the distribution of the collusive behavior with other indicators is warranted. 4. For example, 30.1% of first digits shall be 1, 17.6% number 2, 12.5% number 3 etc. 5. For example, if there is a high threshold above which contracts are published (i.e. low value contracts are entirely cut-off), low value contracts being not part of the distri- bution could lead to the violation of Benford’s law by default. Therefore, the difference between the theoretical and actual distribution of first digits ought to be calculated on a big enough sample of multi-magnitude distribution of contract values for it to be meaningful. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 14 > > > F I G U R E 3 - Distribution of First Digits in Construction Contracts in Sweden vs Benford’s Law (red line)6 Panel A: Stockholm County Panel B: Jönköping County Source: Fazekas and Toth, 2016, p. 74. Another price-based indicator is relative price, that is, the (relative price is above 90 percent) (Morozov and Podkolzina, awarded contract price divided by the initial estimation. The 2013). The literature on collusion risks suggests several price lower the relative price, the greater the savings that could be difference-based indicators and low bid price variance can achieved by competition. Naturally, bid prices - hence contract also be used to distinguish between collusive and competitive prices - might be higher than the initial estimations, as budgeting tender processes and for modeling favor exchanges among for complex projects is difficult ex ante. However, repeatedly colluding suppliers (Ishii, 2009). high relative prices are unusual in an otherwise competitive market: either buyers repeatedly underestimate costs, which The example below shows the distribution of relative prices of is unlikely, or bidders coordinate their bid prices. Relative contracts awarded to companies that were found to participate price is particularly useful when identifying potential bid- in collusion in Spain during and after the proven cartel period rigging schemes. Price increases unrelated to cost changes, (Figure 4).7 It shows that the relative share of contracts having long term price stability at unusually high levels indicate a relative price around 1, that is low savings compared to the market performance problems (OECD, 2014; Oxera, 2013). initially estimated price, is high during the cartel period (blue), Research shows that tenders with large discounts (relative while larger savings, that is when relative price is less than price below 90 percent) are associated with the number, 0.8, became more frequent after the prosecution of cartel capacity and experience of bidding suppliers, whereas these members. characteristics are unrelated to prices if discounts are small 6. The exact product code category assigned to these contracts is ‘construction work for pipelines, communication and power lines, for highways, roads, airfields and railways; flatwork’. 7. The cartel was active between 1996 to 2015, and the calculations are based on contracts awarded to the participating companies between 2005 and 2020. Note that some contracts might not have been rigged, the figure shows all contracts awarded to the prosecuted companies during and after the start of the legal case. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 15 > > > F I G U R E 4 - Relative Price During and After the Proven Cartel Period8 Source: Fazekas and Toth, 2016, p. 74. Risk Group 3: Supplier Characteristics are based on linked datasets such as sanction or debarment lists. Supplier risk indicators generally build on company registry information such as location of registration (e.g. in The risk group of Supplier characteristics comprises a tax haven), financial performance data (e.g. turnover and indicators for features of government suppliers that indicate profitability), and shareholder and management information likely fraudulent or corrupt behavior. Suppliers participating (e.g. names and shares of owners) (Fazekas et al, 2018). in corrupt exchanges act as vehicles of rent extraction and Most risk indicators in this group are directly related to specific distribution. Just as corrupt government contracting differs suppliers with some related to specific individuals, such as a from competitive tendering and contract implementation, shareholder, and then aggregated to the company level. As companies participating in corrupt exchanges are expected with many indicators in the GRAS framework, some indicators to differ from their peers in a number of key features. High are sensitive to national and market conditions and have to be risk supplier characteristics are diverse. Table 3 enumerates tailored to context. For example, unusual profitability requires 18 risk indicators covering 9 different risk patterns. Nearly all setting of appropriate thresholds, taking into consideration indicators in this group require combining company and public normal market or sectoral profit rates, which vary over time procurement indicators and data, in some cases indicators depending on market dynamics. 8. The exact product code category assigned to these contracts is ‘construction work for pipelines, communication and power lines, for highways, roads, airfields and railways; flatwork’. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 16 > > > T A B L E 3 - Individual Risk Indicators in the Supplier Characteristics Risk Group (higher values indicate higher risk, unless otherwise specified) Risk pattern Red flag/ indicator Description 3.1. Unusual size 3.1.1. Contract revenue/ tur- Ratio of the total value of contracts won in a given time period nover ratio compared to company turnover in the same period (very high, especially above 1 = higher risk) 3.1.2. Contract revenue per Total value of contracts won in a given time period divided by employee the number of company employees (values higher than the market average = higher risk) 3.2. Unusual 3.2.1. Unusual profitability Company profit rate in a given time period (values higher than profitability the market average = higher risk) 3.3. Broad scope of 3.3.1. Number of economic Number of economic activities, i.e. distinct detailed market activities activities codes (very high number or activities from different sectors = higher risk) 3.4. Young supplier 3.4.1. Period between incor- Number of days between the date of incorporation and the poration and 1st contract date of 1st contract won (lower values, typically less than 365 = higher risk) 3.5. Non-registered 3.5.1. Contract before incor- Company incorporation date is after the contract award date supplier poration 3.6. Sanctions 3.6.1. Sanctioned company Company under sanctions: past or current 3.6.2. Sanctioned sharehol- Company’s shareholder/legal representative under sanctions: der/ legal representative past or current 3.6.3. Link to another sanctio- Company’s shareholder/legal representative linked to another ned company sanctioned company: past or current 3.6.4. Contracts while sanc- Number/total value of contracts won while under a sanction tioned (applied to the company or to a connected individual) 3.6.5. Sanction relative dura- Duration of the sanction period (company or connected indivi- tion dual) relative to the total time the company has existed 3.6.6. Period between incor- Number of days between the date of incorporation and the poration and 1st sanction starting date of its first sanction (lower values = higher risk) 3.7. Shareholder with 3.7.1. Shareholder has low Company shareholder with extremely low socio-economic low socio-economic socio-economic status status (e.g. registered as social beneficiary, low-income/low- status -skilled employee, or member of poor household) 3.7.2. Status duration Total number of months the shareholder has/had extremely low socio-economic status (e.g. for how long registered as so- cial beneficiary, low-income/low-skilled employee, or member of poor household) 3.7.3. Time overlap between Number of months he/she was simultaneously a company status and company owner- shareholder and with low socio-economic status ship 3.8. Shareholder/legal 3.8.1. Convicted shareholder Company shareholder/legal representative has a criminal representative with conviction criminal record 3.9. Tax haven regis- 3.9.1. Company registered in The company is registered in a tax haven (as denoted by Tax tration tax haven Justice Network’s Financial Secrecy Index) 3.9.2. Shareholder registered A company shareholder is registered in a tax haven (as deno- in tax haven ted by Tax Justice Network’s Financial Secrecy Index) EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 17 One of the most widely used red flags for suppliers is registration Another related risk is the potential loss of tax revenue from of the supplier or one of its significant shareholders in a secrecy the successful supplier through tax evasion or tax avoidance jurisdiction. We identify tax havens using the Financial Secrecy (Fazekas and Kocsis, 2020). We expect a higher incidence of Index of the Tax Justice Network (Tax Justice Network, 2022). risk factors in the procurement cycle when the foreign supplier Awarding a public contract to a company registered in a tax is registered in a tax haven too. Looking at a large Europe- haven presents the risk that anonymous company ownership wide dataset, Fazekas and Kocsis (2020) find exactly this conceals a conflict of interest of a politically connected owner. relationship (Figure 5). > > > F I G U R E 5 - Non-Domestic Suppliers’ Tax Haven Registration (based on FSI score) and the Incidence of Selected Procurement Cycle Red Flags in the European Union (including the UK), 2009-2014 0.34 0.34 0.32 0.31 0.3 0.28 0.28 0.26 0.26 0.24 0.22 0.2 single bidder radio CRI NOT tax haven registered company tax haven registered company Source: adapted from Fazekas and Kocsis (2020) Sanctions also represent a strong signal of potential incidence of red flags of the procurement cycle is about twice wrongdoing, even though the incidence of such red flags as high for debarred suppliers in the United States federal tends to be rare (typically a few thousand flagged cases out of procurement as is for suppliers that have not been debarred millions). Sanctions are highly correlated with more frequently (Figure 6). observed red flags in the procurement cycle. For example, the EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 18 > > > F I G U R E 6 - US Federal Suppliers’ Debarment Status and the Incidence of Selected Procurement Cycle Red Flags, 2004-2015 Source: adapted from Fazekas et al (2022) Risk Group 4: Political Connections most demanding in GRAS: each individual relationship requires linking at least 3 different dataset categories, connecting suppliers (from public procurement data), their shareholders The risk group of Political connections comprises indicators or employees (from company or employment data) and public which capture the relational aspects of corruption, some actors or political organizations (from electoral data, public point directly at conflict of interest while others represent payroll information or asset and interest declarations). As each organization-level relationships such as a company donating risk indicator covers a few variants of possible relationships, to a political party. Corruption in public procurement, due including more complex, indirect links, GRAS requires data to its very nature, involves informal coordination between a from four or five different data sources to screen for all these range of public, i.e. politicians and bureaucrats, and private variants for each indicator.9 These complex relationships are actors (Fazekas et al, 2018). Political connections can be further detailed in Table 5. demonstrated in a number of ways, through political finance such as campaign donations or personal connections such as While most indicators in this group are binary, recording the family ties. Hence, we organize the 10 political connections risk existence or absence of a link, in some cases indicators derive indicators according to 3 risk patterns (Table 4), recognizing from continuous variables (e.g. total value of donations of a the division between politicians and civil servants: political company in a period) which require appropriate thresholds. finance, personal connections to politicians, and personal Small donations are unlikely to be suitable red flags for connections to civil servants. corruption. For example, Fazekas et al (2022) found in the US federal contracting market that only donations above Risk indicators in this risk group are initially assessed at about 11,000 USD have a discernible effect on companies’ the level of relations which are then traced back to specific contracting risks with tendering risks substantially increasing suppliers, for example identifying a former politician employed as donation value increases. The appropriate threshold for by a government supplier. Given the relational nature of political high-risk donations depends on the country and period which connections risk indicators, data requirements are among the the adoption and tailoring of GRAS should take into account. 9. Specific data field and dataset requirements for each individual indicator are specified in Appendix II. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 19 > > > T A B L E 4 - Individual Risk Indicators in the Political Connections Risk Group (higher values indicate higher risk) Risk pattern Red flag/ indicator Description 4.1. Political finance 4.1.1. Donation to electoral Company/shareholder/employee donated/supplied to a politi- campaign cian/political party in a period: Yes/no (variants: direct/indirect link & link to non-elected/elected/in power politician/party) 4.1.2. Value of donation to Value of donations made by the company to a politician/po- electoral campaign litical party in a period (variants: direct/indirect link & link to non-elected/elected/in power politician/party) 4.1.3. Contracts won following Number/total value of contracts won from public bodies (e.g. donation municipality, region, central government body) with politicians who received donations/supplies from the company in period (variants: direct/indirect link & link to non-elected/elected/in power politician/party) 4.1.4. Percent of contracts Percent of contracts won from public bodies (e.g. municipality, won following donation region, central government body) with politicians who recei- ved donations/supplies from the company in period (variants: direct/indirect link & link to non-elected/elected/in power politician/party) 4.2. Personal connec- 4.2.1. Company's personal Company has or had a personal connection to a politician/ tions to politicians connections to politicians political party functionaire in a period: Yes/no (variants: direct/ indirect link & link to non-elected/elected/in power politician/ party) 4.2.2. Contracts won following Number/total value of contracts won from public bodies (e.g. political connection municipality, region, central government body) with politicians who are/were linked to the company in period (variants: direct/ indirect link & link to non-elected/elected/in power politician/ party) 4.2.3. Percent of contracts Percent of contracts won from public bodies (e.g. municipali- won following political con- ty, region, central government body) with politicians who are/ nection were linked to the company in period (variants: direct/indirect link & link to non-elected/elected/in power politician/party) 4.3. Personal connec- 4.3.1. Company's personal Company has or had a personal connection to a public bure- tions to bureaucrats connections to bureaucrat aucrat in a period: Yes/no (variants: direct/indirect link) 4.3.2. Contracts won following Number/total value of contracts won from public bodies (e.g. connection to bureaucrat municipality, region, central government body) with public bureaucrat who are/were linked to the company in period (variants: direct/indirect link) 4.3.3. Percent of contracts Percent of contracts won from public bodies (e.g. municipality, won following connection to region, central government body) with public bureaucrat who bureaucrat are/were linked to the company in period (variants: direct/indi- rect link) EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 20 > > > T A B L E 5 - Possible Sub-Dimensions of Political Connection Indicators Connection domain Nature of connection Type of connection Political finance Donor/Supplier Direct: through the company itself or one of its shareholders Politician/party official personal or other individuals directly linked to it (e.g. legal representati- ves, employees, managers, accountants) Civil Service personal Indirect: through a connected company (common sharehol- der or other individual as listed above), a business associate (shareholder's partner in another company), a relative etc. Company donations to electoral campaigns have received Among political connections risk indicators, one of the most extensive scholarly and policy interest (OECD, 2017, chapter widely studied and probably most relevant, is the employment 1). When a prospective government supplier donates to a of top politicians by companies to gain government favors political campaign, it may intend to support the candidate (Goldman et al, 2013). Former politicians can open doors who, upon winning elections, can pay back the favor through for a future supplier, share insider information or facilitate government contracts. Such a pattern has been identified in bribery in return for contracts. Studies have found that high as well as low integrity countries as diverse as Sweden, suppliers’ connections to political decision-makers increases the US or Brazil, though the scale of impact varies: in Brazil, their procurement revenue. In Serbia politically connected 100 USD party donations leads to an additional 1400 USD suppliers have about 30 percent higher single-bidding rate worth of contracts, while the same number is “only” 250 USD compared with politically unconnected suppliers, a pattern worth of contracts in the US (Boas et al, 2014; Bromberg largely reproduced across the Balkans region (Mineva et al, 2014; Hyytinen et al, 2018). 2023). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 21 Datasets Underpinning GRAS Risk Indicators The comprehensive and refined risk indicator framework of company registries and company financial records. These GRAS requires a range of high-quality and granular datasets are similarly essential and required for many risk indicators. linked to each other. These datasets have been identified Employment relationship data are also among the most based on the practical demands of indicator calculation relevant for GRAS and typically include employer name and across a range of countries and effective operation of GRAS ID, employer location, employee name and ID, employee based on the Brazilian experience. Broadly speaking, GRAS admission date, employee position and remuneration, demands two types of datasets, the public contracting employers’ number of employees. datasets and micro-level data on firms and individuals. Table 6 summarizes the types of data required for GRAS, with a Individual-level information is found in a variety of different general assessment of how essential each of these are for the databases. Government social benefit programs typically indicator framework. include information on the name and ID of beneficiaries, type of benefit (e.g. conditional cash transfer), benefit duration, Public expenditure databases contain details of all or most benefit value, and beneficiary location. This information can be phases of the public procurement cycle, at the contract or used to identify risky company officials (strawmen). Electoral purchase levels. These are essential, first and foremost, for datasets can be used to establish connections between firms all risk indicators in the procurement cycle group. Moreover, and politicians, revealing direct and indirect links associated they are also used for all the other risk patterns assessed by with potential conflict of interest or favoritism. Asset and the system. The data contained in these databases generally interest declarations can be used to identify conflicts of includes: supplier name and unique ID, requesting agency interest and political connections. Electoral records contain name and location, procurement method, contract value information on candidate name, ID and party, election results, and date, proposals details, winning proposal, and contract campaign suppliers’ name and ID, campaign expenses details, amendments details. campaign donor name and ID, campaign donation value. Criminal records databases typically include the names and The second block of datasets includes information on IDs of those convicted, the criminal offense, date and location companies or individuals. Data on companies is found in of criminal complaint. > > > T A B L E 6 - Key Datasets for GRAS and Respective Relevance Level Dataset category Number of related red flags Relevance level 1 Public procurement 60 Essential 2 Employment relationships 17 Essential 3 Corporate and shareholder data 39 Essential 4 Electoral data 7 Important 5 Blacklists 6 Important 6 Asset and interest declarations 6 Important 7 Socio-economic data 3 Useful 8 Criminal records 1 Useful EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 22 3. >>> GRAS in Practice: Implementation and Results in Brazil The World Bank implemented GRAS as a pilot initiative in the states of Mato Grosso and Rio de Janeiro, and the Municipalities of São Paulo and Porto Alegre in late 2022.10 Brazil has made substantial efforts to increase government transparency through the proactive disclosure of public information over the last decade and so offers a favorable data environment for the implementation of GRAS (OECD, 2022). The Brazilian GRAS benefits from the availability of numerous public datasets including trans- actional data on more than 2 million contracts executed in 10 Brazilian states and by the fed- eral government, accounting for over US$ 50 billion in expenditures (Velasco et al. 2020). This detailed public procurement data can be combined with publicly available micro-level data on companies and individuals, drawn from business registration data and datasets on political campaign donations and expenses, sanctioned suppliers and individuals, and conditional cash transfer beneficiaries, among others. GRAS currently operates a large data lake with over 250 million data points. Its data mining algorithms can automatically identify dozens of risk patterns related to public procurement fraud and corruption, at the level of public suppliers, contracting agencies and even individuals. This offers oversight bodies a powerful tool to identify high-risk entities in the public procurement market and allows them to better target their investigations. The red flags processed in the GRAS system correspond largely to those described in Chapter 2, with a few exceptions that have not been incorporated into GRAS as described in this sec- tion. The framework described in Chapter 2 is broader than the Brazilian GRAS to incorporate widely used and validated indicators from other risk assessment tools, such as the European opentender.eu11 or the global procurementintegrity.org (Box 1), or based on relevant academic literature.12 10. This was possible thanks to the resources provided by the Spanish Fund for Latin America (SFLAC); the governments implemented GRAS starting January 1st, 2023. 11. The indicators related to the tender advertisement period or to suppliers’ characteristics are employed on the platform opentender.eu, covering close to 50 million contracts in 33 European jurisdictions. 12. This is the case of a few novel collusion indicators that have been developed and presented in recent literature on cartel detection. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 23 3.1. Data Sources Used by GRAS in Brazil key dataset category that is not part of the Brazilian GRAS is asset and interest declarations, which are not available in this context, but can be an important source of relevant information The system as implemented in Brazil relies on seven out for GRAS implementation in other countries where this data of the eight dataset categories listed in table 6 above. All can be accessed. Table 7 below lists the datasets employed sources used are publicly available, with the exception of data in Brazil. on employment relationships in the private sector. The only > > > T A B L E 7 - Data Sources Employed by GRAS in Brazil Dataset cate- Dataset type Dataset name Responsible agency Scope gory 1. Public pro- 1.1. Public con- Licitações e contratos State Courts of Accounts Selected Brazilian States curement tracting datasets 2. Employ- 2.1. Employ- Relação Anual de Infor- Ministério do Trabalho e National ment rela- ment registra- mações Sociais (RAIS)13 Emprego tionships tion 2.2. Public ser- Servidores públicos (different agencies at all National, sub-national vants government levels)14 3. Corporate 3.1. Fiscal regis- Relação de Instituições Receita Federal do Brasil National and sharehol- tration data Financeiras em funcio- (RFB) der data namento 3.2. Financial Relação de Instituições Banco Central do Brasil National sector compa- Financeiras em funcio- nies namento 4. Electoral 4.1. Candidates Candidatos Tribunal Superior Eleitoral National data profiles (TSE) 4.2. Campaign Prestação de contas de Tribunal Superior Eleitoral National finance data campanha (TSE) 4.3.Party finan- Prestação de contas Tribunal Superior Eleitoral National ce data partidárias (TSE) 5. Blacklists 5.1. Blacklisted Cadastro Nacional de Controladoria Geral da National companies Empresas Inidôneas e União (CGU) Suspensas (CEIS) 5.2. Sanctioned Cadastro Nacional de Controladoria Geral da National companies Empresas Punidas União (CGU) (CNEP) 5.3. Sanctioned Cadastro Nacional de Controladoria Geral da National non-profits Entidades Privadas sem União (CGU) Fins Lucrativos Impedi- das (CEPIM) 5.4. Blacklisted Cadastro de Empre- Ministério do Trabalho e National employers - sla- gadores que tenham Emprego ve work submetido trabalhadores a condições análogas à de escravo CONTINUED 13. Dataset protected by law for data concerning individual person identity and salaries, but released under specific non-disclosure agreement. An anonymised dataset is publicly available. 14. Decentralized data collected through multiple web crawlers from the different agency websites where these are published. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 24 Dataset cate- Dataset type Dataset name Responsible agency Scope gory 6. Socio- eco- 6.1. Conditional Benefícios ao cidadão Ministério do Desenvolvi- National nomic data Cash Transfer mento e Assistência Social, beneficiaries Família e Combate à Fome (MDS) 7. Criminal 7.1. Arrest war- Banco Nacional de Man- Conselho Nacional de Justi- National records rants dados de Prisão (BNMP) ça (CNJ) Data from the different sources were unified into a single of different red flags. Fraud cases often present multiple database, allowing the implementation of algorithms for the suspicious aspects that can be well captured by such red- identification of each individual red flag and thereby increasing flag-based systems, in particular when they are designed to the efficiency of the system. Automatic routines were identify red flags from different perspectives, as is the case developed to clean, transform and validate the data, ensuring with GRAS. Consequently, the system is particularly useful to sufficient quality for the system to run (Velasco et al. 2020). identify “high risk suppliers” that are more likely involved in With regards to its technical architecture, GRAS is a highly fraud, corruption and collusion and so may be prioritized for versatile and flexible system that can operate independently investigation. of any specific hardware or software configurations. A more detailed description of its structure is offered in Appendix III. This feature of GRAS can be demonstrated by navigating a concrete case. An audit team using GRAS in Rio de Janeiro GRAS system offers a comprehensive and user-friendly received a tip on a specific company from one of the largest platform for generating audit reports and searching for municipalities in the state. The company had won 11 contracts public agencies, politicians, or firms. It also provides intuitive with a total value of more than R$ 16 million (US$ 3.2 million). selection filters to identify potential red flags in companies or GRAS was used to dig deeper into the company, and a variety procurement procedures. Users can select any combination of of risk patterns indicating potential fraud were identified on the red flags as well as actual values and ranges for the specific GRAS interface using a selection of specific filters. red flags. Risk pattern #1 - Broad scope of activities. Corporate 3.2. Illustrative Examples and Purposes of the registration data indicated that the company in question had System as main economic activity kitchen and catering services for transport companies and corporate offices. However, its A system such as GRAS enables a range of different registered secondary activities were unusually diverse: trade applications and extensions, depending on the dataset in parts and components of motor vehicles; wholesale trade categories available and the quality and comprehensiveness of computers and computers supplies; hydraulic, ventilation, of the data collected. In Brazil, GRAS was developed in direct and cooling systems; car rental without driver; wholesale trade collaboration with law enforcement and oversight offices in of instruments for medical and surgical use; maintenance selected Brazilian subnational governments, with a strong and repair of vessels; wholesale trade in chemicals and focus on its potential for investigations in connection with petrochemicals; and cleaning services. Such a broad and procurement fraud, corruption and collusion. Other uses and unconnected range of activities - when not related to well- purposes were explored as well. This section offers a few known very large conglomerates - is common for companies examples of these uses, illustrating some of the system’s involved in fraudulent or collusive practices. contributions to investigative initiatives. Risk pattern #2 - Shareholder with low socio-economic Example 1. Identifying multiple red flags for a selected supplier status. Corporate registration data also indicated that the company had two shareholders, one of whom was a beneficiary GRAS relies on a wide range of risk patterns and associated of the cash transfer program Bolsa Família, which targets red flags to identify instances of potential fraud, corruption families with low income. This is a risk pattern indicating a and collusion in public procurement. Each red flag functions potential strawman, someone used to hide the identity of the only as an indication of potential irregularity, and the level of company’s real owner(s). risk rises as individual suppliers are linked to a higher number EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 25 Risk pattern #3 - Personal connections to bureaucrats. Risk pattern #4 - Political finance. GRAS indicated a key GRAS identified the company’s second shareholder as a civil political connection risk, because one of the company’s servant in a federal public hospital in the city of Rio de Janeiro, shareholders, listed as a Bolsa Família beneficiary, had been employed as a social worker. Most of the company’s contracts a supplier to the campaign of an influential local councilor who were related to food supply for hospitals. had recently run for mayor. > > > F I G U R E 7 - GRAS Navigation through Multiple Red Flags Connected to a Single Supplier Red ag: diverse and Red ag: shareholder as a Search for the name of the non-related economic bene ciary of government company and click activity classi cation social program The GRAS User now has a complete and detailed analysis of a company's public sector operations and red- ags Red ag: political Red ag: political connections - shareholder as connections - shareholder as political campaign supplier public servant In this example in particular, GRAS was used as a complement Auditors and investigators can, for instance, rely on their to a tip received by the audit team, but other investigations can qualitative expertise about risk patterns that are commonly similarly be initiated following initial suspicions of fraud and observed in combination in known cases in their jurisdictions, corruption raised from the system’s reports. Through filtering and select groups of risk patterns as filters to identify potential functions and different aggregation options, GRAS offers similar cases. GRAS can also help enforcement agencies users sufficient flexibility to define sets of criteria based on identify multiple risk patterns based on which different potential which higher risk actors can be identified and narrowed down lines of investigation can be pursued for corroborating to be targeted by specific investigations (see Figure 8 below). evidence. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 26 > > > F I G U R E 8 - Examples of GRAS Filtering Options Example 2. Identifying collusion in electronic reverse would have been the case under real competition.16 auctions A number of GRAS red flags from the Collusion risk group can In electronic reverse auctions, suppliers anonymously bid help to identify collusion using a “kamikaze” company: a) as prices down until the auction is complete. This procedure aims a typical bid variance bias, where a high relative bid distance to promote a high level of competition. The electronic reverse between the kamikaze’s and winner’s bids is observed; b) the auction can be gamed through collusive behavior using a so- colluding partner that is set to win the auction will bid to deviate called “kamikaze” (or “rabbit”) company,15 in coordination with the least possible from the third lowest bid, which might also another bidder - the scheme usually involves two colluding result in a high relative contract value; c) connections between partners: One presents an unrealistically low bid to scare the colluding bidders, such as common registration data or a off other competitors from further reducing their initial bids. common shareholder or employee, are likely to be present; The other colluding partner, which is meant to actually win and d) if the kamikaze company engages repeatedly in such the contract, then adjusts its initial bid only to the extent schemes, it might also present the Top Loser risk pattern. necessary to have the second lowest bid. In these auctions, Similarly, the winning company may be identified as a typical the qualification phase typically takes place after the bidding; winner against Top Losers, a risk pattern also made visible the kamikaze partner then purposefully fails to fulfill the by the system (Figure 9). This feature assists users in more qualification requirements, and the colluding partner, after easily identifying potential collusive links between providers. being qualified, is awarded the contract at a higher price than 15. https://3rcapacita.com.br/artigo/licitante-coelho 16. Changes in the regulation of electronic reverse auctions in Brazil were introduced in 2019 in an attempt to address these situations, by requiring all qualification doc- umentation to be submitted beforehand (https://www.editoraforum.com.br/noticias/decreto-do-novo-pregao-eletronico-inibira-fraude-conhecida-como-coelho-nas-licitacoes-afir- ma-especialista/). EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 27 > > > F I G U R E 9 - Information on Providers with a Pattern of Winning Against Top Losers Example 3. Preventive vetting of bidders can also serve as a quick and effective resource for due diligence during the tendering phase based on reliable, third- GRAS relies largely on data from past contracting procedures party data.17 and can be a useful tool for identifying contracting irregularities ex-post. A further application for preventive purposes is 3.3. Impact and Results Achieved also possible. GRAS can be used as a bidder “vetting tool”, allowing purchasing authorities to assess bidder profiles An assessment of preliminary results achieved using GRAS prior to awarding a contract. This review could offer a quick during pilot implementation reveals that the system has been and detailed overview of bidders’ contracting history such effective in detecting a large number of procurement tenders as amount and types of contracts awarded in the past; and and contracts displaying risk patterns. Table 8 below illustrates review of risk patterns related to suppliers’ characteristics as some of the results for selected red flags that the system well as indicators from the procurement cycle. Hence, GRAS reported in 2020. 17. For another, company vetting tool, supporting due diligence, but based on globally available public procurement data, see: https://tenderx.eu/ EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 28 > > > T A B L E 8 - Examples of Cases Identified by GRAS for Selected Risk Patterns Risk group Risk pattern Identified cases 1. Procurement cycle 1.1. Non-competitive proces- 2308 companies that received contracts through direct awards ses 2. Collusion 2.1. Top loser 420 companies that won bids against top losers 2.7. Common shareholder 857 companies that won bids against companies sharing a common shareholder 3. Supplier characte- 3.4. Young supplier Almost 150 companies that were awarded a contract within ristics 120 days after their incorporation 3.6. Sanctions Approximately 800 sanctioned companies that were awarded contracts 3.7. Shareholder with low 450 companies with shareholders registered either as benefi- socio-economic status ciaries of cash transfer programs or as low-skilled employees 4. Political connections 4.1. Political finance Almost 4,500 public suppliers that are either electoral cam- paign donors (as companies or through their shareholders) or suppliers 4.3. Personal connections to 500 firms owned by public servants that received contracts bureaucrats from the same agency where the shareholders were em- ployed These exemplify some typical risk patterns linked to being registered were not uncommon in Brazil. Recently procurement fraud, corruption and collusion in Brazil. Non- created suppliers may be inexperienced, increasing risks for competitive procedures, for instance, are allowed under proper contract implementation. They may have been created certain circumstances, but are sometimes unduly employed as shell companies to be used by corrupt networks. in combination with multiple bidding procedures for similar or almost identical objects, in order to remain under contract Companies that have been formally sanctioned and suspended value thresholds above which competitive procedures would from contracting may present increased risks. GRAS identified be legally required (Santos and Souza 2016). some 800 firms that won new contracts while under sanctions. Procurement regulations require public agencies to check Collusive rings working to simulate competition in tenders databases of sanctioned firms before contracting, but this are a common feature in public procurement. GRAS flagged provision is clearly not preventing sanctioned companies from a number of risk patterns that may indicate collusion. getting new business in the public sector while blacklisted. In Interestingly, links between shareholders are not necessarily one of the states covered in the GRAS pilot, almost 7 percent well hidden, as shown by the hundreds of cases where the of all contracts were awarded to sanctioned companies. system identified “competing” bidders with common owners. By analyzing information from multiple tenders, GRAS Strawmen are used to register companies that bid for and detected bidding patterns that indicate potentially colluding occasionally win procurement contracts. Use of strawmen actors that are not real competitors, i.e. the so-called top to hide the company’s true beneficiary is linked to irregular losers,18 as well as the companies that may have benefitted activity which warrants further investigation. GRAS flags these from collusive action winning contracts against such top cases by profiling the socio-economic status of companies’ losers. GRAS identified hundreds of suppliers associated with shareholders through complementary datasets, indicating these risk patterns in a small sample of Brazilian states. when they are identified as beneficiaries of cash transfer programs or registered as low-skilled or low-pay employees, GRAS also employs multiple indicators related to risky supplier i.e. with a socio-economic status that is atypical for real characteristics. Companies that obtain contracts shortly after business owners. 18. For another, company vetting tool, supporting due diligence, but based on globally available public procurement data, see: https://tenderx.eu/ EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 29 The Brazil pilot identified thousands of suppliers (or their The use of GRAS by Public Prosecutor’s offices in the shareholders) registered as donors or suppliers to electoral pilot states has already actively contributed with input to campaigns or political parties. These companies have been relevant corruption investigations. In one of these states, awarded in total close to R$ 100 billion (US$ 2 billion) in two investigations conducted by the Federal Police were public contracts.19 Indeed, in all of the states covered in informed by targeted analyses conducted with GRAS. GRAS the pilot projects a large share of contracts was awarded to identified risk patterns that led to the uncovering of farms of companies with connections to politicians, in some cases shell companies and a money laundering chain. GRAS used a more than half of all contracts. An even stronger indication specific algorithm to identify ghost public workers in the public of potential conflict of interest are cases of suppliers with payroll in one municipality used by locally-elected officials to politicians or public servants as shareholders, in particular divert public funds (Velasco et al. 2020). This illustrates that when their contracts were obtained from the same agency GRAS can yield concrete results in a short period of time after where those shareholders occupy office or are employed. In implementation. In the Brazilian context, its application at the a single state, GRAS identified 122 companies with politicians sub-national level has been of great relevance to strengthen as shareholders (Velasco et al. 2020). In all the jurisdictions anti-corruption action where oversight and law enforcement covered by the pilot, GRAS identified 500 companies owned agencies are more under-resourced. by public servants that received contracts with agencies where they worked. 19. Corporate donations were allowed in Brazil until 2015, when a ban was introduced and only private donations from individuals remained legal (https://www.idea.int/da- ta-tools/country-view/68/55). As GRAS includes data from earlier periods, corporate donations can still be identified in some cases, and individual donations by company owners remain substantial. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 30 4. >>> A Roadmap for GRAS Implementation Building on the positive results of the GRAS pilot in Brazil, the World Bank seeks to promote implementation of the system in other countries and jurisdictions. This section reviews relevant steps for a feasibility assessment applicable to any context, which, by and large, include: • • • • Collaboration with governmental agencies; Data maturity assessment; Validation and adaptation of the red flag framework; and Feasibility assessment and recommendation of improvements of the data infrastructure. Following this initial assessment, a detailed implementation plan for a pilot stage must be devel- oped together with collaborating government agencies. For this step, there is no standardized model, as concrete steps for implementation in a given jurisdiction will be highly context-specific, with a broad variation in scope of implementation depending on aspects such as: data availabil- ity, data access and data protection issues; extent of support within government and at different government levels; existence of complementary data analytics initiatives; the need for tailoring of the framework to the specific context and data environment, among other relevant factors to be taken into account. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 31 4.1. Collaboration with Governments been instrumental in mapping relevant data sources for the key dataset categories required for full GRAS implementation. Engagement with the government is an essential first step in In some cases, detailed data dictionaries for public and non- GRAS implementation. Government agencies, in particular public datasets were provided as input for the data maturity those responsible for anti-corruption, procurement, law assessment, which added to the robustness of the analysis. enforcement and oversight, as well as public finances, are the Indeed, it is important to highlight that, even though the obvious clients for a governance risk assessment tool. GRAS Brazilian GRAS pilot benefitted from open, fully public and relies on public data collected and managed by governments, machine-readable datasets, this is not necessarily required for their buy-in is essential in securing data access and for GRAS. Where open data is less advanced or data protection efforts to improve data disclosure as part of a transparency regulations preclude individual-level data from being agenda. Collaboration with central governments is a preferred published, for instance, agreements with the responsible strategy, in the sense that an investment in GRAS nationally agencies to access data from internal, non-public government would likely set a risk-assessment infrastructure that can be datasets can deliver the data infrastructure needed for GRAS. either directly used by or at least more easily extended to sub-national levels. Nevertheless, as the Brazilian experience 4.2. Data Maturity Assessment demonstrates, GRAS can also bring great added-value if initially implemented in sub-national jurisdictions. The data maturity assessment provides a detailed overview of the data infrastructure for GRAS implementation. Table 9 In feasibility assessment exercises undertaken by the World below lists the dataset categories required for each of the 23 Bank in Latin America, collaboration with oversight and risk patterns that integrate the GRAS framework. control authorities, as well as procurement agencies, has > > > T A B L E 9 - Dataset Categories Required Per Risk Pattern Risk group Risk pattern Public Employ- Corpo- Electo- Bla- Socio- Cri- Inte- procu- ment rate & ral data cklists -eco- minal grity rement rela- sha- nomic records decla- tionships rehol- data rations der data 1. Procure- 1.1. Non-competitive pro- X ment cycle cesses 1.2. Non-competitive ten- X der results 1.3. Contract implementa- X tion biases 2. Collu- 2.1. Top loser X sion/ Bid-ri- 2.2. Fixed difference bids X gging 2.3. Bid variance biases X 2.4. Unusual contract X value 2.5. High price X 2.6. Require Public Pro- X X curement 2.7. Corporate Data X X 2.8. Require PP, Employ- X X ment and Corporate Data CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 32 Risk group Risk pattern Public Employ- Corpo- Electo- Bla- Socio- Cri- Inte- procu- ment rate & ral data cklists -eco- minal grity rement rela- sha- nomic records decla- tionships rehol- data rations der data 3. Supplier 3.1. Unusual size X X X characte- 3.2. Unusual profitability X X ristics 3.3. Broad scope of activ- X X ities 3.4. Young supplier X X 3.5. Non-registered sup- X X plier 3.6. Sanctions X X X 3.7. Shareholder with low X X X X socio-economic status 3.8. Shareholder/ legal X X X representative with crimi- nal record 3.9. Tax haven registration X X 4. Political 4.1. Political finance X X X X connec- 4.2. Personal connections X X X X X tions to politicians 4.3. Personal connections X X X X to bureaucrats The data maturity assessment entails an initial mapping of of data quality, accuracy and completeness in each of the relevant data sources for each of the 8 dataset categories essential fields for the calculation of GRAS specific risk required in the GRAS framework, comprehensive identification indicators. The assessment entails four steps (see table 10 of individual data fields in each dataset, and an assessment below). > > > T A B L E 1 0 - Country Data Maturity Assessment Process Assessment steps Objectives Relevant questions A. Consultation with Initial mapping of relevant - Does the information required20 exist as a structured, disag- government agencies data sources and collection gregated dataset? of basic information on each - Is the dataset publicly accessible? Is it available as open source/dataset data (i.e. in a machine-readable format)? In which formats is it available? Where can it be accessed? How often is it upda- ted? - What is the legal framework that establishes its publicity? - What agency is responsible for collecting/managing the dataset? - What is the jurisdiction coverage (national/ subnational)? What is the time coverage? - Is there a published data dictionary? Where can it be acces- sed? If not, can one be provided for the assessment? CONTINUED 20. A list of necessary data fields for GRAS should be provided as a reference. This is available in Appendix I. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 33 Assessment steps Objectives Relevant questions B. Review of available Complementing information - Are there other relevant sources for the respective dataset data sources through provided in the consultation category? (in which case the same basic information obtained desk research (filling potential gaps) and from governments in the previous step is collected during the identifying additional/alternati- research) ve sources - What is the level of observation in the dataset? What is its scope (e.g. types of entities/individuals covered)? - Are there any apparent issues with this data source that can be already identified at this stage (e.g. lack of proper unique identifying data, limitations in scope)? C. Detailed variable-le- Verifying whether identified - Are essential data fields covered by the dataset? If not, are vel mapping of identi- datasets contain the data there other relevant fields that could function as proxies? fied sources/datasets fields required for GRAS - Which individual GRAS indicators can be fully/partially im- indicators and assessing the plemented? feasibility of each individual - Is additional information required to complete the assess- indicator ment? D. Individual dataset Assessing the accuracy, com- - Are key data fields correctly filled, complying with data for- assessment prehensiveness and comple- mat requirements? teness of the data available - Is missing data a widespread problem? and identifying relevant gaps - Are recorded values in the datasets consistent with actual transactions and legally binding documents (e.g. contract values in the dataset correspond to actual contract values in signed contracts)? - Are there important data scope limitations such as a class of procurement transactions excluded from government registers (e.g. tenders of state owned enterprises)? First, the assessment starts with a consultation with the Third, each of the datasets identified is then examined to relevant government agencies using a brief questionnaire, confirm the availability of the minimum set of data fields where they should indicate the main data sources in each necessary to produce the 60 indicators specified under the category and provide basic information about each source/ GRAS framework. A reference list of essential data fields is dataset, as covered by the relevant questions listed in the provided in Appendix I. For publicly accessible datasets, table above. Specific data dictionaries can be provided by this step can rely on the analysis of corresponding data the collaborating agencies, in particular those that are not dictionaries where available, or the dataset itself in some publicly available.21 This is an extremely valuable input for the cases, and, when applicable, also on the verification of subsequent steps in the assessment. government public search platforms for the relevant sources, with example searches to illustrate which data is retrieved.22 Second, the information provided is reviewed and validated The analysis is somewhat more challenging for the datasets through desk research. Complementary information is that are not publicly accessible and for which no data collected, and potential additional or alternative sources dictionary can be obtained. In those cases, datasets are are identified. Information on the newly identified sources is in most cases populated by information provided through documented following the same questions that oriented the registration procedures, i.e. channels by which government previous stage. After this initial mapping of the GRAS data agencies collect information from individuals or organizations, sources, a broad picture emerges in terms of relevant data mainly through public digital services, a useful strategy can be environment characteristics, potential gaps at the dataset to conduct the assessment based on documentation on those level and access limitations. data collection processes. For instance, registration forms 21. In a GRAS feasibility assessment done in Peru, for instance, government agencies collaborating with the researchers conducting the study provided several data dictio- naries, even for data that were not publicly accessible. This provided for a more robust and reliable detailed assessment of each dataset later on. 22. In feasibility assessments conducted in Latin American countries, it was observed that some datasets were made available as open and downloadable files, but those versions included only partial data when contrasted to the information obtained in example searches on the corresponding public search platforms. Therefore, it is an important step to check if open data files indeed cover the full underlying datasets. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 34 may be accessed to directly observe the type of information to have broader applicability. The Brazil GRAS pilot relies that is collected. Also official tutorial material (e.g. manuals, not only on established public procurement corruption risk videos) published by the respective agencies can be used as indicators, but also on risk patterns that relate to companies, a reference to identify individual variables contained in those political connections and collusive practices. Such broad datasets. Once the specific data fields have been mapped, range of indicators allow for triangulation and flexibly adapting the feasibility of individual risk patterns and indicators can be GRAS to new contexts. assessed for a preliminary overview of the potential scope of GRAS implementation.23 GRAS will need to be tailored to the particular context where it will operate, incorporating or prioritizing elements that might Fourth, an in-depth evaluation of individual datasets is be relevant for detecting fraud, corruption and collusion risks important to assess whether requirements in terms of data in that context. Indicators may have to be adjusted to better accuracy and completeness are fulfilled for GRAS to operate fit the type of data available,24 or even to incorporate specific as designed (for in-depth data assessments with examples risks relevant for the new context that might not be adequately see: Horn et al, 2021, chapter 2; Cingolani et al, 2016; Czibik covered in the framework currently. This may require the et al, 2015). Even if required data fields exist in the mapped design of more appropriate or additional indicators. Indicator datasets, a detailed examination of their actual content is thresholds and risk value ranges will have to be defined to needed to establish: whether missing rates on key variables reflect national and local specificities of public procurement are sufficiently low or too high for adequate analysis; whether regulations, markets and corruption strategies (Fazekas and common identifiers for individuals and organizations such Kocsis, 2020). Such parametrization and tailoring to context is as company registry IDs are present and follow the required indispensable for prediction accuracy of GRAS. This can be format in order to link different datasets; and whether relevant achieved by using proven positive and negative cases (e.g. fields have unusual or extreme distributions decreasing their machine learning) or exploiting expected correlations among value for risk flagging (e.g. whether a categorical variable established risk factors (Adam et al, 2022). takes one of the possible categories in 99% of the cases). It is expected and highly likely that all the administrative datasets Governments may already have data analytics initiatives for will suffer from some or all of these problems. However, some governance risk assessments in place. Understanding how of them can be remedied as part of a GRAS implementation these function and what synergies may exist with GRAS is plan, for example filling in missing fields from related fields (e.g. an important step in planning GRAS implementation to avoid if the buyer city is available but buyer state is missing, drawing duplication of efforts and to best complement the approaches on a city-state correspondence table from the statistical office already in use. This complementary implementation of GRAS enables reliably filling in blanks). can entail the extension of existing tools to cover new risk patterns or incorporating new sources of data, or even covering 4.3. Validation and Tailoring of GRAS Framework new jurisdictions (e.g. sub-national entities). Pre-existing risk to National Specificities assessment systems are often intended for exclusive internal use by government agencies. Complementary mapping will Even though the GRAS pilot was conceived to work for the likely have to be supported by interviews with public servants specific data environment of Brazil, the GRAS framework in oversight functions, analysis of recent audits and related offers a robust set of established risk indicators that are likely reports already produced by the relevant authorities. 23. Appendix II includes a reference table listing the specific fields required for each individual indicator. 24. One example observed during the preparation of feasibility assessments in selected Latin American countries refers to differences in the socio-economic data available there, in contrast to the type of dataset employed under this category in Brazil. The Brazilian GRAS makes use of individual-level data on actual beneficiaries of a number of focalized cash transfer programs, which indicate exact time periods and amounts received by each beneficiary. This type of data was not found to be easily accessible in the countries examined, but an alternative type of data could be identified that could fulfill the same purpose, namely household classification datasets which indicate specific households, their individual members and a household-level poverty classification. Consequently, the implementation of GRAS indicators originally employing variables from beneficiary datasets would need to be somewhat redefined in order to better relate to how these alternative data sources are structured. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 35 4.4. Feasibility Report and Recommendations The feasibility report represents only a first step towards the elaboration of a detailed implementation plan, which must The feasibility assessment concludes with the production be developed as a second step to address concrete and of a feasibility draft report directed at the collaborating context-specific considerations, as well as more actionable government agencies (or for public disclosure as well). The recommendations for the tailoring of GRAS to the particular assessment includes a context-validated indicator framework context. It could foresee gradual implementation of the system for GRAS implementation, based also on the stage of data according to the scope of feasibility and a proposal for a pilot maturity observed as well as the scope of implementation stage, indicating possible extensions depending on further that it enables. Most importantly, the assessment should developments in the underlying data environment. Indeed, discuss specific, detailed data limitations, especially those depending on the level of engagement and commitment of the referring to the most essential dataset categories required for respective government agencies regarding data transparency, GRAS operation, i.e. public procurement and corporate and positive changes in public data availability may take place shareholder data. It should also refer to other context-specific quite dynamically, requiring the initial feasibility assessment potential difficulties for GRAS implementation, such as data to be revisited occasionally to observe whether conditions for protection regulations, whenever applicable. This should GRAS implementation have substantially changed. be accompanied by recommendations for improvement of collection,25 management and disclosure of datasets. A A GRAS implementation plan should also consider capacity GRAS feasibility assessment may also be included as part of building needs for potential users of the system, depending on a broader agenda to promote transparency - in particular in the the features and the scope of indicators actually implemented. area of anti-corruption or public procurement and corporate This should thus include the necessary activities to ensure that data - and digitization of public services. Report should be GRAS users are sufficiently trained on GRAS functionalities validated with input from government agencies involved, as and data analytics more broadly. well as from local data, public procurement and corruption experts, helping to fill possible gaps in the analysis. 25. In general, GRAS requires the existence of structured datasets generated by electronic data collection systems. In their absence, implementing important parts of the framework may become partly or fully unfeasible. One such example was observed in feasibility assessments conducted for Panama and Ecuador, where data on political finance was available only through financial reports submitted in paper, which created difficulties for the application of risk patterns associated with political con- nections. In those contexts, one important recommendation referred to the introduction of an electronic system for the submission of electoral and party financial reports. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 36 5. >>> Conclusions and Potential for Extensions This report presents a comprehensive fraud, corruption, and collusion risk assessment meth- odology and tool, the Governance Risk Assessment System (GRAS), and its implementation in Brazil. GRAS is a data-driven tool which can improve the detection accuracy of fraud, corrup- tion and collusion, thereby increasing the efficiency and effectiveness of audits and investiga- tions. GRAS presents 60 different red flags, linked to 23 broad risk patterns falling into 4 main groups of risks behaviors. GRAS, like other data-driven risk assessment tools, can only lead to tangible results if it is adequately integrated into an effective, broader anti-corruption framework (Fazekas et al, 2019). Investigations and audits can be improved by data-driven tools only if the responsible institutions are well-resourced, meritocratic and independent and if they are able to collaborate bringing even complex cases to courts. GRAS can be readily implemented in countries with a high degree of data maturity. However, extensive data requirements need not restrict dissemination and implementation of the GRAS in countries with lesser data maturity or a data landscape with different strengths. Based on the current status of public data availability and governance in the world,26 no country and govern- ment is likely to fulfill all the requirements for GRAS. Even in Brazil, with an especially favorable data environment, GRAS implementation is not without its limitations (for specific areas of fur- ther development see below). The data environment assessment can provide an initial roadmap for agencies on what steps can be taken to implement GRAS despite data limitations. Efforts to promote GRAS can and should consider a minimum viable version that can be implemented wherever governments offer the necessary policy support. The core functionality of GRAS can then be gradually expanded and refined. For example, GRAS implementation could be coupled with a transparency agenda, profiting from efforts to improve procurement data quality and pub- lication across the globe. One third of GRAS indicators may be implemented with public procure- ment data alone. Following improvements in the availability of additional datasets and required data fields, the system can be gradually extended. With time, a refined methodology might be needed as well, as fraud, corruption and collusion strategies are likely to evolve in response to enforcement efforts, bringing the need for adaptation and improvement of the framework. The initial adoption and continuous improvements to GRAS should also support the movement to- wards open government data to support risk assessment, the analysis of government spending efficiency and accountability more broadly. 26. https://globaldatabarometer.org/ EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 37 Further Enhancing GRAS While GRAS builds on and encompasses the results of a wide data across borders could greatly enhance investigation and rich law enforcement practice and academic literature, it and audit effectiveness. There are existing platforms, can be improved in future iterations. For example, user groups such the ProACT tool (Box 1), which build on a wide range beyond investigators and auditors can be served. Potential of country datasets and allow for risk assessment across improvements that are being considered are discussed below. different jurisdictions. Adding further countries to GRAS or allowing it to connect to existing cross-country tools will 1. Additional Data. As corruption does not stop at allow users to identify risk patterns and flag cases more administrative and national borders, more comprehensive comprehensively. > > > B OX 1 : P R O C U R E M E N T A N T I - C O R R U P T I O N A N D T R A N S P A R E N C Y P L AT F O R M ( P R O A C T ) ProACT is based on open data from national e-procurement systems from 46 countries and open data on World Bank and Inter-American Development Bank financed contracts for over 100 countries. The data has been collected from of- ficial government procurement portals and standardized into a single data structure by the Government Transparency Institute.27 ProACT allows users to search and analyze specific contracts, tenders, buyers, suppliers and markets. It also offers country-level statistics, including competition, transparency and integrity indicators, which can be further disag- gregated by sector, procurement method, and contract value range.28 ProACT has been developed by the World Bank in collaboration with the Government Transparency Institute, building on European experiences with corruption risk portals such as www.opentender.eu. ProACT is intended for a wide range of users, including procurement officers in national procuring entities; procure- ment specialists and analysts in MDBs and national Public Procurement Authorities; NGOs that work on procurement, integrity, transparency, and open government; and researchers and academia. ProACT allows procurement officers in national contracting agencies to access information from public procurement records outside their own country. This helps them track firms and analyze international market conditions for specific goods, works and services, and can be a complementary tool to GRAS to gain further insights into a specific provider beyond the national market. 27. For more details and recent updates on this dataset see: https://www.govtransparency.eu/gtis-global-government-contracts-database/ 28. For the detailed methodology, see: https://www.procurementintegrity.org/assets/about/ProACT_methods_paper_20220809_final.pdf EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 38 2. Indicator extensions. Corruption schemes constantly percentage is cause for concern, 10%, 30%, 80%, etc. evolve and proxying all widely present corrupt behaviors Such decisions are hard as they fundamentally depend is essential for a comprehensive and reliable tool. on the context and prevailing market or sectoral norms. Among areas where potential new indicators can be Such parameters and interpretation questions can be brought into GRAS, the contract implementation phase addressed by looking at relationships among indicators is critical. Contact implementation is harder to monitor, and contextual variables. For example, when non- less standardized and there are fewer or no competitors competitive procedures are related to overpricing and who watch over the fairness of the process. While GRAS market-specific procedure type distributions, it is possible has indicators on cost overruns and delays, a range of to identify which procedure types are non-open and which indicators could be deployed targeting payments such extent of non-open procedure type use is likely to be as unusual timing and values of payments. The Brazilian risky.29 GRAS already provides a good example of how this can be explored based on e-invoice data. More challenging, 4. Methodology improvements. Allowing users to look but essential, is to incorporate data on quality and across a wide range of individual factors lends them a great quantity of eventually delivered goods and services. deal of flexibility, but forgoes opportunities to combine A host of corrupt schemes look impeccable on paper, individual indicators into a more accurate measurement. but the resulting roads are barely usable, the websites For example, a recent study applying machine learning crash, or the services rendered are irrelevant. Another methods to 78 proven cases of collusion in 7 European extension already implemented in the Brazilian GRAS countries, achieves 80-90% prediction accuracy by is additional context-specific risk patterns regarding combining 5 indicators of collusion. Many of the accurately supplier characteristics: based on fine-grained local identified cases do not score particularly high on individual socioeconomic data, suppliers registered in very humble risk dimensions, only by combining ‘weak signals’ into a locations, especially while having at the same time no comprehensive predictive model does accuracy increase registered employees, can be identified as potential ghost (Fazekas et al, 2023). companies. This has been found to be a very relevant risk pattern already in one of the pilot States where GRAS is 5. Broadening the pool of users. GRAS is well-suited to operational, with 88% of the total high-risk contract value the needs and activities of investigators and auditors in affected by this red flag. Such types of indicators tailored the fraud, corruption, and collusion areas, but there are to specific contexts and harnessing other available additional use cases. First, the preventive use of GRAS data sources in a given jurisdiction are an important could support a range of monitoring bodies. For example, development beyond the initial implementation of GRAS’ public procurement or competition authorities could standard framework. identify markets and areas of high risk and implement broader preventive interventions such as reviewing 3. Indicator design. The interpretability of indicators procurement policies and guidance documents of the can be further improved based on state-of-the-art data relevant public buyers. Improving buyers’ procurement science. The current GRAS framework rests on a wide skills and institutions could lower risks across a wide set of range of indicators that allocate the task of interpretation contracts. Second, GRAS can support policy assessment and parametrization to the user. For example, once a and reform by identifying policies which allow for high-risk user sees the indicator of contract share through non- activities such as too high contract value thresholds for competitive procedures (indicator 1.1.1), which is a mandatory competitive tendering or auction design prone continuous indicator; he or she has to decide which to collusion. 29. The GRAS team is working to incorporate ChatGPT to support users in identifying key red-flags. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 39 >>> References Adam, Isabelle, Fazekas, Mihály, Kazmina, Yuliia, Teremy, Zsombor, Tóth, Bence, Villamil, Isa- bela Rosario; & Wachs, Johannes (2022) Public procurement cartels: A systematic testing of old and new screens. GTI-WP/2022:01, Budapest: Government Transparency Institute. Alexeeva, Victoria; Padam, Gouthami; Queiroz, Cesar. 2008. Monitoring Road Works Contracts and Unit Costs for Enhanced Governance in Sub-Saharan Africa. Transport paper series; no. TP-21. World Bank, Washington, DC. Bajpai, Rajni; Myers, C. Bernard. 2020. Enhancing Government Effectiveness and Transpar- ency : The Fight Against Corruption. Washington, D.C.: World Bank Group. Berger, A., & Hill, T. P. (2015). An introduction to Benford’s law. Princeton University Press. Boas, Taylor C., F. Daniel Hidalgo, and Neal P. Richardson. 2014. The spoils of victory: Cam- paign donations and government contracts in Brazil. Journal of Politics 76 (2): 415–29. Bromberg, Daniel. 2014. Can vendors buy influence? The relationship between campaign contri- butions and government contracts. International Journal of Public Administration 37 (9): 556–67. Cingolani, Luciana; Fazekas, Mihály; Kukutschka, Roberto Martínez B.; and Tóth, Bence (2016) Towards a comprehensive mapping of information on public procurement tendering and its ac- tors across Europe. Cambridge: University of Cambridge. Collier, Paul; Martina Kirchberger, and Måns Söderbom, The Cost of Road Infrastructure in Low- and Middle-Income Countries, The World Bank Economic Review, Volume 30, Issue 3, October 2016, Pages 522–548, https://doi.org/10.1093/wber/lhv037 Czibik, Ágnes; Tóth Bence; and Fazekas, Mihály (2015) How to Construct a Public Procurement Database from Administrative Records? With examples from the Hungarian public procurement system of 2009-2012. GTI-R/2015:02, Budapest: Government Transparency Institute. European Commission (2022) Cohesion in Europe towards 2050. Eighth report on economic, social and territorial cohesion. Luxembourg: Publications Office of the European Union. Fazekas, M. & Márk, L. (2017). Objective corruption risk indicators using donor project and contract data. GTI-R/2017:02, Budapest, Government Transparency Institute, September 2017. Fazekas M, Tóth IJ and King LP (2016) An Objective Corruption Risk Index Using Public Pro- curement Data. European Journal on Criminal Policy and Research 22(3): 369–397. DOI: 10.1007/s10610-016-9308-z. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 40 Fazekas, Mihály and Tóth, Bence, (2016), Assessing the potential for detecting collusion in Swedish public procurement. Uppdragsforskningsrap. 2016:3, Swedish Competition Authority, Stockholm. Fazekas, Mihály, Luciana Cingolani, & Bence Tóth (2018), Innovations in Objectively Measuring Corruption in Public Procurement. In Helmut K. Anheier, Matthias Haber, and Mark A. Kayser (eds.) Governance Indicators. Approaches, Progress, Promise. Ch. 7. Oxford University Press, Oxford. Fazekas, Mihály, Ugale, Gavin, and Zhao, Angelina, (2019) Analytics for Integrity. Data-Driven Approaches for Enhancing Corruption and Fraud Risk Assessments. OECD, Paris. Fazekas, Mihály, and Kocsis, Gábor, (2020), Uncovering High-Level Corruption: Cross-National Corruption Proxies Using Public Procurement Data. British Journal of Political Science, 50(1). Fazekas, Mihály, and Blum, Jurgen Rene. (2021), Improving Public Procurement Outcomes: Review of Tools and the State of the Evidence Base. Policy Research Working Paper;No. 9690. World Bank, Washington, DC. Fazekas, Mihály; Ferrali, Romain & Wachs, Johannes (2022) Agency independence, campaign contributions, and favouritism in US federal government contracting, Journal of Public Adminis- tration Research and Theory, available online. Fazekas, M., Tóth, B. and Wachs, J. (2023). Public procurement cartels: A large-sample test- ing of screens using machine learning. GTI-WP/2023:02, Budapest: Government Transparency Institute. Goldman, Eitan, Jörg Rocholl, and Jongil So. 2013. Politically connected boards of directors and the allocation of procurement contracts. Review of Finance 17 (5): 1617–48. Horn, Peter; Czibik, Ágnes; Fazekas, Mihály; and Tóth, B. (2021): Analyzing Public Procurement Risks: Training manual. Budapest: R2G4P / Government Transparency Institute. Hyytinen, A., Lundberg, S. and Toivanen, O. (2018), Design of public procurement auctions: evidence from cleaning contracts. The RAND Journal of Economics, 49: 398-426. Mineva, Daniela; Fazekas, Mihály; Poltoratskaya, Viktoriia; and Tsabala, Kristina (2023) Rolling Back State Capture in Southeast Europe. Implementing Effective Instruments for Asset Declara- tion and Politically Exposed Companies. Center for the Study of Democracy, Sofia. OECD (2022). Open Government Review of Brazil: Towards an Integrated Open Government Agenda, OECD Public Governance Reviews, OECD Publishing, Paris. OECD (2016) Preventing Corruption in Public Procurement. Available at: https://www.oecd.org/ gov/ethics/Corruption-Public-Procurement-Brochure.pdf (accessed 15 November 2021). OECD (2017) Preventing Policy Capture. Integrity in Public Decision Making. OECD Publishing, Paris. OECD (2019) OECD Integrity Review of Mexico City: Upgrading the Local Anti-Corruption Sys- tem. OECD Public Governance Reviews. OECD Publishing, Paris. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 41 Santos, FB and Souza, KR (2016). Como combater a corrupção em licitações: detecção e pre- venção de fraudes, Fórum, Belo Horizonte. Tax Justice Network (2022) Financial Secrecy Index. 2022 Methodology. Tax Justice Network. See: https://fsi.taxjustice.net/fsi2022/methodology.pdf (accessed on the 16th of March 2023) Velasco RB (2019) Identifying Corruption Risk in Brazil: New Measures for Effective Oversight. In: Rotberg RI (ed.) Corruption in Latin America. Cham: Springer International Publishing, pp. 57–91. Velasco RB, Carpanese I, Interian R, et al. (2020). A decision support system for fraud detection in public procurement. International Transactions in Operational Research 28(1): 27–47. World Bank (2009). Fraud and Corruption: Awareness Handbook. Washington DC: World Bank. https://stock.adobe.com/pt/images/hacker-attack-maintenance-concept-and-hacking-cyber- crime-cyber-security-user-is-using-smartphone-with-warning-triangle-for-error-notification/620- 026942 https://stock.adobe.com/pt/images/entrepreneurs-small-business-sme-independent-men-work- at-home-use-smartphones-and-laptops-for-commercial-checking-online-marketing-packing- boxes-sme-sellers-concept-e-commerce-team-online-sales/621503743 https://stock.adobe.com/pt/images/rio-de-janeiro-june-21-2017-the-selaron-steps-in-the-histo- ric-center-of-rio-de-janeiro-brazil/191264522 https://stock.adobe.com/pt/images/perspective-view-of-stock-market-growth-business-invest- ing-and-data-concept-with-digital-financial-chart-graphs-diagrams-and-indicators-on-dark-blue- blurry-background/610683944 https://stock.adobe.com/pt/images/human-multicolored-iris-of-the-eye-animation-concept-rain- bow-lines-after-a-flash-scatter-out-of-a-bright-binary-circle-and-forming-volumetric-a-human- eye-iris-and-pupil-3d-rendering-background-4k/612500470 https://stock.adobe.com/pt/images/vector-illustrations-of-futuristic-digital-tech-architecture-ab- stract-blue-hi-tech-theme-for-davertising-or-game-artwork-futuristic-concept/600939016 https://stock.adobe.com/pt/images/un-analyste-data-travaillant-sur-des-jeux-de-donnees- depuis-sont-ordinateur-portable/634850584 https://stock.adobe.com/pt/images/modern-neon-cyberpunk-open-space-office-interior-blurred- with-information-technology-overlay-corporate-strategy-for-finance-operations-marketing-gen - erative-ai-technology/628395923 https://stock.adobe.com/pt/images/long-exposure-shot-of-crowd-of-business-people-walking- in-bright-office-lobby-fast-moving-with-blurry-generative-ai/619509578 EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 42 >>> Appendix I. GRAS Data Field Requirements EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 43 Table 11 below displays in detail the relevant data fields for fields contained in the relevant datasets. each dataset category considered in the GRAS framework. Those fields marked in bold are minimum requirements for the Specific numerical identifiers for individuals, companies computation of GRAS indicators, and the other fields listed and agencies adopted should be common to the different refer to additional information useful for data validation or to datasets to allow cross-referencing for the risk assessment. be displayed in the user interface as reference for system Which identifiers are more widely used for registration across users. Depending on specificities of the data sources mapped different databases will also be context-specific, as they vary as part of the feasibility assessment, these may be adjusted from country to country. or complemented to better reflect corresponding or equivalent > > > T A B L E 1 1 - List of Data Fields Required for GRAS in each Dataset Category Dataset category Dataset field 1. Public procurement AgencyID AgencyName AgencyLocation AgencyGovLevel ID Process Date NumberBids ProcurementMethod ItemNumber BidNumber BidValue BidDate AwardBid FirmName FirmID ContractValue ContractDate ContractObject ContractAmendValue ContractAmendDate ContractAmendObject ContractID PublicationDate BidDeadline DeliveryDate(estimated) ImplementationDate(final) ContractDeliveryDelay ContractEndDate ProductCode FirmLocation ImplementationLocation EstimatedPrice TenderFinalPrice CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 44 Dataset category Dataset field 2. Employment rela- EmployerID tionships EmployerName WorkerID WorkerName Date of birth Admission date Termination date Position Remuneration 3. Corporate and sha- FirmID reholder data FirmName EntityType FirmCountry FirmAddress FirmPhonenumber FirmEmail FirmActivityCode FirmActivity Profit Turnover FirmConstitutionDate Year ShareholderID ShareholderName ShareholderEntryDate ShareholderExitDate ShareholderCountry LegalRepresentativeID LegalRepresentativeName AccountantID AccountantName EconomicGroup 4. Electoral data CandidateID CandidateName ElectionDisputed OfficeDisputed Elected PartyName ElectionJurisdiction AffiliationStart AffiliationEnd PartyRepresentationStart PartyRepresentationEnd CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 45 Dataset category Dataset field 4. Electoral data PartyRepresentationPosition CampaignDonorID CampaignDonorName CampaignDonationValue DonorLocation CampaignSupplierID CampaignSupplierName ExpenseValue SupplierLocation Year 5. Blacklists SanctionedID SanctionedName Sanction_date(sta) Sanction_date(end) SanctioningOrgID SanctioningOrgName 6. Socio-economic data BeneficiaryID BeneficiaryName BeneficiaryLocation BenefitDate(first) BenefitDate(last) BenefitValueTotal CashtransprogramType HouseholdID HouseholdLocation HouseholdClassification ClassificationValidStart ClassificationValidEnd ClassificationDate HouseholdmemberID HouseholdmemberName 7. Criminal records PersonID PersonName Crime SentenceDate 8. Asset and interest PersonID declarations Name AgencyID AgencyName Position Year CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 46 Dataset category Dataset field 8. Asset and interest ShareholderCompanyID declarations ShareholderCompanyName RelativeID RelativeName RelativeWorkplace FamilyRelation EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 47 >>> Appendix II. Feasibility Assessment at the Indicator Level EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 48 Table 12 below indicates specific data fields that are required potential adjustments in the data fields identified as relevant in for a feasibility assessment of individual GRAS indicators, as each context and in the indicator framework, should context- they are needed to compute the variables employed in the specific opportunities or constraints require them. related risk assessment. Again, these should also reflect > > > T A B L E 1 2 - Required Data Fields Per Red Flag/Indicator Risk group Risk pattern Red flag/ indicator Dataset category Required data fields 1. Procure- 1.1. Non- 1.1.1. Contract share through 1. Public procure- ID Process; ProcurementMethod; ment cycle -competitive non-competitive procedures ment FirmID; ContractID; ContractValue; processes ContractDate 1.1.2. Contract share after call 1. Public procure- ID Process; FirmID; ContractID; for tenders absent ment ContractValue; ContractDate; Pu- blicationDate; BidDeadline 1.1.3. Contract share after shor- 1. Public procure- ID Process; FirmID; ContractID; tened advertisement period ment ContractValue; ContractDate; Pu- blicationDate; BidDeadline 1.2. Non- 1.2.1. Contract share as single 1. Public procure- ID Process; FirmID; ContractID; -competiti- bidder ment ContractValue; ContractDate; ve tender NumberBids results 1.2.2. High winning rate 1. Public procure- ID Process; FirmID; BidNumber; ment BidValue; BidDate; AwardBid 1.2.3. Contract share in buyer's 1. Public procure- ID Process; FirmID; ContractID; portfolio ment ContractValue; ContractDate; AgencyID 1.3. Con- 1.3.1. Contract share with size- 1. Public procure- ID Process; FirmID; ContractID; tract imple- able cost overruns ment ContractValue; ContractDate; Con- mentation tractAmendValue biases 1.3.2. Contract share with size- 1. Public procure- ID Process; FirmID; ContractID; able delivery delay ment ContractValue; ContractDate; DeliveryDate(estimated); Imple- mentationDate(final); ContractDeli- veryDelay; ContractEndDate 2. Collusion 2.1. Top 2.1.1. Low winning rate 1. Public procure- ID Process; FirmID; BidNumber; loser ment BidValue; BidDate; AwardBid; ProductCode; FirmLocation; Imple- mentationLocation 2.1.2. Number of competitors 1. Public procure- ID Process; FirmID; BidNumber; ment BidDate; AwardBid; ProductCode; FirmLocation; ImplementationLo- cation 2.1.3. Number of wins against 1. Public procure- ID Process; FirmID; BidNumber; Top Losers ment BidDate; AwardBid; ProductCode; FirmLocation; ImplementationLo- cation 2.1.4. Winning rate against Top 1. Public procure- ID Process; FirmID; BidNumber; Losers ment BidDate; AwardBid; ProductCode; FirmLocation; ImplementationLo- cation 2.1.5. Number of Top Loser 1. Public procure- ID Process; FirmID; BidNumber; competitors ment BidDate; AwardBid; ProductCode; FirmLocation; ImplementationLo- cation CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 49 Risk group Risk pattern Red flag/ indicator Dataset category Required data fields 2. Collusion 2.2. Fixed 2.2.1. Number of colluding part- 1. Public procure- ID Process; FirmID; BidNumber; difference ners with fixed difference bids ment BidValue; BidDate; AwardBid; bids ProductCode; FirmLocation; Imple- mentationLocation 2.2.2. Number of bids with fixed 1. Public procure- ID Process; FirmID; BidNumber; difference bids ment BidValue; BidDate; AwardBid; ProductCode; FirmLocation; Imple- mentationLocation 2.2.3. Frequency of fixed diffe- 1. Public procure- ID Process; FirmID; BidNumber; rence bids ment BidValue; BidDate; AwardBid; ProductCode; FirmLocation; Imple- mentationLocation 2.3. Bid 2.3.1. Bid share in low variance 1. Public procure- ID Process; FirmID; BidNumber; variance tenders ment BidValue; BidDate; AwardBid; biases ProductCode; FirmLocation; Imple- mentationLocation 2.3.2. Bid share in high relative 1. Public procure- ID Process; FirmID; BidNumber; bid distance tenders ment BidValue; BidDate; AwardBid; ProductCode; FirmLocation; Imple- mentationLocation 2.4. Unusu- 2.4.1. Contract share with con- 1. Public procure- ID Process; FirmID; ContractID; al contract tract value violating Benford’s ment ContractValue; ContractDate; value Law ProductCode; FirmLocation; Imple- mentationLocation 2.5. High 2.5.1. Contract share with very 1. Public procure- ID Process; FirmID; EstimatedPri- price high relative contract value ment ce; TenderFinalPrice; BidNumber; BidValue; BidDate; AwardBid; Con- tractID; ContractValue; Contract- Date; ProductCode; FirmLocation; ImplementationLocation 2.6. Com- 2.6.1. Number of competitors 1. Public procure- ID Process; FirmID; BidNumber; mon regis- sharing registration data ment ProductCode; FirmLocation; Imple- tration data mentationLocation 3. Corporate and FirmID; FirmAddress; FirmPho- shareholder data nenumber; FirmEmail; LegalRe- presentativeID; AccountantID 2.6.2. Number of tenders with 1. Public procure- ID Process; FirmID; BidNumber; bidders sharing registration data ment ProductCode; FirmLocation; Imple- mentationLocation 3. Corporate and FirmID; FirmAddress; FirmPho- shareholder data nenumber; FirmEmail; LegalRe- presentativeID; AccountantID 2.6.3. Share of contracts won 1. Public procure- ID Process; FirmID; BidNumber; against competitors sharing ment ContractID; ContractValue; Con- registration data tractDate; ProductCode; FirmLoca- tion; ImplementationLocation 3. Corporate and FirmID; FirmAddress; FirmPho- shareholder data nenumber; FirmEmail; LegalRe- presentativeID; AccountantID 2.7. Com- 2.7.1. Number of competitors 1. Public procure- ID Process; FirmID; BidNumber; mon sha- with common shareholde ment ProductCode; FirmLocation; Imple- reholder mentationLocation 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 50 Risk group Risk pattern Red flag/ indicator Dataset category Required data fields 2. Collusion 2.7. Com- 2.7.1. Number of competitors 1. Public procure- ID Process; FirmID; BidNumber; mon sha- with common shareholder ment ProductCode; FirmLocation; Imple- reholder mentationLocation 3. Corporate and FirmID; ShareholderID; Sharehol- shareholder data derEntryDate; ShareholderExitDa- te 2.7.2. Number of tenders with 1. Public procure- ID Process; FirmID; BidNumber; competitors sharing a share- ment ProductCode; FirmLocation; Imple- holder mentationLocation 3. Corporate and FirmID; ShareholderID; Sharehol- shareholder data derEntryDate; ShareholderExitDa- te 2.7.3. Share of contracts won 1. Public procure- ID Process; FirmID; BidNumber; against competitors with com- ment ContractID; ContractValue; Con- mon shareholder tractDate; ProductCode; FirmLoca- tion; ImplementationLocation 3. Corporate and FirmID; ShareholderID; Sharehol- shareholder data derEntryDate; ShareholderExitDa- te 2.7.4. Number of competitors in 1. Public procure- ID Process; FirmID; BidNumber; the same corporate group ment ProductCode; FirmLocation; Imple- mentationLocation 3. Corporate and FirmID; EconomicGroup shareholder data 2.7.5. Number of tenders with 1. Public procure- ID Process; FirmID; BidNumber; competitors in the same corpo- ment ProductCode; FirmLocation; Imple- rate group mentationLocation 3. Corporate and FirmID; EconomicGroup shareholder data 2.7.6. Share of contracts won 1. Public procure- ID Process; FirmID; BidNumber; against competitors in the same ment ContractID; ContractValue; Con- corporate group tractDate; ProductCode; FirmLoca- tion; ImplementationLocation 3. Corporate and FirmID; EconomicGroup shareholder data 2.8. Com- 2.8.1. Number of competitors 1. Public procure- ID Process; FirmID; BidNumber; mon em- with common employee ment ProductCode; FirmLocation; Imple- ployee mentationLocation 2. Employment EmployerID; WorkerID relationships 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate 2.8.2. Number of tenders with 1. Public procure- ID Process; FirmID; BidNumber; competitors sharing an employ- ment ProductCode; FirmLocation; Imple- ee mentationLocation 2. Employment EmployerID; WorkerID relationships 3. Corporate and FirmID; ShareholderID; Sharehol- shareholder data derEntryDate; ShareholderExitDa- te CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 51 Risk group Risk pattern Red flag/ indicator Dataset category Required data fields 2. Collusion 2.8. Com- 2.8.3. Share of contracts won 1. Public procure- ID Process; FirmID; BidNumber; mon em- against competitors with com- ment ContractID; ContractValue; Con- ployee mon employee tractDate; ProductCode; FirmLoca- tion; ImplementationLocation 2. Employment EmployerID; WorkerID relationships 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 52 Risk group Risk pattern Red flag/ indicator Dataset category Required data fields 3. Supplier 3.6. Sanc- 3.6.4. Contracts while sanctio- 1. Public procure- ID Process; FirmID; ContractID; characteris- tions ned ment ContractValue; ContractDate tics 3. Corporate and FirmID; ShareholderID; Sharehol- shareholder data derEntryDate; ShareholderExitDa- te; LegalRepresentativeID 5. Blacklists SanctionedID; Sanction_date(sta); Sanction_date(end) 3.6.5. Sanction relative duration 1. Public procure- FirmID ment 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; LegalRepresentativeID 5. Blacklists SanctionedID; Sanction_date(sta); Sanction_date(end) 3.6.6. Period between incorpo- 1. Public procure- FirmID ration and 1st sanction ment 3. Corporate and FirmID; FirmConstitutionDate shareholder data 5. Blacklists SanctionedID; Sanction_date(sta) 3.7. Share- 3.7.1. Shareholder has low 1. Public procure- FirmID holder with socio-economic status ment low so- 2. Employment EmployerID; WorkerID; Position; cio-econom- relationships Remuneration; Admission date; ic status Termination date 3. Corporate and FirmID; ShareholderID; Sharehol- shareholder data derEntryDate; ShareholderExitDa- te; LegalRepresentativeID 6. Socioeconomic BeneficiaryID; HouseholdID; Clas- data sification; ClassificationValidStart; ClassificationValidEnd; House- holdmemberID; BenefitDate(first); BenefitDate(last) 3.7.2. Status duration 1. Public procure- FirmID ment 2. Employment EmployerID; WorkerID; Position; relationships Remuneration; Admission date; Termination date 3. Corporate and FirmID; ShareholderID; Sharehol- shareholder data derEntryDate; ShareholderExitDa- te; LegalRepresentativeID 6. Socioeconomic BeneficiaryID; HouseholdID; Clas- data sification; ClassificationDate; Clas- sificationValidStart; Classification- ValidEnd; HouseholdmemberID; BenefitDate(first); BenefitDate(last) 3.7.3. Time overlap between 1. Public procure- FirmID status and company ownership ment 2. Employment EmployerID; WorkerID; Position; relationships Remuneration; Admission date; Termination date 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; LegalRepresentativeID CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 53 Risk group Risk pattern Red flag/ indicator Dataset category Required data fields 3. Supplier 3.7. Share- 3.7.3. Time overlap between 6. Socioeconomic BeneficiaryID; HouseholdID; Clas- characteris- holder with status and company ownership data sification; ClassificationValidStart; tics low so- ClassificationValidEnd; Household- cio-econom- memberID; BenefitDate(first); ic status BenefitDate(last) 3.8. Sha- 3.8.1. Convicted shareholder 1. Public procure- FirmID reholder/ ment legal repre- 3. Corporate and FirmID; ShareholderID; Sharehold- sentative shareholder data erEntryDate; ShareholderExitDate; with criminal LegalRepresentativeID record 7. Criminal records PersonID 3.9. Tax 3.9.1. Company registered in 1. Public procure- FirmID haven regis- tax haven ment tration 3. Corporate and FirmID; FirmCountry shareholder data 3.9.2. Shareholder registered in 1. Public procure- FirmID tax haven ment 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; ShareholderCountry 4. Political 4.1. Political 4.1.1. Donation to electoral 1. Public procure- FirmID connections finance campaign ment 2. Employment EmployerID; WorkerID; Admission relationships date; Termination date 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; LegalRepresentativeID 4. Electoral data CandidateID; ElectionDisputed; OfficeDisputed; PartyName; ElectionJurisdiction; Elected; CampaignDonorID; CampaignDo- nationValue; DonorLocation; Cam- paignSupplierID; ExpenseValue; SupplierLocation; Year 4.1.2. Value of donation to elec- 1. Public procure- FirmID toral campaign ment 2. Employment EmployerID; WorkerID; Admission relationships date; Termination date 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; LegalRepresentativeID 4. Electoral data CandidateID; ElectionDisputed; OfficeDisputed; PartyName; ElectionJurisdiction; Elected; CampaignDonorID; CampaignDo- nationValue; DonorLocation; Cam- paignSupplierID; ExpenseValue; SupplierLocation; Year 4.1.3. Contracts won following 1. Public procure- ID Process; AgencyID; AgencyLo- donation ment cation; AgencyGovLevel; FirmID; ContractID; ContractValue; Con- tractDate CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 54 Risk group Risk pattern Red flag/ indicator Dataset category Required data fields 4. Political 4.1. Political 4.1.3. Contracts won following 2. Employment EmployerID; WorkerID; Admission connections finance donation relationships date; Termination date 3. Corporate and FirmID; ShareholderID; Sharehol- shareholder data derEntryDate; ShareholderExitDa- te; LegalRepresentativeID 4. Electoral data CandidateID; ElectionDisput- ed; OfficeDisputed; PartyName; ElectionJurisdiction; Elected; CampaignDonorID; Campaign- DonationValue; DonorLocation; CampaignSupplierID; ExpenseVal- ue; SupplierLocation; Year 4.1.4. Percent of contracts won 1. Public procure- ID Process; AgencyID; AgencyLo- following donation ment cation; AgencyGovLevel; FirmID; ContractID; ContractValue; Con- tractDate 2. Employment EmployerID; WorkerID; Admission relationships date; Termination date 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; LegalRepresentativeID 4. Electoral data CandidateID; ElectionDisput- ed; OfficeDisputed; PartyName; ElectionJurisdiction; Elected; CampaignDonorID; Campaign- DonationValue; DonorLocation; CampaignSupplierID; ExpenseVal- ue; SupplierLocation; Year 4.2. Person- 4.2.1. Company’s personal con- 1. Public procure- FirmID al connec- nections to politicians ment tions to politicians 2. Employment EmployerID; WorkerID; Admission relationships date; Termination date 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; LegalRepresentativeID 4. Electoral data CandidateID; ElectionDisput- ed; OfficeDisputed; PartyName; ElectionJurisdiction; Elected; ElectoralProcess; AffiliationStart; AffiliationEnd; PartyRepresenta- tionStart; PartyRepresentationEnd; PartyRepresentationPosition 8. Asset and inte- PersonID; AgencyID; Year; Sha- rest declarations reholderCompanyID; RelativeID 4.2.2. Contracts won following 1. Public procure- ID Process; AgencyID; AgencyLo- political connection ment cation; AgencyGovLevel; FirmID; ContractID; ContractValue; Con- tractDate 2. Employment EmployerID; WorkerID; Admission relationships date; Termination date CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 55 Risk group Risk pattern Red flag/ indicator Dataset category Required data fields 4. Political 4.2. Person- 4.2.2. Contracts won following 3. Corporate and FirmID; ShareholderID; Sharehol- connections al connec- political connection shareholder data derEntryDate; ShareholderExitDa- tions to te; LegalRepresentativeID politicians 4. Electoral data CandidateID; ElectionDisputed; OfficeDisputed; PartyName; ElectionJurisdiction; Elected; ElectoralProcess; AffiliationStart; AffiliationEnd; PartyRepresenta- tionStart; PartyRepresentationEnd; PartyRepresentationPosition 8. Asset and inte- PersonID; AgencyID; Year; Share- rest declarations holderCompanyID; RelativeID 4.2.3. Percent of contracts won 1. Public procure- ID Process; AgencyID; AgencyLo- following political connection ment cation; AgencyGovLevel; FirmID; ContractID; ContractValue; Con- tractDate 2. Employment EmployerID; WorkerID; Admission relationships date; Termination date 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; LegalRepresentativeID 4. Electoral data CandidateID; ElectionDisput- ed; OfficeDisputed; PartyName; ElectionJurisdiction; Elected; ElectoralProcess; AffiliationStart; AffiliationEnd; PartyRepresenta- tionStart; PartyRepresentationEnd; PartyRepresentationPosition 8. Asset and inter- PersonID; AgencyID; Year; Share- est declarations holderCompanyID; RelativeID 4.3. Person- 4.3.1. Company’s personal con- 1. Public procure- FirmID al connec- nections to bureaucrat ment tions to 2. Employment EmployerID; WorkerID; Admission bureaucrats relationships date; Termination date; Position 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; LegalRepresentativeID 8. Asset and inte- PersonID; AgencyID; Year; Sha- rest declarations reholderCompanyID; RelativeID 4.3.2. Contracts won following 1. Public procure- ID Process; AgencyID; AgencyLo- connection to bureaucrat ment cation; AgencyGovLevel; FirmID; ContractID; ContractValue; Con- tractDate 2. Employment EmployerID; WorkerID; Admission relationships date; Termination date; Position 3. Corporate and FirmID; ShareholderID; Sharehol- shareholder data derEntryDate; ShareholderExitDa- te; LegalRepresentativeID 8. Asset and inte- PersonID; AgencyID; Year; Sha- rest declarations reholderCompanyID; RelativeID CONTINUED EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 56 Risk group Risk pattern Red flag/ indicator Dataset category Required data fields 4. Political 4.3. Person- 4.3.3. Percent of contracts won 1. Public procure- ID Process; AgencyID; AgencyLo- connections al connec- following connection to bureau- ment cation; AgencyGovLevel; FirmID; tions to crat ContractID; ContractValue; Con- bureaucrats tractDate 2. Employment EmployerID; WorkerID; Admission relationships date; Termination date; Position 3. Corporate and FirmID; ShareholderID; Sharehold- shareholder data erEntryDate; ShareholderExitDate; LegalRepresentativeID 8. Asset and inter- PersonID; AgencyID; Year; Share- est declarations holderCompanyID; RelativeID EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 57 >>> Appendix III. GRAS Architecture EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 58 The Governance Risk Assessment System (GRAS) is an GRAS’s Architecture with Docker agnostic system, meaning it can operate independently of any specific hardware or software configurations, making it GRAS’s architecture incorporates four Docker containers: highly versatile. It is designed to be both flexible and scalable. GRAS’s architecture consists of three core components: • database: This is a Postgres container that hosts the system’s database. • Database: Stores public procurement data, registration data, and risk patterns. This component uses PostgreSQL, • api: A container housing the API application, which a robust and reliable database management system. establishes connections between the database and the web application. • API: Acts as the conduit between the database and the • web: This container hosts the web application and creates web interface. The API is developed using Django, a a connection with the API. powerful Python programming tool. • etl: Standing for Extract, Transform, Load, this container is • Web Application: Generates the web pages that users tasked with updating registration databases and connects use to analyze risk pattern data. This user interface is built with the database container. using the JavaScript React framework. Each container operates akin to a virtual machine, serving as GRAS takes advantage of Docker container technology, an an independent server for their respective applications within open-source solution that ensures portability and facilitates GRAS. The containers communicate through an internal easy deployment in various environments, be it local or cloud- Docker network, which bolsters the system’s security by based platforms such as AWS, Google Cloud, Azure, etc. The avoiding the need to open additional ports, except for the web following is a detailed explanation of how GRAS harnesses application. this concept. Deploying GRAS in Docker containers ensures consistent Docker Usage application behavior—termed as ‘uniform application execution’—regardless of the hosting environment. This Docker is an open-source tool that streamlines the creation, provides a sturdy, scalable, and secure architectural solution deployment, and running of applications in containers. It for risk pattern data analysis. provides isolation, maintaining distinct environments for development, testing, and production. This feature ensures consistency, meaning the application behaves the same way across different stages, hence minimizing potential errors or discrepancies. Therefore, by using Docker, GRAS can operate seamlessly across different environments. Apart from standard Docker, GRAS utilizes Docker Compose, an extension of Docker. It simplifies the configuration and operation of applications that involve multiple containers, enabling the definition of dependencies within a single file. EQUITABLE GROWTH, FINANCE & INSTITUTIONS INSIGHT <<< 59