83210 v3 The World Bank and Public Procurement—An Independent Evaluation Volume II: Achieving Development Effectiveness through Procurement in Bank Financial Assistance © 2014 Independent Evaluation Group The World Bank Group 1818 H Street NW, Washington DC 20433 Telephone: 202-458-4497; Internet: www.ieg.worldbankgroup.org E-mail: ieg@worldbank.org Some rights reserved 1 2 3 4 17 16 15 14 This work is a product of the staff of the Independent Evaluation Group (IEG). Note that IEG and the World Bank does not necessarily own each component of the content included in the work. IEG and the World Bank therefore do not warrant that the use of the content contained in the work will not infringe on the rights of third parties. The risk of claims resulting from such infringement rests solely with you. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of IEG, the World Bank, its Board of Executive Directors, or the governments they represent. IEG and the World Bank Group do not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of IEG and the World Bank Group concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of IEG and the World Bank Group, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 Unported license (CC BY 3.0) http://creativecommons.org/licenses/by/3.0. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: IEG (Independent Evaluation Group). 2014. The World Bank and Public Procurement—An Independent Evaluation. Volume II: Achieving Development Effectiveness through Procurement in Bank Financial Assistance. Washington, DC: World Bank. License: Creative Commons Attribution CC BY 3.0 Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by the Independent Evaluation Group or the World Bank Group and should not be considered an official World Bank Group translation. IEG and the World Bank Group shall not be liable for any content or error in this translation. All queries on rights and licenses should be addressed to the IEG, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-3125; e-mail: ieg@worldbank.org. ISBN-13: 978-1-60244-237-5 ISBN-10: 1-60244-237-1 ii CONTENTS Contents ABBREVIATIONS .................................................................................................................................................................. VII ACKNOWLEDGMENTS .......................................................................................................................................................... IX 1. BANK PROCUREMENT AND DEVELOPMENT IMPACT ....................................................................................... 1 Bank Procurement Guidelines: Overall Perceptions and Specific Concerns ............................................................ 4 Bank Procurement Processes ................................................................................................................................... 8 Using Country Procurement Systems ..................................................................................................................... 10 The Bank’s Use of Country Systems Pilot ........................................................................................................................... 10 IEG Analysis of the Bank’s Use of Country Systems Pilot .................................................................................................. 13 Use of Country Systems – the Experience of Other Donors ............................................................................................... 19 Use of Country Systems—Stakeholders’ Views on the World Bank Pilot ........................................................................... 23 Use of Country Systems—Stakeholders’ Views on the Adoption of Country Systems ....................................................... 24 Upstream and Downstream Involvement in Procurement ....................................................................................... 27 Adequacy of Upstream Involvement .................................................................................................................................... 27 Downstream Involvement—Role in Contract Management ................................................................................................ 28 Developing Domestic Suppliers and Offering Domestic Preferences ..................................................................... 30 Present Provisions for Domestic Preferences ..................................................................................................................... 30 Broader Interpretation of Developing Domestic Supply Capability...................................................................................... 31 Environmentally Responsible Procurement—Potential for Bank Use ..................................................................... 33 Bank Policies, Practice, and Experience ............................................................................................................................. 34 International Perspectives ................................................................................................................................................... 35 Achieving Value for Money in Bank Procurement ................................................................................................... 37 Applying Value for Money in the World Bank ...................................................................................................................... 38 2. ADAPTING TO EVOLVING NEEDS....................................................................................................................... 41 Procurement in Community Driven Development Projects ..................................................................................... 41 The CDD Approach and the CDD Portfolio ......................................................................................................................... 42 CDD Lending and Bank Procurement Policies .................................................................................................................... 43 IEG Portfolio Analysis—Performance of All Projects with CDD Elements .......................................................................... 44 IEG Project Reviews—Findings from a Subset of Projects with Community Control ......................................................... 45 Procurement in Fragile and Conflict Situations ....................................................................................................... 50 Analysis of Procurement Issues in FCS—Issues Identified in CASs .................................................................................. 51 Analysis of Procurement Issues in FCS – Review of Project Documents ........................................................................... 54 Analysis of Procurement Issues in FCSs—Staff Questionnaire .......................................................................................... 55 Procurement of Information and Communications Technology .............................................................................. 59 The Bank’s ICT Portfolio and the Scope of the Evaluation ................................................................................................. 59 ICT and Bank Procurement Procedures.............................................................................................................................. 61 IEG Structured Survey: Procurement Staff and Task Team Leaders in a Sample of ICT Projects ................................... 62 IEG Supplementary Interviews: ICT Experts, Procurement Legal Experts ......................................................................... 67 IEG Country Survey Findings—Experience with ICT Procurement .................................................................................... 68 Conclusions ......................................................................................................................................................................... 70 Procurement in Public-Private Partnership Investments ......................................................................................... 72 Public-Private Partnerships in the Bank’s Portfolio ............................................................................................................. 74 IEG Review of Experience in Bank Operations ................................................................................................................... 77 Conclusions ......................................................................................................................................................................... 81 3. MANAGING RISK AND INCREASING TRANSPARENCY ................................................................................... 86 iii Introduction ............................................................................................................................................................. 88 Managing Procurement Risk—Prior Reviews and Thresholds ............................................................................... 88 Concept and Use of Thresholds .......................................................................................................................................... 88 Perceptions of Thresholds—IEG Field Visits ...................................................................................................................... 92 IEG Analysis of Thresholds and Risk Management ............................................................................................................ 95 Results ................................................................................................................................................................................. 95 Conclusions and Implications .............................................................................................................................................. 98 Managing Procurement Risk—Post-Procurement Review ..................................................................................... 98 Managing Procurement Risk—The P-RAMS Instrument ...................................................................................... 102 Managing Risk Before P-RAMS ........................................................................................................................................ 102 The Introduction and Contribution of P-RAMS .................................................................................................................. 107 Perceptions of P-RAMs and ORAF: IEG Field Visits ....................................................................................................... 110 Empirical Analysis of P-RAMS Data .................................................................................................................................. 110 Additional Observations on P-RAMS ................................................................................................................................ 113 Measuring Risk Failure—Misprocurement, Complaints, and Investigations ......................................................... 114 Misprocurement and Procurement Complaints ................................................................................................................. 114 Fraud, Corruption, and the Role of INT ............................................................................................................................. 115 Sanctions, Suspensions, and Corporate Risk ................................................................................................................... 119 Perceptions of the Role of INT—IEG Field Visits .............................................................................................................. 122 4. ACHIEVING EFFICIENCY IN BANK PROCUREMENT ....................................................................................... 124 Procurement Efficiency and Value for Money—Tracking Procurement ................................................................ 124 Main findings ..................................................................................................................................................................... 124 Tracking Contract-Level Procurement Data ...................................................................................................................... 125 Bank-wide Tracking of Procurement: Form 384 ................................................................................................................ 128 Existing Regional Systems ................................................................................................................................................ 130 Future Directions ............................................................................................................................................................... 132 Measuring Procurement Efficiency—IEG Data on Sample Contracts .................................................................. 134 Analysis and Findings: Average Elapsed Times ............................................................................................................... 136 Analysis and Findings: Variation in Elapsed Times—Frequency Distributions ................................................................. 138 Time Taken to Process a Contract—Findings from Regression Analysis......................................................................... 140 Efficiency of Procurement in Africa – IEG Analysis of PROCYS data .............................................................................. 141 Further New Directions for Increasing Efficiency .................................................................................................. 143 Bank Involvement in the Development of E-Procurement ................................................................................................ 146 BIBLIOGRAPHY .................................................................................................................................................................. 151 ENDNOTES .......................................................................................................................................................................... 159 Boxes Box 1.1. Management Review of Bank Procurement Systems ............................................................... 2 Box 1.2. Procurement—Overall Perceptions of Bank Guidelines and Processes ................................... 4 Box 1.3. Procurement Processes—Examples of Inflexibility and Loss of Development Effectiveness ... 9 Box 1.4. Stage II Equivalence Analysis – Bidding Documents Review ..................................................18 Box 1.5. Use of Country Systems—The Bank and the Larger Development Community ......................20 Box 1.6. Using Country Systems—Benefits Perceived by Client Implementing Agencies .....................25 Box 1.7. Using Country Systems—Drawbacks Perceived by the Private Sector ...................................25 Box 1.8. Perspectives on Domestic Preference .....................................................................................32 Box 1.9. Procurement and Energy-Efficient Products and Services ......................................................35 Box 1.10. Asian Development Bank—Loan Agreement Provisions on the Environment .......................36 iv CONTENTS Box 1.11. Incorporation of Maintenance Costs and Life Cycle Costing in Bank Procurement............... 39 Box 2.1. Afghanistan: Emergency National Solidarity Project II ............................................................ 48 Box 2.2. Procurement Issues in the Maranhão Integrated Program...................................................... 49 Box 2.3. Procurement Issues and Procurement Support in FCS—Evidence from Selected CASs ....... 52 Box 2.4. Procurement Guidelines under OP 8.0 and OP 10.0 .............................................................. 52 Box 2.5. Two-Stage and Single-Stage Bidding ..................................................................................... 62 Box 2.6. IEG Review of ICT Procurement—Project Support Provided .................................................. 64 Box 2.7. Messages from the ICT Portfolio Review ................................................................................ 67 Box 2.8. Negotiation to Procure ICT Systems ....................................................................................... 68 Box 2.9. Using Sole-Source Procurement for a System Upgrade ......................................................... 70 Box 2.10. PPP—Missed Opportunities in Other Countries? .................................................................. 80 Box 2.11. India—A Tale of Two PPPs................................................................................................... 82 Box 3.1. Procurement Risk Management in Recent IEG Evaluations ................................................... 86 Box 3.2. The Bank’s Taxonomy of Procurement Risk ........................................................................... 89 Box 3.3. Setting Method Thresholds to Optimize Competition at the AfDB ........................................... 93 Box 3.4. The Procurement Risk Assessment and Management System ............................................ 103 Box 3.5. Integrating Procurement and Project Risk—Tanzania Second Transport Corridor Project ... 107 Box 3.6. Perceptions on New Risk Management Tools—P-RAMS and ORAF ................................... 111 Box 3.7. The Office of Sanctions and Debarment—Two-Tier Sanctions System ................................ 120 Box 4.1. Benchmarking Municipal Procurement Activities—An Example from Finland ...................... 126 Box 4.2. Procurement Tracking Systems at the World Bank—Early Efforts ........................................ 128 Box 4.3. Procurement, Disbursement, Contract Accounting, and Form 384 ....................................... 129 Box 4.4. Proposed New Systems Architecture: The Integrated Procurement Plan ............................. 133 Box 4.5. New Procurement Methods—Framework Agreements and Reverse Autions ...................... 144 Box 4.6. E-Procurement ...................................................................................................................... 145 Tables Table 1.1. Summary of Results of the Bank’s Use of Country Systems Pilot (2005–10) ....................... 12 Table 1.2. Bank UCS Pilot Participants—Average Scores for MAPS Pillars and Indicators.................. 14 Table 1.3. Bank Requirements for NCB and Derogations in Select Countries ...................................... 16 Table 1.4. Bank Support During Procurement Design and Planning..................................................... 28 Table 1.5. Contract Management and Implementation ......................................................................... 29 Table 2.1. CDD Projects: ISR Procurement Performance Ratings and Overall Outcomes ................... 46 Table 2.2. IEG Review of ICT Procurement—Project Characteristics ................................................... 63 Table 2.3. IEG Review of ICT Procurement—Reported Use of Bank Procurement Methods .............. 65 Table 2.4. IEG Sample of Bank PPP Projects Reviewed (by region FY04–12)..................................... 78 Table 2.5. Financing Structure of Bank PPP Projects ........................................................................... 79 Table 3.1. Prior Review, Procurement Method and Clearance Thresholds ........................................... 90 Table 3.2. ICB Prior Review Contracts for Goods and Civil Works (numbers and value) (FY00–12) ... 91 Table 3.3. Prior Review Procurement Thresholds: Correlations with Indices of Governance................ 96 Table 3.4. Share of Contracts Designated for Post Procurement Review (2002–10)............................ 99 Table 3.5. Risk Categories and Individual Risk Identification in Pre-P-RAMS PADs by Period .......... 105 Table 3.6. P-RAMS Compliance Rates by Region .............................................................................. 109 Table 3.7. Data on INT Investigation Cases by Region Conducted in FY09–12 ................................. 116 v Table 3.8. Correlation of INT Cases with Project, Procurement, and Financial Management Risk......118 Table 4.1. Summary of System Contributions to Tracking Procurement Performance ........................131 Table 4.2. “No-Objection” Steps for Prior-Reviewed Contracts............................................................135 Table 4.3. Average Elapsed Time between Steps in the Procurement Process (days) .......................137 Table 4.4. Average Elapsed Time and Contract Value by Procurement Category ..............................137 Table 4.5. Average Elapsed Time: Issue of Specific Procurement Notice to Contract Signature ........138 Table 4.6. Africa: Numbers of Iterations and Average Response Times per Procurement Transaction Request, 2009–12 ...............................................................................................................................141 Table 4.7. Africa: Response Times for Procurement Categories by Clearance Level, 2009–12..........142 Figures Figure 1.1. Bank UCS Pilot Participants—Average Scores for MAPS Pillars and Indicators .................15 Figure 1.2. Bank UCS Pilot Participants – Approaching the World Bank System ..................................18 Figure 2.1. CDD and Other Projects—Performance in Low Governance Environments .......................45 Figure 2.2. Procurement Performance and Development Outcomes by Type of CDD ..........................46 Figure 2.3. Trends in IBRD/IDA and RETF Support to FCS (2003–12) .................................................51 Figure 2.4. World Bank Group: ICT Portfolio 2003–10 ..........................................................................60 Figure 2.5. World Bank PPP Projects—July 2002 to December 2011, by FY........................................74 Figure 3.1. Country and Project-Level Prior Review Thresholds by Region—Civil Works and Goods ..97 Figure 3.2. Procurement Risk Ratings for Projects Above and Below Country-Level Prior Review Thresholds .............................................................................................................................................97 Figure 3.3. Distribution of Risk Ratings in Post-Reviewed Contracts...................................................100 Figure 3.4. Comparison of Project Procurement Risk (from P-RAMS) and PPR-Based Risk Ratings .101 Figure 3.5. Procurement Risk Categories, Risk Mitigation, and Risk by Sector—A Sample ................106 Figure 3.6. P-RAMS Completed by Month (2009–12) .........................................................................109 Figure 3.7. Summary Results from P-RAMS Applications, 2010–12 ...................................................112 Figure 3.8. P-RAMS Risk and P-RAMS Risk Mitigation.......................................................................112 Figure 3.9. Time Distribution of Complaints Resolution .......................................................................115 Figure 3.10. INT Complaints by Stages in Procurement and Project Implementation .........................117 Figure 3.11. Correlation of INT Cases with Project, Procurement, and Financial Management Risk (cases with at least one risk rating)......................................................................................................119 Figure 4.1. Number of Observations by Country, Fiscal Year, Procurement Category, and Sector.....136 Figure 4.2. IBRD and IDA Countries—Elapsed Procurement Times by Procurement Method (days) .139 Figure 4.3. Distribution of Days from Issue of Specific Procurement Notice to Contract Signature .....139 Figure 4.4. Borrower Delays and Bank Response, 2010–12 ...............................................................143 Appendixes for this volume are available at https://ieg.worldbankgroup.org/evaluations/world-bank-group-and-public-procurement vi Abbreviations ADB Asian Development Bank MAPS Methodology for Assessing AfDB African Development Bank Procurement Systems CAS Country Assistance Strategy MCC Millennium Challenge Corporation CDD community-driven development MDB multilateral development bank CPAR Country Procurement Assessment NCB national competitive bidding Report NTLA non-lending technical assistance CPIA County Policy and Institutional OECD Organisation for Economic Co- Assessment operation and Development- FCS fragile and conflict-affected situation Development FICID International Federation of Consulting OECD-DAC Development Assistance Engineers Committee, Organisation for IAD Internal Audit Vice Presidency Economic Co-operation and ICB international competitive bidding Development process OPRC Operational Procurement Review ICT information and communications Committee technology OPSOR Operations Risk Management IDA International Development Agency Department, Operations Policy and IDB Inter-American Development Bank Country Services Vice Presidency IEG Independent Evaluation Group ORAF Operational Risk Assessment IFC International Financial Corporation Framework IFI international financial institution PAD Project appraisal document IIFCL India Infrastructure Finance Company PPP public-private partnership Limited P-RAMS Procurement risk assessment IGR Institutional Governance Review management system INT Integrity Vice Presidency RPM Regional procurement managers IPR Independent Procurement Review SEPA Procurement plan execution system ISN Interim Strategy Note UNCITRAL United Nations Commission on ISR Implementation Status Report International Trade Law WBI World Bank Institute vii Acknowledgments This report of the Independent Evaluation Group (IEG) was prepared by a core team led by Anjali Kumar, with major contributions provided by Nils Fostvedt, Prem Garg, Gita Gopal, Ian Hume, Eduardo Maldonado, Arvind Nair, Carla Pazce, Juliane Piecha, Robert Rothery, Swizen Rubbani, Justin Sigman, Kathryn Steingraber, Nadine Tushe, Christine Wallich, and Clay Wescott. Eleven country case studies were undertaken by Gian Casartelli, Jorge Claro, Ajay Guha, Rodolfo Hernandez, Jean- Jacques Lecat, Robert Rothery, Paul Schapper, and Christine Wallich. Additional contributions were provided by Jeanmarie Fath Meyer and S. Janakiram. Corky de Asis, Aimée Niane, Agnes Santos, and Lily Tsang provided administrative support. The team gratefully acknowledges the support of managers and staff throughout the World Bank, especially in Operations Policy and Country Services departments and from both former and current regional procurement managers. IEG appreciates the many individual procurement staff throughout the Bank who have been generous with their time for surveys and interviews on aspects of the Bank’s procurement system. Particular thanks are offered to the field-based procurement staff that provided assistance for field visits in Azerbaijan, Bangladesh, Ethiopia, Indonesia, Mexico, Morocco, the Philippines, Peru, Senegal, Tanzania, and Turkey, arranging meetings with key officials as well as private persons. IEG also extends its thanks to other areas of the Bank consulted during the course of this evaluation: the Controllers unit, Economic Management, Institutional Integrity, Internal Audit, and Legal units. IEG also appreciates exchanges with the African Development Bank, the Asian Development Bank, the European Bank for Reconstruction and Development, the European Commission, the European Investment Bank, the Inter-American Development Bank, and the Organisation for Economic Co-operation and Development, who shared their perspectives with the evaluation team, and its exchanges with academia at George Washington University, as well as Harvard University, the University of Florida and Nottingham University, UK. The team also thanks client government counterparts, private sector suppliers, project implementation units, personnel from implementing agencies, and members of civil society who participated in IEG’s surveys and interviews. The evaluation benefitted from constructive advice from external advisers and peer reviewers. Peer reviewers included Praful Patel (former Vice President, South Asia Region, and former Director, Infrastructure, Africa Region); Christiaan Poortman (former Vice President, Middle East and North Africa Region, World Bank, and current Senior Advisor at Transparency International and Chair of the Board of the ix ACKNOWLEDGEMENTS Construction Sector Transparency Initiative); and Dr. Peter Trepte (Senior Fellow in Public Procurement Law, University of Nottingham). An external advisor provided overall guidance to the study—Professor Daniel Gordon, Associate Dean for Government Procurement Law, at the George Washington University Law School. The evaluation was conducted under the guidance of Nicholas York, Director, and Ali Khadr, Senior Manager, IEG Country, Corporate, and Global Evaluation, and Caroline Heider, IEG Director General. x 1. Bank Procurement and Development Impact Introduction to Part II—Focus and Rationale As described in the first part of this two-part evaluation of the World Bank and public procurement, conducted by the Independent Evaluation Group (IEG), the World Bank has a dual role with regard to public procurement in its client countries. First, because good national public procurement practices are a major determinant of the effectiveness of public expenditure, building client countries’ capacity for better public procurement within their own national systems is an essential element of the poverty reduction focus of the Bank, and in accordance with global principles of aid effectiveness. The Bank’s contributions to country procurement capacity building are therefore the focus of Volume I of this report. Second, World Bank–financed investment lending operations constitute a significant element of public procurement in client countries; therefore, the Bank’s own procurement policies and processes affect the development impact of Bank lending. This volume therefore focuses on the second aspect of the Bank’s dual role: the extent to which the Bank’s own procurement guidelines and processes help achieve the Bank’s development goals, that is, to ensure that its funds are used effectively and efficiently and for the purpose intended. IEG therefore evaluates the extent to which the guiding principles of the present Bank Guidelines are achieved, in terms of transparency, competition, economy, and efficiency, and the development of domestic markets. The evaluation parallels an intensive multiphase review by management of the Bank’s procurement function, intended to pave the way toward significant future changes (Box 1.1). IEG’s evaluation is intended to inform management’s review in terms of specific proposals for change; also seeks to address specific queries and concerns raised by the Bank’s Board members regarding the Bank’s procurement systems and processes. Accordingly, this chapter first selectively reviews areas where country client and staff stakeholders have drawn attention to the need for review of the current guidelines and processes. Second, it examines evidence on core themes proposed by Bank management for inclusion in its future procurement framework (the use of country systems, the potential for engagement in upstream and downstream aspects of procurement, and, centrally, the overarching value for money proposition). Third, IEG selectively reviews aspects of Bank procurement policy of interest to the Board, for example, the scope for sustainability or environmental considerations or the scope 1 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT for domestic preferences. The balance of the report addresses the core themes of the extent to which current guidelines and processes meet the new business needs of the Bank (Chapter 2), the management of fiduciary risk in procurement (Chapter 3), and the efficiency of the current procurement system (Chapter 4). Box 1.1. Management Review of Bank Procurement Systems Management’s review of the Bank’s procurement systems began with a detailed memorandum discussed with the Bank’s Board in February 2012 that pointed toward changes in the relevance of the present system, away from traditional infrastructure projects for which its procurement systems were designed. These trends have paralleled global changes in client country capacity, supplier patterns, and new practice emerging in public procurement. The goal of the review, as stated by management, is to position the Bank’s procurement policies and procedures in the context of its modernization agenda, moving away from “one-size-fits-all” and at the same time reflecting its leadership role in promoting best practices in public procurement. Management points out that there has not been a fundamental review of the Bank’s procurement policies, starting from first principles, since the Bank’s founding. In its most recent discussions with the Board (April 2013), the Bank presented an overall vision for its proposed future reforms: Procurement in Bank operations supports clients to achieve value for money in delivering results. Developmental results achieved are core, and the overarching goal of achieving value for money is underpinned by specific areas of change: an enhanced but selective use of country systems at the request of the borrower, under a new best-fit-for-purpose approach; a changed, nonmandatory role for the Bank Guidelines and standard bidding documents; introduction of new procurement methods; a greater strategic procurement planning function; and the maintenance of fiduciary assurances despite a smaller number of prior-reviewed contracts, through a more rigorous risk- based approach. At borrowers’ request, particular methods and procedures of the procurement systems of the implementing institution of a specific operation could be selectively used under the new fit for purpose approach. Sources: World Bank 2012c, 2013b. Main Findings  On the whole, country clients, the private sector, and Bank staff agree that the Bank’s present procurement guidelines for goods and works are reasonably successful in securing fairness, competition, and transparency in Bank procurement. Nevertheless, there may be scope for improvement in certain details, in addition to 2 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT areas of complex procurement. There are greater perceptions of problems with the consulting guidelines.  There is considerably less comfort with Bank procurement processes, especially with regard to time taken, flexibility, and consistency, which lead to losses in development effectiveness. Focus on compliance in transactions may distract from a focus on outcomes and need not ensure the containment of systemic fraud and corruption.  Future options for the use of country systems in Bank procurement can draw lessons from its recent pilot, an exercise characterized by rigor but also by minutiae in terms of detailed of areas of compliance.  Some areas of difference between the Bank and its clients touch on core principles (competition, market access); others impact such principles less. A review of the materiality of such differences would inform the adoption of an incremental approach toward use of country systems, in line with other multilateral development banks (MDBs).  Country stakeholders and the private sector do not invariably prefer the use of country systems, especially for the kind of large value contracts for which Bank systems were designed.  Regarding upstream involvement, there is scope for more Bank support in acquisitions planning and in the integration of procurement in public budget management.  There is a clearly perceived need for greater involvement downstream in terms of contract management. The roles of different players need review relative to international practice, legal obligations, and task team leader incentives.  Most countries have elements of domestic preference in their procurement policies. Although the Bank Guidelines endorse the principle of developing domestic manufacturing and contracting capabilities, the prescribed domestic preference scheme is restrictive in scope, which may explain its limited use. To the extent that the Bank retains the objective to develop domestic capacity through its procurement system, it may wish to consider a broader approach.  With regard to sustainable and environmentally aware procurement, Bank Guidelines and practices already offer many avenues to incorporate “green” considerations, although they are not systematically used. There is scope for more 3 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT explicit incorporation of environmental factors in technical specifications and bid evaluation criteria.  The principle of value for money in procurement is being increasingly incorporated into public procurement worldwide, with varying emphasis. Many aspects are implicit in Bank procurement; some are frequently used. Stronger direction could be given to staff and borrowers to fortify the adoption of such practices. Bank Procurement Guidelines: Overall Perceptions and Specific Concerns By and large, both country clients and other users of the Bank’s procurement guidelines view them positively (Box 1.2). Procurement under Bank Guidelines is seen to produce good results in terms of achieving open and fair competition, integrity, transparency, quality, and price—often better than national systems operating within the context of national oversight and accountability (Appendix Tables A.1–A.6). Some country clients perceive that serious bidders are more likely to show interest if a project is Bank financed and under Bank procurement policy, as opposed to national procedures. The Bank was deemed to balance different considerations and not limit decisions to price factors—although these factors play a significant role. Box 1.2. Procurement—Overall Perceptions of Bank Guidelines and Processes In Morocco, private sector representatives have confidence in the Bank’s procurement system for reasons ranging from safe financing and payment to transparency and integrity to possibility of recourse. In Azerbaijan and Tanzania, Bank Guidelines were thought to bring better quality by eliminating unqualified bidders and because payments under Bank-supervised projects were generally faster. Discussions also suggested that bidders appear to take fiduciary concerns more seriously in internationally funded procurement. There was general consensus that Bank Guidelines work well for straightforward procurement, especially in infrastructure projects. In Azerbaijan, for example, project implementing units in charge of infrastructure projects cited the guidelines positively in terms of overall content and levels of participation and also voiced appreciation for the quality of standard bidding documents. By contrast, project implementing units in health and education projects were critical of the Bank’s rigid approach and its insistence on international competitive bidding process for goods and works, even for small items or projects, when clients deemed national competitive bidding to be equally effective. Source: IEG field visits. Yet despite broad acceptance, respondents to the IEG surveys and field visits pointed to specific areas of Bank Guidelines that may merit re-examination or revision. These 4 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT areas are discussed below. Less positive messages were conveyed in terms of Bank processes: country clients felt that there was at best modest flexibility to respond to special circumstances, and there were suggestions of delays in contract award. Bid Submission—The Two-Envelope System The two-envelope system used by entities such as the Asian Development Bank (ADB) is not currently a World Bank process. It refers to the simultaneous submission of technical and price proposals in separate envelopes, with sequential opening and evaluation. The Bank requires single-envelope bid submission (except for consultants), perhaps because of concerns about the fraud and corruption or perhaps because of possible lack of transparency. Procurement staff point out that the single- envelope system acts as a deterrent for high-quality firms, encouraging too many bids with a least cost focus. This perception is shared by some other donors, who also believe that the one-envelope system leads to outcomes where quality is largely ignored (evaluators are reluctant to reject technically inferior, lowest priced bids once prices are known). The ADB permits the two-envelope system. In Bangladesh and the Philippines, country procurement processes include single- stage two-envelope procurement. In Ethiopia and Tanzania, governments also preferred a two-envelope, postqualification system, as described in the preceding paragraph and used by other entities, rather than the Bank’s lengthy prequalification system with one envelope. Governments perceive the latter as very time consuming and limited in terms of screening out poor quality bidders/bids. They view the one- envelope system as militating against value for money. In Ethiopia, there was also substantial discussion about the Bank’s policy of requiring prequalification of bidders (rather than postqualification) and one-envelope bidding—an approach that makes it very difficult to turn down a low-price bidder that may not have the capacity to perform or that may be known to have performed poorly on a previous contract. 1 One option for the Bank could be to review existing bid management procedures and permit the two-envelope system, if deemed adequate. In terms of bid criteria, some countries request permission to use bills-of-quantities (that is, with estimates of quantities of materials, parts and labor to be used and their respective prices—also referred to as price-percentage bidding), which may be useful in a less-sophisticated environment. 2 Bid Evaluation—Third-Party Participation In Mexico, national legislation permits the use of “social witnesses” to participate, on a nonvoting basis, in bid evaluation; this practice is considered important for raising transparency. Respondents in Bangladesh point out that third-party participation has 5 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT been selectively used there, as in the Philippines. The Bank could consider how to make this selectively available, for example, through technical experts, to the extent that confidentiality considerations can be protected. 3 Some country clients have also requested that the Bank consider participation in bid evaluation, as a silent or nonvoting observer, to help ensure the integrity of the process. Although potentially value adding, this method nevertheless raises questions of possibly conflicting roles for the Bank, in terms of provision of advice versus provision of oversight. 4 New Procurement Methods—Framework Contracts, Using Negotiation There are many variants of framework contracts, and all allow the multiple use of suppliers, without rebidding each purchase. They can save cost and time for small but frequent purchases or for repeat or related purchases. The Bank’s current policy, which introduced the use of framework contracts in 2011, limits their use to common- use goods, simple nonconsulting services, or small-value emergency works, with a contract duration of up to three years and a value not exceeding the national competitive bidding (NCB) limit. Several countries confirmed that framework contracts are permissible to some degree under local law (Bangladesh, Mexico, Perus, and Tanzania), although their use has been limited under the Bank’s Guidelines. Several country respondents felt that there should be more flexibility to negotiate with a bidder, selected through a competitive process, on contracts for goods and works, as is the case with certain consultant selection methods (quality-based selection, for example). Country respondents said that this would increase clarity and provide an opportunity to tailor and correct minor omissions. Postbid negotiation was permitted in some countries, for example, in Bangladesh. Such practice makes it difficult to ensure transparency and is therefore not widely accepted. However, other options to introduce a degree of dialog into the procurement process, such as the competitive negotiation method offered in specific circumstances by the United Nations Commission on International Trade Law (UNCITRAL), may be considered. 5 The Bank’s position on negotiation options was considered restrictive (see Chapter 2). In Azerbaijan, Bank bidding processes were perceived as overly emphasizing the need to keep bidders and buyers at arm’s length from each other, allowing dialog only through formal meetings. Direct dialogue between buyer and bidder was thought to be not sufficiently encouraged, thereby restricting the exchange of important information that could improve bids. Decisions related to the degree and methods of negotiation could be made on a case-by-case basis in a country-specific context. 6 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Cofinancing Several borrowers complained about the imposition of Bank Guidelines even when the Bank finances only a small share of a project. In Morocco, in a project cofinanced by the African Development Bank (AfDB), the AfDB Board had to be requested to allow nationals of nonmember states to participate in the entire procurement, although the Bank was a minority contributor. In another case, a sector-wide approach, the Bank’s requirement to use its own procurement guidelines was rejected by the government (especially with regard to allowing the Bank to audit suppliers’ accounts), leading to protracted delays and eventual project restructuring. Consultant Selection Although the Bank Guidelines for procurement are generally seen as reasonable and value adding, its consultant’s guidelines appear to be less accepted. Many concerns surfaced during country visits: the lengthy quality and cost-based selection process and the volume of information sought from bidders, which were seen as expensive to produce and time consuming to evaluate. It was felt that better consultants refrained from expressing interest in quality- and cost-based selection processes. There were also concerns about the outcomes, which were perceived to lead to the selection of the “least worse” firm, thereby affecting the quality (that is, value) of services. Clients pointed toward undue attention to and incorrect use of proposal prices, apparently reflecting Bank insistence on quality- and cost-based selection rather than allowing greater use of quality-based selection. Other difficulties have been mentioned, for example, geographical diversification in short-listing criteria in situations where only local firms were likely to bid. In a quality- and cost-based selection process, the Bank correctly discourages any negotiations of the scope/quality of services and of prices, because the quality/price of the proposals is compared in the bids. Yet in essence this problem arises because quality- and cost-based selection is quite often adopted when it should not be, that is, when services are complex and contract finalization requires discussion between client and bidder. It has also been pointed out that the post- contract-award negotiation system in consultant contracts at the Bank, following a quality and cost-based selection, is not widely accepted as good practice. Greater use of a quality-only system was urged. 6 Negotiations are allowed under quality-based selection to define contract scope and contract price. 7 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Bank Procurement Processes Delays in Prior Review and “No Objections” Although the Bank Guidelines are generally seen as reasonable and value adding, the Bank’s procurement processes are perceived to be problematic and time consuming. Delays in getting no objections in the prior review process was a familiar theme in almost all country visits. Overall, the Bank was seen as placing more emphasis on safeguarding against risks to the integrity of the process than on efficiency and time. Countries expressed concern about the Bank’s lengthy process of approvals, especially when reference is made to the regional procurement manager or to the Operational Procurement Review Committee (OPRC) (Appendix Table A.6). Conflicting Guidance In several country visits, clients raised issues regarding inconsistencies in the advice and decisions among different Bank staff. Respondents in Mexico commented on the inconsistencies in advice from procurement personnel and task team leaders, compounded by frequent rotation among the latter. Consistency in comments was a significant issue for an implementing agency in Tanzania. Document “churn” was also a concern. Even when response times meet business norms, the Bank sometimes requests more documentation or clarification, extending clearing time. 7 Written communication requirements particularly affected decentralized projects (for example, in Indonesia). Implementing agencies commented that delays are more common on Bank-financed contracts than in those financed by ADB, AfDB, or the Inter-American Development Bank. Inflexible Interpretation of Guidelines Although the Bank Guidelines were deemed to work reasonably well and to incorporate sufficient provision for flexibility, it was generally felt that they were inflexibly applied (see Box 1.3), with the Bank not able to respond quickly to special circumstances (Appendix Table A.5). Clients urge focus on underlying intent and the application of reasonable interpretation. Finally, many stakeholders—Ethiopia, Indonesia, Morocco, and Tanzania were notable examples—believe that focus on transactions displaces focus on achieving outcomes. It is pointed out that even with perfect compliance, there can be fraud and corruption, which is sometimes endemic. Compliance may sometimes give a false sense of security and may even obscure malpractice. It was urged that efforts be made to tackle fraud and corruption issues at a systemic level rather than through individual Bank projects. 8 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Box 1.3. Procurement Processes—Examples of Inflexibility and Loss of Development Effectiveness In Indonesia, the implementing agency involved in a power sector procurement, the State Electricity Company, did not advertise on the United Nations Development Business, as required by the Bank. The electricity company, an experienced entity, had advertised on its own website. Suppliers were aware of the tender, and competition was reasonable. In any event UNDB is little used by potential suppliers. However, the false step resulted in a one- year delay. The Bank considered declaring misprocurement but subsequently allowed the contract to go ahead. Task team leaders and country management concurred that it was obvious that the Bank should have waived the oversight, quickly, and proceeded with procurement. Two examples were offered in Mexico. The first referred to a failed loan for the supply of vaccines during an influenza epidemic. Contract review by the procurement Board at headquarters pointed out that required anticorruption clauses were not included in the invitation to submit bids. Authorities pointed to the urgency of the situation and requested flexibility from the Bank. And in Mexico’s energy efficiency lightbulb exchange program, proposed procurement solutions that could have made the program sustainable over time were rejected by the Bank because they required contracting with specific established businesses, deemed to be a noncompetitive process. In Tanzania the U.S. Agency for International Development had contracted with a social marketing firm to market a premixed nutrition formula in different parts of the country. The firm was seasoned and had developed an effective campaign. According to the country client, Bank financing was unable to piggyback on the existing contract, although rebidding would have implied that two parallel social marketing contracts would be in place, with duplication of effort, training, materials development, and corporate overhead. However, according to the task team leader, single-source procurement would not have been a sensible option for this consultancy, as it was not a continuation of existing work but rather different approaches to the same intervention in different parts of the country. Examples also illustrated inflexibility in terms of bid securities and bid securing declarations. To ensure that bidders honor their bids if they win, the guidelines offer borrowers the option of requiring bank guarantees (with the amount and form to be specified in the bidding documents). However, the guidelines also give borrowers the option of requiring a bid securing declaration. Instances arose (Morocco) where Bank staff have been demanding in terms of the wording of the guarantee, leading to the disqualification of bidders. And in Tanzania, suppliers complained that Bank procurement staff insisted on bid security, instead of the bid securing declaration. In Indonesia, a bid was rejected because the bid security validity was two days less than prescribed, which Bank staff considered a de minimis deviation. Source: IEG field visits. 9 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT To summarize, although overall perceptions of Bank Guidelines are fairly positive, there is often discomfort with the interpretation of the guidelines and with the procurement process, which is seen as cumbersome, time consuming, prone to delay, and inflexible. Using Country Procurement Systems Given the pros and cons of current Bank procurement rules and processes, to what extent would it be advisable for the Bank to move toward greater use of country systems, and if so, in what manner should it proceed? In view of the centrality of this topic to the ongoing debate on future directions for the Bank, IEG presents a detailed analysis, beginning with a review of the Bank’s use of country systems pilot and analysis of its outcomes, followed by an examination of practices in other bilaterals and international financial institutions (IFIs), and finally, reviewing perceptions presented by the Bank’s stakeholders, specifically on the Bank pilot as well as on the general proposal to move toward country systems, based on evidence gathered from field visits. THE BANK’S USE OF COUNTRY SYSTEMS PILOT Beginning in 2005, the Bank sought to further increase its reliance on country systems, in particular for the remaining frontier of international competitive bidding (ICB) (World Bank 2005b). At the request of the Board, Bank management prepared a methodology on how to identify countries and projects to participate in a limited piloting program. It cited the core rationale for the use of country systems in its presentation to the Board in June 2007: maximizing development impact, increasing country ownership, 8 facilitating harmonization, and reducing transaction costs. 9 The Board approved the methodology in April 2008, and formal implementation of the program began in FY09 (World Bank 2009c). Twenty countries expressed initial interest; 17 remained engaged at the close of the pilot program. Over a period of two years, the program would identify a group of 8–10 countries and a set of projects within those countries where national procurement systems were of sufficient quality and capacity to achieve results comparable to the World Bank’s procedures. The first year progress report (World Bank 2009c) suggested delays. Early assessments revealed that all participating countries needed some further development of their procurement systems to meet Bank standards. Management extended the program to June 2011 and proposed that it introduce country-level capacity development and interim mitigation measures for projects to address deficiencies, allowing countries conditional approval for the pilot, subject to compliance with a Country Development Action Plan. 10 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT A core element of the methodology was the use of the new instrument of the Development Assistance Committee of the Organisation for Economic Co-operation and Development (OECD–DAC), introduced for the benchmarking of procurement systems: the Methodology for the Assessment of National Procurement Systems (MAPS)—devised with considerable Bank input (see Volume I, Chapter 3) (OECD 2005b, 2006b). It offered a platform for the benchmarking of countries’ procurement systems and was therefore used by the Bank for the establishment of baselines for its analyses of the extent of countries’ readiness to use their own procurement systems in World Bank-financed projects. 10 The piloting program methodology had three stages: Stage I: Selection of potential pilot countries and assessment of the overall quality of each country’s procurement system through a MAPS exercise Stage II: Assessment of the consistency and equivalence of the country’s procurement system, compared with the Bank’s procurement policies Stage III: Assessment of compliance, performance, capacity, and fiduciary risks at the sector/executing agency level. Fifteen countries completed the Stage I evaluation of country procurement systems using MAPS. The Bank drew up precise guidance on the passing scores required to meet Bank standards for each subindicator. Thus, 17 core subindicators bearing on key transparency and economy principles of the Bank and procurement policy were required to receive the highest score of 3. Another 10 required a score of 2, subject to an action plan to achieve a 3 during implementation. An additional 25 subindicators could earn a score of 2 without corrective measures. There were also two subindicators where only a score of 3 or 0 was possible, regarding the regulatory body and complaints review body. 11 IEG calculates that this implied a passing score of a total of 128 out of a possible 162 points. 12 Stage II, completed by nine countries, analyzed whether candidate countries’ procurement policies and procedures were consistent and equivalent to those of the Bank. The assessment was independent of Stage I, notwithstanding similar sources of information. One element of Stage II included a further stage of analysis, to see whether pilot countries’ bidding documents could be used for international competitive procurement. 13 A checklist of clauses consisted of 44 subcategories for instructions to bidders and an additional 37 for general conditions of contract. 14 Stage III entailed an assessment of compliance, performance, capacity, and fiduciary risks of the executing agencies of proposed Bank pilot projects. No countries completed Stage III of the Bank’s use of country systems pilot. 11 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT No countries were unconditionally cleared to participate at the close of the piloting program. The second progress report in December 2010 admitted that the piloting program had proved challenging and had “not been a success” from the perspective of its original objective—use of country procurement systems in individual projects (World Bank 2010e). It was evident that no project using country procurement systems would be approved before the scheduled end of the program on June 30, 2011, and the Bank decided that the program would not be extended beyond that date. Of the four countries conditionally cleared to move to the project phase, only one, Brazil, was moving forward with Phase III at the time of management’s final progress report. African regional and country management teams elected not to proceed with project identification in Mauritius, Rwanda, and Senegal because of the uncertainty of the future of the program, and also so they could focus on the proposed new Program for Results agenda (Table 1.1). Table 1.1. Summary of Results of the Bank’s Use of Country Systems Pilot (2005–10) Particip- Participants Stage achieved ants (nos.) (countries) Countries cleared to proceed to project phase subject to CDAPs 4 Brazil, Mauritius, Rwanda, and Senegal Countries/Agencies that completed Stages I and II and were 3 Morocco, India Powergrid, the recommended to continue to pursue procurement reforms with support Philippines from the Bank Other Countries completing Stages I and II 2 Macedonia, Poland Countries that had only completed Stage I at the end of the UCS Pilota 6 Burkino Faso, Colombia, Ghana, Indonesia, Jordan, and Panama Countries pursuing other prioritiesb 2 Bhutan, Turkey Total 17 Source: IEG analysis of World Bank data on use of country systems pilot. Note: CDAP = country development action plan; UCS = use of country systems. a. Though Indonesia is included as a participant in Bank documents, its situation seems ambiguous. Indonesia conducted a MAPS exercise, though it was not formally a part of the UCS pilot. b. Bhutan’s small size and nature of engagement with donors prompted it to first focus on the harmonization of donor and country policies, rather than UCS; Turkey, as an European Union accession candidate country, opted to give greater emphasis to the harmonization of its procedures with those of the European Union, with support from the European Union under its SIGMA program. The second progress report acknowledged certain factors that contributed to the limited outcome of the pilot program, notably, “the uniqueness and specificity” of some Bank standards that were a particular obstacle to Stage II; high costs and uncertain benefits for participating countries and Bank staff; skills needs on system assessments; and in some countries, the inapplicability of the concept of a country system, especially in large federal countries, in the presence of variations between states, sectors, and implementing agencies. Yet the exercise had been a learning process. The methodology allowed the Bank to bring together baseline assessments 12 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT of multiple countries’ procurement systems in a single, comprehensive framework and exposed ways that diagnostic instruments, MAPS in particular, could be improved. The process of dialog with participating countries, other MDBs, and within different Bank departments was also valuable. IEG ANALYSIS OF THE BANK’S USE OF COUNTRY SYSTEMS PILOT IEG’s analysis of the Bank’s use of country systems piloting program aims to (i) review the quality of participating countries’ procurement systems, to understand to what extent Bank expectations could not be met; (ii) review Bank criteria for countries’ compliance under the piloting program, to assess how easy or difficult it was to satisfy those standards; and (iii) examine the extent to which the Bank could have provided more flexibility in these requirements in order to enable moving forward with the use of country systems experiment. This was undertaken by a detailed analysis of all diagnostic material emerging from the experiment, for each of the two principal stages—the Stage I and Stage II assessments—and a review of Bank Country Procurement Assessment Reports (CPARs) and loan documents in sample countries, to examine areas in which countries’ national procurement systems have not met Bank norms. Stage I MAPS Assessments—Review of Use of Country Systems Documents The Stage I analysis focused on scores of each country for the 54 subindicators of MAPS, in comparison to the minimum passing scores designated by the Bank’s piloting program (Appendix A, section on MAPS indicators and scores). Of the 16 countries that produced reports for the Stage I analysis, IEG calculated the average country scores for each indicator and the number of countries receiving passing scores. 15 Pillar I, the Legislative and Regulatory Framework, was both the strongest performing in absolute terms (with an average score of 2.4 across all subindicators for all countries) but also the weakest in terms of numbers of countries receiving passing scores—around 11 countries. This reflects the fact that Pillar I contained a disproportionate number of “core” subindicators requiring the highest score of 3 in order to advance in the piloting program.16 The second and third major pillars, Institutional Framework and Management Capacity and Procurement Operations and Market Practices, had average scores at the required average of 2.2, but only about 12 countries met this target—although some exceeded it. One area where there were significant discrepancies in Pillar III was with regard to contract administration and dispute resolution provisions. The Bank’s average passing score, at 2.7, exceeded the country average of 2.2, and only 9 13 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT countries passed on this score. Under Pillar IV, Integrity and Transparency of the Public Procurement System, the efficiency of access to appeals mechanisms was a difficult area, with a required average passing score of 2.8, and an achieved average of 2.1 (Table 1.2). Table 1.2. Bank UCS Pilot Participants—Average Scores for MAPS Pillars and Indicators Average number of Average Average countries with score passing score passing score (out Pillar and indicator received required of 16) Pillar I: Legislative and Regulatory Framework 2.4 2.6 11.1 Indicator 1: Public procurement legislative and regulatory framework 2.5 2.9 10.5 achieves the agreed standards and complies with applicable obligations. Indicator 2: Existence of Implementing Regulations and 2.3 2.2 11.8 Documentation. Pillar II: Institutional Framework and Management Capacity 2.2 2.2 12.1 Indicator 3: The public procurement system is mainstreamed and 2.2 2.0 13.3 well integrated into the public sector governance system. Indicator 4: The country has a functional normative/regulatory body. 2.5 2.5 13.3 Indicator 5: Existence of institutional development capacity. 1.8 2.0 10.0 Pillar III: Procurement Operations and Market Practices 2.2 2.2 11.3 Indicator 6: The country's procurement operations and practices are 2.0 2.0 12.3 efficient. Indicator 7: Functionality of the public procurement market 2.3 2.0 11.5 Indicator 8: Existence of contract administration and dispute 2.2 2.7 9.7 resolution provisions Pillar IV: Integrity and Transparency of the Public Procurement 2.2 2.4 11.3 System Indicator 9: The country has effective control and audit systems. 2.0 2.0 12.4 Indicator 10: Efficiency of appeals mechanism. 2.1 2.8 9.6 Indicator 11: Degree of access to information 2.4 2.0 14.0 Indicator 12: The country has ethics and anticorruption measures in 2.4 2.6 11.4 place. Sources: World Bank UCS country documents. Performance of individual countries was mixed, ranging from scores of 1.4 for Bhutan to 2.7 for Turkey and Morocco (Figure 1.1). 17 None met all required benchmarks for all subindicators, yet half (Brazil, Macedonia, Mauritius, Morocco, the Philippines, Rwanda, Senegal, and Turkey) had total scores greater than the total required score. For the three weakest subindicators, fewer than half of the countries met the required benchmark. 18 Notable difficulties were encountered in the area of nondiscriminatory participation and selection, where 5 of 16 countries received a passing score. 19 Issues clustered around two areas: state-owned enterprises and regional/domestic preferences, especially the former. In contrast to the MAPS standard of requiring rules for participation of state-owned enterprises to promote 14 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT fair competition, four countries (Ghana, Poland, Rwanda, and Senegal) allowed unrestricted participation, while the rules in a fifth (Turkey) were overly lenient. 20 Figure 1.1. Bank UCS Pilot Participants—Average Scores for MAPS Pillars and Indicators Sources: IEG analysis of World Bank UCS pilot country documents. Price preferences given to domestic supplies were another area of contention. The MAPS benchmark on domestic price preferences limited such preferential treatment to a reasonable amount (for example, 15 percent or less). A number of countries (Burkina Faso, the Philippines, Senegal, and Turkey) allowed higher levels of preferential treatment. Price preferences for domestic firms, minimum locally manufactured content, or required association or purchasing from domestic firms also served in some cases to lower scores. Another area of common failing was subindicator 1(h) on complaints arising from the failure to have a truly independent review body (Indonesia, Morocco, and the Philippines). Only 7 of 16 countries received a passing mark. IEG undertook a further analysis of the extent to which such issues are encountered in other Bank borrower countries, beyond those participating in the use of country systems pilot. IEG’s review covered all additional selected sample countries for this evaluation (Appendix Table A.8). Given that the Bank has sought to expand the use of country systems through the use of NCB, IEG reviewed recent projects documents as well as CPARs, in terms of their provisions for procurement under the NCB method. 21 According to Bank Guidelines, country procurement rules can be used under NCB as long as they are broadly consistent with the Bank’s principles. 22 In specific areas where local laws and practices do not comply, the Bank details measures to be taken to enable country procurement rules to be used. These modifications are typically included in the loan/credit agreements, and where applicable, in bid documents. 15 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT A review of such additional requirements by the Bank for the use of NCB revealed a list of nearly 40 problem areas (Appendix Table A.8). 23 Many were common to those that emerged from the pilot, although with somewhat more granularity and specificity. The most common areas requiring modification included the Bank’s prohibition on regional and domestic preferences, eligibility restrictions (including, for example, the need for foreign firms to enter into joint ventures with local firms), advertizing and free availability of bidding documents, time permitted for bid preparation, 24 the use of appropriate bidding documents, 25 and the award of contract to the lowest priced responsive bidder 26 without further negotiations. 27 Overall, the analysis confirmed that there are a number of areas in which many Bank client countries follow practices that are different from those of the Bank. Some of these may be more important than others, such as the principle of non-discrimination and the level of domestic preferences. Others may however be easier to accommodate, such as the Bank’s requirement for single envelope bid submission. Some derogations that allow greater use of negotiation are used elsewhere, though they are generally discouraged in the Bank. Stage II Equivalence Analysis—Review of Guidelines In Stage II, the piloting program sought to establish, in key areas of countries’ national procurement systems, equivalence to key features of the Bank Guidelines (Table 1.3). IEG prepared a mapping among key sections of the Bank Guidelines, corresponding policy requirements for the piloting program, and the evaluated performance of participating countries. Scores were assigned: 3 for areas where no problem was noted, 2 for areas where country policy deviated in a minor or easily remedied way, and 1 for a significant nonconformity. 28 Despite some overlap in subject matter (for example eligibility), the relative scores of countries varied from Stage I (Figure 1.2 and Box 1.5). A list of all areas assessed and their scores for conformity to the Bank Guidelines can be found in Appendix Table A.9. Table 1.3. Bank Requirements for NCB and Derogations in Select Countries No of reports No of countries reviewed reviewed where where the Issue the issue is Bank requirements for NCB is noted noted (35) (26) There may be no regional /domestic preferences regarding the sources of labor and 18 15 material. Prospective bidders must be allowed at least 30 days for bid preparation. 19 14 Appropriate standard bidding and prequalification documents must be used. 15 14 Eligibility cannot be restricted based on nationality of bidder and/or origin of goods. 17 13 Award must be made to the lowest evaluated qualified and responsive bidder. 15 13 16 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Minimum requirements must be explicitly stated in the documents. 12 12 Bidding opportunities must be advertised in the local press. 11 11 Bidders are not generally required to register with a local or federal authority except under 13 11 certain conditions. The procurement process cannot be cancelled, all bids rejected, and/or rebidding 13 11 conducted without approval. Public bid opening is required. 11 9 Bid evaluation criteria other than price may be allowed only if quantified in monetary terms. 8 8 Price negotiations may not be conducted with “winning” bidders prior to contract signature. 10 8 Parastatals may only be allowed to bid under certain conditions. 9 7 Bids may not be rejected based only on a comparison with the procuring entity's estimate; 11 7 invitations to bid shall not establish minimums and maximums. Audits and inspection of records related to bid submission and performance of the supplier 8 7 are permitted. “Two-envelope” bid opening procedure is permitted for procurement of goods or works 6 6 under specific conditions, notably if domestic law precludes use of one envelope. Foreign firms' eligibility cannot be conditioned on joint ventures with local firms. 7 5 Qualification criteria shall be applied on a pass/fail basis. 5 5 Extension of the time period to prepare bids may only be allowed under exceptional 5 5 circumstances. An inflation clause is recommended for contracts over a year. 7 5 Bidding documents are freely available. 7 4 Bid security shall be in the form of a letter of credit or bank guarantee from a reputable 5 4 bank. Joint venture partners must be jointly and severally liable. 6 3 There may be no restrictions on the means of delivery of bids. 3 3 Prequalification should be used only for large works projects. 5 3 Award must be published. 4 3 Bidding documents and contract shall include provisions on sanctions for fraud and 3 3 corruption. No preference may be given to suppliers or contractors based on region or locality of 4 2 registration, small size, ethnic ownership, and so forth. Source: IEG data compiled from loan agreements and CPARs. Note: NCB = national competitive bidding. A common problem area in Stage II, again, related to the restriction of participation of state-owned enterprises. Another problem area under eligibility was the lack of definition of conflicts of interest. Other issues that scored below 2.5 out of 3 included provisions on two-stage bidding and turnkey contracts; notification and advertisement; and examination, evaluation, and comparison of bids. 17 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Figure 1.2. Bank UCS Pilot Participants – Approaching the World Bank System Senegal Poland Bhutan Morocco Philippines Macedonia Colombia Rwanda Mauritius Brazil 2.4 2.5 2.6 2.7 2.8 2.9 3 IEG Scores Source: IEG analysis of World Bank UCS pilot country documents. Box 1.4. Stage II Equivalence Analysis – Bidding Documents Review Of the 11 countries for which IEG reviewed Stage II, 7 conducted a specialized bidding documents review. Four (Bhutan, Mauritius, Rwanda, and Senegal) had bidding documents modeled after World Bank standard bidding documents. Such documents for these countries were found to be acceptable (although with some exceptions and clarifications). Among the three others, one (Macedonia) had many problem areas; the other two (the Philippines and Poland) had limited issues that were comparable to countries with bidding documents modeled after the Bank’s. Bidding documents of all seven countries had some deviation from the Bank Guidelines/best practice, though many problem areas were not necessarily viewed as incompatible with the Bank’s requirements for the pilot program. Recurring issues included: • Differences in provisions on eligibility and sanctions. Firms sanctioned by the Bank were not necessarily excluded by the pilot countries (Mauritius, Poland, and Senegal). Descriptions of conflicts of interest in the case of state-owned enterprises (Macedonia, Poland, Rwanda, and Senegal) were also a problem area; sections on bidder’s nationality and related cross reference to eligible countries were frequently omitted. • Inadequate fraud and corruption provisions. There were no such provisions for Macedonia and Poland (although reports note that in Poland this is covered by the Polish Penal Code; in Macedonia, this is indirectly covered under other provisions), and they were lacking in detail in Rwanda. The definition of “obstructive practices” was different from the Bank standard in the case of Senegal and was missing in Rwanda. • Neglect to require statements of source of funds (Bhutan, Macedonia, and Poland). • Need for more comprehensive guidance concerning the use of bidding documents (for example, Macedonia and Rwanda). Source: IEG field visits. By the end, four pilot countries (Brazil, Mauritius, Rwanda, and Senegal) were cleared to proceed to the project phase subject to action plans (World Bank 2011f). Five (the India Powergrid, Macedonia, Morocco, the Philippines, and Poland) were not cleared after Stage I and Stage II assessments. Of these, the Bank suggested that Morocco, the Philippines, and the India Powergrid could continue to pursue 18 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT procurement reforms with Bank support. Six countries (Burkino Faso, Colombia, Ghana, Indonesia, Jordan, and Panama) had completed only Stage I at the close of the pilot program. And two (Bhutan and Turkey) completed the assessment process but chose alternative paths to meet their needs. In the end, even the four countries conditionally cleared to move to the project phase did not do so. Brazil was expected to move forward at the time of the final progress report but shifted course after deciding that the conditions imposed as a requirement for the use of country systems, which effectively created a hybrid, modified national procurement system, were too onerous. Meanwhile, because of a desire to focus on the proposed new Program for Results agenda and the uncertain future of the program, regional and country management teams elected not to proceed in Mauritius, Rwanda, and Senegal. USE OF COUNTRY SYSTEMS – THE EXPERIENCE OF OTHER DONORS The World Bank was not alone in the obstacles it encountered in the use of country systems experiment (Box 1.4). Retrospective looks at the movement toward this note the limited capacity of national procurement systems as a common theme (Pallas and Wood 2009; Ministry of Foreign Affairs of Denmark 2008) across donors. Institutional policies, fiduciary responsibilities, and concern for the speed of disbursement all contributed to the reluctance to rely on country systems despite the consensus of the Paris Declaration. 29 Interestingly, however, the OECD has noted that there is no clear relationship between the use of country systems and the quality of national procurement systems, suggesting that the greatest hurdle to greater use of country systems may be political instead of technical (OECD 2011c). The United States’ government’s posture, in the policy of its Millennium Challenge Corporation (MCC), is pragmatic: [T]he use of country systems for elements of compact implementation can deepen country ownership. However, MCC recognizes that the desirable goal of country ownership achieved through the use of country systems should not be pursued at the expense of program results or fiscal responsibility and accountability (Blue and Eriksson 2011). In this respect, MCC is further constrained by the pressure to finish projects and obtain measurable results within the five-year time frame of a compact. If host government inefficiency, lack of experience, or outright corruption weakens MCC’s ability to finish the job, MCC chooses to take greater control of implementation (Blue and Eriksson 2011). 19 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Box 1.5. Use of Country Systems—The Bank and the Larger Development Community Impetus toward increased use of country procurement systems started in the larger development community, to lighten the burden on recipient countries of navigating specific procurement rules of each donor along with their own system, ease cofinancing, and motivate countries to bring their systems up to an acceptable international standard, thus serving development goals by improving all government expenditures, not just those funded by donors. Several donors, including the World Bank, adopted policies to promote the use of country systems. Yet there was little consensus in terms of what use of country systems implied or how it would be implemented. The Organisation for Economic Co-operation and Development report on OECD: Percentage of Aid using progress in implementing the Paris Declaration Country Systems, 2005–10 reported that use of country procurement systems by donors overall increased from 40 percent in 2005 to 44 percent in 2010. For the World Bank, use of country systems rose somewhat more, from 42 percent to around 54– 55 percent. These statistics are, however, clouded by the Paris Declaration’s ambiguous and broad definition of the use of country systems.a Results obtained for the World Bank reflected, in part, its increased use of national competitive bidding (NCB) for those goods and services and in those markets that would be unlikely to attract foreign competition. However, under NCB, borrowers are required to make adjustments to be in compliance with the Bank’s guidelines (World Bank 2011e, 2007b). NCB as a proportion of procurement methods has risen over time as countries’ domestic supply capacities mature. Other forms of Bank procurement also make greater use of national procedures, such as direct contracting, the use of force accounts, and loans through financial intermediaries. The Bank’s new Program for Results instrument proposes to make intensive use of country systems, based on an agreed capacity building plan.b In addition, development policy lending by the Bank is effectively channeled entirely through national systems, and an increased share of development policy loans would be reflected in the above statistic. Sources: OECD 2011b; World Bank 2008, 2013d. a. Depending on the sample of benchmark countries, the increase could be measured as 54 or 55 percent. In terms of definition, the report stated only that “[c]ountry systems and procedures typically include, but are not restricted to, national arrangements and procedures for public financial management, accounting, auditing, procurement, result frameworks and monitoring.” b. The Program for Results, as it is commonly known, built in turn on the Bank’s Sector-wide Approaches and its output-based lending, both of which necessarily make larger use of countries own systems. See World Bank 2013f for a discussion. The African Development Bank (AfDB was similarly concerned that the “use of country systems creates additional risks of delays, poor procurement and inadequate 20 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT financial reporting” (AfDB 2011). It adopted a cautious approach to the use of country systems through a three-phase program. Phase I is based on the understanding that the use of country procurement systems is acceptable for all non-ICB contracts for goods and works if a country’s systems for domestic procurement are judged acceptable and provide for the progressive use of these systems. The AfDB has done a rigorous assessment for about 45 countries in Africa (representing approximately 97 percent of annual average approvals) and found their procedures (regulatory and institutional systems) acceptable for Phase I. AfDB does not require a change or waiver to existing policies (procurement guidelines), as it relies on existing provisions allowing use of NCB procedures, similar to those found in the Bank Guidelines. As fiduciary responsibility remains ultimately with AfDB, in Phase I AfDB applies its complaints redress policy. Phase II of the program moves to full use of national procedures, extending reliance on the regulatory and institutional systems of the country to include the judicial and complaint redress systems. Although these two phases are limited to non-ICB contracts, they will cover a majority of contracts financed by AfDB. Phase III will extend the program to full use of country systems for all contracts, including ICB. Implementation is not expected in the near term. The Inter-American Development Bank’s (IDB) guide for use of country systems lays out its approach (IDB 2010). It describes the criteria and procedures by which IDB assesses a country’s procurement system related to acceptability, conditions of use on IDB-financed contracts, and required improvements, if any. This program applies only to non-ICB procurement of contracts that are below the ICB threshold. A country is approved to use its systems through a waiver to IDB policies. 30 Accordingly, no changes to its policies were required. Country systems are assessed against MAPS requirements. Further requirements are added, where necessary, to elaborate IDB-specific needs (for example, where MAPS requires annual procurement plans, IDB elaborates plan contents, such as budget, schedule and method of procurement). The IDB’s stated objective is to use a country’s procurement system to contribute to project outcomes. It recognizes that the system may not meet all MAPS or IDB-specific requirements and may still be used, albeit with limitations. Moreover, it offers avenues for partial compliance and a progressive approach to increasing broader use. Noted flexibilities include: • It is not necessary to assess or approve the entire country system— subsystems or components may be approved. For example, approval might be limited to administrative entities such as national government agencies, state- owned enterprises, sectors such as infrastructure or health, or a particular state agency. It could also be limited to particular types of procurement, such as goods 21 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT and works or consulting services, or to particular procurement methods such as competitive tendering, but not other methods such as direct contracting. • It is possible to approve a system or component of a system if the intent of the requirement is met through other means. For example, a country may not have the required arbitration law, but other acceptable legislation or regulations may provide a legally valid tool for resolving contractual disputes. • Where a mandatory requirement is not met, it can be accepted conditionally. For example, if proper advertising is not mandatory under a country’s system but is done anyway, the system can be approved conditional on the practice being made mandatory, with the conditionality attached to the loan agreement. • Where nonapproval runs counter to the objective of contributing to project outcomes, a system may be approved with an attached requirement to address a deficiency. For example, if a required information system is seen as substandard, approval can be granted on the understanding that the information system is upgraded, possibly with IDB assistance. The IDB program of use of country is progressing. IDB informed IEG that as of July 2013, four countries (Brazil, Chile, Ecuador, and Mexico) had been approved for advanced use, and eight countries are awaiting approval for partial use. 31 The IDB and AfDB use of country systems programs contrast with the Bank’s pilot, which strove for full compliance to a set of criteria for the full range of procurement circumstances, including ICB (except consultant recruitment). Furthermore, with the exception of the assessment of India’s state-owned Powergrid, the Bank’s intention was to approve national systems for use on all projects in a country. IDB and AfDB avoided the “all-or-nothing” approach, striving to gradually move to full use of country systems though a measured process involving partial use, conditional acceptance, agreed improvements, and considerations of intent and objectives. Donors told IEG that the Bank led the MDBs in the endeavor to use country systems, that its methodology was detailed and rigorous, and that it evolved through extensive discussion with a number of stakeholders. However, its pilot appeared to many to set an overly high bar. This meant that in countries where official development assistance was not a significant portion of government expenditure, there was little incentive to engage in time-consuming dialogue and to possibly adjust systems to comply with Bank requirements. Such countries may have thought it more pragmatic to continue using their own systems for government expenditure and to ring-fence systems for Bank-financed projects. In contrast, countries where official development assisstance is high are typically those with weak capacity. Perhaps knowing that their systems had very little chance of being accepted, they had limited incentive to join the Bank’s pilot. 22 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT It was further observed that the Bank’s approach to the use of country systems had to balance conflicting interests. Although some countries felt that criteria were overly strict, the private sector in many developed countries was skeptical of the regulatory and judicial systems of many developing countries—they believed that their chances of winning international tenders would be lower if country systems were used. Donor countries gave mixed signals. It was not uncommon to see the bilateral arm of a country pushing for increased use of country systems, although its representatives to the MDBs advised caution. USE OF COUNTRY SYSTEMS—STAKEHOLDERS’ VIEWS ON THE WORLD BANK PILOT Among the countries evaluated for the Bank's pilot, three (Morocco, the Philippines, and Senegal) were part of the 11-country survey conducted as part of this review, although none fully qualified for the program in the end. Several other surveyed countries explored the possibility of participating in the pilots but did not. The reasons given, in case study countries, varied: • Bangladesh and Ethiopia did not think they were ready or thought that the requirements would be too difficult to achieve. Azerbaijan was judged by a recent CPAR as not ready. • Indonesia said that the pilot methodology was flawed and that it was too mechanistic and would not lead to a mutually acceptable harmonized system. • Two countries had other priorities at the time. Mexico wanted to focus on addressing recent CPAR recommendations, and Turkey wanted to focus on meeting European Union requirements. 32 Among the pilot countries, Morocco and the Philippines commented that requirements were too stringent—in the Philippines the four or five items of noncompliance were seen as minor or easily ring-fenced. Development partners in the Philippines further reported that, although the government had seen the pilot as a valuable exercise, it was disappointed by the outcome. The government invested significant financial and human resources and had an expectation of successfully moving toward full use of country systems, but the final decision-making process was opaque, and it was never fully explained why the Philippines did not pass Stage II. Morocco, the Philippines, and Senegal all said that the use of country systems should not be an all-or-nothing process, that it is highly possible to take incremental steps toward greater use of country systems even if countries could not adjust their systems in their entirety. If the objective of the pilot was the identification of countries where loans could be made that would be implemented entirely through country systems, the Bank’s approach was too onerous and detailed to be practical. However, it illustrated a 23 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT number of areas in which country systems do fall short of Bank standards and that would require some compromise in terms of the underlying principles of Bank procurement. Not all were of equal importance, and in some cases the issues that were identified could also be interpreted as country priorities (preferences for domestic providers, regions, specific suppliers). Other differences reflect countries’ use of modes of procurement that are little used in the Bank. The discussion shows that there is little consensus on the meaning of the use of country systems. In practice it appears that there is a continuum of use of country systems, rather than an absolute presence or absence of such use. USE OF COUNTRY SYSTEMS—STAKEHOLDERS’ VIEWS ON THE ADOPTION OF COUNTRY SYSTEMS Beyond the discussion of the Bank’s approach to its pilot, IEG reviewed a broad spectrum of stakeholder views on the more general theme of moving toward the use of country systems in Bank procurement. IEG found prominent differences between views of borrowers (government procurement offices, government counterparts, and implementing agencies) and private suppliers. Government Client Views From the perspective of most government clients, moving toward country systems has potential advantages (Box 1.6). First, it can ease the need to learn duplicate procurement process for implementing agencies. Learning one set of rules and processes instead of two saves human resources. Second, it eases layering in clearances, hence reducing delay. In some countries, national procurement offices oversee public procurement contracts and also give clearance. This leads to the need for two sets of “no objections” for executing agencies. Such clearance and oversight may also be delegated from a national office to a financial intermediary or other financing agent, which leads to similar layering of “no objections.” Third, it is pointed out that thanks in no small measure to support by the Bank over preceding decades, in many client countries the procurement regime is already very similar to that of the Bank. Yet there were caveats. Country systems themselves may not be homogenous, especially in federal governments, as pointed out in Mexico and Ethiopia. Although the majority of Bank-financed procurement occurs at the federal level, the use of country systems ultimately must take into account state regulations. Ideally, federal and state procurement legislation should be harmonized but typically the federal level has no jurisdiction over state procurement. In Mexico, among 32 states, only 2 apply federal procurement legislation. Ethiopia’s 13 provinces each have their own procurement regimes. Some clients perceive that suppliers are more responsive to tenders under Bank guidelines, especially in ICB, and there is the perception that 24 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT maintaining some procurement under the mantle of the Bank would give a competitive advantage. Box 1.6. Using Country Systems—Benefits Perceived by Client Implementing Agencies Especially in countries where Bank investment lending has been a declining part of its support, as in Mexico, even institutions well versed in national procurement legislation have to learn a new “procurement language” for a very small part of the procurement they perform, leading to inevitable confusion and delay. In Morocco, a Bank client starting its first project financed by the Bank mentioned that the necessity to mobilize eight people dedicated to a Bank project within its organization and to train them to apply the Bank’s procurement system represents a heavy burden. Bangladesh Bank staff noted that using country systems would reduce the requirement for procurement agents to learn two systems. Similar concerns were echoed in Ethiopia, Senegal, and Tanzania. Tanzanian clients pointed out that capacity constraints in terms of learning two systems are compounded in projects with several implementing agencies (an example was a project involving 18 municipalities and 6 sectors). Requirements to obtain no objections from the government tender board and from the Bank added significantly to processing time. Senegal also noted that the double prior review process adds time and cost. In Indonesia, implementing agencies expressed support for greater use of country systems to reduce confusion and limit procedural delays. Source: IEG procurement country surveys. Private Sector and Supplier/Contractor Views Contrary to views of Bank country staff and government stakeholders, private sector contractors and suppliers clearly favored the use of Bank systems over country systems, as they perceived them to have embedded safeguards. Suppliers point out that despite frustration with the implementation processes of the Bank, the Bank Guidelines are broadly acceptable (albeit with some areas where modifications could be useful, as discussed earlier in this chapter) and help maintain a level playing field. Reasons private agents offered included greater transparency and integrity, more timely payment, and effective recourse (Box 1.7). Some suppliers also claimed greater administrative burdens in government processes. Box 1.7. Using Country Systems—Drawbacks Perceived by the Private Sector In Morocco, the private sector expressed confidence in the Bank’s procurement system, saying it provided assurance of timely payment, transparency and integrity, and a recourse mechanism. Companies emphasized that the Moroccan system lacked an independent complaints procedure. In Peru, the predominant view was that efficiency, economy, and 25 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT transparency would suffer if Peru’s procedures were applied. Similarly, in Turkey, private firms, including national consulting firms, expressed their preference for the Bank’s procurement system and believed local systems overemphasize price. They consider the administrative requirements of the national system to be burdensome (especially for small firms); procedures are not always transparent; and the handling of complaints protracted, with dispute resolutions lasting years. There was also the issue of debarred firms that particularly affected national consultants; compared to the limited number of firms debarred under the Bank, national lists are very lengthy, as they include firms debarred for performance reasons, such as delays. Local consultants prefer to bid on contracts that fall in sectors outside the scope of the Public Procurement Law – such as the defense sector. International consultants reported abstaining from participating in competitions conducted under Turkish regulations. The private sector in Senegal also did not favor the full use of country procurement systems, citing the lack of transparency (especially in the light of the changes made to the scope of the procurement code by the government in 2011) and risk of fraud and corruption because of the lack of an independent and efficient complaint mechanism and court system. There was also concern about local audit capacity and the extent to which it could effectively ensure the integrity of the system. Although contracts are between a supplier and the government and therefore should not be affected, there is a perception that the Bank has powers of persuasion when its own systems are applied, which may be reduced if country systems are used. Source: IEG procurement country surveys. Conclusion—Partial Use of Country Systems Countries questioned whether the issue is Bank systems versus country systems per se or the implementation of such systems. In several countries, the transition to the country system is not considered difficult, as the differences are typically minor. The more critical challenge is ensuring integrity, minimizing interference, and achieving governance outcomes—the principal issue is reliance on country governance systems rather than focusing on procurement systems (identified as being the procurement procedures and rules). In Bangladesh, stakeholders stressed the importance of Bank oversight and its contribution to contract formulation and execution; they also saw it as reducing political interference and bribery. Greater clarity is needed on the extent to which the Bank would be able to maintain oversight in the procurement process, if country systems are to be used. Currently, Bank Guidelines/bidding documents/contracts give the Bank the right to inspect and audit. Bank management has maintained that Bank legal remedies—for example, declaring misprocurement—would still be available, and also that existing rights and 26 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT obligations with regard to fraud and corruption would not be altered. Yet it is not clear how complaints, dispute resolution, or observance of sanctions lists would operate, or whether the Bank’s role in moral suasion, for example, to ensure timely payment, could still be exercised. Given also that in practice many countries have multiple systems, in many areas—as has already occurred with NCB—the Bank could adopt a more flexible approach, making incrementally greater use of country systems, though continuing to work with countries as necessary to uphold best practice. In particular, the Bank could scrutinize the areas of difference that are observed systematically across different countries and prioritize those that could benefit from more flexible approaches by the Bank, compared to those where, on the basis of the Bank’s principles, further dialogue with countries and further capacity building may be desirable. Upstream and Downstream Involvement in Procurement ADEQUACY OF UPSTREAM INVOLVEMENT One of the propositions explored in Bank management’s preliminary vision of a new procurement policy is the need for greater Bank involvement in both upstream and downstream aspects of the procurement process. IEG’s exploration of stakeholder views concerning the need for greater involvement first reviewed perceptions of current levels of Bank upstream involvement, in the planning, design, and procurement preparation phase. Bank upstream engagement was positively perceived in most cases, especially in terms of the provision of advice on the preparation of terms of reference, technical specifications, and the use of the Bank’s standard bidding documents (Table 1.4). 33 More involvement would have been welcomed in some instances (for example, in Azerbaijan, because of the limited capacity of some project implementing units and consequent difficulties preparing technical specifications, drafting bid documents, and so forth). Such problems were compounded in complex areas such as information and communications technology (ICT) (Azerbaijan and Turkey). There were some client references to limited Bank staff capability. These frequently referred as much to task team leaders as to procurement staff. In Ethiopia, task team leaders were sometimes perceived as having limited technical knowledge and unable to support clients in the writing of technical specifications needed for bidding documents. 27 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Table 1.4. Bank Support During Procurement Design and Planning Proc Bank Country Question Avg staff TTLs clients In terms of project preparation, how effective is the Bank's procurement design 3.4 3.2 3.4 3.5 /planning process at contributing to overall project outcomes? With respect to the Bank's requirement that the borrower prepare a procurement plan 3.6 3.4 3.8 3.6 that is updated regularly throughout the project, to what extent does this requirement contribute to procurement outcomes? Source: IEG questionnaire. Note: 1 = negligible; 2 = modest; 3 = substantial and 4 = high. TTL = task team leader. Upstream involvement at the acquisitions phase need not be only in terms of design details for individual contract packages. It refers also to the entire procurement planning process, and at an even higher level, the integration of the procurement function in overall budget management and the public finance architecture, where, so far, the Bank’s role has been limited. 34 DOWNSTREAM INVOLVEMENT—ROLE IN CONTRACT MANAGEMENT Some clients voiced concerns that current Bank involvement in contract management may not be enough, especially because of limited country capacity to appropriately supervise contract management. In the Philippines, Bank procurement staff pointed out that procurement constitutes “asset acquisition”; for this reason, they felt that procurement input is required beyond contract award and up to the point that works are turned over or goods are delivered and installed. In Bangladesh, procurement staff noted the need to support contract management if there are payment issues, contract termination issues, and the like. Support was expected from Bank task team leaders, Bank technical experts, or consultants, as much as from procurement staff. Clients attributed limited Bank engagement (in Bangladesh, Ethiopia, Peru, Tanzania, and Turkey) to limited task team leader time, skill constraints, frequent rotation, and sometimes inadequate engagement. Bank task team leader supervision was deemed ‘’spotty” or “nominal” at best and in some countries was seen as restricted to semiannual review missions. Limited team leader contributions were thought to partly reflect limited incentives for more task team leader engagement in downstream supervision aspects, especially those involving procurement. Although there is a procurement accreditation option for team leaders, interest is limited because of the lack of rewards or incentives. There is some perception that accreditation may conversely lead to increased responsibilities as well as increased exposure to risk, with limited additional returns. 28 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Bank support to implementing agencies in the postcontract award period was considered necessary to seriously combat fraud and corruption in project implementation. In Indonesia, where fiduciary risks were a continuing matter of concern, there is a view that although risks during the procurement process can be addressed to some extent, risks during contract management remain. The scope for large-scale fraud and corruption during project implementation is exacerbated by inadequate record maintenance, poor tracking of construction quality, absence of quality assurance plans, and inappropriate contract variations, which may be caused by incomplete design, inappropriate specification, delays in processing payments, or collusive practices. It was also claimed that when procurement ends with contract award, limited integration of internal controls implies that much important information pertaining to contract management (such as actual quantities of goods delivered, actual amounts paid, change orders, revisions, cancellations, and so forth) is not captured, resulting in limited knowledge of actual results. Bank procurement staff point toward a need for guidelines to staff on contract management, to ensure adequate controls and avoid reputational risks (Table 1.5). The Bank could prepare project management manuals to help executing agencies or require them to develop suitable procedures. Mention was also made in some countries, such as Bangladesh, that extending the period of engagement to upstream and downstream phases needed to be considered in tandem with adjustments to the nature of Bank engagement. The Bank could take a more mentoring than clearing role, with more hands-on engagement. Table 1.5. Contract Management and Implementation Country Bank proc Bank Question Avg mgmt staff TTLs Once a contract is awarded, is there still a need/role for a procurement staff or 2.8 2.9 3.1 2.4 accredited staff to oversee contract management /implementation? Or are TTL skills adequate? Source: IEG questionnaire. Note: 4 =s trong need; 1 = negligible need. TTL = task team leader. These discussions reflect some of the tensions concerning the role of the procurement function of the Bank in downstream areas of procurement implementation. Although there clearly are perceived needs for such support, it is not clear whether or to what extent the support should be provided by Bank procurement staff, Bank task team leaders, or technical experts, or by national institutions that can provide support; it is also unclear whether hands-on (project implementation agencies) are preferred or through oversight functions (such as a national audit office). 29 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Among Bank staff, there is a need to review the present incentive structure and to consider the nature of the engagement as much as its duration. In addition, there are legal issues. Contracts are made between a borrower/implementation agency and contractor, and the Bank is not a party to this. Thus, the role and extent of the Bank’s procurement function will need to be considered relative to the roles of other agents in the country. In Bangladesh, for example, procurement staff opined that more engagement with contract management represents an oversight and advisory function for national execution, rather than a transfer of responsibility to the Bank. Most other countries and international documents do not address contract performance or consider it a part of procurement. 35 The Bank could also fortify its support to countries (currently offered on a limited scale through the World Bank Institute [WBI]) to develop civil society oversight of public procurement and contract execution. Developing Domestic Suppliers and Offering Domestic Preferences PRESENT PROVISIONS FOR DOMESTIC PREFERENCES As stated in the Bank Guidelines, for decades the Bank has seen procurement as a tool to encourage the development of domestic contracting and manufacturing. It is one of four considerations that guide current procurement rules and procedures. Although provisions for domestic preference are built in to the guidelines, they allow domestic preference only for ICB and only for goods and civil works. For goods, borrowers may give a price preference of 15 percent to domestically manufactured goods. 36 For civil works, in qualifying countries (those with per capita income of $1.25 or less), domestic contractors may be given a bid preference of 7.5 percent over foreign bidders. Analysis undertaken by the Bank’s procurement anchor found that domestic preference provisions are seldom used, and when they are used, they seldom determine the outcome of bidding (World Bank 2012o). From FY99 to FY09, this provision was used on 0.3 percent of contracts (0.4 percent by value) and was estimated to have potentially affected the outcome of the bidding on only 12 of 57,000 contracts. Bank management points out that the use of domestic preferences is at the discretion of the borrower and the executing agency and may not always be invoked because of its potential for inflation of the bid price. A background paper (World Bank 2012o) concludes that it is doubtful that the domestic preference policy continues to benefit local industries; it also points out that any preferential 30 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT treatment, by definition, discriminates among suppliers, which goes against open competition and has the potential to be divisive. Rather, the Bank could focus on helping governments to encourage domestic industry and develop mechanisms, like e-procurement, to enhance transparency and fairness and lower costs for all bidders. Capacity building efforts thus become even more paramount. BROADER INTERPRETATION OF DEVELOPING DOMESTIC SUPPLY CAPABILITY IEG’s 11-country survey polled stakeholders on the usefulness of domestic preferences to client countries. Among Bank management, results varied by country, with more developed countries considering this less important. Domestic preference, as provided in the Guidelines, was considered substantially or highly justified in poorer countries: Ethiopia, Indonesia, and Tanzania. In contrast, justification was considered low in more advanced countries such as Peru and Turkey. Bank procurement staff in the field and task team leaders see some form of domestic preference as worthwhile, even though it is little used because of a sense that there are sometimes developmental and/or project-specific benefits to be derived from favoring local contractors and manufacturers. Clients point out that the Bank’s current domestic preference scheme appears ineffective, as it is narrow in scope and thus limited in relevance. In Indonesia, attention was drawn to differences between the Bank’s domestic preference scheme and that found in national procedures. Bank Guidelines do not allow preferences under NCB or methods of procurement other than ICB, and bidding must be open to all qualified firms, whether local or foreign. By contrast, the Indonesian preference scheme excludes foreign bidders for contracts below a specified value ($5 million for works, $1 million for goods, and $0.5 million for consulting services). In other countries, contractors are ineligible under Bank policy because of the country income threshold, though they are eligible under national policies. Other aspects of domestic supply were discussed in Bangladesh, Ethiopia, Indonesia, Tanzania, and Turkey, although they were outside the purview of the Bank’s current domestic preference policy. With regard to consultant contracting, it is perceived that the Bank’s usual insistence on securing geographic diversity (that is, not more than two bidders per country in consultant contracting) is considered to limit good development outcomes in countries where there are strong local consultant capabilities. Respondents think that the geographic diversity requirement screens out capable local bidders and thus discriminates against domestic consultants. Foreign consultants often do not bid in markets with good domestic capability, as 31 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT they fear that their price will not be competitive. Unintended consequences are less total competition and the screening out of qualified domestic firms. Management points out that there are flexibilities for using alternative shortlisting methods, or for including national experts explicitly in the scoring process. But these appear to be little known and little used by Bank staff and clients. Another unintended consequence is that domestic firms sometimes associate with foreign firms to meet geographic requirements, when they could have expressed their interest directly, at lower cost, without the foreign partner. Ethiopia, Indonesia, Tanzania, and Turkey all provided examples of such cases. Domestic consultants that have won joint venture contracts in such situations maintain that they then undertake the majority of the subsequent work. Country respondents suggested that the Bank drop the two-firm per country limitation, allowing more domestic consulting firms to bid. They also favored dropping the six-firm limit on the short list, to allow for more overall competition. As many borrowers apply preference schemes within national systems (Box 1.8), the Bank needs to understand the objectives and mechanisms involved, especially as the Bank moves to greater use of country systems. A broader review of the issue of encouraging domestic suppliers would have merit, not limited to the scope of the current and narrowly defined domestic preference policy. It would appear that there could be scope for more broad-based support to local suppliers, while taking account of competition and open markets. Box 1.8. Perspectives on Domestic Preference Internationally, different instruments may guide governments in conducting public procurement, including provisions for domestic preference, although this is one of the more controversial aspects of procurement policy. In addition to the Bank Guidelines, the UNCITRAL Model Law allows evaluation criteria to include a margin of preference for the benefit of domestic suppliers or contractors or for domestically produced goods. It also allows for any other preference, if authorized or required by procurement regulations or other provisions of state law. However, the World Trade Organization’s Government Procurement Agreement and European Union Directives do not allow such preferences. They maintain the principles of nondiscrimination and equal treatment among members and thus do not allow preferences (with some very narrow exceptions). Both the Agreement and European Union Directives say that socioeconomic considerations cannot be used as 32 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT barriers to restrict access to bidders of member states. As reported in a background paper prepared by the Bank as input to its Procurement Policy Review (World Bank 2012o), many countries pursue domestic preference objectives (such as development of small and medium-size enterprises), though their procurement policies vary. These findings are based on a survey of the experiences of the European Union and other countries (Australia, Canada, Chile, Japan, Malaysia, Mexico, South Africa, the United Kingdom, and the United States). The greatest number of formal preference and technical assistance programs are in Australia, Canada, Malaysia, South Africa, and the United States; they target women, indigenous populations, racial and ethnic minority populations, disabled populations, and small and medium-size enteprises. The tools used include explicit preferences such as set-asides for specific groups of vendors, award preference during the bidding process, and locale-specific targeting of preferences within specific regions of a country. Proposed new changes to the European Union Public Procurement Directive make several suggestions to promote small and medium-size enterprises, which could also be useful for the development of domestic suppliers: removing barriers such as turnover or years-of-experience requirements, simplifying bidding requirements (for example, allowing the use of self-declarations), packaging procurement to make it easier for small and medium-size enterprises to apply, and direct payments to subcontractors (Appendix A, last section). Sources: World Bank 2012o, IEG 2013. Environmentally Responsible Procurement—Potential for Bank Use There is no consensus on what environmentally responsible procurement—sometimes referred to as “sustainable procurement” or “green” procurement—constitutes. One Bank background paper defines green procurement as "the purchasing of products that provide environmental and related socioeconomic benefits" (World Bank 2012f). This broad definition would suggest any procurement practices geared toward limiting pollution or toxicity, recycling, increasing energy efficiency, protecting biodiversity, assessing carbon footprints, and possibly more when considering social benefits. The background paper reports life-cycle analysis as a way of capturing environmental costs and benefits through the procurement process, adding that a rigorous methodology for tracking overall fiscal costs is lacking. IEG points out that it is especially difficult where choosing the greenest product does not lead to the lowest price, that is, where the choice cannot be quantified or justified 33 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT monetarily. Ecolabeling can be used to specify goods that meet certain criteria, but problems exist with agreed standards. 37 Although green procurement is very much discussed, there appear to be challenges in definition and methodology and thus in developing commonly accepted policies and practice. BANK POLICIES, PRACTICE, AND EXPERIENCE The Bank does not have an explicit policy on green procurement. Bank Guidelines do not mention it. Indirectly, however, environment-related provisions can be found in the Bank’s standard bidding documents. Evaluation criteria provide for bid price adjustments based on life cycle costing, for example. Such criteria can be used to specify energy efficiency requirements and could be extended to other “green” specifications. The form of contract based on International Federation of Consulting Engineers (FIDIC) contracts includes a clause on protection of the environment. It reiterates that contractors must comply with environment-related technical specifications and are further bound by local laws (that may require environmental clearances, for example). These measures provide for inclusion of environmental concerns in the procurement process in practice, although without any guiding policy requiring strategic use of procurement to meet environmental or other socioeconomic objectives. IEG's 11-country survey found limited views on the topic. In 10 countries, Bank procurement staff responded to a question on how effectively the Bank’s procurement regulations and practices contribute to government policies related to green procurement. None said it was high, though in three countries (Azerbaijan, Bangladesh, and Tanzania) staff said it was substantial. The remaining seven responding countries said it was modest, low, or negligible. As the Bank has no explicit policy on green procurement, IEG assumes that these views derive from practices, perhaps related to requirements for environmental impact assessments or to bidding and contract provisions noted above. In the Philippines, the only country where clients raised the topic, it was said that green procurement was a policy area that the government wanted to pursue but that the Bank had shown little initiative. The Bank's procurement practices accommodate some of what is seen as necessary for effective procurement of environmentally friendly services and products. Put differently, although there are areas in need of strengthening (public-private partnership [PPP] contracting, for example), there is nothing in the Bank's policies or practices that prohibits employing life cycle costing or using environmentally efficiency technical specifications. Box 1.9 provides examples of Bank projects where 34 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT such factors are incorporated. Bank practice accepts imposition of national environmental laws in technical specifications and contract terms and conditions (FIDIC based). This relates mainly to regulations on mitigating environmental damage during construction. It is not clear if it would extend to imposing environmentally friendly regulations. IEG suggests more specificity in Bank policies (procurement or environment) on how national regulations are to be treated in this regard in Bank-funded procurement. Box 1.9. Procurement and Energy-Efficient Products and Services Two Bank publications evaluate the role of procurement to address energy efficiency—the first deals with obtaining energy efficient services, the second with obtaining energy efficient products. The first (World Bank 2010g) focuses on contracting with private sector energy service providers. Issues raised, primarily related to long-term contracting for privately provided services, are similar to those found in PPP procurement (Chapter 2); IEG supports adjustments that help achieve better outcomes. With respect to procuring energy-efficient products, a later publication (World Bank 2012f) recommends greater use of labels or certification, preferably shown to be more economical through life-cycle cost analysis. Life-cycle costing is accommodated in current Bank procurement practices through the bid evaluation criteria found in the Bank’s standard bidding documents. However, both publications recommend explicit policies that mandate procurement of energy efficient goods and services, as found in many national jurisdictions (the U.S. General Service Administration Acquisition Regulations, for example). Bank policies do not do this, and do not provide guidance where procuring the greenest product does not lead to lowest cost. Specifying products based on energy-efficient labeling or certification is controversial, as it must consider the stature of the certifying authority. The Bank accepts internationally recognized certification requirements as a valid technical specification, but in many instances requires “or equivalent” to be added, to not bias procurement to a particular supplier or country of origin. UNCITRAL, the World Trade Organization Government Procurement Agreement, and European Union approaches provide guidance on working within internationally accepted norms that consider fairness, competition, and trade regulations. Sources: World Bank 2010g, 2012f. INTERNATIONAL PERSPECTIVES Different international instruments can guide governments in the conduct of public procurement, including provisions related to socioeconomic and environmental 35 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT concerns. 38 Both the Government Procurement Agreement of the World Trade Organization and European Union Directives allow for specific reference to environmental factors in technical specifications, evaluation criteria, and contract conditions. Because of their trade-facilitation orientation, the Government Procurement Agreement and European Union Directives say that socioeconomic considerations cannot be used as barriers to restrict access to bidders of member states. However, the UNCITRAL model law recognizes and accepts socioeconomic policies in national legislation. These are defined as “environmental, social, economic and other policies authorized or required by the procurement regulations or other provisions of law.” Reviewing the environmental policies and policy frameworks of some European Union countries (mainly Finland, Norway, and Sweden) and six other countries (Australia, Brazil, Canada, Chile, Mexico, and the United States), IEG notes that all countries have policies that attempt to use procurement to achieve energy efficiency and environmental protection objectives. Two features stand out: the use of environmental standards (ecolabeling) in technical specifications and incorporating green considerations into the procurement process via life-cycle cost/benefit analysis. These features maintain the principles of competition and contract award based on technical compliance and most economically advantageous offer, while at the same time using procurement strategically to address environmental policy objectives. Box 1.10. Asian Development Bank—Loan Agreement Provisions on the Environment The ADB guide(ADB 2007) requires that its loan agreements include the following clause: …The Borrower shall ensure that the Goods, Works, and Consulting Services procured under this Loan Agreement are produced/rendered in a responsible manner with a view to resource efficiency, waste minimization, and environmental considerations, as set out in the Environmental Management Plan, Initial Environmental Examination, or other Project-related documents, as applicable. In this connection, the Borrower and the Project Executing Agency shall furnish, or cause to be furnished, to ADB all such reports and information as ADB shall reasonably request to verify compliance…. Source: ADB 2007. The Bank was a member of a working group on environmentally responsible procurement, through the Heads of Procurement of the MDBs. The working group did not produce harmonized guidance, although the ADB proceeded to prepare a guide for ADB use (Box 1.10). 39 The ADB guide works within the existing policy framework- policies related to procurement and to environment. It does not propose policy 36 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT changes, but guides staff on how to introduce environmental considerations into the procurement process, within existing policies. Its two main points refer to the incorporation of environmental standards (ecolabeling) in technical specifications and the use of life cycle cost analysis to factor environmental benefits into bid evaluation. Thus it appears that the ADB has integrated environment-related procurement matters into its project design and formulation more explicitly than the Bank. Achieving Value for Money in Bank Procurement Achieving value for money is intended to be a cornerstone of the Bank’s new procurement policy. The Bank’s ongoing procurement review proposes a shift to value for money principles, although details as to what this would involve have still to be spelled out. IEG’s review of the application of value for money in other jurisdictions provides some guidance to factors that the Bank could take into account. Although there is no commonly accepted definition of value for money in public procurement, a typical one would be “what a government judges to be an optimal combination of quantity, quality, features and cost, expected over the whole of the project’s lifetime” (Burger and Hawkesworth 2011). This definition incorporates the widely accepted elements of quality, features, and life-cycle costs. Beyond this, value for moeny is frequently broadened to include social and environmental externalities or factors such as employment creation, support for vulnerable groups, or local content. 40 Such definitions are difficult. What is of value to one person or government may not be valuable for another. The core issue is to determine what bid evaluation factors may be taken into account; especially controversial is the matter of how to apply nonprice factors. Strategies to introduce value for money concepts to public procurement must consider a number of sometimes competing factors: • Effectiveness of markets to deliver desired results related to cost, quality, innovation, and risk, considering such things as capacity of the procuring agent and levels of competition • Trade policy, in particular its relation to international competition and preference schemes, and alignment of procurement policies and practices with internationally accepted norms • Public accountability requirements of transparency and fairness • Methods to assess life-cycle costs and benefits, especially when imputing environmental and socioeconomic considerations • Mechanisms to control fraud and corruption. 37 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Austrailia, the European Union, the United Kingdom, and the United States are four jurisdictions that have taken a long-term and systematic approach to integrate VfM concepts into their public procurement practices (Appendix A has details on the application of value for moeny in six public jurisdictions). 41 The U.S. “best value” approach also addresses the matter of incorporating non-price factors to achieve a best value outcome. The European Union’s proposed updates to its public procurement framework further emphasize the role of value for money (Appendix A) with more explicit instructions on life-cycle costing, as well as the incorporation of social, environmental, and labor laws. Additionally, the Asia-Pacific Economic Cooperation forum (which includes 21 Pacific Rim countries) has endorsed value for money concepts since 1999, and the International Atomic Energy Agency is one United Nations agency that has adopted these principles (IAEA 2011). IEG has reviewed the application of value for money in each of these jurisdictions (Appendix A). Based on IEG’s review of how it is applied in other jurisdictions and agencies, the following observations are offered on how the Bank’s current policies and practices align with value for money concepts as practiced by others, and where there is scope for modifications in practice, if this practice, in its narrow sense, is to be reflected. APPLYING VALUE FOR MONEY IN THE WORLD BANK Accommodation of Nonprice Factors in Procurement Decisions With value for money, as defined by the European Union, MEAT (Most Economically Advantageous Tenders) directives procurement may consider such factors as quality, technical merit, aesthetic and functional characteristics, environmental characteristics, running costs, cost-effectiveness, after-sales service and technical assistance, delivery date, and delivery period, in addition to price. These are typically evaluated through life-cycle costing and present value assessments of costs and benefits. The Bank’s procedures, which prescribe selection of the lowest evaluated bid, provide for this, and it is commonly done (Box 1.11). For example, technical specifications described in evaluation criteria can stipulate required environmental characteristics, and criteria can be developed to take into account operations and maintenance costs over the life of a facility. 38 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT Box 1.11. Incorporation of Maintenance Costs and Life Cycle Costing in Bank Procurement Two projects in Turkey provide examples of the incorporation of certain value for money principles in present Bank procurement, including life-cycle costing for the evaluation of energy-efficiency technical specifications. • Supply and Installation of Submarine Power Cable: Technical specifications for energy losses and for five years servicing costs have been built into the financial evaluation criteria for the underwater cable, thus providing examples of how life-cycle efficiency and long-term maintenance costs have been taken into account. • Construction of Natural Gas Storage Facility: The project incorporates a 15- year life-cycle cost analysis by imputing the present value of operations and maintenance costs in the bid price. Source: World Bank procurement documents. Although the Bank’s policies and practices (as found in the standard bidding document) do not preclude this aspect of value for money, there is no explicit recommendation to exploit the potential benefits offered; stronger direction could be given to staff and borrowers to reflect the practice within the Bank’s procurement system. Expanded guidance is also required on how to incorporate factors into a quantifiable cost benefit analysis and further on how to accommodate nonprice factors. The Bank’s requirement to award contracts to qualified bidders that submit technically responsive bids and whose offered price is the lowest evaluated does not allow consideration of non-quantifiable costs and benefits. Generally speaking, for procurement of goods and works, bid evaluation factors must be quantifiable in monetary terms (an exception relates to ICT procurement, where weighting of technical offers means that contracts may be awarded to higher priced offers that are scored as technically “better”). Accordingly, technical offers are assessed for compliance, and among all compliant offers, the contract is awarded to the firm offering the lowest evaluated price. Evaluated price may consider various factors (operations and maintenance costs and energy-efficiency benefits, for example), but these factors must be financially quantifiable. Technical compliance provides further value for money opportunities, as does the process to qualify bidders, either in advance of bidding (prequalification) or as part of the bid evaluation process (postqualification). For example, it is common practice to require a supplier to have been in business for a specified number of years, have a minimum annual turnover, and be solvent. Qualification requirements may be further extended to the manufacturer (who is not necessarily the bidder). Common 39 CHAPTER 1 BANK PROCUREMENT AND DEVELOPMENT IMPACT examples include requiring that the goods must have been in manufacture for a specified number of years, perhaps at a specified volume, or that the factory that produces the goods must have an operational history, and so on. Such measures are effective at ensuring quality and mitigating supply chain risks. However, beyond these examples, the Bank’s requirements do not easily accommodate nonquantifiable criteria—for example, the accommodation of merit points or other scoring systems— where the assignment of weights given to different factors could be arguable and where there is increased need for judgment in bid evaluation. It is sometimes questioned if these procedural approaches can truly achieve desired value for money outcomes, especially in complex procurement. Such outcomes are implicitly assumed to follow from, first, an appropriate specification of contractual requirements and, second, from the prescribed bidding and selection process for the preferred supplier. Expecting contracting authorities to comprehensively specify all aspects of high-value, high-risk projects in advance of any selection process may not be realistic: “Contracting authorities that carry out particularly complex projects may without this being due to any fault on their part find it objectively impossible to define the means of satisfying their needs or of assessing what the market can offer” (Kruger 2009). As noted, some jurisdictions extend nonprice factors to preferences for certain bidders and products (domestic firms, locally produced goods, firms owned by ethnic minorities, and so forth). Although this broader definition of value for money is not discussed here, a summary of experience in with regards to certain broader aspects is provided in Appendix A. 40 2. Adapting to Evolving Needs As pointed out by Bank management, at the time of introduction of the Bank’s current procurement policy, Bank lending followed a different model, focusing largely on infrastructure financing (World Bank 2012c). Client countries’ financing needs have since changed, and so have patterns of Bank lending. Are the Bank’s present procurement modalities equipped to deal with the evolving pattern of its lending? This chapter selectively explores some new directions of lending that present particular challenges for procurement, community-driven development (CDD) lending, lending in fragile and conflict-affected situations (FCS), lending for the acquisition of ICT, and lending for PPPs. In each area, concerns have been raised about the extent to which present procurement modalities are appropriate. Overall, IEG finds that in CDD projects and FCS situations, Bank Guidelines and processes per se are not problematic. Difficulties appear to stem from limited capacity among country staff, limited use of permitted flexibilities, and, for FCS, difficulties in the overall enabling environment. But in the more complex and newer areas of ICT lending and PPPs, although the Bank has made recent efforts toward incorporating the complexities presented by these challenging new areas, much remains to be done to align Bank procurement with prevailing best practice. Procurement in Community Driven Development Projects • Almost a quarter of Bank projects have some CDD elements, according to present nomenclature, and these appear to have a higher incidence of procurement issues than other projects. • Looking more specifically at a subset of projects that actually deploy CDD methods—that is, community management of resources—procurement does not appear to be unusually problematic, in terms of difficulties in implementation. • Reported problems have frequently related to the non-CDD components of those projects, reflecting difficulties in procuring (centrally) the consulting services and technical assistance needed to support the implementation of the CDD components. • The Bank Guidelines appear to have appropriate flexibility for such projects. Bank front line staff have been timid in the use of flexibility, because of an internal culture of risk aversion. 41 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • Although there may be potential for fraud and corruption in CDD projects, in view of their scattered nature and associated difficulties in monitoring, there are also cases where inflexibilities applied in Bank processing have led to misplaced efforts at “compliance”—and subsequent allegations of fraud and corruption. THE CDD APPROACH AND THE CDD PORTFOLIO Community-driven development (CDD) is an approach to development that gives control over planning decisions and investment resources to community groups and local governments. CDD programs are thus perhaps the farthest, in some sense, from the ICB model for which Bank procurement was originally designed. 42 CDD programs operate on the principles of participatory governance, demand responsiveness, community empowerment, and downward accountability. Initiated during the 1990s to help address problems of low capacity and lack of mutual trust, especially in postconflict situations, the CDD approach has emerged as an important element of the Bank’s lending programs over the last decade, with more than a quarter of IBRD/IDA lending during FY02–11 making at least partial use of the CDD approach. 43 Several defining features of the CDD approach have important implications for procurement processes—the numerous, small-value subprojects, often in remote locations, and dispersed over large geographical areas and decentralized implementation through local communities with limited access and exposure to formal market institutions. To what extent are Bank procurement policies and procedures, designed for large value purchases and implementation through experienced agencies, able to adapt to this structure, or do they need to undergo significant customization to support efficient and cost-effective implementation of CDD projects? In this section, IEG reviews the effectiveness of the present system, in terms of the following: • Enabling delivery of intended results including inter alia strengthening of country capacity for such procurement Assuring transparency and efficiency in the use of project funds during implementation. The evaluation triangulates findings from four main data sources: (i) relevant Bank documentation related to procurement in CDDs; (ii) a comparative and comprehensive review of portfolio data; (iii) a detailed selective review of a purposive sample of 34 CDD projects; and (iv) semistructured interviews with task team leaders/procurement staff in the Bank on procurement-related issues in CDDs. 42 CHAPTER 2 ADAPTING TO EVOLVING NEEDS IEG’s portfolio review covered all evaluations with CDD elements over FY02–11. It reviewed overall outcome ratings as well as risk ratings in project/program documents as well as Implementation Status Reports (ISRs) and examined overall development outcomes in relation to project procurement performance for CDD and non-CDD projects. Its sample of in-depth review of 34 CDD projects was intended to lead to an understanding of the nature of procurement problems affecting implementation and outcomes of CDD projects. CDD LENDING AND BANK PROCUREMENT POLICIES The most noteworthy feature of CDD procurement is that, unlike other projects with many small contracts (for example, rural development projects implemented by government agencies and/or project management units), the primary procurement implementation responsibilities rest with the benefiting communities themselves. Often their role extends even further: they may contribute some resources toward implementation of subprojects (in cash or in kind) and they may also be the “contractor/supplier” for some of the works and services needed for the subprojects. These multiple roles, coupled with usually limited familiarity with formal fiduciary processes, pose the twin challenges of keeping the processes simple and within capacity constraints, while at the same time having adequate checks and balances to minimize fraud and corruption. Addressing these twin challenges has meant allowing further flexibility in procurement procedures applicable to CDD projects but with continuing attention to the imperatives of “economy, efficiency, transparency and fair competition.” 44 Detailed guidance issued to staff asks that the application of the guidelines be “scaled to the project and the community’s capacity” and “take into account local culture and norms” while assuring a “high degree of transparency and accountability” (World Bank 2012d). As a general rule, CDD procurement is expected to: • Be simple enough as to be understood and implemented by the community using simple, standardized documentation. • Be sufficiently transparent to promote real competition among suppliers and to facilitate control in the selection of contractors and use of funds. • Balance risks with efficiency considerations. Given the nature of CDD projects, ICB, NCB, or international shopping are seldom used (outside components managed by the central implementing units—for example, for overall project coordination and monitoring and evaluation). Procurement methods commonly used include local shopping, local bidding, direct contracting/off-the-shelf purchases, and community force accounts. Prior reviews 43 CHAPTER 2 ADAPTING TO EVOLVING NEEDS are rare, and postreviews, as with other projects, are done on a sample basis and are often outsourced to third parties. Beneficiaries participating in the management of subproject funds are expected to provide important peer control over the use of funds. IEG PORTFOLIO ANALYSIS—PERFORMANCE OF ALL PROJECTS WITH CDD ELEMENTS Extensive previous analyses of CDD implementation by the Bank and by IEG have mixed findings with regard to the presence of procurement issues (Appendix B, summary section). To investigate the extent to which CDD projects encounter procurement difficulties, IEG first undertook a review of Bank investment lending, comparing both procurement difficulties and project performance outcomes for CDD and non-CDD projects, to assess the extent to which there may be differences in the two groups. 45 The portfolio included a total of 2,044 distinct investment lending projects, of which 552 used the CDD approach at least to some degree. Of these, among the CDD group, as of December 31, 2012, 287 projects had been completed and closed; the remaining 265 were still under implementation. In the overall sample of CDD and non-CDD projects, performance and results for 501 projects have been validated by the IEG through its Implementation Completion and Results Reports. For ongoing projects, IEG used data on performance from staff self- reported Implementation Status Reports (ISRs). Data for all projects together suggest that performance on project procurement is closely associated with overall project success (as discussed in Volume 1, Chapter 1). An analysis of the subset of project with CDD elements (as defined by the CDD anchor) shows further that these relationships hold equally strongly for CDD projects (Appendix B, portfolio analysis). 46 The analysis also suggests that CDD projects appear to have a higher propensity for procurement problems than non-CDD projects, and the performance gap appears to have widened over the past five years (Figure 2.1). The apparent downward trend in procurement performance for CDD projects brings into question the effectiveness of efforts made in recent years to simplify and streamline procurement processes related to the CDDs. Outcomes are largely invariant to region, sector, or loan size. CDD procurement tends to be more problematic in countries with lower levels of governance—FCS countries and countries with low Country Policy and Institutional Assessment (CPIA) scores (3 or lower). Interestingly, additional financing projects with CDD components show fewer procurement problems and better outcomes—a result perhaps of the self-selection associated with additional financing and of the learning and capacity building embedded in them. Finally, CDD projects aimed at improving 44 CHAPTER 2 ADAPTING TO EVOLVING NEEDS local governance or service delivery are more prone to procurement problems than those with the primary objective of community empowerment or improving livelihood security (Figure 2.1 and Figure 2.2). 47 Figure 2.1. CDD and Other Projects—Performance in Low Governance Environments Source: IEG analysis. Note: Projects with an ISR performance rating of moderately unsatisfactory. IEG PROJECT REVIEWS—FINDINGS FROM A SUBSET OF PROJECTS WITH COMMUNITY CONTROL The portfolio-wide analysis was complemented by a review of project cycle documentation (Project Appraisal Documents, ISRs, and where available Implemenation and Completion Results Reports) for a sample of projects. Three considerations guided sample selection. First, the sample was restricted to the subset of CDD projects with community control and management of investment funds, because from the procurement perspective, the most significant differences between CDD and non-CDD projects arise when communities have the responsibility for managing the procurement and implementation of subprojects. According to the Social Development Department database, this was the case for 237 of the 552 so- called CDD projects approved by the Bank during FY02–11. 45 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Figure 2.2. Procurement Performance and Development Outcomes by Type of CDD Source: IEG analysis. Note: For closed projects only. In a second stage, the review focused on the subset of the sample in which procurement problems were manifest. Thus, projects obtained after the first and second stages of screening were further stratified into four groups based on their procurement performance ratings by staff in ISRs (Table 2.1). 48 Table 2.1. CDD Projects: ISR Procurement Performance Ratings and Overall Outcomes Group I: 26 projects Group II: 57 projects Procurement rating of Moderately Procurement rating of Moderately Unsatisfactory or worse for Satisfactory or better for procurement procurement performance in the last problem in the last ISR and a rating of MU ISR or below in one or more earlier ISRs Unresolved procurement problems Procurement problems at some stage during implementation that were eventually resolved. Group III: 16 (closed) projects Group IV: 131 remaining projects Procurement performance rated No procurement problem reported in any Moderately Satisfactory or above ISR. Outcomes moderately satisfactory or throughout implementation; but with better. unsatisfactory overall outcomes. Source: World Bank. Note: ISR = Implementation Status Report. To maximize learning, the review focused on Groups I–III. 49 In accordance with the rest of the evaluation, a regionally balanced sample was drawn, based on the focus countries for this evaluation, as described in Chapter 1. 50 The final sample included 46 CHAPTER 2 ADAPTING TO EVOLVING NEEDS 34 projects: 17 in Group I, 12 in Group 2, and 5 in Group III. Thus, by design this sample is not meant to be representative of the CDD portfolio as a whole, as it is focused on understanding the problematic segment of the portfolio. In Groups I and II the evaluation focused on understanding the nature of the procurement problems affecting implementation, their actual/potential impact on intended project outcomes, measures taken by the Bank to help resolve them, and lessons for minimizing these problems in future. In Group III, the evaluation focus was on understanding the role, if any, that procurement might have played towards the unsatisfactory outcomes of the respective projects. Where necessary and feasible, the findings from the documentary reviews were supplemented by discussions with the respective task teams. Appendix B (project reviews) provides the relevant extracts from documentation for the 34 projects together with project-specific summary judgments on the procurement aspects of these projects. The main systemic findings are as follows: • Contrary to the patterns emerging from the portfolio-wide data, procurement in projects using CDD guidelines does not appear to be unusually problematic. In several cases of reported problems in Groups I and II, the procurement problems actually related to the non-CDD components of those projects. 51 More commonly, the reported problems reflected difficulties in procuring (centrally) the consulting services and technical assistance needed to support the implementation of the CDD components. 52 Of the 29 projects reviewed in Groups II and III, difficulties in implementing the CDD guidelines were an issue in less than a third of the cases. Thus, rather than a CDD procurement guidelines issue, this is a project design issue. • Overall, procurement performance of community-managed components does not appear to be a major constraint to effectiveness of the CDD projects (see Box 2.1, for example). • Where CDD procurement became an issue, it had less to do with the rigidities of the guidelines per se and more to do with how they were interpreted and used. In some cases, the difficulties arose because the client country’s own guidelines (for example, in Brazil; see Box 2.2) were more restrictive than what was allowed in the respective loan agreements with the Bank. In most cases, however, difficulty arose because task team leaders were reluctant to use the flexibility available under the guidelines because of excessive risk aversion. For example, requiring three quotations for shopping is still the norm in many projects, despite alternative options available in situations where that is not the appropriate 47 CHAPTER 2 ADAPTING TO EVOLVING NEEDS solution. 53 Staff training and leadership from the Procurement Board in disseminating the spirit of the CDD guidelines may help in this regard. Box 2.1. Afghanistan: Emergency National Solidarity Project II The Afghanistan Emergency National Solidarity Project II CDD project (2007, $120 million) was launched in challenging circumstances, where challenges were anticipated and largely overcome. Procurement issues did not relate to CDD components. The project supported community-level governance and management of subprojects aimed at improving access to infrastructure services. Because the subprojects were widely scattered, the Bank hired a procurement oversight consultant who supervised the process and built in-house capacity at the implementing ministry. Facilitators were hired on performance-based contracts to assist with social mobilization, subproject preparation and implementation, and capacity building for communities in target districts. This was accompanied by simplification of procedures at the community level and continuous training. Results were positive: 71 percent of the implemented subprojects targeted rural areas, and 73 percent of targeted rural families experienced improved access to basic services like power, roads, drinking and irrigation water, and education. Subprojects implemented during the project period generated direct employment for the beneficiary families, equivalent to 10.6 million labor days. Residual procurement problems. One challenge was that financial management and procurement responsibilities were split between various entities. Procurement problems related to non-CDD components and required procurement by a central agency, for example, the hiring of consultants or contracting of classroom construction. A sample of 25 subprojects for Kabul Province was reviewed, and documentation for 19 did not indicate major deficiencies. Source: IEG. • Potential misuse of project funds through fraud and corruption is a risk noted in almost all projects, and they emphasize transparency through proactive disclosure and accountability to the communities as a remedial measure. Such abuse, if it occurs, could be a reputational risk to the continuing viability of the CDD approach. 54 Sound post-reviews remain important vehicles for assuring fiduciary probity through checks and balances. 55 However, much of the postreview work focused on procedural compliance and not on outputs and outcomes. • Effort devoted to Bank supervision of fiduciary aspects is of major concern to task team leaders. Many of them believe that in the wake of the India Detailed Implementation Review, a disproportionately large share of supervision in CDD 48 CHAPTER 2 ADAPTING TO EVOLVING NEEDS projects is going into monitoring and oversight of fiduciary aspects—at the cost of more substantive project design issues (World Bank 2007a). This reflects the current input-focused approach to oversight of fiduciary aspects in CDD projects seeking compliance with myriad processes in innumerable subprojects. Shifting the CDD paradigm so that communities are expected to deliver agreed outputs and outcomes rather than trying to control their procurement of the inputs may be a way around this dilemma. Box 2.2. Procurement Issues in the Maranhão Integrated Program Although the Maranhão Integrated Program (2004, $30 million) built on good experience from an earlier project, problems arose with procedural compliance. Post-review exercises in 2008 looked at 100 contracts for 77 community subprojects approved in 2007. Evidence was found of systematic misapplication of Bank procurement guidelines and red flags of possible fraud. Problems detected included invalid quotations or quotes that did not meet the requirements of agreed shopping procedures. According to the Implementation Completion Report, …concurrent Bank missions on financial management … noted that (i) these procurement irregularities were not related to financial management practices, and (ii) the investment subprojects identified in April 2008 physically had been completed to the satisfaction of the beneficiaries. In other words, problems were related specifically to the misapplication of procurement procedures. Closer examination revealed no evidence of systematic fraud. But procedures had been misapplied, apparently because the implementing agency and the communities did not understand them properly. In addition, there was misunderstanding on whether the Brazilian law or Bank procurement guidelines took precedence. The project was closed in December 2008 after the allegations of fraud added to allegations of corruption and subsequent political instability at the state and municipal level. Up to this point the project financed 626 community subprojects—short of the 1,200 targeted at appraisal—36 percent of the loan was already disbursed, and beneficiaries were satisfied with project results. Despite the good pace of delivery of results, the project outcome was rated Unsatisfactory, mainly because at the moment of cancellation, 64 percent of the loan had not been disbursed and because of the allegations of fraud during the procurement process. Source: IEG. 49 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Procurement in Fragile and Conflict Situations • Increasing Bank engagement with countries in FCSs, often as the fulcrum of multidonor trust funds, heightens Bank responsibility for effective procurement in these environments. • Although issues related to procurement are pointed out in a number of country strategy reports in FCSs, their frequency is not greater than similar issues raised for all countries. • The Bank offers flexibilities in procurement in such environments, many of which have been helpful and widely used. Conversely, such flexibilities have also been used in operations that are not under special procedures. • Many flexibilities afforded for FCS procurement—such as greater delegation to field staff, greater choice over procurement methods, or higher thresholds—are in line with Bank-wide reform proposals and consistent with the findings of the present report. • The need for special regimes for such operations and clients may diminish over time. Around 30–35 countries have been included each year within the scope of the Bank’s definition of FCS, including approximately 1.5 billion people. 56 These areas are plagued by fragility, instability, and conflict, as well as delayed economic growth. Given the particular vulnerabilities and numbers of people affected, the international community is paying close attention to these countries to find means of restoring or reinforcing economic progress, as well as to improve the mechanisms through which international organizations offer assistance (World Bank 2007c, 2011e; OECD 2011a). The Bank has become a focal point, not only on account of IBRD/IDA financial support, but because of its leveraged collaborations with bilateral and multilateral aid organizations. Bank collaborations have helped harmonize approaches to working in FCSs and mobilize additional resources, notably through multidonor trust funds administered by the Bank (IEG 2006b). Although IBRD/IDA commitments to FCS over the period FY03–12 (Figure 2.3) show no sustained trends, funding to FCS from Recipient Executed Trust Funds has been increasing both in absolute terms and relative to total commitments from such funds over the same time period (Figure 2.3). 57 Given that the Bank administers not only its own resources but also those contributed by other donors, its fiduciary role assumes increased importance. Bank policy toward FCS has aimed to balance fiduciary controls with an appreciation of the unique operating environment that often requires rapid response and operational flexibility in countries with low capacity levels. Because most lending to FCS occurs through the investment lending vehicle, 50 CHAPTER 2 ADAPTING TO EVOLVING NEEDS the efficiency, transparency, and effectiveness of Bank procurement processes assume increased importance. Figure 2.3. Trends in IBRD/IDA and RETF Support to FCS (2003–12) Source: IEG analysis. Note: FCS = fragile and conflict-affected stat; RETF = Recipient Executed Trust Funds ANALYSIS OF PROCUREMENT ISSUES IN FCS—ISSUES IDENTIFIED IN CASS IEG’s analysis of Country Assistance Strategies (CASs)/Interim Strategy Notes (ISN) began with a sample of CASs from 10 FCSs over FY06–11 to identify the nature and extent of procurement-related issues. 58 Around two-thirds (16 of 25) mention procurement problems that limit development effectiveness. Eleven of the 25 CAS reports reviewed had specific procurement-related monitoring indicators in the CAS matrix, and 6 identified procurement-related outcomes. The problems referred to stemmed mostly from capacity bottlenecks. It is interesting that IEG’s review did not highlight Bank procedures as being a major impediment to procurement in FCS (Box 2.3). Thus two key areas emerged: • Lack of procurement capacity—The most cited procurement problem, mentioned in 13 of 25 CASs. Low procurement implementation capacity is compounded by weak public administration systems, including flawed legal systems for procurement and difficulties with financial management that compound challenges in operating environments. • Integrity and transparency—Nine of the 25 CAS reports mentioned issues related to corruption in procurement. 51 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Box 2.3. Procurement Issues and Procurement Support in FCS—Evidence from Selected CASs Capacity issues are frequently mentioned. Four FCS CAS/ISN reports referred to problems in the legislative and regulatory framework related to procurement. In Liberia, “Weak procurement legal framework and practices lead to inefficiencies in the use of public funds.” Low levels of country capacity impact procurement capacity; for example, “Haiti’s financial management and procurement systems continue to show significant shortcomings, while institutional capacity is weak and corruption prevalent.” The following CAS/ISN reports also mention weak procurement capacity: Nepal (FY04 and FY09), Haiti (FY07), Liberia (FY09), Burundi (FY05), Lao PDR (FY05), and Sierra Leone (FY10). Integrity and transparency, specifically corruption have also been highlighted. For instance, in Nepal, “…one of the growing concerns is that public procurement processes are being undermined at times by acts of intimidation to deter competition.” Corruption and lack of transparency were also mentioned in the following reports: Timor Leste (FY06), Nepal (FY04), Haiti (FY05, FY07, and FY09), Burundi and Afghanistan (FY09). Planned CAS support—Procurement-related outcomes and monitoring indicators: • Eleven reports had procurement-related monitoring indicators in the CAS/ ISN matrices and six had procurement-related CAS outcomes. In some reports, the Bank’s planned support to procurement was included with other policy initiatives. The FY05 Sierra Leone CAS, for example, had both outcomes and monitoring indicators related to improving transparency, accountability, and efficiency of procurement operations in the country. • The Haiti, Lao PDR, Sierra Leone (FY05), Liberia, Nepal (FY09), and Sierra Leone (FY10) CAS/ ISN reports included both procurement-related outcomes and monitoring indicators in the country strategy matrix. The Afghanistan (FY06), Timor Leste (FY06 and FY10), and Burundi (FY02 and FY05) reports included only monitoring indicators but no outcomes. Source: IEG. Box 2.4. Procurement Guidelines under OP 8.0 and OP 10.0 OP 8.00 Procurement Guidelines OP 10.0 Procurement Guidelines Described procedures for designating emergency Not Mentioned staff—Rapid Response Committees with an assigned procurement staff member to provide fiduciary support. Project Preparation Facility thresholds increased Project Preparation Facility from $2 million to $5 million. thresholds increased from $5 52 CHAPTER 2 ADAPTING TO EVOLVING NEEDS million to $10 million. Retroactive financing limit is increased from 20 Unchanged percent to 40 percent. Allows for increased clearance thresholds (field Unchanged staff can approve no objections for projects that were previously only approved by RPMs). Allows for the borrower to contract with United Unchanged Nations agencies, nongovernmental organizations, regional institutions, and other multilateral/bilateral donors to undertake project implementation—including procurement. Provides for simplified procurement arrangements Unchanged offering greater flexibility, speed, and simplicity by delegating procurement review and clearance authority to task teams and allowing methods other than NCB or ICB. Refers to the Fiduciary Principles Accord an Unchanged agreement between the Bank and the United Nations that allows agencies that are signatories to rely on its own fiduciary requirements when implementing trust funds administered by any other signatory Provides for a standard form of agreement for Unchanged supply of health related goods through the United Nations, United Nations Provision of technical services. Allows for procurement agents and project management agents. Increases the threshold to $500,000 subject to review Requires Consultant and clearance by RPM for CQS, describes Qualification Selection appropriate use of Direct Contracting in FCS. contracts greater than Shopping threshold of $500,000 for goods and $300,000 be cleared by RPMs $1,000,000 for simple works. Unchanged Shortlist of consultants where fewer than six firms have expressed interest and where a wide geographic spread of shortlisted consultants cannot be achieved. Provides for the use of Positive Lists, the reduction Unchanged of commodity customs and the use of Force Account. Additionally, OP 8.50 recommends that Unchanged 53 CHAPTER 2 ADAPTING TO EVOLVING NEEDS procurement teams working in FCS consider the following: Taking a sequenced approach, allowing low risk items to move ahead; Increasing prior review thresholds—dependent on the circumstances; Using a monitoring and/or procuring agent; providing for preshipment inspection especially for commodities; and intensifying Bank supervision. Source: World Bank. Note:CQS = Consultatnt’s Qualifications Selection; FCS = fragile and conflict-affected state; ICB = international competitive bidding; NCB = national competitive bidding; NGO = nongovernmental organization; RPM = regional procurement manager. Though significant, these numbers should be viewed in perspective. IEG’s overall review of country CASs, to gauge the extent to which procurement-related issues are identified (undertaken in Chapter 2 of Volume 1), highlighted similar proportions of CASs which mentioned procurement related issues, and in these, too, capacity- building issues were the primary concern (Box 2.4). ANALYSIS OF PROCUREMENT ISSUES IN FCS – REVIEW OF PROJECT DOCUMENTS IEG next undertook an analysis of Project Appraisal Documents based on a sample of 58 Bank projects selected from 10 FCSs between FY06 and FY11; IEG sought to assess the extent and the manner of utilization of procurement flexibilities under OP 8.0 in World Bank operations in FCSs, to gauge their usefulness and to see whether any supplemental measures may be needed to increase procurement performance in these environments. 59 For each country, IEG selected at least four projects that represented lending under both emergency and normal processes. 60 To identify which projects were more likely to be emergency projects, IEG used the following criteria: • Projects with a stated objective that was emergency related OR • Projects explicitly identified as emergency recovery loans. Findings from this analysis were primarily that: • The use of procurement flexibilities as laid out in OP 8.0 in FCSs was not limited to emergency projects. • When OP 8.0 flexibilities are used, strong justification is rarely provided. • The most frequently used flexibility was simplified procurement arrangements such as shopping, single-source selection, and community participation in procurement. 54 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • Procurement agents were used in countries with severe capacity constraints regardless of whether the projects were considered emergency projects. • Most of the project ISRs did not include discussions of the procurement flexibilities applied during project implementation. ANALYSIS OF PROCUREMENT ISSUES IN FCSS—STAFF QUESTIONNAIRE In addition to examining project-level information, IEG sought to incorporate feedback from Bank procurement staff with extensive experience in FCSs. IEG administered a questionnaire to 15 procurement specialists and hub coordinators who, combined, have experience in at least 45 such countries or territories. The questionnaire asked respondents about procurement in FCS, including but not limited to the following topics: the extent to which lending operations used OP 8.0, whether Bank staff used the flexibilities allowed or avoided risk; the nature of the specific streamlined procurement procedures that are most frequently utilized; the impact of OP 8.0 on economy and efficiency; impacts of borrower procurement capacity; and Bank engagement with other donors and multidonor trust funds. Overall, respondents point to extremely low or scarce procurement capacity; low private sector capacity in parallel; and lack of technical expertise and difficulty in attracting personnel, including Bank procurement specialists to relocate to FCS.61 Low levels of market competition and limited bidder interest—especially bidders from outside the country—are also pointed out. 62 Most respondents did not think that delays in implementation were caused by procurement issues, and most respondents thought that the Bank’s processes in FCS were at least substantially efficient. Extent to Which OP 11.0 or OP 8.0, Respectively, Are Used in FCS The extent to which OP 8.0 was used in FCS was mixed. For example, although South Sudan conducted all operations under OP 8.0, Cameroon exclusively used OP 11.0. This has been the practice in the West Bank and Gaza too, but for a short period. In Nigeria, only multidonor trust fund projects were prepared under OP 8.0, whereas in Timor Leste, only one project was undertaken using OP 8.0. Notwithstanding their preparation under OP 8.0, projects in some FCS (the Democratic Republic of the Congo, Guinea Bissau, and Liberia) rarely took advantage of the flexibilities provided for in OP8.0 and instead applied the normal OP 11.0 processes. 55 CHAPTER 2 ADAPTING TO EVOLVING NEEDS OP 8.0 Flexibilities That Were Used the Most Among respondents, there was consensus on the frequent use and benefits of higher approval authority to procurement staff in the field, often up to regional procurement manager level, and greater use of higher thresholds for NCB and prior review; greater use of rapid procurement methods such as direct contracting (reportedly used most frequently); simple shopping or use of prequalified consultants; and greater use of extension of contracts. Respondents indicated that three countries—the Democratic Republic of the Congo, South Sudan, and Sudan— regularly used accelerated bidding and streamlined procedures and found them to be highly effective. However, there was a caution on higher NCB thresholds in low-capacity environments. In some cases “roving consultants,” provided by the Bank to troubleshoot, have usefully supplemented government capability through just-in- time support—for example, in Ethiopia and Sudan. It was also cautioned that higher thresholds under OP 8.0 (or its equivalent) are rendered irrelevant if all country thresholds are raised. It was also pointed out that putting senior and more experienced staff in place amounted effectively to raising thresholds. Respondents also pointed out that some provisions such as selection based on consultant qualifications and direct contracting are useful but that they don’t necessarily need an OP 8.0. Contract extensions were widely used in Iraq, South Sudan, and Timor Leste, for example. Also, in Timor Leste, single sourcing under other nonemergency projects was used widely, mainly through twinning arrangements with quasigovernment entities and universities in the region. Borrower concerns about perceptions and potential accusations of corruption arising from the use of single source selection partly explain why this flexibility was not put to use in FCSs such as Iraq. Finally, the use of United Nations agents in FCS operations was mentioned as one possible way of adding procurement resources, mainly by the procurement specialists from the Africa Region, but the results were mixed. Country Capacity Is the Most Limiting Factor Limited country capacity to implement procurement was cited as a major factor in 8 of 15 cases—most respondents indicated that the Bank tried to address this by providing more expert staff and resources in the field offices, with an emphasis on quality and experience, as opposed to numbers of staff. 56 CHAPTER 2 ADAPTING TO EVOLVING NEEDS One respondent indicated that country capacity was a more significant factor than prior review, methods, or clearance thresholds. Countries’ greatest need was the provision of guidance in the process, to the extent of helping clients prepare bidding documents or advising on bid evaluation. The Bank has tried to provide both better trained staff as well as training for client procurement staff. Difficulties persist in hiring and retaining well-trained local procurement staff. Risk Aversion Among Bank Staff Opinions on risk aversion among Bank staff are mixed. Some respondents indicated that Bank staff, task team leaders, and procurement specialists are perceived as inexperienced and risk averse and recommended more training; others indicated that there was no evidence of risk aversion. Two respondents indicated that the flexibilities, though available, have not been used because of staff risk aversion. Risk aversion is more evident among junior staff, who are hesitant to make decisions that require judgments; there is limited incentive for them to do this and considerable downside risk in the event of any subsequent issues. In Africa, extensive efforts have been made to increase the amount of Bank staff with substantial procurement experience. This has increased capacity on the Bank side of the procurement process. According to one respondent, thresholds were effectively raised by raising the seniority and decision making authority of Bank staff. Bank Engagement with Donors in Procurement in FCS Bank engagement with donors in FCSs is usually in the context of Bank-administered multidonor trust funds. One exception was the Democratic Republic of the Congo, there was no joint engagement between the Bank and other donors there, especially in the health sector, which had more than 20 actors. A meeting was called to try and remedy the situation. The result was the “Kinshasa agenda,” which was based on the Paris declaration and Accra workshop recommendations. Differences in approaches do not appear to have been an issue, as in such cases the Bank administers the totality of funds, under Bank Guidelines and processes. This has generally been acceptable to other donors. The Impact of OP 8.0 on Economy and Efficiency Respondents did not feel that the use of OP 8.0 measures led to a loss of competition and thus less economy. In FCS circumstances, often the presumption of competitive markets is not correct. In terms of efficiency, there was a general endorsement that the flexibilities, when used, added to efficiency in the contracting process. However, there is not always a clear distinction, in terms of circumstances of application and 57 CHAPTER 2 ADAPTING TO EVOLVING NEEDS eligibility, between OP 11.0 and OP 8.0. And the main overall issues remain: lack of country capacity and lack of sufficient private sector capability or interest. Summary and Conclusions Overall, IEG found that in fragile situations, procurement capacity among country counterpart staff may be a greater binding constraint than the Bank Guidelines. Although procurement flexibilities under OP 8.0 have been useful, most could also have been used without formal recourse to a separate set of guidelines. The documentation of permitted flexibilities is nevertheless useful, particularly for less experienced staff, who may have the greatest hesitation in taking risks. There is little incentive for risk taking and a substantial downside if decisions are erroneous. Increases in review thresholds (in line with the recommendations of this report) are under consideration in several Bank regions. In Africa it is claimed that some OP 8.0 procurement flexibilities are being rendered irrelevant by the raising of all thresholds (clearance, prior review, and methods thresholds) in FCS countries. The Latin America and the Caribbean Region is moving in a similar direction, currently developing new thresholds for Haiti. And one staff member noted that putting more senior or experienced personnel in the field has also been tantamount to raising thresholds. One surprising finding to emerge is the unevenness of the application of permitted flexibilities. Thus, many eligible projects have not used such flexibilities, and conversely, they have also been used in contexts where OP 8.0 has not been specifically invoked. Going forward, given the loose application of the regime so far, fine tuning of the flexibilities may not necessarily be needed; what would be useful is the ability to exercise discretion in drawing on them. Going forward, greater overall flexibility in Bank Guidelines, should this occur, is likely to reduce the need for resorting to special provisions. It is clear that the greatest constraint in FCS environments is not so much the Bank’s procurement policies, which do have provisions for needed flexibility, as implementing capacity constraints. Reinforcing capacity among country staff, but also raising awareness of Bank staff and supplementing that as appropriate with external resources, are priorities. Very recently, the Bank adopted a new policy directive towards procurement in Fragile States, as part of the investment lending simplification and reform (World Bank 2013e). While processes remain broadly similar to OP 8.0, the scope of applicability has been widened to include, beyond situations of urgency, situations of low capacity including, for example, small or remote countries. 58 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Procurement of Information and Communications Technology • Bank systems for information and communications technology (ICT) procurement have improved in the recent past, and the Bank Guidelines have permitted new flexibilities and clarifications that are well suited to the ICT process. Bank standard bidding documents for ICT projects, with their scoring process, Bank acceptance of two-stage bidding, and Bank approvals, when needed, of sole source, and its clarification of conflicts of interest for consultants, are welcome moves. • Legacy difficulties remain, reflecting the adaptation of the present system from one designed for a very different purpose. The Bank’s standard bidding documents are not user friendly and are not aligned with industry norms. • Although the system has flexibilities, many are discretionary and the complex chain of clearance for no objection can lead to significant delay. • Task team leaders and procurement staff have a limited awareness of available flexibilities and hesitate to use them, in part because of risk aversion, but also because of a limited understanding of their benefits. • Better project design is an issue, and this requires training and awareness building among the team leaders. • Country capacity is sometimes—though not always—a constraint. • Although the Bank has consistently offered advice and support for contract implementation, its interventions have been somewhat skewed toward the project planning, rather than implementation phase. THE BANK’S ICT PORTFOLIO AND THE SCOPE OF THE EVALUATION ICT procurement differs from other procurement of goods and works in that it typically encompasses a mixture of professional services and goods—supply of hardware and software, typically involving customized development of specialized applications. The unique challenges of ICT procurement derive from the complexity of the systems being procured, which continuously evolve and expand in parallel to changes in user needs and advances in technology. Moreover, such projects are typically implemented by agencies not knowledgeable in the relevant technical areas or in complex procurement (a central bank, for example). Many ICT projects are in the government administration and finance areas and have a broad stakeholder base that extends beyond the implementing agency. Decision making is further complicated as they typically involve change management that can disturb vested interests (tax reform, for example). The Bank’s ICT portfolio is significant (Figure 2.4), and ICT components are found in a range of sectors. International Finance Corporation (IFC) ICT projects have also grown significantly and now dominate the Bank portfolio in numbers. Complex information technology systems are frequently included within projects related to public administration 59 CHAPTER 2 ADAPTING TO EVOLVING NEEDS (budget management systems) or the financial sector (central bank systems) or in seemingly unrelated areas such as the rural sector (cadastral surveys) or the human development sector (health modernization, for example). IEG’s evaluation focuses on the effectiveness of the Bank’s procurement system for large and complex information technology systems, typically involving the procurement of hardware systems and customized software. 63 IEG’s evaluation draws on (i) a review of the Bank Guidelines and standard bidding documents for ICT procurement; (ii) structured interviews/surveys of procurement specialists and task team leaders, based on a sample of projects in the ICT portfolio; and (iii) semistructured interviews with a wider range of Bank staff with specific experience in ICT transactions, including information technology specialists from the Bank’s Transport, Water, Information, and Communications Technology unit, from the procurement anchor and the Bank’s legal department. 64 IEG also undertook a review of previous IEG, Quality Assurance Group, and Bank reports that refer to Bank procurement structures for ICT (Appendix Box B.1). Figure 2.4. World Bank Group: ICT Portfolio 2003–10 Sources: IEG ICT database, based on IEG (2011b). 60 CHAPTER 2 ADAPTING TO EVOLVING NEEDS ICT AND BANK PROCUREMENT PROCEDURES ICT procurement, as with all other procurement, is governed by the Bank Guidelines and one set of its standard bidding documents that is specially designed to meet the needs of ICT. 65 One main feature of the bidding documents for ICT procurement is that they provide, as an option, evaluation criteria that score the technical proposals of bidders and then combine the scores with prices to derive an overall best bid. This is unlike the standard bidding documents for procurement of goods and works, which specify award of the contract to the lowest evaluated priced offer among technically compliant bids. This feature recognizes that procurement of information technology systems involves not only procurement of hardware and software, but also technical services, which require an evaluation of technical capability as well. The information technology bidding documents give guidance on developing technical specifications tailored to ICT systems. The contract terms and conditions similarly address ICT-specific requirements, especially in the areas of intellectual property rights and software licensing. Although earlier versions of the guidelines included clauses that recognized the unique procurement requirements posed by projects involving ICT, in 2011 revisions were introduced to specifically address ICT-related procurement concerns: 66 • Two-stage bidding for procurement of complex systems. 67 The guidelines specifically mention ICT procurement as an area where rapid technological advances make it difficult to prepare technical specifications in advance of the bidding. In straightforward procurement of works, a step-by-step process is followed in which an engineering design is first completed to develop technical requirements. Bidding documents are then prepared and a contractor is selected, who completes the works according to the design. For an ICT system, the bidding process will likely turn up unanticipated requirements. During contract implementation, technology may change and new requirements surface, both leading to the need to adjust the contract. In such cases, the guidelines say, the Bank may require the borrower to use two- stage bidding (see Box 2.5). • Direct purchase for backward compatibility. 68 A particular problem for ICT procurement is that it is often necessary to procure equipment that is compatible with existing systems (Windows operating system, for example). The guidelines make it clear that for ICT systems, compatibility issues may require specification of a particular brand of equipment, but that in such instances, Bank no-objection is required. • Framework agreements. 69 These are long-term agreements that set out arrangements for ongoing procurement over the term of the agreement. Although ICT is not specifically mentioned in the clause, it is possible that 61 CHAPTER 2 ADAPTING TO EVOLVING NEEDS framework agreements could be used to procure, for example, simple software maintenance or servicing. Box 2.5. Two-Stage and Single-Stage Bidding The guidelines suggest that for complex ICT procurement, rapid technological advances can make it difficult to prepare complete technical specifications in advance of bidding. In such instances, two-stage biddng may be appropriate. Under two-stage bidding, un-priced technical proposals are invited in a first stage. These are typically based on a conceptual design or performance specifications. During the first-stage evaluation, bids are subjected to technical and commercial clarifications, and requirements are adjusted to incorporate information learned through the process. In the second stage, amended bidding documents are issued and bidders are requested to submit final technical and priced bids. An IEG portfolio review found reluctance among task team leaders to use two-stage bidding—in 15 projects across 10 countries it was used in 5 projects in 4 countries (Bangladesh, Egypt, Russia, and Vietnam). The task team leader of a large information technology tax administration project said that single-stage bidding, where technical and price proposals are submitted and evaluated in a single process, was seen as quicker and equally effective. Procurement specialists with ICT experience disagree, saying that the time associated with two-stage bidding can avoid the much longer process of rebidding when single-stage bidding fails. Moreover, two-stage bidding allows purchasers to learn about available technology and to align their technical requirements with the best solutions offered by the market. The specialists say that single-stage bidding is best suited to situations where the purchaser has the capacity to specify the technical requirements to the detail that allows bidders to price all required items. Source: IEG survey of ICT procurement. IEG STRUCTURED SURVEY: PROCUREMENT STAFF AND TASK TEAM LEADERS IN A SAMPLE OF ICT PROJECTS IEG undertook discussions, on the basis of a structured questionnaire, with procurement specialists and task team leaders who had experience with the procurement of information technology systems in 15 major projects where information technology accounted for a large part of project value. 70 The intent was to review recent projects, under implementation, for procurement-related issues. Sample characteristics, including IEG scores, are shown in Table 2.2, together with perceptions on the quality of Bank support. Information on procurement methods used is in Table 2.3. 62 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Table 2.2. IEG Review of ICT Procurement—Project Characteristics Quality of Bank support during: Preparation Implementation Amount ($ Approval Nature of project Country millions) (FY) Type Quality Type Quality Health insurance administration Egypt 75 2010 HQ/SC Medium none na Central bank payments Bangladesh 350 2009 HQ/SC nc HQ High Education research network Bangladesh 81 2009 HQ/SC nc SC nc Financial management Vietnam 60 2009 HQ High none na Flood forecasting and monitoring Poland 184 2007 HQ/SC High HQ/SC High Insurance administration Poland 89 2006 HQ/SC High HQ/SC High Financial Reporting and Audit Pakistan 84 2006 HQ/SC High Note 1 na Public service administration Uganda 70 2006 HQ/SC High HQ/SC High Local government administration Uganda 55 2008 HQ/SC High HQ/SC High Internet connectivity Romania 60 2006 HQ Medium SC nc Power system control ( SCADA) Turkey 66 2005 HQ/SC High none na Land registration Serbia 30 2004 HQ Medium none na Customs administration Russia 140 2003 HQ/SC High none na Tax administration Russia 100 2003 HQ nc none na Financial management Vietnam 54 2003 HQ/SC Medium HQ/SC Medium Source: IEG ICT procurement staff survey. Note: HQ = Bank HQ procurement specialist; SC = specialist consultant; nc = no comment provided; na=not applicable; High = support timely and competent; Medium = timely and competent support, acceptable but with noted deficiencies; Low = incompetent, unresponsive, and/or ineffective.Implementation did not proceed (procurement cancelled). HQ = headquarters. SCADA = Supervisory Control and Data Acquisition. Most Bank procurement specialist support was provided in the preparation phase (Box 2.6 and Box 2.7). More Bank staff assistance was provided to review bidding documents and the bid process (evaluation) than to assist with implementation, by a factor of more than two to one. Bank headquarters procurement specialists provided support to all 15 projects during the preparation stage, but only on 6 during implementation. Eleven projects also utilized technical specialist consultants to assist during project preparation, and 7 used such consultants for project implementation. Qualitative comments in the IEG survey are positive—procurement specialist assistance was rated from “adequate” and “sufficient” to “very helpful.” There were no reported cases of Bank support being unresponsive or lacking in technical expertise. 71 63 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Box 2.6. IEG Review of ICT Procurement—Project Support Provided The Bangladesh Central Bank strengthening project is cited by procurement specialists involved with the project as a positive example where Bank technical support was provided throughout the project, including both the preparation and implementation phases. After an initial procurement failure, a repackaging exercise delayed the information technology component by about two years. Subsequent legal problems with one package introduced further delays. However, the system was eventually procured and successfully implemented. Another positive example cited was the Vietnam Financial Management Reform Project. For this project, the implementing agency received strong support from headquarters procurement specialists, specialist consultants, and procurement specialists in the field. In another country, the consultant who was hired by the Bank to assist with project preparation was reportedly competent and effective, but was not allowed to later work for the government to assist with implementation, as this was seen as a conflict of interest. The problem reported here is possibly due to interpretation of the pre-2011 guidelines that indicated issues with conflict of interest if the same parties contributed to both preparation and implementation. The 2011 revisions to the guidelines make it clear that conflicts between services related to project preparation and subsequent works associated with project implementation only pertain to provision of non-consulting services. This is another recent change, which could be brought to the notice of task team leaders and procurement staff. Source: IEG. Bank procurement procedures are seen as adequate, but some potentially helpful features are not frequently used. The overall perception of those interviewed is that the Bank’s procurement system is effective, with deficiencies noted in only three projects. All but two projects made use of the point scoring system. Further observations on the effectiveness of the procurement system are offered below: • Prequalification was used in only two projects. This might be explained by a sometimes limited number of suppliers for complex information technology systems. 72 • There is a preference for single-stage bidding. Two-stage bidding was used in only five projects. • Country procedures are sometimes easier to use. On two projects in Poland it was commented that using country procedures would have been preferable, as the government and bidders were unfamiliar with the Bank’s procedures, leading to extensive clarifications. Similarly, in Vietnam, Bank procedures were seen as overly complicated and difficult for the procuring agency to use. 64 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Table 2.3. IEG Review of ICT Procurement—Reported Use of Bank Procurement Methods Nature of project Procurement methods Country Amount Approval PQ 2Stage IT SBD Scoring Effectiveness ($ millions) (FY) (perception) Health insurance Egypt 75 2010 ✓ ✓ ✓ ✓ High administration Central bank payments Bangladesh 350 2009 ✓ ✓ High Education research network Bangladesh 81 2009 ✓ ✓ High Financial management Vietnam 60 2009 ✓ ✓ ✓ No response Flood forecasting & monitoring Poland 184 2007 ✓ ✓ Medium Insurance administration Poland 89 2006 ✓ ✓ Medium Financial Reporting and Audit Pakistan 84 2006 ✓ ✓ High Public service administration Uganda 70 2006 ✓ ✓ ✓ High Local government Uganda 55 2008 ✓ ✓ High administration Internet connectivity Romania 60 2006 ✓ ✓ High Power system control Turkey 66 2005 ✓ ✓ High (SCADA) Land registration Serbia 30 2004 ✓ ✓ High Customs administration Russia 140 2003 ✓ ✓ ✓ High Tax administration Russia 100 2003 ✓ High Financial management Vietnam 54 2003 ✓ ✓ ✓ Medium Source: IEG ICT procurement staff survey. Note: PQ =p requalification used; 2Stage = two-stage bidding used on at least one contract; IT SBD = the Bank’s standard bidding documents for information technology used on at least one contract; scoring = combined technical and financial scoring used on at least one contract; High Effectiveness = Bank’s procurement methods fully acceptable with only minor concerns; Medium Effectiveness = Bank’s procurement methods acceptable but with noted deficiencies that negatively impacted procurement process. SCADA = Supervisory Control and Data Acquisition. The bidding process and bid award were seen as generally satisfactory. However, problems exist with low competition and delays in evaluating bids. Nonacceptances of bids for procedural matters were limited to one late bid and one nonconforming bid security. There were no instances of cancelled bidding or rebidding. 73 The level of complaints was typical of any procurement and was adequately dealt with. Clarifications during bid preparation were few and did not result in significant changes to technical specifications. Clarifications were minor or procedural, not affecting system specifications. There was only one reported instance of the need to negotiate contractual terms after contract award, and this related only to software licensing terms (Serbia). Contract award prices were usually close to estimates. For six of the nine projects on which price data are available, award prices were equal to or below the estimates—although they were higher than estimate on the other three projects. 65 CHAPTER 2 ADAPTING TO EVOLVING NEEDS The largest reported overrun was the Russian customs system, which was approximately 60 percent above the estimate. However, generally price increases during bidding do not appear to be an issue, and this is consistent with the point above that there were few reported changes to technical requirements during bidding. And misprocurements or complaints seem to be only as frequent as in a general selection of projects. There was one reported Integrity Vice Presidency (INT) investigation resulting from a complaint from a losing bidder in one of the surveyed projects. Perhaps more substantively: • Competition was frequently low. More than half of the respondents said that competition was weak, or reported fewer than three bidders. The explanations given were political instability in the country (Egypt), few firms capable of supplying the required systems (Serbia), and limited interest because of the nature of the procurement (expansion of an existing system in Turkey). These explanations are reasonable. 74 • Two projects showed significant delays in contract award (out of only five answers to this question). The two reported delays were significant (60 days in one instance and 6 months in the other). It was noted that the project that incurred the six-month delay (Vietnam Financial Management system, 2003 project) incurred a further four-year delay for the contract to be signed following contract award. This was attributed to changes in technical design and scope of work and possible political issues. Contract implementation in the majority of projects is satisfactory, although there are some significant implementation delays. Eight of twelve projects reported delays from minor to three years, while four reported no implementation delays. In a number of projects, implementation is ongoing—final results are not known. So far, seven projects report no price variations exceeding 10 percent of contract value, though four projects report variations exceeding 10 percent. The largest, 33 percent, was reported for the Vietnam Financial Management system (2003 project). Causes of delay include: • Coordination problems between different contracts for supply of hardware and software. • Start-up problems arising from organizational changes within the borrower, and frequent changes in the Bank’s task team leaders. • Changes to scope of work during implementation. The Bangladesh Research and Education system reported a delay in commencement of bidding of three years because of difficulty finalizing technical specifications. This has delayed the entire project but is attributable more to project design factors than to procurement. 66 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Box 2.7. Messages from the ICT Portfolio Review Sixteen procurement specialists and task team leaders, interviewed for the portfolio review offered their general views on challenges encountered in procuring ICT systems. They broadly agreed that delays derive from the technical complexity of the systems being procured, requiring longer times than procurement of less complex products and works. ICT systems more frequently encounter changes to technical design and scope of work. Lengthy preparation of technical specifications delays procurement. This is compounded by the low capacity of implementing agencies. The Bank needs to support projects with adequate technical resources, such as ICT-knowledgeable procurement consultants. Moreover, applications are often politically sensitive, adding possible political interference. Project design and implementation typically require broad support from a range of stakeholders on features and functionality. Inadequate stakeholder input leads to systems that do not meet requirements or expectations. This in turn leads to ongoing modifications to technical requirements. There are problems in the formulation of the procurement methods, especially where exceptional methods are involved, such as sole-source negotiation. Such methods are sometimes the most appropriate, but Bank approval is a lengthy process that can negatively impact project outcomes. Source: IEG survey. IEG SUPPLEMENTARY INTERVIEWS: ICT EXPERTS, PROCUREMENT LEGAL EXPERTS IEG undertook additional, semistructured interviews, primarily with senior procurement specialists and ICT staff. On the positive side, some experienced procurement staff hail the ICT bidding documents as highly flexible. There are variations within the documents to cover simple goods (printers and peripherals, for example), services (including service contracts and professional services under consulting procedures), and complex systems that involve a combination of hardware and services (including both two-stage and single-stage procedures). These give procuring agents a range of options to fit different circumstances and substantially align with industry practice. Nevertheless, a number of specific issues were raised in terms of the appropriateness of the Bank’s current ICT bidding documents (Box 2.8). • The documents are voluminous, complex, and sometimes intimidating to borrowers and Bank staff. Though larger suppliers probably understand them, smaller or newer suppliers might not. Suppliers complain of the difficulty in bidding on Bank contracts. 67 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • The terms and conditions of contract do not align with industry norms. Bank documents evolved from the infrastructure procurement documents (FIDIC based) and are not typical of industry practice. • The Bank does not easily allow some procedures that are commonly used in the private sector. Sole sourcing and direct or competitive negotiation are examples, especially where a purchaser may already use a particular technology or product and therefore must purchase the same technology for expansion projects (Box 2.8). 75 Framework contracting, where the client can purchase from a list of qualified contractors, is another. Box 2.8. Negotiation to Procure ICT Systems Greater use of negotiated contracting has been advocated (by the Transport, Water, Information, and Communications Technology unit, for example) as an effective procurement method and as more in line with industry practice. Direct negotiation with a single supplier (sole source) may be used when, for example, an expansion of an existing system is required. Many factors need to be considered, such as compatibility of equipment and training of staff. Although the guidelines provide for sole-source procurement (clause 3.7) specifically in such circumstances, Bank no-objection is required. Competitive negotiation involves simultaneous negotiation with more than one supplier; the process typically culminating with interested bidders submitting a “best and final” offer. The method is provided in the UNCITRAL model on procurement. The European Union rules provide for competitive dialogue, and this is commonly used by governments and the private sector. Some Bank staff advocate the adoption of one of these models as more efficient, albeit with transparency and fairness concerns. To a certain extent, negotiation of technical and commercial factors can be accomplished under the Bank’s two-stage bidding procedure (see Box 2.5), although negotiation is only allowed in the first stage, at which point prices are not known. The second stage (price proposal stage) must be conducted without negotiation. Source: IEG. IEG COUNTRY SURVEY FINDINGS—EXPERIENCE WITH ICT PROCUREMENT IEG field visits indicated that present Bank Guidelines for ICT procurement are problematic, with only a “modest” score from country clients and Bank procurement staff, in terms of effectiveness. In five IEG case study countries, there was experience with ICT procurement. The following views were noted: 68 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • In all five countries (Azerbaijan , Indonesia, the Philippines, Tanzania, and Turkey), procurement of ICT was said to cause delays that negatively impact achieving project outcomes. • Development of technical specification is a common problem. System requirements are fluid and difficult to manage. This is partly a procurement issue, as specifications are seldom fully understood when tender documents are initially being prepared. It is also a contract implementation issue, as requirements will inevitably change during implementation and flexible processes are needed to accommodate this. • The guidelines and standard bidding documents have limitations when it comes to procurement of complex systems. They are seen as effective for straightforward procurement, but less so for procurement of complex systems. Two-stage bidding is held out as a good option for procurement of complex ICT systems, as it gives added flexibility to enter into dialogue with bidders on technical requirements. However, in both Indonesia and Turkey, two-stage bidding was criticized for adding time to the procurement process and not adding value. • There is an excessive emphasis on competition, where this may be problematic. In Turkey, the task team leader thought that there was a good rational for direct contracting so as to obtain technology that fit with an existing system, but was unable to get it approved (Box 2.9). In Tanzania, it was also commented that sole sourcing should be given greater consideration. • Skill shortages are a common problem, for both Bank and implementing agency staff, especially in projects being implemented through local governments. Because they lack knowledge, task team leaders and Bank staff do not utilize the flexibility that the Bank’s procedures allow. Consultants can sometimes but not always help. In Indonesia, consultants were used extensively and Bank headquarters staff was heavily involved, yet the procurement failed. The project implementing unit accused the Bank of being partial to certain suppliers, requiring arbitration and further delaying procurement. • ICT systems typically involve change management with attendant political complications. In Indonesia, one of the projects involved tax reform, which threatened vested interests. Moreover, the implementing agency was not technical (as compared to an electric utility for example). In the view of the task team leader, these were the primary factors contributing to the failed procurement. It implies that the problems that arise during procurement are often rooted in project design and other factors. 69 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Box 2.9. Using Sole-Source Procurement for a System Upgrade Turkey provides an interesting example of when to use sole-source procurement, as well as the difficulties in obtaining Bank approval—but also, the value added that Bank advice can offer. Two projects were undertaken to expand and upgrade a supervisory control and data acquisition system by the operator of the electric power utility. The first project expanded the existing system. The implementing agency initially wanted to sole source the procurement to the supplier of the existing system. The Bank encouraged competitive bidding, arguing that other suppliers could provide the expansion equipment and that it would function technically with the existing system. The implementing agency agreed. The supplier of the original system won the bid, but at a price lower than the estimate. The competition was effective in obtaining a lower price. The second project is currently being implemented to upgrade the entire system. The Bank initially insisted on competitive bidding, but the implementing agency argued that a system upgrade affected the entire system operations. To change equipment to a new vendor would require extensive changes to operations including retraining of staff. The Operational Procurement Review Committee agreed to the sole-source procedure, but it took one year to get Bank approval even though procurement specialists and the task team leader supported it from the outset. The example highlights certain issues. First, it is difficult and time consuming to obtain exceptions to normal procedures, to the point of serious project delays, and even with widespread support for the exception among Bank staff. Second, there is a reluctance to trust the implementing agency, even a technically knowledgeable and competent one like an experienced power utility. It raises the question of whether the Bank has superior expertise to override or even advise the purchaser. Last, the example highlight’s the Bank’s focus on competitive bidding and the difficulty it has in approving noncompetitive procedures. Source: IEG analysis. CONCLUSIONS Based on the above evidence, IEG concludes that Bank systems for ICT procurement have improved in the recent past, for example, with regard to its ICT bidding documents and their scoring process, two-stage bidding, concessions to the need for sole source, and clarification of conflicts of interest for consultants. Yet difficulties remain, reflecting the adaptation of the present system from one designed for a very different purpose. As a result, the standard bidding documents remain cumbersome and confusing (to users and bidders), and the Bank is not aligned with industry norms. 70 CHAPTER 2 ADAPTING TO EVOLVING NEEDS A second set of issues arises from the need to better educate task team leaders and procurement staff to better understand and more effectively use the present system. Although the system has flexibilities, many are discretionary and the complex chain of clearance for no-objection can lead to significant delay. Country capacity is sometimes—though not always—a constraint. Better project design is also an issue. Finally, though the Bank has offered advice and support for contract implementation, its interventions have been somewhat skewed toward upstream transactions. The limited present standard bidding documents for ICT are noted by clients. The clients note that the Bank should undertake an independent study of the practices commonly used by industry and governments (including the European Union) to align the Bank’s documents with best practice. It is noted that European Union directives do not include standard bidding documents; however, the European Commission has developed some for its own use. The Multilateral Development Bank Heads of Procurement has developed harmonized model documents for ICT procurement, but the Bank has not adopted them. 76 Limitations in the possibilities for dialogue between bidders and procuring agents are also referred to. Although the Bank’s two-stage bidding procedure provides the opportunity for a dialog to adjust technical requirements and commercial terms in the first stage, during evaluation of non-priced technical proposals, this is not in the second stage, during evaluation of price proposals. However, the review found that two-stage bidding is not frequently used for procurement of complex ICT systems because of perceptions that benefits do not outweigh the longer time involved. To enhance opportunities for dialog, the Bank could review options for the adoption of competitive negotiation / competitive dialog processes, similar to that of the UNCITRAL model law, and European Union Directives, and subject to borrower capacity. Greater use of direct negotiation with preferred suppliers (sole source) might be used less restrictively where circumstances warrant. Also echoed here, to a greater degree, were process difficulties caused by rigidities in oversight. Obtaining Bank no-objections can be difficult and time consuming because of low approval thresholds and the “chain of command” approach. Mechanisms are needed to fast track no-objections and approvals of exceptions. 77 At lower levels, inexperience may explain, in part, a reported rigidity of oversight. Better incentive structures are needed to address the risk-averse culture. Task team leaders, consultants, and especially field procurement specialists and others involved in project design would benefit from training, including in options for complex ICT procurement—for example, the use of two-stage bidding, direct 71 CHAPTER 2 ADAPTING TO EVOLVING NEEDS contracting, management contracts, framework contracts, and so forth, with guidance on how to apply them. 78 Implementing agency capacity is frequently limited, and there is a need to take measures to build capacity of the implementing agency well ahead of procurement, ideally as part of an overall ICT capacity-building strategy, prior to or during project preparation. Priority could be given to those ministries or departments with a long- term outlook of implementing many ICT projects and with an established procurement function. In some cases, consultants could have an important role to play in agencies that are technically weak. Bank specialists’ advisory role can come into conflict with the Bank’s oversight role and the need to remain neutral—for example, in Indonesia, where it was reported that the borrower accused Bank headquarters staff of favoring certain suppliers. Good design leads to better project outcomes, more so for projects involving large and complex ICT systems. Unique aspects of ICT projects that can impact procurement include a plurality of shareholders, for example, a financial management system with links between treasury and audit. Synthesizing competing demands and handling political interests is challenging, but if not done properly at the design phase. Unusual procurement methods—for example, sole source—can be built in to the procurement plan. Appropriate contract packaging—keeping procurement packages to a manageable number, ensuring packages are independent of one another, and ensuring single responsibility by avoiding splitting hardware and software contracts—would be beneficial. The Bank could also examine the extent to which contracting for information technology services could be used. And during implementation, discipline is needed to freeze the design so as to avoid continuous design modifications that keep adding features and functions as the system is implemented. These problems are not unique to the Bank, and IEG recognizes that every procurement system struggles with these challenges in ICT procurement. Gartner Research reports that the success rate in large private sector information technology projects is about 50 percent, and Bank outcomes should be viewed with realism (Gartner Group 2012). Procurement in Public-Private Partnership Investments Traditionally the Bank has loaned to client countries’ governments to finance public sector projects. But today, even in areas of traditional lending such as infrastructure, government agencies partner with private sector parties, which contribute financial 72 CHAPTER 2 ADAPTING TO EVOLVING NEEDS resources to deliver a public service or asset. This helps leverage scarce resources and can also have the advantage of bringing market discipline to the project. Yet lending to the private sector poses procurement-related challenges. The private sector typically uses commercial market practices to enter into deals that are financially attractive and bankable. It is also risk averse—it is not only the expected profit but also the certainty of the profit that ensures a total return. Risk includes factors such as strategic business partnerships, supply chain reliability, confidentiality of commercially sensitive data, dealing with developer-initiated proposals, working with preestablished concessions, and guarding against conflicts of interest. Public sector procurement faces a different set of specific factors: public accountability, open competition, fair treatment of prospective partners, and minimized subjectivity. When the private sector uses public funds, including Bank lending, the challenge is to strike a balance between public sector principles and commercial market practices. To the extent that the Bank’s public sector-oriented procurement processes do not match the demands of public-private partnership (PPP) projects, this impedes uptake of PPP investments. In view of the need to leverage Bank resources in a world where private investment is a growing share of the total, better procurement systems for such situations are critical for the Bank. IEG finds that although there has been recent improvement, there significant hurdles remain. • The Bank’s recent draft on procurement in private sector projects and transactions (World Bank 2013a) still has areas that lack clarity, especially with regard to the scope and standards of the Bank’s review of private investor procedures; situations where the Bank enters late, and is not involved in the design (including the design of procurement processes); situations where the Bank might be a minority investor; on-lending arrangements; and unsolicited proposals. • The Bank is seen as slow and ill equipped to adapt to the procedures of others, especially where quick response is needed, for example, to round out a financing consortium. Unclear guidance (for example, on competitive procedures, unsolicited proposals) and arrangements that rely on “acceptability” and where deviations require "waivers" are not a good option and lead to a lack of predictability and delay. • In the same vein, the Bank does not adequately recognize commercial confidentiality concerns that may make large players unwilling to share some information with a minor player, if it is considered commercially sensitive. 73 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • Even competitively selected concessionaires and other private sector partners can enter into conflicted downstream agreements, and Bank guidance in this regard is unclear. • Bank policy, although very similar to those of other MDBs, is not fully aligned with European Union practices; one important difference is the degree to which negotiation is permitted to arrive at a PPP concession agreement. PUBLIC-PRIVATE PARTNERSHIPS IN THE BANK’S PORTFOLIO Figure 2.5 illustrates Bank activity in PPP projects over the past 11 fiscal years. 79 The number of projects has shown a trend increase of about 70 percent over the period, from 19 to 32 per year. Financing commitments rose substantially after 2008, from $2.7 billion, peaking at $7.2 billion in 2011, possibly in response to the global financial crisis; they have since dropped to $4.8 billion in 2012. The data understate the Bank’s PPP activity, as some of the Bank’s work is advisory, leading to viable projects, but without Bank financing. IEG was told that the Bank would likely do more PPP loans, but it is constrained in part by procurement requirements. Figure 2.5. World Bank PPP Projects—July 2002 to December 2011, by FY Source: IEG database. Note: PPP projects for FY10–12 were compiled by and under the guidance of Get PPP and expanded to include noninfrastructure projects. The PPP projects for FY 2002–09 were estimated by IEG. IEG’s evaluation is based on four elements: (i) a portfolio review of 46 PPP projects over the years FY04–12; 80 (ii) interviews with Bank procurement staff from all regions and the anchor, with specific experience in PPP transactions, the Bank Get PPP anchor, the legal department, and IFC staff; (iii) comparisons with other countries; and (iv) IEG’s 11-country field-based survey of procurement. 81 Although earlier versions of the Bank’s procurement guidelines included clauses that recognized the unique procurement requirements posed by projects involving the 74 CHAPTER 2 ADAPTING TO EVOLVING NEEDS private sector, in 2011 revisions were introduced to increase flexibility (World Bank 2011f). The revisions make explicit reference to PPP projects. The revised policies are less prescriptive, tending more toward agreeing to “acceptable” procedures, typically defined by the principles spelled out in Section I of the guidelines (open competition, quality and transparency). These revisions are reflected in clauses 3.13, 3.14, and 3.15, which modify clauses 3.12 and 3.13 of the previous version of the guidelines. • As in earlier versions of the guidelines, concessions, or similar forms of private sector arrangements, are ideally selected competitively. If they are, the concessionaire or entrepreneur may procure according to its procedures and need not follow Bank procedures. The former requirement to select the concession using ICB was, however, relaxed to the more general “open competitive bidding.” Also, consistent with earlier versions of the guidelines, where a concession is not selected competitively, procurement by the concessionaire or entrepreneur (called “downstream procurement”) is to be competitive. However, the former requirement to use the Bank’s ICB procedures was relaxed to include invited bids and for smaller valued contracts, “methods acceptable to the Bank.” Consulting services were also added to items that the Bank may finance, procured according to the Bank’s consultant guidelines. • A new clause in the Bank Guidelines provided for, on an exception basis, Bank financing of incumbent concessions (including state-owned) and for the concessionaire or entrepreneur to apply its procurement procedures where Bank determines they are “acceptable.” The clause applies certain guideline- specific conditions related to conflict of interest and arrangements of state- owned enterprises and reiterates guideline requirements to ensure fair competition, transparency, economy, quality, and efficiency. It also requires the Bank to conduct postreviews during implementation to assure that procedures remain acceptable and are followed. • As in earlier versions, the revised guidelines address situations where the Bank funds an intermediary that on-lends to a beneficiary; the overriding requirement for procurement is the use of well-established procurement methods and commercial practices “acceptable to the Bank.” Where the beneficiary is investing in a PPP project, revisions to the 2011 guidelines direct the reader to specific clauses related to PPP procurement (summarized above), which helps define “acceptability” in terms of awarding PPP concessions. However, the responsibility for assessing procurement arrangements and oversight is transferred to the intermediary, under terms spelled out in the project documentation. 75 CHAPTER 2 ADAPTING TO EVOLVING NEEDS In the recent revisions to the guidelines, circumstances related to financing ongoing operations (incumbent concessions) are addressed. In addition to the guidelines, guidance on what are considered “acceptable” procurement arrangements is provided in a Guidance Note of September 2010 (World Bank 2010c). Although the Guidance Note predates the issuance of the 2011 revised procurement guidelines, it anticipates the revisions. For example, an appendix details what is considered “open competitive procedures acceptable to the Bank.” Guidance is also given in such areas as dealing with unsolicited proposals and how to establish oversight when working though financial intermediaries to address the quick response aspect commonly found in PPP on-lending arrangements. The Bank plans to update the Guidance Note to more fully reflect the revised guidelines. In January 2012 the Operations Risk Management, Operations Policy and Country Services Vice Presidency issued a draft note to aid the ongoing procurement policy review with respect to private sector projects. The note covers PPP as one of four categories of private sector engagement in projects, the others being direct lending to private sector entities, financial intermediary lending, and guarantees. The note proposes ways to fully utilize current features of the Bank Guidelines—it does not propose changes to Bank policy. This is a valuable initiative to improve PPP procurement within the existing policy framework.82 However, certain areas for concern remain: • The extent to which the Bank would review and approve procurement capabilities and procurement actions of private borrowers is not always clear, nor are the criteria for such review. A practical concern among procurement specialists in the field and among task team leaders, as found by the IEG review, is the lack of clarity about the extent to which they should get involved in the procurement practices of concessionaires or private sector parties. The approach proposed in the note does not address this. • The note does not address circumstances where the Bank is not able to exercise oversight and influence in a step-by-step procurement process. It does not address special circumstances, such as where the Bank is a minority player in a project, in which case the major investors may not agree to the Bank setting terms for procurement. Similarly, there are situations where the Bank comes into a project late, after all procurement arrangements are in place or some procurement completed. Furthermore, the Bank is sometimes asked to work under an umbrella arrangement, where it needs to approve numerous small investments quickly. 76 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • The note does not address sanctioning, debarment, and right-to-audit provisions, matters that the IEG review identified as needing attention, especially where Bank requirements come into conflict with local laws. • The note does not elaborate how the Bank’s approach differs from that of IFC or why this is so. For example, the Bank applies a different level of review of private borrower practices and retains a discretionary ability to review certain critical transactions. It would also be helpful to explain the extent to which the Bank differs from other IFIs such as the European Bank for Reconstruction and Development and the European Investment Bank, and how Bank policies deviate from those of the European Union, which is especially important for clients working under European Union directives. • The note proposes the formation of a Business Review Committee to bring consistency of approach, and provide better access to Bank experts. While this could be a positive move, care is recommended to not add another layer of review. IEG reviewed the extent to which current Bank proposals are in line with other MDBs’ or client countries’ national policies (details are in Appendix B). Not surprisingly, there is little variation in the PPP procurement policies of the MDBs, reflecting 10 years’ efforts to align guidelines. Yet there are noteworthy differences between the Bank and IFC. IFC does not have documented requirements regarding procurement in its PPP investments. It reviews a concession agreement (and key contracts in a typical transaction) as part of its overall assessment of credit, development, and reputational risks in the transaction, rather than as part of a self- standing procurement exercise. IFC takes a commercial approach, assessing if the PPP arrangement was the result of an open and competitive process; if prices are reasonable; if the project complies with local laws; if it is financially viable; and if the parties are reputable, competent, and financially sound. This reflects the commercial orientation of IFC. Unlike the World Bank Group, which separates private and public sector lending operations between two different organizations (IFC and IBRD), most other MDBs have an in-house private sector lending window. Their operations include departments dedicated to private sector lending, yet they manage to accommodate this private sector lending function within the terms of procurement policies that are almost identical to the Bank’s (details are discussed in Appendix Box B.2). IEG REVIEW OF EXPERIENCE IN BANK OPERATIONS The IEG portfolio desk review looked at 46 projects approved in an eight-year period, up to December 31, 2011. The intent was to review projects that were approved 77 CHAPTER 2 ADAPTING TO EVOLVING NEEDS relatively recently but that were under implementation, as these would exhibit procurement-related issues. IEG selected the sample from a list of 73 projects based on data from financial and private sector development and Get PPP. 83 Of those 73, 25 related to creating an enabling environment and pipeline development and were not reviewed. The remaining 46, which involved Bank investment, were subjected to the review. Almost half (22) were from Africa, whereas Latin America and the Caribbean and South Asia accounted for nine and seven, respectively. 84 East Asia and Pacific accounted for four. Europe and Central Asia and the Middle East and North Africa accounted for two each (Table 2.4). Table 2.4. IEG Sample of Bank PPP Projects Reviewed (by region FY04–12) FY Region AFR EAP ECA LCR MNA SAR Total 2004 1 1 2006 1 1 2008 1 1 2009 1 1 2010 6 1 5 1 2 15 2011 12 2 1 4 4 23 2012 2 1 1 4 Total 22 4 2 9 2 7 46 Percent 48 9 4 20 4 15 Source: IEG portfolio analysis. Note: PPP = public-private partnership. Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; MNA = Middle East and North Africa; SAR = South Asia. IEG’s main findings are summarized below. • The Bank is typically the lead financier, and it uses investment lending to finance government investment in a PPP. The portfolio review showed that virtually all the projects used investment lending. 85 In 31 of the projects (67 percent), the Bank was the lead financier, and in the remaining 15 (33 percent) the Bank provided between 8 and 50 percent of the financing. In 36 loans (78 percent) Bank lending financed government investment in the PPP. In only six loans did Bank lending go to a financial intermediary for on-lending. 86 Four loans were to a special purpose vehicle. IFC was involved in four projects, three as advisor and one as equity investor.87 • In the majority of these projects the Bank is involved from the beginning and sets the procurement arrangements. In 41 (89 percent) of the projects, work did not commence until Bank financing was approved. The Bank was a late entrant in less than 5 percent of projects reviewed. Only one had significant 78 CHAPTER 2 ADAPTING TO EVOLVING NEEDS prior investments to which the Bank had to adapt. In 44 projects (96 percent) the Bank was involved in the design and was able to establish procurement arrangements—in the remaining two, the Brazil N 2 O Emission Reduction Project and the Uruguay Wind Power Farm Project, the Bank had to adapt to prior established procedures. These low numbers likely reflect the problems that the Bank has in entering after procurement arrangements are in place— were it not so difficult, the Bank would probably be able to take on more of these projects. Table 2.5. Financing Structure of Bank PPP Projects Financing structure Projects % Investment loan to cover government investment 36 78 Loan to financial intermediary for onlending to PPP investments 6 13 Investment loan to special purchase vehicle 4 9 Total 46 Source: IEG portfolio analysis. Note: PPP = public-private partnership. • Procurement problems were reported in over a third of ISRs. Of the 21 projects in which ISRs reported procurement issues or delays, the issues were directly or indirectly related to it being a PPP project in 17 (37 percent of all projects). • There are a significant number of PPP concessions awarded to incumbent concessionaires. Eight projects (18 percent) involved incumbent concessionaires. Of these, three were not competitively selected. IEG’s country surveys included interviews with a range of stakeholders to obtain their views on the suitability of the Bank’s procurement procedures for PPP projects. On average, across countries, and across interlocutors, procurement provisions in Bank guidelines for PPPs were perceived as “modestly adequate,” with an average score of 2 out of 4. In 8 of the 11 countries surveyed, the Bank team had no PPP operational experience, although in some instances there is a high level of interest in PPP (Box 2.10). The following views were noted: • In the Morocco Ouarzazate Project, a 500 MW solar energy complex, the government observed that the Bank tried to adapt its standard bidding documents but that this proved unsuitable for the project. Washington-based Bank staff, rather than country procurement staff, advised the implementing agency, but they did not have appropriate experience to design bidding documents, instructions to bidders, or contract conditions. A procurement expert has been hired to help resolve these issues. 79 CHAPTER 2 ADAPTING TO EVOLVING NEEDS • In Tanzania, the Bank assisted with developing a PPP policy, a PPP law, and regulations. A private company is in advanced stages of discussion with the government to invest in a water project. It is thought that the Bank’s requirements of competitive bidding, at a stage when the private partner is already in advanced discussions with a supplier, would not allow Bank financing. The company is hesitant to reopen the process and has suggested that the Bank evaluate the transaction on its merits to assess whether it meets the standard of a competitive outcome. The company has suggested that IFC could be approached, as the transaction could meet IFC’s standard of a competitive commercial deal, in terms of outcome (result, price). • In Bangladesh, concern was expressed over the Bank’s engagement with a power project. The Bank was approached to provide a relatively small share of the financing, but it entered the process when bids were already evaluated and ready for award, yet would have required a “retrofitting” to Bank Guidelines. • In Peru, there is a general view that the Bank’s current procurement procedures are not adequate to meet the needs of more sophisticated projects or the use of new lending modalities such as PPP. The Bank has no PPP portfolio in Peru. Furthermore, task team leaders were of the view that Bank procurement staff are not sufficiently conversant with PPP requirements. More generally, they were critical of the Bank’s insistence on competitive bidding, suggesting that an IFC approach, which looks at reasonableness of price and results, is more practical. They added that the Bank’s PPP procurement guidance is inadequate, particularly in such areas as contracting on the basis of lowest subsidy, or for service delivery contracts. There is also a general view that the Bank’s Guidelines do not accommodate many common modern approaches to PPP; for example, in the Philippines the Build- Operate-Transfer law mandates “Swiss challenge” procedures for all unsolicited PPP offers. In Tanzania, respondents pointed out that procurement of toll roads on a “least-subsidy basis” is now very common. Box 2.10. PPP—Missed Opportunities in Other Countries? In the majority of field visit countries the Bank had not initiated any PPP lending. Yet there is a prevailing view that PPP is important. The following comments were extracted from field visit reports: 80 CHAPTER 2 ADAPTING TO EVOLVING NEEDS Indonesia: There is no experience with PPP among Bank staff (including task team leaders and procurement staff), government (including implementing agencies and regulatory staff), or civil society (including consultants and contractors). The government views PPP as promising. Philippines: The Bank is not investing in any PPP projects in the Philippines, and implementing agencies have no experience working with the Bank on PPP projects. The unsuitability of the Bank’s Guidelines for small-scale decentralized PPPs, especially those supported via on-lending from the Development Bank of Philippines, was noted. The Bank’s procurement guidelines were seen as too strict for the private sector and made on-lending uncompetitive with other PPP financing sources such as private banks. Yet there are a number of PPP initiatives that do not involve the Bank, in education, health, power, and infrastructure. Noted was the department of education initiative to construct 10,000 schools over the next three years under a build, lease, and transfer scheme, as well as a (Manyland) water concession supported by ADB, and a Manila water concession supported by IFC This has ADB support. The Bank’s PPP activities are limited to supporting services—consultancy to provide technical support to a local government to develop a PPP water treatment plant project, for example. This is support for technical design and specifications, not financial or procurement advice. Azerbaijan: The Bank is not investing in PPP projects in Azerbaijan, and implementing agencies have no experience working with the Bank on PPP projects. Bank management notes that there is no existing legal framework in the country for PPPs. PPP initiatives in power and urban transportation, which do not involve the Bank, have been proposed. Turkey: The Bank is not investing in PPP projects in Turkey. The Ministry of Development (formerly the state planning organization) has been struggling with drafts of its PPP law, attempting to make it compatible with European Union legislation. Although deemed to be finalized by June 2013, it has still to be enacted. Many implementing agencies have limitations that would affect the quality and processing of PPPs. Nevertheless, Turkey has launched an ambitious 10-year PPP program and is currently managing a number of large PPPs, independent of Bank involvement, with 3 major infrastructure PPPs signed over the past year. Source: IEG field visits. CONCLUSIONS There is consensus that PPP is an important instrument for the Bank, though it is also widely thought that the Bank’s public sector-oriented procurement processes do not 81 CHAPTER 2 ADAPTING TO EVOLVING NEEDS match the demands of PPP projects. The guidelines and guidance notes tend to assume that the PPP project will be stand alone, that the Bank will be a major player, and that the Bank will be involved in the design of procurement processes from the beginning. This impedes uptake of PPP investments, especially in areas such as gap financing or situations where the procurement arrangements have either been put in place earlier, where the Bank is a minority or late contributor, or where the Bank operates through a financial intermediary. The Bank is seen as ill equipped to adapt to the procedures of others. Additionally, the Bank may need to respond quickly, to round out a financing consortium for example. Here, too, it is seen as unable to respond quickly to assess procurement arrangements. This is the experience of the India Infrastructure Finance Company Limited Project (see Box 2.11). Although current Bank policies, such as those introduced through recent revisions to the guidelines and articulated in the Bank’s guidance note, provide added flexibility, there is a need for more clarity in regard to review of “acceptable” procedures. Recent revisions may have shifted difficulties from the rules themselves to how they are interpreted and applied. Much now depends on how intent individuals may be to "do what they can to make it work." Box 2.11. India—A Tale of Two PPPs Although India was not included in the country surveys, IEG contacted India-based procurement staff, given the Bank’s considerable experience with PPP financing in India, and discussed their experiences related to two projects: (i) the Karnataka State Highways Improvement Project (KSHIP), providing Bank financing to the government for highway construction and operation concessions; and (ii) the India Infrastructure Finance Company Limited (IIFCL) project, where the Bank finances a government facility that provides gap financing in infrastructure projects, on an on- lending basis to IIFCL, a minority cofinancier. The investment reviewed was for a power transmission line concession. The two projects offer different experiences. With KSHIP, the Bank has had a long- term involvement and more than a decade of learning and adaptation. In the case of IIFCL, a new agency, the Bank is a minority player. IIFCL financing is limited to 20 percent at most of the value of a project (a “gap financing” model), and it typically relies on the lead financier for due diligence. For the IIFCL project, the Bank provided retroactive financing. Power Grid, though highly experienced at procurement and familiar with Bank’s procedures, used its own procedures, not knowing of the Bank’s participation at the time of procurement. Lessons learned 82 CHAPTER 2 ADAPTING TO EVOLVING NEEDS from these projects are: • Aligning government documents with Bank requirements is difficult, but adds value. In KSHIP, the implementing agency used model documents (RFQ, request for proposal, model concession agreement) issued by the government planning commission. Documents were redrafted to comply with Bank policy. Bank procurement staff feel that the implementing agency benefitted from adjusting to Bank requirements (for example, by securing the trust of bidders), although, unlike for investment projects, the Bank does not have standard documentation or a standard concession agreement for PPP procurement. • Getting involved early is important. In both IIFCL and KSHIP, staff emphasized the importance of Bank involvement in the design of processes, development of transaction documents, and building implementing agency capacity, at an early stage. • Clarify that procurement provisions specific to fraud and corruption (including right to audit) are “non-negotiable.” The government agreed to incorporate Bank policies in the procurement process and made the necessary amendments. In the case of IIFCL, this was done subsequent to the contract via a side letter. The Bank also insisted that Bank-sanctioned firms be barred from bidding but would not agree to allow use of India’s sanction list. Some disagreements between the Bank and implementing agencies relate to good practice, for example, in the handling of complaints, dispute resolution, and arbitration. Implementing agencies required bidders to submit a declaration that they would not challenge the outcome, as a condition of bidding. The Bank, appropriately, required proper complaints, dispute resolution, and arbitration systems. Other areas of disagreement are unique to Bank requirements and derive from its guidelines. For example, the Bank objected to eligibility criteria that invoke national security concerns (possibly giving preference to domestic companies), the involvement of state-owned entrerprises, and joint ventures of bidders with the implementing agency (Power Grid). National preferences, national security, and contracting with state-owned enterprises (including joint ventures) are common in many countries (see Chapter 3). Other guideline-specific disagreements arose regarding the implementing agency limiting the number of prequalified bidders and its use of two-envelope bidding (standard practice in India, and in many other countries, but not allowed by the Bank Guidelines), although eventually the Bank agreed. These issues are general to all procurement and not PPP specific, but illustrate the difficulties of imposing Bank policies into a new environment that involves parties other than government, where 83 CHAPTER 2 ADAPTING TO EVOLVING NEEDS the Bank does not necessarily drive the process. A management update was provided in June 2013 on the ongoing restructuring of the IIFCL Project, downsizing the loan amount from $1.195 billion to $195 million. New flexibilities have been added and the Bank has now approved a risk-based approach toward procurement due diligence. IIFCL is now allowed to follow a structured framework assessment of concessioning authorities that looks into various aspects such as governance structure and procedures for prequalification, bidding, and contracting. The Bank will not carry out a prior review of subprojects or IIFCL’s due diligence. These steps will make IIFCL’s due diligence of subprojects more streamlined and efficient. The Bank’s fiduciary safeguards will be to carry out a comprehensive annual audit of subprojects; a pre-assessment of IIFCL to ensure that it has the capacity to discharge its responsibility as agreed and a periodic capacity assessment to ascertain the established capacity is sustained over time; agreement with IIFCL on the necessary next steps to enhance its capacity to the level consistent with its new roles and responsibilities; and the right to carry out an Integrity Audit of the Developer, as done by IFC, to ascertain that the principles of procurement are adhered to in the selection of the developer. Source: IEG field visits. Although there are no standard approaches to procurement in PPP projects, there is scope for more tailor-made model procurement arrangements. In India, the lack of standard documentation was identified as a key constraint that the Bank might address. Skills are limited; few procurement specialists have experience in this area. The Bank could also make better use of dialogue during the selection process to arrive at a PPP concession agreement, for example, as the European Union allows. Countries following European Union practices might employ “competitive dialogue,” where a dialogue is conducted with bidders throughout the procurement process, aimed at obtaining the best cost and technical terms for the PPP agreement. The Bank can also make efforts to accept certain commercial considerations, such as the unwillingness of larger players to share all information with a minor player, if the information is considered commercially sensitive. This reportedly happened on one Bank PPP investments (India). When a concession is awarded on the basis of competitive bidding, the guidelines allow procurement to follow the procedures of the concessionaire. Yet conflicts of interest can still arise, for example, when a concessionaire awards over-valued contracts to an affiliate. It is conceivable that competitively selected concessionaires enter into conflicted downstream agreements, whereas those not competitively 84 CHAPTER 2 ADAPTING TO EVOLVING NEEDS selected pursue best value procurement. The Bank might consider other methods to control conflict of interest, such as through due diligence of the concession arrangements, as undertaken by private parties. It seems unlikely that the Bank can take a purely private sector approach to PPP, yet, somewhat like IFC, the Bank too needs to be willing to adapt to the legal framework of the project country, in part to avoid putting other parties in a situation of conflicting requirements. One complication for the Bank is that IFC might act as financial advisor. With its stronger orientation toward the private sector, it may not provide procurement-related advice that meets Bank policy. To the extent that IFC and the Bank are seen from the outside as the same agency, this can confuse clients. PPP contracts are output based and of long duration. They are designed to deliver a service (transport services, for example) and might persist for periods in the range of 20 years. This makes them quite different from a project to provide inputs (to construct an airport, for example) that only last long enough to complete construction. Over the duration of a PPP contract, assumptions of outputs (volume, pricing) and economic conditions may need revising. To manage PPP project outputs, contract management becomes as important as the process by which the contract is arrived at (procurement), and guidance on good practice in contract oversight may be needed. 85 3. Managing Risk and Increasing Transparency The Bank has a highly developed framework to manage procurement risk (Box 3.1). 88 Ex ante controls are present at the transactional level with procurement staff involvement in the prior review of a large proportion of individual transactions. Ex ante controls also operate through the assessment of procurement risks at the overall project level. The Bank also employs ex post controls on procurement risk, on samples of post-reviewed contracts. There are mechanisms for complaints and allegations of wrongdoing in procurement, and in select cases for investigations of allegations by its Integrity Vice Presidency (INT). Box 3.1. Procurement Risk Management in Recent IEG Evaluations IEG’s Review of IDA Internal Controls: An Evaluation of Management's Assessment and the IAD Review (2009c) found that there were generic weaknesses in controls over financial management and procurement processes at the Bank, in part because of regional variations in process. IEG found that no controls explicitly addressed the risks of fraud and corruption in IDA operations. IEG’s subsequent evaluation of the Bank’s remediation program (IEG 2010d) acknowledged the strengthening of fiduciary controls through the introduction of new risk-management tools but pointed out that the new systems would take time to be effective. The World Bank Country Level Engagement on Governance and Anticorruption (IEG 2012d) pointed to a lack of consistent and systematic treatment of corruption risks related to procurement. It cited the focus on ring-fencing methods in some countries but not in others as a reflection of the Bank’s inconsistency in setting risk tolerances. The evaluation also reported that the initiatives designed to manage the Bank’s reputational risks relating to governance and anticorruption were not necessarily the same as those that would help countries take on calculated development risks. Sources: World Bank 2011g; IEG 2009c, 2010d. These systems were reinforced in recent years following increased focus on fraud and corruption and reports of deficiencies in procurement risk management. This chapter reviews present procurement risk-control measures to inform the Bank’s proposals to move toward procurement processes that may require new trade-offs between risk and efficiency. IEG’s evaluation complements a recent review of procurement risk undertaken by the Bank’s Internal Audit Vice Presidency (IAD). Whereas IAD examined the level of compliance with project-level procurement risk-assessment tools, IEG reviews qualitative aspects of their performance in terms of preventing procurement risk. 89 This analysis 86 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY focused exclusively on process (or control) risk, because there are no indicators or tracked data on outcome risk. The Bank’s norms and directions in terms of procurement risk, and fighting fraud and corruption, need to be cognizant of international trends in this regard. Governance in public procurement is increasingly on the agenda of international organizations, underlining the need of countries to adjust to international norms if they expect to be part of the international economy. 90 Main Findings  Viewing all aspects of present procurement risk management, the overall conclusion is fairly positive, albeit qualified. In terms of design, the Bank has a highly articulated, and recently strengthened, set of risk-management instruments. However, they could be sharpened in focus, better integrated, and better used, in terms of data input and analysis of findings. The balance between specific tools could be revisited, in the light of a risk-efficiency trade-off.  In terms of results, most current measures of risk failure appear to be within acceptable levels of risk tolerance, based on numbers of misprocurements declared each year; procurement complaints, relative to bid awards; and INT referrals in relation to perceived fraud and corruption indicators in the procurement process. Ratios fall in ranges below 1–2 percent of contracts by number and value. Cases where the evidence from Post-Procurement Reviews/Independent Procurement Reviews (IPRs) shows that some countries to be seriously at odds with Bank procedures are already being monitored and managed.  The Bank puts considerable emphasis, in terms of its present risk-management framework, on ex ante risk controls through mechanisms such as prior review. Prior review instruments only partially reflect country or project risk. Overall, they could be relied on less as a risk control mechanism.  The Bank effectively applies its most restrictive and thus slowest review process (prior review of ICB contracts) to its most competitive and therefore least risky procurement method. 91 There is likely to be scope to reduce the risk efficiency trade- off by reducing the share of contracts that are prior reviewed and focusing prior review on high-risk contracts, as opposed to value based thresholds. “Methods” thresholds could incorporate better use of market information.  The findings of Post-Procurement Reviews and IPRs could be more strategically used.  Procurement risk was already being managed to a generally high standard prior to the introduction of the Procurement Risk Assessment Management System (P- RAMS). It does not appear that the correlation between risk identification and risk mitigation has been much improved, although P-RAMS is a well-intended effort to 87 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY offer more focused and standardized assessment and mitigation of procurement risks.  In principle, a key positive feature of P-RAMS is its dynamic aspect, now beginning to be articulated. Yet excessive frequent sequencing may be counterproductive. However, the P-RAMS template and process can be cumbersome and time consuming, especially with multiple implementing agencies, and it may foster a rigid “check the box” approach that limits added value. Risks identified in P-RAMS are not weighted or prioritized and can lead to index aggregation errors. Although the P- RAMS system has a facility for procurement staff override of automatic weightings, there is little evidence that this is exercised. Finally, the P-RAMS template could further sharpen its focus on fraud and corruption by featuring that risk as an additional risk factor.  Fraud and corruption risk is systemic and more Bank focus on the overall country level, rather than the transactions level, could increase effectiveness.  Client capacity issues remain a key risk area. Introduction Procurement risk in the Bank is not clearly defined or clearly distinguished from overall project risk (Box 3.2). Moreover, Bank focus, in terms of risk, tends to be on process risk (that is, non-compliance), rather than on outcome risk (that is, the risk of certain procurement objectives, or wider development objectives, not being achieved). IEG’s analysis necessarily also focuses on process risk, in the absence of instruments for or measures of the latter. The analysis focuses on the Bank’s principal present instruments to manage and to measure procurement risk, beginning with its core system of procurement thresholds, principally for prior review; its post-review system, and its risk assessment systems, notably the relatively recent Procurement Risk Assessment Measurement (P- RAMS) system. IEG also reviews available information on a series of measures of risk failure (misprocurement, complaints, fraud and corruption indicators) and the integration of procurement risk management with other areas of Bank risk management. Managing Procurement Risk—Prior Reviews and Thresholds CONCEPT AND USE OF THRESHOLDS A central element in the Bank’s ex ante risk management framework for procurement has been the establishment of an interlocking system of thresholds, defined in terms of contract value, which determine the level of scrutiny a procurement contract receives from Bank staff as well as the method of contracting applied (Table 3.1). Thresholds are critical tools for risk management, as they represent key pivot points in the trade-off 88 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY between procurement risk and efficiency. Thresholds are intended to reflect perceptions of risk to which the contract may be subject, in terms of the broad achievement of the Bank’s fiduciary objectives. Other factors reflected are conditions in the client country, in terms of country-level procurement capacity, the state of development of the private sector, and the quality of governance. At the project level, Bank procurement thresholds also take into account capacity of implementing agencies, based on their prior experience and record of performance, assessed in a procurement capacity assessment. Box 3.2. The Bank’s Taxonomy of Procurement Risk Bank procurement guidelines (OP/BP 11.00) do not define procurement risk. Some Bank reports refer to project risk, which has many nonprocurement components as “inherent risk,” whereas procurement risk is called “control risk.” In its turn the Bank’s procurement risk assessment measuring system, the P-RAMS template (which deals exclusively or mainly with procurement risk), calls its main risk element “project risk.” The Bank could benefit from more clearly defining a risk hierarchy, differentiating between types of procurement risk and distinguishing procurement risk from general project risks. Benefit would also be obtained from a unified risk taxonomy, so that all risk instruments whether in procurement or elsewhere, use a common terminology. The P-RAMS template should be amended to refer to procurement risk as “assessed risk” (premitigation) and “residual risk” (after mitigation). “Project risk” could be retained, but should be defined to include all inherent risk factors, including non-procurement factors. Source: IEG. Thresholds for prior review occupy the most important role in risk management, as they determine contract sizes, above specific values, that require the Bank to intervene upstream in the procurement process. Although the implementation of procurement in the Bank system rests with the borrower, the Bank specifically provides its no-objection at specific stages (see Appendix C, section on “prior review” and “clearance” for further details). These include, in the broadest terms: no-objections to draft bid documents or requests for proposals, bid evaluation report and recommendation for award, and any contract amendments. No-objection stages depend on the type of items procured (usually referred to as the category)—goods, works and consultant services. Prior review thresholds have been defined at a Bank-wide level, to set global maxima, and below this, at a country level, determined by regional procurement managers. These in turn are intended to serve as maxima at the level of individual projects, where project thresholds are determined by the task team leader and procurement specialist concerned, based on procurement risk assessments and the assessed capacity of the implementing agency as set forth in the procurement plan (World Bank 2011f). 92 89 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY Table 3.1. Prior Review, Procurement Method and Clearance Thresholds Type of threshold (revisions) Description Threshold set by Prior Review Threshold Level at which all contracts with an There are three different prior review thresholds: (1997, 2008, 2009) aggregate value above the threshold must Bank-wide prior review thresholds set by be prior reviewed by Bank procurement staff OPSOR; regional prior review thresholds set at and receive a “no objection” at pre the country level by the regional procurement established steps in the procurement managers; and project-level prior review process. Thresholds will vary by thresholds set by task team leaders and procurement category and method. procurement specialists taking into consideration the results of the procurement capacity assessment and assessed capacity of implementing agency. Procurement Method Threshold Level at which all contracts with an Method thresholds are set by the regional (Depends on Region) aggregate value above the threshold must procurement managers and they also can be set be procured through a specific procurement at the project level by the task team leaders and method. The most common threshold is for procurement specialists. NCB/ICB but there are also thresholds for Shopping and Short List National Consultants. For some countries the prior review thresholds set by the RPMs are the same as the procurement method thresholds. Clearance Level at which all contracts with an Thresholds/Mandatory Review aggregate value above the thresholds must Thresholds (1992, 2002, 2005, receive no objections by PS/PAS, RPM or OPSOR 2009) the OPRC. Source: IEG compilation. Note: ICB = international competitive bidding; NCB = national competitive bidding; OPRC = Operational Procurement Review Committee; OPSOR = Operations Risk Management, Operations Country Policy and Services Vice Presidency; PA = procurement assistant; PAS = procurement accredited staff; TTL = task team leader. In addition, prior review contracts are also subject to clearance thresholds, which determine who, in the Bank’s procurement hierarchy, gives the relevant no-objections to prior review contracts. At the lowest level, contracts are typically cleared by field office- based procurement staff or procurement accredited staff, and at one level up, by hub coordinators for a cluster of countries, especially in some regions such as Africa. Larger value contracts, usually above $25million but below $50 million, are cleared by regional procurement managers for each of the six regions. The largest contracts, above $50 million are subject to the clearance of the OPRC—under the office of the chief procurement officer, at the Bank’s center. 93 Related to prior review thresholds and sometimes coincident with them are methods thresholds that determine whether specific contracts will be undertaken through its ICB procedures; NCB; or less formal procurement systems, such as shopping. 94 Methods thresholds, however, depend not only on the risk perceptions of project or country entities, but also on the state of development of local markets, from the point of view of goods to be procured. The greater the extent to which local suppliers are able to meet a given need of the contracts, the higher the method threshold is likely to be. In some 90 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY regions and countries, at specific points in time, prior review and methods thresholds have coincided—thus contracts undertaken by the ICB method have been those also subject to prior review. However, in most countries, large NCB contracts can also be subject to prior review. Table 3.2 shows, within the group of contracts subject to prior review, that ICB contracts have declined sharply as a proportion of the number of contracts, and many contracts under other methods are also being prior reviewed, although ICB clearly dominates in terms of the total value of prior reviewed contracts. Table 3.2. ICB Prior Review Contracts for Goods and Civil Works (numbers and value) (FY00–12) Value of ICB No. of ICB No. of all prior ICB contracts No. of ICB Prior review contracts contracts as % of contracts as % of FY reviewed (G&CW) (G&CW) ($ (G&CW) (G&CW) ($ billions) prior review prior review contracts (000) billions) contracts (000) contracts contracts 2000 6.6 5.0 5.2 2.8 79 56 2001 6.5 4.6 5.2 2.5 81 55 2002 6.4 5.0 5.0 2.6 77 51 2003 6.5 4.8 4.8 2.4 74 50 2004 7.3 4.7 5.1 2.4 70 51 2005 8.0 4.5 6.0 2.1 76 47 2006 6.7 3.8 5.2 1.8 77 48 2007 8.4 4.1 6.7 2.0 81 51 2008 10.4 3.9 8.0 2.0 77 51 2009 9.6 4.6 7.1 2.0 75 43 2010 11.4 5.0 8.9 2.1 78 42 2011 12.3 4.7 9.5 2.0 77 42 2012 12.1 4.2 10.5 2.1 87 50 Total 112.1 58.8 87.3 28.9 78 49 Source: World Bank. Note: These data refer to goods and works alone. Adding consultant services reduces the proportion of ICB. Thresholds have been revised, though on an ad hoc basis, and systematic information on their evolution, and on previous levels of thresholds, is limited, partly because most such decisions have been taken at the regional procurement manager level on a country basis. 95 Looking at the past five years, Bank-wide prior review ceilings were found in 2008, set in terms of three risk categories (high, moderate, and low); revised shortly after in 2009 to a four-point risk scale (high, substantial, moderate, and low). The 2009 Bank-wide prior review thresholds are still in effect. Historical regional/country level prior review and method thresholds are available, in varying degrees of comprehensiveness, for each Bank region. 96 Clearance thresholds have been changed at least four times in the past decade. In 2002 the first Decision Authority Matrix that layed out clearance responsibilities for prior review as well as for single-source contracts, rebidding approval, and approval of request for extension of bid validity was used. A 2005 revision has Bank-wide thresholds for 91 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY clearance by regional procurement managers (RPMs) and by the OPRC, by procurement category. A further 2009 prior review revision outlines clearance thresholds for procurement assistants/procurement-accredited staff, regional procurement managers, and OPRC by procurement category (and by method for direct contracting and single- source contracts). 97 Although thresholds have been a central device for managing procurement risk and achieving transparency, they illustrate clearly the trade-off between risk management and other guiding principles: economy and efficiency. Lower prior review thresholds may reduce risk but are time intensive and increase procurement staff work load, possibly detracting from staff time spent on other aspects of procurement work, such as capacity development, as argued by the Bank’s procurement anchor. 98 Clients point to delays in the procurement process, especially when contracts have higher clearance thresholds and need Washington-based approval (as discussed in Chapter 4). Lower method thresholds may arguably increase potential participation of international bidders (thereby increasing competition) but could imply lost opportunities for local suppliers and markets, in addition to adding to time spent on procurement. In practice, depending on market capabilities, lower method thresholds may not even attract more international suppliers if there are a wide range of competent domestic producers (Box 3.3). PERCEPTIONS OF THRESHOLDS—IEG FIELD VISITS Review Thresholds and Clearance Thresholds Bank procurement staff acknowledge that current review procedures were perceived to be helpful for controlling fiduciary risk—reported by Bank procurement staff in Bangladesh. Private sector respondents (for example, in Azerbaijan) point out that reviews serve to limit mistakes overlooked by Project Implementation Units. Nevertheless, on balance, there was a view that current prior review threshold could be raised with beneficial impacts on workload and clearance time, and only limited impact on risk or competition. Recent increases in thresholds in many countries have had a positive reception among Bank procurement staff (Ethiopia, Mexico, Morocco, and Peru). And respondents from Indonesia, Azerbaijan, Bangladesh and Tanzania indicated that they should be raised. Respondents in Azerbaijan, Bankgladesh, the Philippines, Senegal, and Turkey also stressed the need to raise procurement method thresholds. In the Philippines, Bangladesh, Mexico, Turkey, and Azerbaijan, it was suggested by country respondents that insistence on ICB for procurement of goods and works as well as for the selection of consultants, when good domestic capacity exists, may produce 92 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY limited gain as ICB has a cost in terms of considerably longer process time required for no-objection. Insistence encourages local firms to take up joint ventures with foreign agencies where often the local partners beat the brunt of the burden of execution. Also in many parts of the Bank, ICB is synonymous with prior review; an approach questioned by Bank procurement staff who did not feel that large value necessarily implied high risk. Finally, there was a sense that risk aversion in the award of ICB contracts led to an over- emphasis on price, sometimes leading to questionable outcomes. 99 Calls were made for raising NCB thresholds accordingly (Azerbaijan and Tanzania) and also for making use of other methods. Box 3.3. Setting Method Thresholds to Optimize Competition at the AfDB Are there ways to raise methods thresholds, to enhance procurement efficiency, while minimizing the possible losses in terms of risk or competition? The AfDB shows that the trade-off can be quite accurately quantifiable, using relevant contract data. Using contract data from its accounting system, AfDB has shown that in works contracts foreign bidders entered bids on contracts for only those contracts in the top quintile (20 percent) of contract value. The AfDB concluded, accordingly, that to set the ceiling contract for ICB at any level lower than 80 percent of highest value would involve no increase in competitive bidding but would involve significantly more review work; put another way, if thresholds on ICB for works contracts were raised to around 80 percent of maximum contract value, there would be significant cost savings in processing time, but little if any loss of competitiveness, because foreign bidders would not in any case enter the market for the lower value contracts. The AfDB has conducted an analysis of all its methods thresholds on a similar conceptual basis which resulted in a new set of recommended thresholds for both works and goods, grouped into three sets of countries ranked by risk ratings. The World Bank has the capability to undertake analyses of methods thresholds, as illustrated above, even based on present Form 384 data. IEG’s analysis of the Bank’s methods thresholds also suggests that the bottom quintiles have a high share of local winners, although somewhat less, on average, than at AfDb (Appendix C). Overseas firms are also better represented in the top 5 percent. More disaggregated analyses would provide methods for a better quantification of the risk-efficiency trade-off. Systematic operational use could be made of such analyses, for example, for broadly comparable groups of countries and categories of goods and services Source: AfDB 2012. It was pointed out (Azerbaijan and Turkey) that there is value in increasing clearance thresholds so that the quantity of contracts that can be reviewed by procurement 93 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY specialists as opposed to RPMs would lower processing times. Bank task team leaders also subscribed to this view and pointed out that raising the country procurement office level of clearance would expedite the no-objection process for bidding awards. Responses also point to the dilemmas of revising thresholds. On the one hand, it was argued that with the present decentralized structure of the application of thresholds at the country level, there are potential differentials in the treatment of perceived risk across countries. One example offered by staff interviewed was that of Haiti, where all contracts are subject to prior review, compared to other arguably equally fragile states such as Burundi and Rwanda, where this has not been the case, although in the views of some procurement staff, country capacity levels may have been lower. Yet Bank procurement staff also perceive a need for more local flexibility in applying thresholds. In Indonesia, for instance, Bank procurement staff complained about the lack of flexibility for adjusting thresholds according to the capabilities of implementing agencies. Tanzania, by contrast, was pointed out as a clear example of threshold flexibility where task team leaders reported that the Bank raised prior review thresholds on two occasions to accommodate country circumstances and context. Procedures for handling complaints also appear to involve referrals to higher clearance levels. Field office procurement staff in some countries (Bangladesh) point out that referring even small, and sometimes technical, complaints to the regional procurement manager for clearance is time consuming and requires detailed prior preparation. Moreover, if not localized, there is inadequate contextualization and less understanding of the issues. Questions were raised about the value of the threshold mechanism for containing risk. Some procurement staff felt that implementing agencies see no objections as a requirement largely to meet audit objectives. In some instances, it was also felt that the review process made country clients overreliant on the Bank, diluting capacity development and giving clients a false sense of security. Several respondents pointed to the trade-off between risk containment and efficiency, showing a lack of clarity in the clearance process, especially because of clearance thresholds that restrict decision making by local procurement staff, and the absence of clear service standards at the Bank in terms of the response process. Country management perceived concern about fiduciary risk as substantially detracting from efficiency in project implementation. Though this was less of a concern to other stakeholders, there was a perception by all that ICB was used somewhat more than needed, from the point of view of achieving a balance between competition and the quality and efficiency of project execution. There were also concerns that although the Bank had good procedures for controlling procurement risk, this may not be enough for 94 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY the overall containment of fraud and corruption, much of which is thought to occur during implementation, that is, in the post-contract-award period (for example, bribery, short payments, contract variations, and so forth). More importantly, the Bank’s procurement processes, although ruling out the worst abuses (direct/noncompetitive contracting and the like), cannot overcome social and political systems where such practices may be commonplace. IEG ANALYSIS OF THRESHOLDS AND RISK MANAGEMENT As discussed in Chapter 4, feedback from country case study questionnaires suggested that current thresholds could be raised with beneficial impacts on workload and clearance time, with only limited impact on risk or competition. IEG undertook its own analysis of the application of procurement prior review and procurement method thresholds across Bank borrower countries, with the objective of seeing whether and to what extent current thresholds, especially for prior review, have been commensurate with risk exposure. IEG first measured the extent to which country-level prior review thresholds for goods and works correlated with select broad parameters of country governance /country fiduciary risk, including the Bank’s CPIA indicators, as well as external indices (Transparency International’s Corruption Perceptions Index and the Kauffmann and Kraay Governance Indicators) on the quality of governance. 100, 101 Second, IEG examined project-level thresholds in relation to project-level risk. Two samples were constructed, one from 2008 and one from 2011, for the comparison of Bank thresholds with external measures of country risk/ country governance. There are two caveats to the sample construction: the setting of thresholds is not standardized across regions; and the gaps in available information vary considerably across regions. Correlations between country-level prior review thresholds and governance indicators were undertaken in clusters, reflecting different practices in setting thresholds across regions. (Details are available in Appendix C, which describes data sources, sample construction, and correlation methods used in the presence of diverse threshold-setting and risk-rating practices across regions). RESULTS Regional Prior Review Thresholds—Correlations with Governance Indicators The overall result is that there is a weak statistical relationship between borrower country risk or country governance (as measured by the selected risk vectors) and Bank prior review risk thresholds. Table 3.3 reports results for all countries in aggregate; the main findings are summarized below. 102 • While the majority of the correlation coefficients are positive, most are moderate to weak in strength, at best around 0.5; there are also a substantial number of much lower correlations. 95 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY • Results are similar across different governance indicators, though somewhat higher for the overall CPIA, the CPIA for budget and financial management, and for the Kauffmann and Kraay indicator on regulatory quality. Results are the poorest for the Kauffmann and Kraay indicator on control of corruption, and also poor for the CPIA subindicator 16 on transparency, accountability, and corruption in public service. • There are somewhat lower correlations for NCB goods and works than for ICB goods and works. • Correlations in 2011 tend to be lower than those in 2008. These might suggest that inconsistent practices in setting thresholds are not improving over time. Table 3.3. Prior Review Procurement Thresholds: Correlations with Indices of Governance CPIA Sub- CPIA sub-indicator Transparency indicator 13. 16: transpar., International - Kauffmann Kauffmann quality of budget account. and Corruption and Kraay - and Kraay - CPIA - and financial corrup. in public Perceptions regulatory control of overall management sector Index quality corruption 2008 ICB goods (68) 0.51 (68) 0.45 (68) 0.32 (67) 0.46 (68) 0.57 (68) 0.25 2008 ICB works (68) 0.46 (68) 0.41 (68) 0.18 (67) 0.30 (68) 0.43 (68) 0.16 2011 ICB goods (117) 0.33 (117) 0.32 (117) 0.20 (123) 0.12 (123) 0.28 (123) 0.15 2011 ICB works (116) 0.41 (116) 0.36 (116) 0.24 (120) 0.24 (122) 0.40 (122) 0.20 2008 NCB goods (51) 0.39 (51) 0.39 (51) 0.25 (51) 0.27 (48) 0.45 (48) 0.14 2008 NCB works (54) 0.39 (54) 0.36 (54) 0.00 (54) 0.11 (54) 0.35 (54) -0.04 2011 NCB goods (32) 0.29 (32) 0.09 (32) 0.26 (37) 0.14 (37) 0.44 (37) 0.29 2011 NCB works (33) 0.38 (33) 0.26 (33) 0.27 (38) 0.21 (38) 0.51 (38) 0.35 Source: IEG analysis. Note: Numbers in parentheses indicate numbers of observations. CPIA = Country Policy and Institutional Assessment; NCB = national competititve bidding. Although these findings are subject to caveats on interpretation, because of some regional data being unavailable and certain difficulties in comparison across regions, they suggest that at best it is difficult to reflect country procurement risk (as reflected in indices of governance and corruption), in the present system of value based indices, with consistency across the Bank. Project-Level Prior Review Thresholds It can be argued that country-level risk thresholds are not the ones that actually apply to individual projects; instead, project level thresholds are used, based on risk specific to the individual implementing agency. IEG therefore also analyzed project level procurement thresholds, and their association with project-level risk (Appendix C). An initial review of prior review thresholds at the project level demonstrates that although project-level prior review thresholds are usually set at the level of the regional prior review threshold (some 50–60 percent in the sample below), there are some 96 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY exceptions, especially projects with thresholds set below the regional/country threshold (15–30 percent) and perhaps more interestingly, a few (20–25 percent) projects that have prior review thresholds set above the country threshold (Figure 3.1). The Europe and Central Asia and East Asia and Pacific Regions have had more project-level flexibility, with projects below or above the country prior review thresholds, compared to greater uniformity in South Asia and Africa. Figure 3.1. Country and Project-Level Prior Review Thresholds by Region—Civil Works and Goods Source: IEG analysis. Note: For civil works n = 65, for goods n = 67. Regions: AFR = Africa; EAP = East Asia and Pacific;ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. Next, data were collected on the residual procurement risk rating at the level of each project (Figure 3.2). This information was associated with information on project level risk thresholds, to see if there was a relation between, for example, relatively lower project risk levels (compared to country risk) and relatively higher project risk. Figure 3.2. Procurement Risk Ratings for Projects Above and Below Country-Level Prior Review Thresholds Source: IEG analysis. Note: For projects with project threshold > country threshold n = 23, for projects with project threshold < country threshold n = 26. 97 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY Contrary to assumptions about project risk levels and relative thresholds, projects with thresholds above country-level thresholds tended to have somewhat higher levels of procurement risk (more projects in the substantial risk category) than projects with thresholds set below the country prior review threshold. The finding suggests again that in aggregate, the application of project-level prior review thresholds has a blurred association with procurement risk. CONCLUSIONS AND IMPLICATIONS The Bank’s prior review thresholds, as presently defined in value terms, do not seem to have a consistent or strong relationship with governance or risk, whether at the country or the project level, although there is some weak positive association. How can these findings be explained? One factor is the likely poor association between value thresholds and risk levels. Large contracts may not necessarily be more risky, and setting thresholds in value terms may capture statistical noise. Other reasons could be the use of different implicit scales in different regions or countries that are not due to differences in risk. There may be a tendency, for example, to “scale” thresholds to loan size or country income, rather than country risk alone. One implication of these findings is that setting an ever-finer grid of value-based thresholds is not likely to significantly improve the management of risk. Some simple broadbanding may be adequate, to the extent that value-based thresholds are to be maintained. A second implication is that differentials in treatment of risk across regions suggest that a uniform Bank-wide system, based on transparent criteria, would be useful. Third, consideration could be given to moving away from value-based thresholds and focusing instead on risk-based criteria. Such a risk-based focus would have least scope for error if determined at the level of individual contracts, instead of broad categories such as goods or works. Nevertheless, the Bank should be cognizant of the increased potential for complaints of arbitrariness in judgments that may be leveled as a result. Managing Procurement Risk—Post-Procurement Review Post-Procurement Reviews assess whether procurement conducted by the client’s implementing agencies, without Bank involvement, has been executed in compliance with Bank procedures. More rapidly processed, but riskier than prior reviews, procurement post reviews are conducted on an agreed sample of all contracts designated for post-review. Designations are made by project staff, agreed with the borrower, and then written into the procurement plan. Most often the sample comprises 20 percent of the total number of designated contracts, but this varies by project, depending on procurement specialist views of project risk. Project staff often use the sample size for post-review as an instrument of risk management (increasing the sample for riskier 98 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY projects) and may also require more frequent than the usual annual postreview cycle. In the sample of 69 project appraisal documents (PADs) that were reviewed covering the period 2002–10, 24 projects explicitly declared a post-review sample below). Of these, 15 required a 20 percent sample but many others varied between 5 and 40 percent. Two projects asked for a 100 percent review. IEG reviewed both completion rates of Post-Procurement Reviews and their findings (in terms of procurement malpractice or suboptimal outcomes). 103 The Bank’s annual procurement reports give considerable emphasis to Post-Procurement Review completions by the regions, and compliance has generally improved in recent years: The FY11 report shows that overall the Bank was 99 percent compliant (that is, the number of actual postreviews, compared to the designated number subject to postreview), with three regions (Latin America and the Caribbean, the Middle East and North Africa, and South Asia) being 100 percent compliant; Africa, East Asia and Pacific, and Europe and Central Asia were 97 percent compliant. 104 Table 3.4. Share of Contracts Designated for Post Procurement Review (2002–10) Percent review required 5 15 20 30 40 100 Numbers of projects (total: 24) 1 2 15 3 1 2 Source: IEG analysis. The format and content of the Post-Procurement Review contains information that is operationally significant: it includes an assessment of procurement procedures employed; compares these to the loan agreement; sees to contract management issues (assesses technical compliance and physical completion of the contract); assesses the capacity of the implementing agency and its systems; identifies noncompliances in executed procurement; and recommends remedies. The format also includes a requirement to complete two key risk ratings, going forward: the assessed risk for the procurement process and the risk expected for implementation of contract management. In some respects (for example, contract management) this is a more rigorous agenda than is contained in the prior-review process, and it has the advantage of occurring in real time (as against ex ante, before project start up), so it benefits from an element of actual implementation. To this extent the Post-Procurement Review is a useful monitoring tool to ensure the integrity of the Bank’s procurement processes, and it appears to be used as such within some regions. The Middle East and North Africa Region, for example, developed consolidated follow-up action plans, to remedy identified deficiencies. The Post-Procurement Review system was recently improved by enabling electronic uploading of its content (including the risk ratings) into the Operations Portal, to link to both P-RAMS and the ISR. It is not clear whether these linkages are fully operational yet, but it is clear that they are a key subject in the integration project that is being worked on by the Bank’s procurement anchor intended for full operation by end of FY14. 105 99 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY The Bank’s annual procurement reports, as regards Post-Procurement Reviews, focus more on compliance than operational content. 106 This limited treatment underrates the potential importance of Post-Procurement Reviews as a risk-management tool, and its potential for directly checking and validating (or discounting) whether procurement performance under postreviewed projects is of a higher or lower standard than that under prior-reviewed contracts, despite ample data being collected at the regional level. Part of the reason is that most regions do not track and analyze consolidated Post- Procurement Review data. From two studies that did so (both in South Asia: Bangladesh and India), there was information that showed that (for India) of those contracts post- reviewed, 24 percent by number and 32 percent by value of contracts were not in compliance with Bank procurement guidelines or the loan agreement. In Bangladesh, findings were similar: 33 percent of contracts by number and 26 percent by value were found to be subject to “major observations.” 107 In the Middle East and North Africa Region a summary report on Post-Procurement Reviews/IPRs undertaken for FY09 and FY10 shows major and concerning deviation from Bank procedures in some countries (for example, the Republic of Yemen), whereas in other countries (Morocco) noncompliance is of a more routine and relatively minor nature. In yet other countries (Turkey, Ukraine) Post-Procurement Reviews in FY11–12 revealed that post-reviewed procurement practices were entirely exemplary. Appendix Tables C.5 and C.6 contain a listing of the Post-Procurement Review reports consulted for this review and a summary of exceptional irregularities found in the case of the Yemen Post-Procurement Review. IEG conducted an analysis of mandatory risk ratings in Post-Procurement Reviews. Procurement staff have to rate the expected future risks of procurement, as well as, interestingly, contract administration. Data from Post-Procurement Reviews suggest only moderate risk ratings going forward (Figure 3.3). Figure 3.3. Distribution of Risk Ratings in Post-Reviewed Contracts Source: IEG analysis of select PPRs. 100 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY IEG also analyzed the extent to which risk ratings under post-reviewed contracts were higher or lower than procurement risk ratings in the P-RAMS database. Figure 3.4 compares evidence on procurement risk ratings from two sources: first, from 520 projects contained in the Bank’s Post-Procurement Review database, which show risk ratings for procurement process and contract administration; second from a general sample of 542 projects for which 668 P-RAMS templates were completed during June and July 2010. The distribution of risk ratings is significantly different. In the Post-Procurement Review risk ratings for procurement process 345 of 520 ratings (66 percent) were rated moderate or low risk; and, similarly, for contract administration 349 out of 520 ratings (67 percent) were in these lower-risk categories. By contrast, in the general project sample, procurement risk was rated high or substantial in 402 of the 668 projects in the sample (60 percent of the total). Figure 3.4. Comparison of Project Procurement Risk (from P-RAMS) and PPR-Based Risk Ratings Source: IEG analysis. Note:Bbased on a sample of 668 completed P-RAMS templates for 542 projects in the period July 2010 to June 30 2012; and review of a sample of PPRs. PPR = Post-Procurement Review; P-RAMS = Procurement Risk Assessment Management System. A first caveat is that the two data sets are not comparable, because the Post-Procurement Review sample is essentially contract-based from postreviewed projects only, and the general sample refers to project risk, and contains both prior- and postreviewed contracts. A second caveat may be that P-RAMS-based risk assessments are made ex ante whereas the Post-Procurement Review assessments are made during implementation, with the benefit of at least some track record of how implementing agencies have been performing. Taken overall, therefore, the evidence emerging from the findings of Post-Procurement Reviews offers mixed results: at least in some consolidated country reports, Post- Procurement Reviews have shown that post reviewed contracts have been noncompliant with Bank procedures in some 24–30 percent of the sample, with “serious observations” in other cases. By contrast, a much larger sample of Post-Procurement Review-based risk 101 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY ratings suggests that procurement specialist staff perceive expected risks of failures (in procurement or contract administration) to be distinctly lower than in a general sample of all projects. As with the analysis of thresholds, it suggests that a more detailed Bank-wide review of its risk instruments and their correlations would be useful and might be a basis to revisit current country risk ratings. Finally, mention is made of the IPR, an elective procurement review tool that regional RPMs can use to review procurement performance in a single project or batch of projects, where there may be specific concerns. IPRs are often done to supplement Post- Procurement Reviews; their results are recorded at the regional level and are used for follow-up by the RPM and the procurement staff. IPRs review the performance of both the Bank and the borrower and have been used to provide feedback and guidance on procurement risk-management issues to task teams and implementing agencies alike. In FY11 there were 54 IPRs completed in 14 countries covering 54 projects and 725 contracts. These have not been analyzed by IEG. Managing Procurement Risk—The P-RAMS Instrument MANAGING RISK BEFORE P-RAMS The Procurement Policy Framework The identification of potential procurement-related risk has been recently formalized in the Bank through P-RAMS, with 11 specific risk indicators (Box 3.4) (World Bank 2010f). A recent review by the Bank’s Internal Audit Department examined levels of completion of the P-RAMS instrument. IEG’s analysis seeks to undertake a qualitative review of the extent to which the new instrument has actually led to an improvement in the Bank’s ability to measure and track procurement risk and assess implications for overall project risk. This analysis therefore undertakes a comparison of risk measurement in the period before and after the introduction of the P-RAMS instrument. IEG also reviews evidence from country case studies on the design, user friendliness, and perceived value added of the instrument. 102 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Box 3.4. The Procurement Risk Assessment and Management System The P-RAMS Users Guide segregates risks into two types: inherent risks and control risks. Risks are measured starting with the overall country and institutional setting (strategic and operational risks that most projects have to take as given), moving to the agency level (where there is scope for the project team in support of their country counterparts to influence the level of risk), and at the project procurement level (where there is the most scope for affecting fiduciary risk levels through establishing compliance and reporting measures). P-RAMS was introduced in response to the IDA Controls review, which noted deficiencies in the management of procurement risk. Its integration into the Bank’s overall risk management framework is envisaged. IEG’s evaluation of the progress made with remedial measures acknowledged the strengthening of fiduciary controls through the introduction of P-RAMS, but pointed out that the new system would take time to be effective. The 11 risk factors contained in the P-RAMS template: 1: Accountability for Procurement Decisions in the Implementing Agency 2: Internal Manuals and Clarity of the PR process 3: Record Keeping and Document Management Systems 4: Staffing 5: Procurement Planning 6: Bidding Documents, (pre-) qualification, short-listing, and Evaluation Criteria. 7: Advertisement, Pre-bid/Proposal Conference and Bid/Proposal Submission 8: Evaluation and Award of Contract 9:Review of Procurement Decisions and Resolution of Complaints 10: Contract Management and Administration 11: Procurement Oversight. Aggregated Risk Measures In the P-RAMS template (completed by the risk assessors on line) each risk category contains several individual questions (as few as three or as many as 10) for a total of 60 individual questions. If the assessor identifies other risks not contained among these 60 questions, he/she can add these new risks by clicking on a “customize risk” facility. A rating is ascribed to each risk factor and these ratings are aggregated by the template to give a calculated risk rating for each risk category. If the assessor disagrees with the overall calculated rating, he/she can override the rating and insert a new one, but this requires an explanation in a text box provided. So, although the template provides implicit equal weighting to all 11 risk factors, the assessor can in principle change this by inserting different weightings. The template also contains a Risk Assessment Summary section, including proposed mitigation measures. Sources: World Bank 2010b, 2012i. 103 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY Approach and Method The empirical basis for the analysis was a sample of PADs. 108 IEG reviewed the sections that dealt with project and procurement risks and, making a qualitative assessment as to their content and quality, focused around two key components of the procurement annex and the procurement plan: the project’s implementing agency capacity assessment (herein referred to as “risk assessment”) and the procurement risk-mitigation plan (herein referred to as a “mitigation plan”), which contains the specific measures to be taken to address and manage the risks found in the risk assessment. 109 The assessment compared risk assessment and management in two groups of projects: before and after introduction of P-RAMS. The results of these groups were then compared. The analysis on the pre-P-RAMS sample was conducted mainly on PADs; the analysis after P-RAMS was introduced was conducted mainly on the content of the P- RAMS template. The pre-P-RAMS projects were approved during the years 2002–10; those with P-RAMS were for projects approved after July19, 2010, the date when P-RAMS became mandatory for all projects at the project concept note stage. The assessment began with the pre-P-RAMS group and focused specifically on the capacity assessment and risk mitigation plans in each project, but also collected data on other related aspects, including country context, links between procurement risk and overall project risk, and use of thresholds. A scorecard was constructed for the risk assessments, consisting of risk categories most commonly found in the PADs. A similar scorecard was constructed to score mitigation measures in the mitigation plan for each project. In theory, P-RAMS also provides for “custom” risks to be identified (that is, those not contained in the specified 11 risk factors), but in practice (judging from the sample) it does not appear that this feature was much used. Analysis and Findings • Procurement risks have generally been well identified, and this has improved in recent years, even before the introduction of P-RAMS, although there are some deviations from this norm. This assessment was made based on a review of: the number of risk categories addressed, the number of specific risks identified and assessed, the linkage of procurement and project risk, and the number of items contained in the procurement risk-mitigation plans. The average number of risks addressed per project was about 12; seven projects addressed more than 20 risks (23 was the highest number); 14 PADs addressed fewer than 10 risks, of which 2 (3 in Latin America and the Caribbean and 1 in Europe and Central Asia) addressed no specific risks (Table 3.5). 110 104 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Table 3.5. Risk Categories and Individual Risk Identification in Pre-P-RAMS PADs by Period 2002–06 2007–10 2002–10 Number of risk categories 4.7 7.2 6.3 Number of individual risks 8.3 14.9 12.3 Fraud and corruption risk Items 0.7 1.9 1.4 Source: IEG calculations from scorecard from a sample of 39 pre-P-RAMS PADs. Note: Average Incidence per PAD. PAD = project appraisal document; P-RAMS = Procurement Risk-Assessment Management System. • Implementing agency capacity is the dominant risk category, both before and after P-RAMS. However, the single most frequent risk category before P-RAMS was the catch-all category “other,” reflecting individual project differences in risk profiles, but also nonspecification in the template. 111 • Elements of fraud and corruption risk were frequently identified in the project sample (Table 3.5 and Figure 3.5). Fraud and corruption risk was the third most frequent category of risks identified. 112 • In terms of the mitigation plans, the most frequent remedy proposed was the appointment of consultants and training of implementing agency staff. Of the 39 sampled projects, 31 contained measures to employ consultants in the project implementing unit and 30 involved training of local procurement staff. Nineteen projects were implemented by project implementing/management unit. • Measures to ensure transparency were also often featured in the mitigation plans (for example, to publicize awards, open or use project-related websites, and use of civil society in oversight). • Comparing infrastructure (higher value) and social sector (typically lower value) projects, infrastructure projects contained a higher share of lower-risk projects (Figure 3.5). This appears to illustrate that large value may not reflect high project risk. Another possibility is that greater use of more competitive procurement methods that use more prior review (ICB contracts are more common in infrastructure than in social sectors) may imply that such projects are viewed as less risky. • PADs make few explicit cross references to Country Procurement Assessment Reports (CPARs) when discussing risk or capacity issues. Explicit reference to CPARs completed in given countries was mentioned in a limited number of cases in the project sample, although they often are sources to highlight risk factors in local implementation of project procurements. From the sample of 39 pre-P-RAMS projects (2002–09), only 15 mentioned the CPAR. In the 24 post-P-RAMS projects (2010–12), 5 mentioned CPARs. 113 105 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY Figure 3.5. Procurement Risk Categories, Risk Mitigation, and Risk by Sector—A Sample Source: Project PADs from 39 projects in 20 countries approved before P-RAMS (2002–10). Note: Only 17 of the 19 infrastructure projects and 16 of the 20 social sector projects in the pre P-RAMS sample had initial procurement risk ratings – these were used for the second set of graphs. Integrating Procurement and Project Risk: Although procurement performance is closely associated with overall success in project implementation, most projects give limited regard to the impact of procurement risk on overall project risk (see Chapter 1). Evidence of this was found by tracking how frequently procurement risk issues were featured in the Critical Risks Table of the PAD: in total, the 39 pre-P-RAMS projects in the sample listed 378 critical risk items, of which only 58 (about 15 percent) were related to procurement. However, there were also projects in which procurement risks were prominently featured, together with thoughtful risk-mitigation measures (Box 3.5). A key purpose of the proposed link between P-RAMS and the Operational Risk Assessment Framework (ORAF) is to provide an automatic platform to display procurement risk as part of overall project risk, but this has yet to become universal practice. 106 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Box 3.5. Integrating Procurement and Project Risk—Tanzania Second Transport Corridor Project The PAD for this project contained a highly articulated and clearly documented account of how procurement risks could impact the overall success of the project and how these risks were closely integrated into overall project risk, as follows: it had a complete, itemized listing of initial risk ratings, descriptions of specific mitigation measures, and corresponding residual risk ratings. It used project covenants to ensure hiring of procurement specialists; it used advanced procurement to avoid delays; it required post- qualification to ensure quality; and had penalties for operators who failed to deliver service (that is, contract management clauses). Of 25 critical project risk factors, 13 were procurement related, of which 8 were rated high or substantial and none low. Mitigation of procurement risks (10 had residual risk rated as moderate or low; none was rated high) contributed significantly to mitigating overall project risk. High Substantial Moderate Low Project risk Initial rating 10 5 8 2 residual rating 7 6 12 Procurement risk initial rating 5 3 5 residual rating 3 4 6 Source: Tanzania Second Central Transport Corridor project, April 2008. To conclude, prior to P-RAMS, the treatment of procurement risk assessment and risk- mitigation measures was generally of good quality, but also with a considerable degree of variability. The treatment of procurement was only loosely standardized in the PADs. PADs generally captured core risk items such as lack of capacity among implementing agency staff, lack of experience with Bank procedures, or poor record keeping, but the data also show a wide dispersion of “other” risk items. There was a discernible increase in the focus on fraud and corruption risk as part of procurement after 2007. THE INTRODUCTION AND CONTRIBUTION OF P-RAMS Intended Contributions—Sharper Risk Focus and Dynamic Aspects Pre-P-RAMS measures addressed procurement risk in project PADs, that is, mostly upstream during project design and preparation. Even though Procurement Plans (in principle, updated annually) and ISRs were intended to track progress with procurement as part of project implementation, a need was expressed for a more downstream and dynamic risk management tool. The response was P-RAMS, mandated for all projects with a Project Concept Note written on or after July 19, 2010. P-RAMS was not intended to either replace existing practices for managing procurement risk or necessarily add new information; the principal purposes were to sharpen the focus 107 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY of risk assessment and the content of the mitigation plan and to codify and standardize the treatment of both aspects. Under P-RAMS, the risk assessment has been replaced by the Procurement Risk Assessment Questionnaire, and the mitigation plan has been replaced by a Mitigation Measure Action Plan. 114 Comparing this with the risk-assessment questionnaire, residual risk ratings are ascribed, based on the perceived likely effectiveness of the mitigation measures. An overall residual risk rating is thus assessed for each implementing agency and thence for the project as a whole. In principle, P-RAMS provides a way to monitor risks and remedial actions over time. Where projects may have more than one implementing agency, a separate Procurement Risk Assessment Questionnaire and Mitigation Measure Action Plan is conducted for each. 115 After the introduction of P-RAMS, as part of the Bank’s efforts at Investment Lending reform towards a more risk-based approach—an overall ORAF was designed, to integrate all key risk dimensions relating to Bank projects into a single online platform in the Operations Portal. It was a logical step to link P-RAMS to ORAF, as a means of including procurement risk as one of the prominent risk elements. The online link systems are mostly in place. ORAF includes fiduciary information (in indicator #3 “Implementing Agency Risk,” which also includes fiduciary risk), though input of this information is not mandatory. Although there is a link to “Procurement Risks” as a tab in ORAF (only for risks rated high or substantial), staff often are not aware of its existence and represent that there is little reflection of procurement risk in the overall project risk management framework. 116 P-RAMS risk ratings are intended to be conducted in several sequences throughout a project’s life, and also to be integrated into project ISRs. There is also a link in the P- RAMS template intended to be informed by outcomes of the periodic Post-Procurement Reviews of contracts that fall into the sample for a given project, so that these various tools form a dynamic monitoring chain that can provide an integrated system of risk monitoring. The dynamic feature of P-RAMS has been stated to be its central attribute. Implementation and Sequencing of P-RAMS Despite its mandatory nature (required for all investment loan projects in the Project Concept Note phase on or after July 19, 2010), P-RAMS was not adopted universally across all regions and it has taken some time to build compliance across the Bank. At the end of FY12 the number of projects for which at least one P-RAMS template had been completed was 542 of 794 projects that were eligible for P-RAMS, a compliance rate of 68 percent. 117 These less than complete compliance rates were built up after a slow start, and reflect a major “compliance push” in many regions before the end of FY12 (Figure 3.6). P- 108 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY RAMS remains a new tool, and staff are not fully familiar with its attributes. Table 3.6 shows compliance rates by region. Figure 3.6. P-RAMS Completed by Month (2009–12) Source: IEG calculations from P-RAMS database provided by OPSOR. In terms of its dynamic contribution, IEG analysis shows that 90 projects of the 542 had completed P-RAMS by June 30, 2012, with more than one sequence and 25 projects that had more than two sequences. This suggests that it will take time for the dynamic aspect to be more fully developed. A question may also be raised as to the required frequency of repeated P-RAMS sequences. The spacing between P-RAMS sequences appears quite short: in some observed cases, only three months. It appears unlikely that risk factors would change so rapidly and it also brings into focus whether procurement efficiency is well served by such frequent sequencing. The dynamic attribute of P-RAMS may need to be more carefully articulated. Table 3.6. P-RAMS Compliance Rates by Region AFR EAP ECA LCR MNA SAR TOTAL Eligible projects 255 147 108 87 72 125 794 Percent completed 58 80 81 62 61 72 68 Number of projects with more than one P-RAMS sequence completed Sequences: Two 12 14 19 7 3 10 65 Three 4 1 3 2 2 2 14 Four/Five 0 3 1 1 0 5 11 Source: OPSOR P-RAMS database. Note: Compliance is defined as at least one P-RAMS sequence being completed. AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. 109 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY PERCEPTIONS OF P-RAMS AND ORAF: IEG FIELD VISITS Bank staff and country clients gave mixed reviews to new risk management tools, such as P- RAMS, and their relation to a wider project risk management framework--ORAF (Box 3.6). 118 EMPIRICAL ANALYSIS OF P-RAMS DATA Comparison of P-Rams and Pre-P-Rams Risk Information IEG compared data gathered for the pre-P-RAMS control group (Figure 3.5) with a sample of projects using P-RAMS (Figure 3.6). The sampled P-RAMS suggest a high rate of response to questions; 89.4 percent. However, “other” risk factors are rarely added by staff, if they have arisen. IEG analyzed data on underlying risk identified in the P-RAMs questionnaires. Figure 3.7 describes, for individual risk areas, the proportion of responses that indicate that implementing agencies do have adequate capacity to deal with the specified risk. Thus, a higher score indicates higher capability, and hence, lower risk. Among the higher risk factors identified are bid advertising; the bid evaluation and award area (which at 57 percent “yes” had the lowest ratings); staffing capacity; and record keeping, process clarity, and agency accountability. The pattern of information contains both similarities and some contrasts with the pre-P-RAMS analysis: implementation capacity (mainly staffing issues) is a common thread. In both, procurement oversight and contract management were among the less risky areas— which raises a query for proposed additional emphasis on this area. Because of limited sample sizes, these comparisons should be interpreted with caution. They do not suggest that P-RAMS has added any particular insights, although systematic analysis of the data—not only at the level of individual projects, but strategically aggregated by region, sector and over time—may have findings. 110 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Box 3.6. Perceptions on New Risk Management Tools—P-RAMS and ORAF While perceived as useful for raising overall awareness of procurement risk as part of project design and administration (Morocco) and modestly effective for identifying risk through early detection (Azerbaijan and Bangladesh), views were more guarded in terms of its effectiveness, with cautions regarding too mechanistic an approach. In Bangladesh, P-RAMS was perceived as being “too mechanical”—a view shared by some Bank task team leaders who see P-RAMS as an additional compliance tool that adds little value to project outcomes. Furthermore, P-RAMS was perceived as failing to pick up important issues reflected in the Project Concept Note stage of the process, pointing toward integration issues between procurement staff and task team leaders. In Ethiopia, Indonesia and Tanzania, P-RAMS was perceived as useful conceptually, especially by country clients, but not very helpful at tackling fraud and corruption risk, especially in contract implementation. In the Philippines, P-RAMS was criticized for being “too subjective and not evidence based.” In Senegal, P-RAMS was also criticized for not being user friendly. In turn, ORAF was perceived by country clients as only being relevant to the Bank and not to executing agencies. In Ethiopia, Bank task team leaders perceived ORAF as being “too theoretical” and failing to address “real risks.” Bank procurement staff in Tanzania view it as too generic, not adequately differentiated across projects. Country management seems to have shared this view. In Indonesia, ORAF was perceived as “something that was a good idea but is no longer effective.” It was deemed to be “…so inclusive and comprehensive that almost everything is seen as a risk….” Thus issues that are truly important are not highlighted. Finally, from the point of view of procurement, and the integration of procurement into the overall risk management framework, Bank procurement staff and Bank task team leaders in Indonesia, Morocco, and Peru perceived ORAF as lacking a comprehensive section on procurement, and procurement staff in Turkey also considered ORAF not particularly effective for procurement risk management. Source: IEG field visits. Risk-Mitigation Action Plans under P-RAMS A principal rationale for introducing the P-RAMS tool was to help ensure a more focused mitigation plan, including specific responsibilities and targeted timelines for mitigation actions, to address the main risks identified. A qualitative reading of the sampled 30 mitigation action plans does reveal that many mitigation action plans indeed contain well-specified mitigation measures and accompanying risk ratings for each risk factor, 111 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY completed for each agency and in some cases for several agencies. A health project in Brazil had 17 agencies (with three P-RAMS sequences) and associated mitigation action plans. Also, where the risk questionnaires show higher risk ratings, more items are specified in the mitigation action plans, as would be expected. Figure 3.7. Summary Results from P-RAMS Applications, 2010–12 Source: IEG analysis of P-RAMS questionnaires. Note: P-RAMS = Procurement Risk Assessment Management System. However, there was also a significant degree of variability or even mismatch in the placing of mitigation measures. In many cases where a risk factor was rated high or substantial in the risk questionnaire, there were no mitigation measures, and others in which mitigation was addressed to risk factors rated moderate or low, as evidenced in the data by the relatively loose correlation between the two instruments (Figure 3.8). These findings raise questions as to whether the goal of “sharpening the focus” of risk mitigation is being well met. Figure 3.8. P-RAMS Risk and P-RAMS Risk Mitigation Source: IEG analysis of P-RAMS data from 2010–12. Note: P-RAMS = Procurement Risk Assessment Management System. 112 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY IEG undertook additional analyses of successive P-RAMS to see whether the differential between initial and residual risk increased (that is, whether risk declined) over time (Appendix C). Results show that differentials are largely static over the periods of time for which observations are available. If, for example, initial and residual risk are reported as high and moderate in the first P-RAMS sequence, subsequent sequences show similar findings. ADDITIONAL OBSERVATIONS ON P-RAMS • P-RAMS operational links remain limited. Although PADs do reflect risk issues pointed out in the P-RAMS template, acknowledgement of P-RAMS is rare: of 22 PADs examined for the period 2010–12 only one gave mention to P-RAMS in the PAD. Fifteen mentioned and made use of ORAF. Systematic linkage of information (including Post-Procurement Reviews when available) will enhance the effectiveness of P-RAMS as a risk-management tool. • P-RAMS risk hierarchy is unclear. Project risk, in P-RAMs, refers to procurement risk focused on the implementing agency. However, project risk also embodies other (nonprocurement) risk elements and the separate treatment of procurement risk, as an element of all risks facing a project may be preferable and reduce overlaps with risk concepts in ORAF. • P-RAMS, as a tool to address fraud and corruption risk, could be sharpened. Although the P-RAMS Risk Assessment Questionnaire does have, among its 60 questions, some that explicitly refer to fraud and corruption–related issues (five in all): they are subsumed under a number of separate areas, which reduces transparency once they are aggregated. 119 However, focus could be more potent with a specific risk factor related to fraud and corruption. Compared to the enhanced treatment of fraud and corruption risk in project PADs, already evident in after 2007–08, it is not clear that P-RAMS has added much value in this area. • Addition of new issues, aggregation. The fixed template may add rigidity: although there is a “custom” facility to add additional risks, this does not seem to have been used by the procurement specialist. Moreover, the template weighs all risks equally. But risks need to be prioritized, and aggregation based on equal weights can give a misleading picture (as pointed out by field staff, in instances where a single high risk has significant weight). Though the template is modifiable in principle, changing weights is not transparent. Finally, there are three questions where a “no” response actually denotes a positive feature, which also leads to some difficulties in aggregation that could be easily addressed. 120 To summarize, P-RAMS offers a useful standardization of risk factors and risk-mitigation plans and provides an ongoing tool of dynamic content to monitor procurement risks throughout the project life-cycle. Althought it is still too early to draw definitive conclusions, there are indications that some improvement may be needed. Despite its short lifespan, some evidence points to a lack of adequate matching of higher risk and 113 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY mitigating actions. Sequencing is clearly important, but the spacing should be kept realistic, with due regard to efficiency. There are questions about whether the whole P-RAMS process is overly cumbersome for the benefits it brings. Both risk assessments and mitigation plans were highly developed before P-RAMS was introduced. Consideration should be given to making some minor but important modifications to the template, to allow for greater flexibility in risk coverage, permit a relative weighting of risk factors, reorient certain questions, and reduce aggregation issues. Consideration could be given to formulating a separate risk factor for fraud and corruption, to give more emphasis to those risks. Integration of P- RAMS applications with other operational tools (ISRs, Post-Procurement Reviews, ORAF) and more systematic use of the information it generates could help not only individual projects but also broad strategic directions to be taken by Bank procurement. Measuring Risk Failure—Misprocurement, Complaints, and Investigations Among the Bank’s armory of instruments to guard against procurement risk are the tracking of misprocurement and procurement complaints; as well as preventive and investigative work undertaken by the Bank’s Integrity (INT) Vice Presidency. IEG’s brief review of evidence in each of these areas suggests that the Bank is currently operating under broadly acceptable levels of procurement risk. MISPROCUREMENT AND PROCUREMENT COMPLAINTS Each year the Bank has a number of contracts under which misprocurement is declared, which arise from a variety of sources. Bank data on misprocurement, measured against the total number of contracts awarded annually, show that misprocurement rates represent 3–3.5 cases per thousand, which appears low. 121 The Bank also maintains a central complaints database, made by bidders (mostly losing bidders) in relation to contract awards. 122 Following recommendations made under the IDA 14 Internal Controls Review the Bank has improved the quality and rigor of its central complaints database, which is now kept current, including mandatory recording of all case closings. Data show that the number of complaints has varied little from an annual number of 300- 400 over the past decade, on all contracts, whether prior reviewed or not. When standardized by the number of procurement contracts awarded each year, the number remains quite stable at about 2 percent of the prior-review contracts, or below a half percentage of all contracts per year.123 One issue with complaints is their potential to impose delays in the procurement process. The data for complaints in the database in FY12 show that the average time for resolution of complaints has been about 150 days, though some were resolved sooner than that, and 114 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY a significant number have taken much longer to resolve. Figure 3.9 shows the distribution of cases according to the length of time to their resolution. IEG’s discussions in field visits indicated that in some cases frivolous complaints may be lodged by competing bidders, as a deterrent to competition. More speedy dispatch of such complaints would improve the process. With rates of complaints—which are a normal part of the process—at such low levels, there is little evidence to suggest that there is much risk failure involved, though the speed of resolution is an area for possible concern. Figure 3.9. Time Distribution of Complaints Resolution Source: World Bank database. Note: 2,157 observations. FRAUD, CORRUPTION, AND THE ROLE OF INT To what extent is there evidence of fraud and corruption risk in procurement, and what risk controls address this? One indicator is the number of “red flag” referrals that the operating regions have made to INT in recent years. Once INT receives a “referral,” it is treated as a complaint. Complaints number around 500 per year. Though seemingly high, not all of these complaints are evidence of control failure, or of fraud and corruption. 124 Furthermore, the complaints relate to both prior-review and postreview contracts. Given that the total number of prior-review contracts per year is around 18,000 (2011), alleged fraud and corruption represents a very small percentage of prior review contracts. In addition, a complaint received in one fiscal year may relate to a prior review contract awarded some years before. Yet some of INT’s Detailed Investigation Reports show that even when the Bank’s procurement procedures are adhered to, and even in ICB contracts prior reviewed by the Bank, fraud and corruption have sometimes been found. 125 Detailed Investigation Reports have detected instances of possible fraud and corruption that are significantly greater than what has been detected by the government or the World Bank, illustrating the limitations of enhancing fraud and corruption controls through tighter procurement alone. 115 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY INT’s Preventive Services Unit offers tools and training to the operating regions in managing fraud and corruption risks in Bank projects, based especially on its Fraud and Corruption Awareness Handbook, the Red Flags Toolkit, which alerts task team leaders and procurement specialists to a number of potential fraud and corruption risks in the procurement process, although some corruption flags may also arise at other stages or during contract management. 126 Some evidence collected by INT as part of a review of Final Investigative Reports and Active Investigations shows that most (89 percent) of identified irregularities (red flags) occur as part of procurement, defined as initial procurement notice to contract award. 127 One percent occurred during project design, and 5 percent each during contract management and financial management, albeit based on less information on contract management, as this is beyond the purview of the Bank. IEG partnered with INT to undertake an investigation of the extent to which complaints matured into cases deemed worthy of further investigation—around 150 of the 500 or so annual complaints. According to INT, the decline in the number of cases per year (Table 3.7) is due to ongoing efforts by INT to resolve more cases at a preliminary stage prior to full investigation and efforts to redirect limited investigative resources toward cases with a greater likelihood of significant impact. Table 3.7. Data on INT Investigation Cases by Region Conducted in FY09–12 Operating Regions (and IFC) FY09 FY10 FY11 FY12 AFR 36 55 21 30 EAP 23 28 7 8 ECA 28 30 12 17 LAC 10 24 9 8 MNA 7 18 5 6 Other (INT/IFC) 1 5 4 TOTAL 139 195 73 80 Source: IEG analysis of INT data. Note: Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa. INT = Integrity Vice Presidency. The recent INT review of its final investigative reports shows that the incidence of fraud and corruption varies according to the type of misconduct. Fraud (that is, cases where bidders claim to meet the criteria set out in the bidding documents, when they in fact do not) appears to be the dominant mode. Corruption (that is, bribing or otherwise inappropriately influencing officials) appears also to be evident in about half the reports reviewed. Collusion (that is, anticompetitive arrangements between bidding entities) appears also to be present but much less frequent. 128 According to INT, such misconduct often goes hand in hand with corruption. IEG further undertook a review of individual complaints in the INT database for FY10 and FY11 to classify complaints at the intake phase in INT according to the stage of the 116 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY project cycle to which the complaint referred (Figure 3.10). 129 Among identifiable procurement stages, the bid submission stage of the cycle had the highest number of complaints; these mostly involved allegations of collusion among the bidders. Fifteen percent of the 2010 complaints, 18 percent of the 2011 complaints, and 17 percent of overall complaints referred to the bid submission stage. Complaints at the contract award stage of the procurement cycle were the second highest. These complaints usually related to accusations that the contract was awarded to a bidder who was not necessarily the lowest bidder or one who did not meet the specified technical criteria. Echoing findings from P-RAMs, it was found that just 10 percent of the complaints across the two years were related to contract award, and 5 percent per year, on average, during contract implementation. A caveat regarding these findings is that 17 percent of complaints across the two years that were deemed to be procurement related could not, by the nature of the complaint, be attributed to a specific stage in the procurement cycle. Figure 3.10. INT Complaints by Stages in Procurement and Project Implementation Source: IEG analysis of INT data. Note: INT = Integrity Vice Presidency. INT investigations have found that some identifiable red flags may go undetected, sometimes through the inexperience of project implementing unit or Bank staff, or concerns about raising sensitive issues, such as a perceived need for more evidence or possible implications for staff if they report the matter to INT. INT’s draft paper on potential red flag issues offers a number of suggestions to operations staff on how to improve their use (Appendix Box C.6). Of relevance to the present evaluation is their likely recommendation to better define a risk basis for transactions, as opposed to a value based threshold, as suggested by IEG (see section on threshold analysis earlier in this chapter), and to build capacity in implementing agencies to detect the most common indicators of fraud and collusion. 117 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY IEG undertook a further exercise on INT data on cases under investigation, over the years FY10 and FY11, to investigate the extent to which it may be possible to predict those cases likely to be identified by INT as fraud and corruption cases, by reviewing the extent to which PAD risk ratings (overall project risk, procurement risk, and financial management risk, as identified in PADs) are correlated with INT cases—especially PAD procurement risk ratings. Data were available for 350 cases that provided at least one risk rating. Table 3.8 provides the frequency and percentage of available PAD risk ratings in each category. Table 3.8. Correlation of INT Cases with Project, Procurement, and Financial Management Risk Risk rating (number) Frequency Percent Cases with a project risk rating (296) High 33 11 Medium 261 88 Low 2 1 Cases with a procurement risk rating (318) High 176 55 Medium 133 42 Low 9 3 Cases with a financial management risk rating (243) High 54 22 Medium 177 73 Low 12 5 Source: IEG analysis of INT data. Note: Based on 350 cases that have at least one risk rating (project, procurement, or financial management). As shown in Table 3.8, the majority of cases (318 of 350) had a procurement risk rating, and more than half (55 percent) were rated as having high risk. By contrast, the majority of cases that had project or financial management risk ratings reported them to be medium— 88 percent and 73 percent, respectively (Figure 3.11). The bulk of procurement risk ratings fell in the high or medium categories. These findings suggest that levels of procurement risk ratings are somewhat more closely associated with potential fraud and corruption risk than with project or financial management risk. Together with the findings of Chapter 1 that good procurement outcomes are correlated with high levels of development effectiveness, risky procurement may also imply a greater risk of irregularities. 118 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Figure 3.11. Correlation of INT Cases with Project, Procurement, and Financial Management Risk (cases with at least one risk rating) Source: IEG analysis of INT data. Note: PAD risk ratings of INT cases—350 cases with at least one risk rating. SANCTIONS, SUSPENSIONS, AND CORPORATE RISK Bank lending processes require task teams (procurement specialists or procurement accredited staff and task team leaders) to check information on suppliers, as part of the due diligence process, prior to making procurement decisions. 130 Mandatory among these are the list of suspended firms, available to Bank staff on the intranet and to Borrowers via Client Connection, and the list of debarred firms, which is public. Firms on these lists are ineligible to receive Bank-financed contracts; firms debarred for more than one year are also subject to sanction by other MDBs, under a cross-debarment arrangement (some debarments therefore originate from other MDBs). 131 The lists of suspended and debarred firms are maintained by the Bank’s procurement anchor and are updated by the Office of Suspension and Debarment, the Sanctions Board and INT to reflect the results of the Bank's administrative sanctions proceedings. The Bank has a two-tier sanctions system for the review and disposition of cases arising from INT’s investigations. The Office of Suspenstion and Debarment represents the first tier of the system, and the Sanctions Board represents the second (appellate) tier which has the function of a higher authority in cases where the decisions by the Office of Suspension and Debarment become contested by respondent companies. 132 Since 2009, INT has been permitted to request that the Office impose an “early” temporary suspension on the subject of an ongoing investigation for up to six months, with the possibility of extension to 12 months, if it determines that there is already sufficient evidence that the firm engaged in sanctionable misconduct. 133 Box 3.7 provides a summary of key features of the debarment process in operation, including with selected data for adjudications by the Office of Suspension and Debarmant in recent years. The sanctions process is an outcome of the 150 or so investigations that INT decides to pursue from the 350–500 complaints it receives each year. As shown in Box 3.7, in recent years the cases have resulted in an average of 47 suspensions and 44 debarments each year. Cases may be completed within a year, but in 119 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY recent years less than 1 percent have been completed within one year, while 25 percent took between one and two years, and 74 percent took more than two years to complete. Box 3.7. The Office of Sanctions and Debarment—Two-Tier Sanctions System The two-tier Sanctions Board (about 40 percent of cases) mechanism External members (4) + Bank staff (3) Independent review (not bound by OSD) Decisions are final, no appeal OSD (about 60 percent of cases) Evaluates evidence submitted by INT in SAE Issues notice of sanctions proceedings to respondent Temporarily suspends respondent Sanction becomes effective if respondent does not contest Process There is a strict division of function between INT (investigation only) and OSD/Sanctions Board (adjudication only). INT decides, after investigation reveals sufficient evidence, to send an SAE. If OSD finds that evidence is lacking (overall, or for specific allegations) it may send the SAE back to INT for further investigation. Once evidence is agreed by OSD, OSD sends notice of proceedings to the respondent. If respondent wishes to contest (within 30 days) it must send a letter to Sanctions Board. Five forms of The standard form of sanction is debarment from eligibility to be sanction awarded World Bank contracts, the default penalty being a three-year debarment, with conditional release. Mitigating factors (for example, severity of infringement; cooperation with INT investigation; internal reform and/or severance of offending personnel) can reduce the term of debarment, based on prestated criteria and amount of relief. The five forms of sanction are • Debarment • Debarment with conditional release • Conditional nondebarment • Public letter of reprimand and restitution • Negotiated resolution. Where there is compelling evidence of sanctionable activity the OSD may impose an Early Temporary Suspension on a company/individual. In such cases the respondent’s name does not appear on the debarment list. This creates a “grey area” which may lead to confusion over whether a suspended party may or may not be awarded a contract. 120 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY Statistical outcomes FY 09 FY 10 FY11 FY12 FY13 Average (FY09–13) Case load Cases sent by INT to OSD 39 43 27 25 33 33 Cases OSD sent to respondents 10 29 33 33 25 26 Settlement agreements - - 11 16 8 Sanctions results Temporary suspensions 32 51 55 58 41 47 Debarments/other sanctions 13 45 34 83 47 44 Duration of cases Mean duration (days) 945 716 433 535 465 < 1 Year Years >2 Years TOTAL Cases submitted to OSD (09-13) 9 41 112 162 Source: OSD statistics. Note: INT = Integrity Vice Presidency; OSD = Office of Suspension and Debarment; SAE = Statement of Actions and Evidence. INT, on behalf of the Bank, also maintains a Company Risk Profile Database—a list of suppliers against which complaints have been received and a case opened, although not concluded. As part of the commitment to due diligence, prior to issuance of no-objections for prior review contracts, staff are mandated to check potential suppliers against the Web-based database. This process seems to suffer from errors of commission and omission. On the one hand, small differences in the way a name is entered can lead to false negative results. On the other hand, if a positive result ensues, the action required of task team leaders is not clear. Technically, such firms are not debarred by the Bank. Thus, task team leaders are not prohibited from approving contracts to suppliers on the database but are advised to consult INT to get more insights. If such additional review points to a high level of concern, a decision may be taken to withhold a no-objection. Staff point out that the process of obtaining feedback from INT can add weeks to processing time. Overall, limited integration of the separate checks that need to be made—the debarment list, temporary suspension list, company risk database, and anti money laundering—add to cumbersome and slow processing. 134 INT affirms that the Corporate Risk Profile Database will be enhanced in FY14, to include a more robust search engine along with being integrated into the forthcoming electronic no-objection system. Links between INT and Bank operations were recently audited by IAD. IAD was of the opinion that despite initiatives taken to improve outreach and feedback, weaknesses remain in terms of the use of information from INT investigations to feed back into operational learning, and the Bank is still not using its INT-generated knowledge to enhance fraud and corruption risk management to best effect (World Bank 2013b). 121 CHAPTER 3 MANAGING RISKS AND INCREASING TRANSPARENCY PERCEPTIONS OF THE ROLE OF INT—IEG FIELD VISITS Field visits described the frequent occurrence of differences between the Bank’s list of debarred firms and the list of firms debarred under national sanctions regimes. The eligibility of such firms to bid on Bank-financed projects has been a frequent concern to client governments (for example, in Ethiopia, Mexico, Tanzania, and Turkey). Yet private sector suppliers often prefer the use of the Bank’s list of debarred firms, especially when country lists are long. In Turkey, debarred firms number some 3,000 and reasons are broad based and include performance factors. This raises the wider question of whether the Bank should use performance factors to screen potential contractors and suppliers. Task team leaders point out that even if not debarred, or under a “negative list,” it may be worthwhile for the Bank to maintain a “positive list” of good performers.135 The issue of firms that are under investigation by INT, or within the World Bank Office of Suspension and Debarment process, but not yet debarred, also arose. Staff prefer to wait until uncertainty ends, often leading to protracted delay. However, this raises the possibility of unreasonably excluding firms that may not be wrongdoers and also leads to the awkwardness of explaining the situation to clients as the firms are not officially debarred, leading to complaints of arbitrariness on the part of the Bank (as occurred in Turkey and Bangladesh). 136 As is noted on the CRPD website, “The existence of the name in the database is not a basis to withhold the issuing of a [no-objection letter].” By contrast, ADB does not maintain such a list and so does not have this problem. ADB does have a nonperforming contractor list. However, the intrinsic effectiveness of debarred lists has also been challenged. In Indonesia, respondents observed that the debarment process has limitations, as firms can regroup under a new name. Some respondents were critical of INT and its contribution to identifying and mitigating procurement risk, especially its sense of proportionality and judgment in terms of “what is important,” particularly if a project is delayed by the actions of INT. In contrast, the preventive aspect of INT’s work has been more appreciated than its investigative function. Staff point out that the process of obtaining feedback from INT can add weeks to processing time. Overall, limited integration of the separate checks that need to be made—the debarment list, temporary suspension list, the company risk profile database and anti-money laundering—results in cumbersome and slow processing. Finally, many respondents, especially those from the private sector, raise questions as to how much can be achieved from project-level or transaction-level controls. Even if issues are detected and corrected in the context of specific transactions, they could well occur in 122 CHAPTER 3 MANAGING RISK AND INCREASING TRANSPARENCY others. In some environments, where issues of governance and fraud and corruption are endemic, the Bank could consider more system-level engagement, for example, through country-level dialogue or programs that need not be linked to procurement. 123 4. Achieving Efficiency in Bank Procurement An important aspect of overall efficacy of the procurement process is the extent to which it makes efficient use of resources. Although it is recognized that the review of procurement by borrowers goes beyond compliance towards a policy and quality review, timeliness, process efficiencies and delays in procurement have been raised as a prime concern in country consultations, undertaken by both management and IEG. IEG’s review and analysis of some factors that affect the process efficiency of Bank procurement identifies factors that will remain relevant to the procurement function and framework. Procurement Efficiency and Value for Money—Tracking Procurement MAIN FINDINGS  Currently, Bank procurement tracking systems are not equipped to provide key information needed to monitor procurement objectives of economy, efficiency, risk management, transparency, or value for money. The three regional “tracking” systems each have different objectives and architecture, as well as their own merits and limitations. For those regions that do not have a tracking system, practices are diverse and data are mostly maintained in field offices. Collecting unified Bank-wide information on procurement parameters is difficult.  Nevertheless, at a Bank-wide level efforts have been made to develop individual modules of a future system architecture. Today there is a renewed agenda within the procurement anchor at the Bank’s Operations Country Policy and Services Vice Presidency to integrate aspects of these systems.  IEG’s analysis of procurement process efficiency suggests that average time taken is clearly much longer than Bank standards, but also, that there is a high level of variability in processing times, with a “long tail” of contracts that take considerably longer than average times. There is variation across procurement methods. National competitive bidding, even when prior reviewed, is notably quicker than ICB, and conversely, consultant contract processing through quality- and cost-based methods is particularly time consuming. The size of a contract, in terms of its value, is a significant determinant, and country capacity and governance appear to matter.  Looking ahead, new procurement methods such as framework agreements can offer a means toward increasing efficiency, not only for the Bank but also for its clients. Many Bank client countries have introduced provisions for framework agreements in their own procurement systems. Although the 124 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Bank introduced the use of framework contracts from 2011, in practice these have been little used.  E-procurement can offer considerable scope for improving procurement efficiencies, among other things also increasing transparency and lowering potential for corruption. Such platforms have won global acceptance and are being incorporated into new procurement legislation in jurisdictions such as the European Union. TRACKING CONTRACT-LEVEL PROCUREMENT DATA As pointed out in IEG’s field surveys, and in management’s proposals for new directions in procurement, one of the core issues raised by clients and staff alike concerns Bank procurement efficiency—notably, the time taken in the Bank’s procurement process and the frequency of delays. This motivates IEG’s review of procurement tracking systems and their data. Second, used effectively, contract data could collect information not only on the contract process but also on the prices paid and whether the Bank is achieving value for money. Bank lending-generated contract data on transactions and prices could provide a wealth of data if properly harnessed. IEG therefore reviews Bank procurement tracking information systems and the data they yield to evaluate the extent to which procurement monitoring objectives are met. Specific monitoring objectives would include the following: • Fulfilling Bank fiduciary objectives—ensuring that funds are used for intended purposes and contributing information needed for disbursement 137 • Making informed choices about markets and suppliers, for example, for setting (methods) thresholds • Monitoring the extent to which core principles of procurement guiding the process have been used—considerations of competition, economy and efficiency; transparency and equity; and domestic market development • Providing management information on the performance of the procurement process and identifying bottlenecks in procurement execution (elapsed times, clearance levels and sequences); project execution (proportion of expenditures contracted/disbursed; “burn” rates); and agent execution (tracking the client/borrower, task team leader, procurement specialists, and private contractors, if the system embraces contract management). • Enabling borrower/client monitoring of procurement and project execution • Increasing market transparency and price discovery—generating information for a wider group of market agents with the potential of getting better value for money, not only for Bank projects but also for overall public sector efficiency in client countries and for other agencies of development (as in the example in Box 4.1). 125 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Box 4.1. Benchmarking Municipal Procurement Activities—An Example from Finland In their 2003 study Kivistö, Virolainen, and Tella gathered procurement price data from nine municipalities and three hospital districts in mid-sized Finnish cities and some of the countries’ larger rural communities, to analyze differences in procurement outcomes between these public organizations. Though most of the organizations claimed to have a procurement strategy, the report shows that when interviewed, they clarified that this meant merely the existence of procurement guidelines. Competitive tendering was used in two hospital districts and two municipalities, with only two organizations publishing requests for proposals on their Internet home page. Procurement Price for Peeled Potatoes in Different The study analyzes, for Municipalities and Hospital Districts example, prices of paper Purchasing Price Quintiles towels and finds big price 100 differences even for Purchasing Price Quitnile 80 identically specified 60 products—that is, after 40 adjusting for either one-ply or 20 two-ply paper. 0 Another example looks at the 1 2 3 4 5 6 7 8 9 10 purchasing price of peeled Source: Kivistö & Virolainen 2003 Municipal Public Entity potatoes. The results show variations in purchase prices ranging from 10–180 percent—with procurement volume not being a particularly significant determinant. The study indicates that variations in procurement competence are a main factor behind these results, depending on whether tendering is purely operational/opportunistic or also strategic. Benchmarking showed that major differences can exist across organizations, on a scale beyond what can be explained even by logistics costs. Benchmarking of different types of data can take many forms at an organization such as the World Bank: internal best practices; best practice in other MDBs/IFIs; best practices among a supplier pool in a country /region; benchmarking outcomes in different organizations within a client country; and benchmarking using different procurement methodologies and criteria, and so forth. Sources: Kivistö, Virolainen, and Tella 2003; Thai and others 2005; Korpela and Tuominen 1994. 126 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT IEG finds that the Bank’s current procurement information systems are far from being able to fulfill the above objectives. Although some basic fiduciary information is collected (on prior review contracts only, through Form 384), the Bank does not have a unified central system to track, organize, and report on its procurement process, to monitor its efficiency, or to centrally track rates of implementation. In recognition of the need for such systems, three of the Bank’s six regions (Latin America and the Caribbean, Africa, and the Middle East and North Africa) launched their own regional systems around 2008, each of which partially contributes towards the range of monitoring needs. None has the ability to provide all the above data, and all are very distant from the ultimate goal, albeit not necessarily an easy one, of capturing information on unit price or value, in a manner that would enable comparisons across contracts, markets, suppliers, or over time. Management recognized the Bank’s need for a centralized system to track contract data some years ago; initially efforts were made to build such a system, but these were abandoned in the hope that integration would come later as part of the modernization agenda. The Bank’s three regional systems each has different objectives and architecture, with their own merits and limitations. These regional systems are inherently unsatisfactory from the perspective of having a Bank-wide system to track procurement transactions. Other regions decided they would wait for a universal Bank-wide system to be developed. For those regions that do not have a tracking system, practices are diverse. Data are mostly maintained in field offices, sometimes by procurement staff and sometimes by sector units. Collecting unified information on procurement parameters is frequently difficult. Today there is a renewed agenda within OPSOR to move gradually to adapt and/or integrate aspects of these systems into a unified approach. This is clearly a priority. Further expansion is needed to include systematic information on price and value, if economy, efficiency and value for money are among the Bank’s procurement objectives (see Box 4.2). Current systems and their limitations are briefly described below, and a more detailed account be found in Appendix D. 127 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Box 4.2. Procurement Tracking Systems at the World Bank—Early Efforts In 2004–06 a pilot document submission system was created inside the Client Connection portal. The pilot was abandoned for a variety of reasons: system speed (client systems were too slow); system reliability (data quality); difficulties in client interface with Bank document filing systems; the likely cost of developing such systems, among competing resources, and prioritization given to developing other key system flags and variables. Coming at a time (following the 2005 Paris Declaration) in which it seemed clear that the Bank was moving towards adopting the use of country procurement systems, moving away from the notion of a unified Bank-wide procurement information system appeared to logically complement an overall move away from centralized procurement and controls. At the same time, the Bank opted to end “contract accounting” in the Bank's Loan Department, whereby disbursement was linked to specific awarded procurement contracts. This further diminished OPSOR incentives to maintain central procurement information systems. OPSOR made a decision to confine its focus to simplifying the collection of procurement data in its Form 384 and to focus its efforts on developing individual procurement system modules (for example, the postreview system; a complaints database; select INT databases; and an integrated Procurement Advertisement and Award Notice system). Source: IEG interviews. BANK-WIDE TRACKING OF PROCUREMENT: FORM 384 Since 1988, the principal Bank-wide source of procurement information tracking has been the Bank's Form 384, a Web-based interface for entering contract information for IDA/IBRD Bank-funded contracts (Box 4.3). 138 A key purpose of the Form 384 has been fiduciary; to ensure that a no-objection and signed contract exist before disbursements are made. Form 384 also records the procurement method and contract award information including supplier names, nationality and eligibility status. The form has undergone several revisions since data collection started, which have successively reduced the amount of information collected. As a tool for meeting the objectives described above, it has significant limitations. To begin with, it does not cover all contracts but only those at or above the applicable prior review threshold. In addition, the link between procurement and contract based disbursement was further weakened from 2007 when the Bank's loan department abandoned the concept of contract accounting (Box 4.3). Although the Bank can declare untoward disbursements as ineligible, it does draw the fiduciary potency of Form 384 into question. 128 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Although contract award data on the winning contract are available to the public after contract signature, some fields of data remain restricted and are available only to Bank staff, not to the borrower or to other external parties. This tends to limit its potential contributions to transparency or to facilitate Bank/borrower joint participation in implementation of project procurement plans. Further, in terms of contract or project processing, it captures only the final stage of the procurement process: the date of the Bank's no-objection to the contract award and the date of contract signature. For this reason it does not provide information capable of tracking processing time at different stages of the procurement cycle, a requirement vital to measuring efficiency. It does not capture unit price data. And in terms of fiduciary objectives, the form does not separate the part of a contract that is Bank financed, compared to other sources, but reports instead on the total value. To the extent that the borrowing government or other donors finance a part of the contract, the role of the Bank is not captured. Finally, questions have been raised about the quality and reliability of its information. Box 4.3. Procurement, Disbursement, Contract Accounting, and Form 384 Prior to 2007, the Bank used contract accounting, that is, it monitored payments made against contracts awarded. In 2007 the decision was made to abandon contract accounting, because of the time consumed in manual checking of payments contract by contract, and with due consideration to the respective roles of the Bank’s Controllers Office (which is not an accounting office) and the borrower. Disbursements are tracked not by individual contracts but by categories of expenditure. If a category has headroom for further disbursements, payment for a given contract can in principle be made. In one recent case, significant overpayment occurred on a contract in Kosovo. The question was raised whether the 2007 decision might have led to a systemic control issue in procurement risk management. The Bank Controller’s Office points out that though serious, this occurred because of a series of misunderstanding at several steps in the procurement-disbursement chain that confused two contracts with a consultant in the same country. A number of controls that were in place failed, on the side of the client as well as the Bank. Further, as frequently happened, Form 384 was not updated and kept current as to payments made. Concerned that the episode might reflect a systemic issue, the Controller’s Office conducted a survey of 236 contracts to check their disbursement accounts for irregularities. They found only one that was not in full compliance—that is, funds had been overdisbursed—evidence they took to be satisfactory of systemic security. The Controller’s Office does not deny that overdisbursements are possible under the current system. 129 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT IEG believes that the 236:1 survey result is encouraging and suggests that this is not a widespread case of risk failure, though monitoring is desirable. As to remedies, cases of this sort would be detected as part of the financial management and audit process, adding impetus to the idea that the Bank should encourage tighter integration of procurement and financial management processes. Source: IEG interviews. EXISTING REGIONAL SYSTEMS The three regional systems that have been developed since 2007 comprise: the Procurement Plan Execution System (SEPA) in the Latin America and the Caribbean Region (with implementation expanding to some countries in Europe and Central Asia), the Procurement Portfolio Dashboard in the Middle East and North Africa Region, and the Procurement Cycle Tracking System in Africa (Table 4.1). SEPA: Procurement Plan Execution System SEPA, launched in 2007, is a Web-based system that focuses primarily on the monitoring and execution of procurement plans related to Bank-funded projects. 139 SEPA’s objective is to promote transparency in Bank operations and to offer a procurement management tool to the Bank and to borrower governments. A part of the SEPA system is open to the public—its information on procurement plans and contracts financed for Bank projects. 140 Fourteen countries in Latin America and the Caribbean and four in Europe and Central Asia have implemented SEPA for use with World Bank-funded projects; work on pilots for India and Brazil is under way. 141 In some countries, for example, in Argentina and Nicaragua, use of SEPA is included in the loan agreements for selected projects. SEPA is shared with the Inter-American Development Bank, which has helped its uptake in Latin America. As a management tool, it helps both the Bank and its borrowers to ensure that detailed procurement plans are prepared, and to identify variations between planned and actual procurement activities. SEPA is a stand-alone system and integrating its information with other tracking systems (Form 384, SAP) can be burdensome, as staff have to enter data multiple times. Data in SEPA are populated by the borrower; although some data are mandatory for procurement plan approval, other data hinge on the readiness of the borrower to enter additional information, and there is a high level of variability in information by project and country. Yet its external availability and especially the sharing of the system between Bank and borrower give SEPA a positive rating for transparency and for building client ownership in the procurement process. 142 130 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Table 4.1. Summary of System Contributions to Tracking Procurement Performance PR performance objectives F384 SEPA PROCYS MNA DASH Fiduciary Discipline YES NO NO NO (Disbursements for But partial (only prior Not a disbursement tool Not a disbursement Not a disbursement tool intended purposes) reviewed contracts); tool weakened since 2007 Risk indicators NO NO NO YES PPR and other risk ratings Transparency NO YES YES NO No public access; not all Public section But partial: prior Portfolio based not transaction contracts Borrower access review contracts only based; access only to Bank staff Efficiency (Processing NO YES, but YES NO time) Tracks only final stages at Can track processing Designed specifically Not transaction based contract signing time but only if Borrower to track processing enters actual dates times Economy, competition YES limited. YES NO NO Information on bid award information on bidders to by country of origin the extent borrower provides Client/borrower NO YES YES NO monitoring Not available to the client Designed to be Designed as Designed as in-house interactive with Borrower interactive tool management tracking tool including with borrower Price discovery NO NO NO NO Contract management NO YES YES YES Updating of procurement Tracking of ISR trends PPR compliance rates and plans results Domestic market NO NO NO NO development Overall procurement NO YES NO YES performance Status of procurement Has aggregate portfolio plans tracking Source: IEG. Note: MNA DASH = Africa Procurement Portfolio Dashboard; ISR = Implementation Status Report; PPR = Post-Procurement Review; PROCYS = Africa Procurement Cycle Tracking System; SEPA = Procurement Plan Execution System. Procurement Cycle Tracking System—Africa Region The Procurement Cycle Tracking system developed in the Africa Region is a platform of communications between the principal parties involved in the procurement process—the borrower, the Bank task team leader, and Bank procurement staff—on procurement processes for contracts subject to prior review. Each interaction or stage in the process is recorded, in terms of the number of days taken from the previous to the present stage. It thus tracks, for example, not only the total elapsed time between a borrower's first request for a no-objection and receipt of the Bank's final no objection, but also the numbers of iterations between the task team leader and borrower, between the team leader and different levels of procurement staff; from the 131 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT field procurement specialist to the regional procurement manager and the Central Procurement Board. The system was piloted in late 2008 and has been in use since 2009. It currently covers more than 460 projects in more than 40 countries in the Africa Region. The system is principally used as a management information system that measures responsiveness of different participants in the procurement process. It is partially integrated into Bank-wide systems, in that no objection notices and related materials are automatically filed into the Bank’s document database. However, its unique transaction request numbers cannot be linked to contract identifiers used in other Bank systems tracking contract data, such as Form 384. It is a Lotus Notes-based system and will face difficulties when the Bank moves away from this platform. Procurement Portfolio Dashboard—Middle East and North Africa Region The Middle East and North Africa procurement portfolio dashboard was also designed as a tool for the management of the Middle East and North Africa procurement unit’s resources. However, its emphasis, unlike the contract and transaction focused approach of the Africa system, is on the execution of the region’s loan portfolio and individual projects within that portfolio. Although SEPA and the Africa Region tracking system provide a platform for interactions between Bank staff and borrowers and the processing of procurement stages through the system, the Middle East and North Africa dashboard is not an interactive or automated system. Information is reported by Bank project team members, and staff in the Middle East and North Africa headquarters update the dashboard on a monthly basis. Unlike the other systems, it provides information on the overall status of project disbursements, on Procurement Post-Review complaints, ISR procurement ratings, and procurement risk ratings. It also provides information on the status of procurement plans (whether they have been updated or not), misprocurements, and cases sent to INT. Bank project team members’ time spent on supporting procurement related activities can be aggregated. This dashboard is available to Bank staff for download from its intranet site. FUTURE DIRECTIONS Given the fragmented nature of its procurement tracking systems, the uneven way in which they contribute to the Bank's procurement objectives, and other policy imperatives (fiduciary controls, the modernization initiative, and so forth), management has now committed to return to building an integrated tracking and controls system (Box 4.4). 132 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Box 4.4. Proposed New Systems Architecture: The Integrated Procurement Plan OPSOR has created a conceptual framework (a work program architecture) to help it build an integrated platform beginning with the integration of several interacting information technology systems. As described, the architecture rests on two pillars: alignment with the Bank's Modernization Initiatives and delivery of client-centric solutions. The key factors described on the side of modernization are: integration—of fiduciary and risk management systems in projects; risk management—with better tools for better and more timely quality control; transparency (open data, open contracting, big data); and knowledge management (delivery systems for continuous knowledge management and learning). The client-centric solutions include integrating data exchanges at the country systems level, including with other MDBs; flexibility (connectivity options between client and Bank systems); capacity building (developing an information technology toolkit for capacity building, that is, interactive bid documents); and contract management (implementing solutions to help clients execute contracts under Bank projects). The integrated procurement plan is intended to draw on existing individual tracking systems, plus new elements, such as an electronic no objection that will activate within Bank systems. At present this element is for the final no-objection only, prior to contract award, as tracked in Form 384. Whether it will later embrace earlier steps in procurement processing (that is, moving through decision stages) and data tracking remains to be seen. Tools will be applied on line in the Operations Portal, so once the data are entered and saved they are instantly also sent to related tools (ORAF, ISR, and so forth) tracked to integration points and archived. Some of these modules already exist, though their interlinkages are at best only partially operative. The integration project is in large measure an effort to bring those linkages into effective operation. Source: IEG interviews. Steps toward an integrated system began in November 2012 with the launch of a pilot of an electronic no-objection system, in two or more countries per region. 143 Piloting is scheduled to go to the end of FY13. The next step will involve developing an integrated document submission module. IEG supports the urgent need to move the new architecture forward. Yet integrating existing modules will not suffice for objectives that would help achieve value-for-money. Links with other Bank areas need also to be established (for example, linking risk indicators with INT data, and so forth) and it will be necessary to build some entirely new modules, especially in areas such as benchmarking of information and other elements of open data, open 133 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT contracting, and knowledge management. Efforts required to bring these elements together should not be underestimated. Periodic reviews of progress actually achieved would be desirable. Given the emphasis on sound monitoring in the Bank’s advice to its country clients, the Bank needs to position itself for a leadership role in this area. Measuring Procurement Efficiency—IEG Data on Sample Contracts As illustrated in the IEG case studies as well as in Bank management’s reviews, stakeholders in the Bank’s procurement process have been less critical of the Bank’s procurement rules than of its procurement processes. Delays in the execution of procurement are a core element of concern. IEG has therefore undertaken a first attempt to examine the extent of such delays and to understand the factors that increase the time taken to process a contract. IEG emphasizes that this is only one dimension, albeit an important one, of efficiency in the procurement process. IEG’s work is illustrative rather than conclusive, and apart from the findings, it demonstrates forms of analysis that may be useful for achieving economy and efficiency in procurement going forward. Given the fragmented and decentralized information on procurement processes across different Bank regions and field offices, IEG initiated its own request for data on prior-review contracts, based on a standardized template, in each field visit country. Data requested included, primarily, information on the dates of each step in the procurement process for which the Bank is required to provide a “no-objection” to a client, in order to be able to track elapsed time between and across successive phases of procurement (Table 4.2; field visit questionnaire). 134 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Table 4.2. “No-Objection” Steps for Prior-Reviewed Contracts Simple goods and works Two-stage goods and works Consultant services (ICB/NCB) (ICB/NCB) (QCBS) Specific Procurement Notice Specific Procurement Notice Expression of Interest Terms of Reference/Short List Draft Bid Documents Draft Bid Document Short List and Draft Request for Proposals Bid Documents as Issued Bid Documents as Issued Request for Proposals as Issued Bid Opening/Minutes Technical Bids Opening of Technical Proposals/Minutes Stage 1 Evaluation – Evaluation of Technical Proposals Report/Minutes/Amended Bid Documents Invitation to Stage 2 as Issued Stage 2 Bidding Opening of Financial Proposals/Minutes Bid Evaluation Report and Bid Evaluation Report and Combined Evaluation Report and Draft Negotiated Recommendation for Award Recommendation for Award Contract Signed Contract Signed Contract Signed Contract Contract Amendment Contract Amendment Contract Amendment (in some cases) (in some cases) (in some cases) Contract Completion Contract Completion Contract Completion Source: Bank procurement anchor. Note: Steps requiring a “no objection” are italic. ICB =international competitive bidding; NCB = national competitive bidding; QCBS = quality- and cost-based selection. The sample of 502 contracts obtained covers the period FY07–12; information received is concentrated around FY10–12. 144 Contracts for infrastructure account for a large proportion of observations by sector. By procurement category, data are distributed more evenly, with consultant services and goods and civil works each accounting for roughly one-third of observations. In terms of procurement methods, ICB contracts form around 40 percent of the sample, whereas NCB and quality- and cost-base selection each contribute about 22 percent of observations (Figure 4.1). 135 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Figure 4.1. Number of Observations by Country, Fiscal Year, Procurement Category, and Sector 300 By Sector 250 200 150 100 50 0 Source: IEG analysis of sample of contract data. ANALYSIS AND FINDINGS: AVERAGE ELAPSED TIMES To explore the time taken to process a contract, estimates of elapsed time were constructed between key procurement steps, from the Borrower Issue of Specific Procurement Notice to contract signature. Measured from the time when the borrower first submits draft bidding document to the Bank, results show that, looking at all contracts together, it takes on average 286 days until a prior reviewed contract is signed (Table 4.3). 145 Separating contracts by category, civil works take the longest (307 days), though there are similar overall processing times for goods (287 days) and consultant services (290 days). 136 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Table 4.3. Average Elapsed Time between Steps in the Procurement Process (days) Borrower first submission Borrower Bank no- to Bank of Bank final no Borrower Borrower bid submission to objection to draft bid objection to draft issue of bid (PreQ) Bank of bid bid Date of (preQ) bid (PreQ) (PreQ) opening evaluation evaluation contract Procurement process steps documents documents documents date/minutes report report signature Issue of Specific Procurement 171.9 169.0 135.1 126.7 226.4 253.0 269.5 Notice Borrower first Submission to Bank 56.6 66.0 121.8 224.3 252.9 286.1 of draft Bid (preQ) Documents Bank final No Objection to draft Bid 24.6 77.7 178.3 199.8 231.8 (PreQ) Documents Borrower issue of Bid (PreQ) 59.7 170.6 190.4 223.3 Documents Borrower Bid (PreQ) Opening 118.4 144.9 178.0 date/Minutes Borrower Submission to Bank of 37.4 73.7 Bid Evaluation Report Bank No-Objection to Bid 48.2 Evaluation Report Source: IEG analysis of a sample of contract data. Note: All contracts in the IEG sample. Multilayered procurement processes and contract approval within client countries can play a significant role in delays after the Bank’s no-objection, to the time the country signs the contract. Therefore, looking at critical shorter intervals, only up to the final “no-objection,” on average, the Bank’s no-objection to the bidding documents is issued 56 days after the borrower submits a first draft. This is more than five times longer than what Bank procedures recommend as a business standard for reviewing or providing comments to bidding documents. 146 Documents presented for review may be returned for revision several times, which affects duration. Although more than 45 percent of the contracts reviewed underwent just a single iteration of draft bidding documents review, another 44 percent required two or more rounds of review, and some (less than 10 percent in IEG’s sample) required three, four, or five iterations. Table 4.4. Average Elapsed Time and Contract Value by Procurement Category Average elapsed time from borrower Number of contracts in dataset first submission to Bank of draft bid Procurement (where contract value is (preQ) documents to contract signature category available) (nos) (days) Average contract value ($) Civil works 137 307.2 15,000,000 Goods 124 287.8 4,709,600 Cons. serv. (QCBS) 99 290.5 2,804,025 Source: IEG analysis of a sample of contract data. Note: QCBS = quality- and cost-based selection. Looking at variations in findings by category, method, and sector, one finding is that procurement processes takes more time, on average, for the selection of consultant 137 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT services using quality and cost-based methods, compared to ICB or NCB, used mainly for goods and works. 147 On average, processing times (to procure consultant services) with quality- and cost-based selection are more than twice as long for contracts awarded using NCB (Table 4.5). Table 4.5. Average Elapsed Time: Issue of Specific Procurement Notice to Contract Signature Procurement method Number of contracts Average elapsed time (days) All contracts 327 270 ICB 155 266 NCB 68 173 QCBS Selection of cons. serv. 82 379 Source: IEG analysis of a sample of contract data. Note: ICB = international competitive bidding; NCB = national competitive bidding; QCBS = quality- and cost-based selection. IEG also analyzed the relative performance of IBRD and IDA countries. 148 Average processing times are longer for contracts in IDA countries, and most of the difference in processing time stems from the very long time taken to procure consultant services (Figure 4.2). 149 Together these results highlight that delays in Bank processes are a reality, and some procurement methods, notably for consultants, are more prone to delays. They also highlight that, as may be expected, country capacity factors are likely a contributing factor. ANALYSIS AND FINDINGS: VARIATION IN ELAPSED TIMES—FREQUENCY DISTRIBUTIONS Average elapsed times do not tell a complete story; the degree of variation around the average describes the proportion of contracts that may take longer to process. For example, although on average 253 days were needed from the borrower’s first submission of draft bidding documents to the Bank’s final no-objection to the final bid evaluation report of the contract, this variable displayed a high dispersion from the average. 150 Half the contracts completed this process in less than 208 days, and the mean time taken is 253, but contracts at the 75th percentile of the distribution take 331 days and contracts at the 95th percentile take 611 days. The five largest processing times in this sample took from 690 to 941 days to process. The degree of variability in processing time also appears to vary by procurement method. National competitive bidding (NCB) contracts show little variability, processing time for both ICB and quality and cost-based selection contracts is widely dispersed, with some contracts taking three years or longer to signature from the issue of the specific procurement notice (Figure 4.3). 151 Overall, the finding that remains true across time intervals is that although most contracts are processed within a reasonable period of time, there is a significant “long tail” of outliers that presumably lead to frequently voiced concerns about delay. One 138 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT likely contributing factor is the degree of complexity in the technical specification of contracts, which is was not possible to proxy because of difficulties of measurement. Delays can also arise from INT processes, outside the procurement anchor. Figure 4.2. IBRD and IDA Countries—Elapsed Procurement Times by Procurement Method (days) 434 IDA 209 342 250 IBRD 173 309 0 100 200 300 400 500 QCBS NCB ICB Source: IEG analysis of a sample of contract data. Note: The time interval measured is from the borrower’s first submission to the Bank of draft bid (prequalification) documents to contract signature. ICB = international competitive bidding; NCB = national competitive bidding; QCBS =quality- and cost-based selection . Figure 4.3. Distribution of Days from Issue of Specific Procurement Notice to Contract Signature All Contracts QCBS 15 60 Number of Contracts Number of Contracts 10 40 5 20 0 0 0 500 1000 1500 2000 0 500 1000 1500 2000 Number of Days Number of Days NCB ICB 30 25 20 Number of Contracts Number of Contracts 20 10 15 10 5 0 0 0 500 1000 1500 2000 0 500 1000 1500 2000 Number of Days Number of Days Source: IEG analysis of a sample of contract data. 139 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT TIME TAKEN TO PROCESS A CONTRACT—FINDINGS FROM REGRESSION ANALYSIS A limitation of the preceding analysis is that although many factors can affect procurement process times, it is difficult to know what the most important determinants of elapsed time may be. IEG therefore attempted a simple multiple regression to look at the relative influence of different factors that affect procurement processing time, including contract attributes (procurement method, category of good or service procured, major sector, and contract value) and country-specific control variables (governance indicators as proxied by the Bank’s CPIA, levels of gross domestic product, and the poverty rate). 152 Two time intervals are examined: The time taken from the issue of the specific procurement notice to contract signature (Appendix Table D.2) and the submission of draft bidding documents to the final no-objection to the bid evaluation report (Appendix Table D.3). One key relationship that emerged from both specifications is that processing time is associated with increased contract value. For every $10 million increase in contract value, processing time increases on average, by 10–14 days. This may be explained by the fact that larger contract values require higher clearance thresholds. Field procurement officers have to request clearance from hub coordinators, regional procurement managers, and the center. The significant correlation between contract value and elapsed times is present even while controlling for regions, sectors, country attributes and gross domestic product. Other variables may also matter, and data limitations may prevent statistically significant results. It is also true that contracts procured using NCB appear to take less time than those using ICB. Finally, governance considerations in the countries concerned may make a statistically significant difference to processing time. 153 To summarize, although only illustrative, the preceding analysis suggests a number of factors that may explain process efficiency and may pave the way for more comprehensive and systematic work in this area. First, average time taken is clearly much longer than Bank norms; second, there is a high level of variability in processing times, typically with a long tail of contracts that take considerably longer than average times. Third, in terms of procurement methods, NCB, even when prior reviewed, is notably quicker than ICB; conversely, consultant contract processing through quality and cost-based methods is particularly time consuming. Fourth, the size of a contract in terms of its value is a prominent determinant of elapsed time. This is likely explained by the implied clearance thresholds. And finally, country capacity and governance matter. Countries with lower CPIAs appear to require longer processing times. 140 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT EFFICIENCY OF PROCUREMENT IN AFRICA – IEG ANALYSIS OF PROCYS DATA Using the Africa Region’s comprehensive database on elapsed time in the procurement process—the only one maintained in the Bank—IEG was able to gain further insights into procurement efficiency, corroborating many of the above messages (Appendix D has full details of the analysis). Elapsed Times in the Procurement Process Each step in a procurement process that involves an exchange between participants in the procurement process is an entry in PROCYS. In 2012, 3,043 requests were logged. 154 On average, there are two to three iterations both between client and task team leader, and team leader and procurement specialist, at each step in the procurement process (Table 4.6). One interesting finding is that in terms of numbers of days of elapsed time per iteration, the client takes the most time with an average— almost 11 days—whereas the task team leader and procurement specialist take around 8 days each. This confirms the importance of country-level factors in terms of determining time taken in procurement. Over time, there is some suggestion of increased efficiency among all parties; the number of days per iteration shrunk considerably for clients, from 12 in 2010 to 5 days in 2012; from 9 to 5 days for task team leaders, and from 8 to 7 days by the procurement specialist (Appendix Table D.4). The analysis also confirms the importance of looking at not just average elapsed time, but its distribution, and confirms that there is a long tail of outlying transactions that take considerably longer than average and that contribute to overall outcomes. The average numbers of days taken by client, task team leader, and procurement specialists was considerably greater than the median of the distribution; for instance, although average days per iteration taken by the procurement specialist per iteration is seven, half the transactions (the median) take fewer than three days. 155 Table 4.6. Africa: Numbers of Iterations and Average Response Times per Procurement Transaction Request, 2009–12 Average numbers of iterations Average number of days taken per iteration Number of Client - TTL—Procurement Client TTL Procurement Year records TTL specialist specialist 2009 1,970 2 2 14 10 8 2010 2,540 3 3 12 9 8 2011 2,778 3 3 12 9 8 2012 3,043 2 2 5 5 7 Avg/Total 10,331 2.5 2.5 10.7 8.2 7.6 Source: IEG analysis of Africa Region data. Note: TTL = task team leader. 141 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT The analysis confirms that clearance at the OPRC level implies long delays in terms of time, with 27 days per iteration. Clearly this partially reflects the increased complexity of such cases. Moreover, the borrower does not always have the capacity to prepare technically adequate documents. Such factors are difficult to reflect in the present analysis. Clients, however, do not take longer to respond to OPRC. There are also clear differences in the numbers of client and task team leader iterations, depending on the clearance level, with an average of five iterations for requests at the OPRC level, four at the regional procurement manager level, and three at the procurement specialist level (Table 4.7 and Appendix Table D.5). Table 4.7. Africa: Response Times for Procurement Categories by Clearance Level, 2009–12 Average numbers of days per Procurement category Clearance level iteration PS RPM OPRC (Number of records) Goods, works, and nonconsulting services 2,690 221 40 Consulting services 4,333 250 4 Client Goods, works, and nonconsulting services 11 16 8 Consulting services 10 13 10 TTL Goods, works, and nonconsulting services 7 7 13 Consulting services 7 10 5 Procurement specialist Goods, works, and nonconsulting services 7 8 16 Consulting services 6 8 25 Source: IEG analysis of Africa Region data. Note: PS = procurement specialist; RPM = regional procurement manager; TTL = task team leader. Delays and Idle Time—Borrower, Bank, and Procurement Staff Besides tracking elapsed times and iterations at each step of the procurement process, the Africa Region database also records “idle” time in between steps. As an example, once bidding documents have been cleared by the Bank, the borrower requires time to prepare for the next step: receiving and evaluating bids, and preparing a bid evaluation report. The procurement office in Africa considers the normal time required for these two steps to be 70 days for consultant contracts and 90 days for goods, works, and nonconsulting services. To the extent that the time in between steps exceeds these norms, it is counted as a borrower delay, which is thus the difference between the “idle time” and the “normal time.” Findings show that borrower delay over and above norms is more important than time taken by the task team. As pointed out by Bank management, multilayered procurement processes in countries also contribute to delays.There are no clear trends over time and considerable variation from one quarter to the next, suggesting that individual transactions may play a large role. 142 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Figure 4.4. Borrower Delays and Bank Response, 2010–12 70 60 50 Number of days 40 30 20 10 0 2010-Q1 Q2 Q3 Q4 2011-Q1 Q2 Q3 Q4 2012-Q1 Q2 Q3 Q4 Borrower Delay Task Team Time Procurement Staff Time Source: Africa procurement office database Note: Delays recorded in PROCYS are defined as the average time taken by the borrower beyond normal processing time (70 days for consultants and 90 days for goods, works, and services). Better tracking systems are just a first step toward making procurement more efficient, in conjunction with other steps. Ideally, tracking systems should integrate potential delays in the Bank that lie outside the procurement anchor, for example, those caused by efforts to control fraud and corruption by INT or the Office of Sanctions and Debarment. There is also scope for increasing efficiency in current Bank processes by streamlining and simplifying the selection process. Current European Union proposals (Appendix A), for example, include shortened time limits for bidding, and IEG found that several client countries have shorter bid times than the Bank requires. With modern communications and more of the process on-line, shortening some periods could be considered. Greater built-in flexibilities—for example, provisions to modify contracts during implementation—could also be considered. The Bank could also consider the introduction of stricter performance penalties for suppliers that have substantially exceeded time limits. Further New Directions for Increasing Efficiency New procurement methods can also offer a means toward increasing efficiency. One example is the use of framework agreements (Box 4.5). Framework agreements can save cost and time, for example, for small but frequent purchases or for repeat phases or purchases that are related to each other. The Bank’s current policy, which introduced the use of framework contracts in 2011, limits their use to common-use goods, simple nonconsulting services, or small-value emergency works contracts, with a contract duration of up to three years and a value not exceeding the NCB limit. In practice, they have been little used under Bank projects, although several 143 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT countries visited confirmed that framework contracts are permissible under local law (Bangladesh, Ethiopia, Mexico, Peru, and Tanzania). 156 Discussions during IEG’s field visits revealed that in Bangladesh, the government developed processes for using framework agreements for the health sector under its own laws. Framework agreements are provided for in Ethiopian law, at the federal and provincial levels. The government of Ethiopia moved forward with its introduction largely unassisted by the Bank, and has entrusted a new agency with responsibility for this—the Procurement and Property Disposal Agency. In Mexico, the Bank facilitated the process of introducing changes into the legal system to provide for the design and formulation of framework agreements. Bank staff played an integral role in establishing a knowledge exchange partnership between the government of Western Australian and the government of Mexico. Box 4.5. New Procurement Methods—Framework Agreements and Reverse Autions Framework agreements, also known as indefinite delivery, indefinite quantity contracts, provide for an indefinite quantity of services during a fixed period of time. These arrangements allow the purchaser more flexibility in both volume and detail of goods or services contracted for. Framework agreements can involve multiple suppliers, allowing the purchaser flexibility to select the best value for each purchase. In a role reversal from a traditional auction, reverse auctions allow bidders to offer the lowest price on the good or service sought by the government purchaser. This is most commonly done on line in real time, creating downward price pressure that often results in a better value to the purchaser. One recent diagnostic report undertaken with the help of the Bank, the Colombia MAPS, discusses reverse auction as well as small-value contracting, shopping, and procurement through commodity markets. Source: IEG. Globally, procurement systems are emphasizing means of improving efficiency that make more intensive use of electronic platforms, and beyond this there is an increasing emphasis on modernization and simplification. The European Union, in its upcoming revisions to its public procurement directive, has proposed several measures to promote and mandate e-procurement—for example, through the use of mandatory electronic notices and communications—a switch to the use of electronic document submission. Also under consideration are proposals to introduce electronic reverse auctions, dynamic purchasing systems, and electronic catalogs. 144 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT Though the Bank permits the use of all these systems, it could go further and encourage client countries to move toward their adoption. Box 4.6. E-Procurement There is extensive evidence from the European Union, the Organisation for Economic Co-operation and Development and individual countries that e- procurement has the potential to strengthen governance through its capacity to enhance transparency and improve access to management and audit information. The Aberdeen Group reported (2008) that public sector enterprises have significantly improved their performance as a result of e-procurement initiatives with lower transaction costs, lower incidents of randome and unaccountable or unjustifiable spending, and lower transaction cycle times. These results were echoed in a 2007 World Bank survey of 14 countries, where e-procurement had been introduced, who recounted benefits in terms of transparency, efficiency and market intelligence. Efficiencies for government are also efficiencies for the private sector: the value of e- procurement for suppliers is in terms of efficient market access, ease of bidding, document transfers, business collaboration, efficient transactions, and market information. Potential savings from the successful implementation of e- procurement framework have been estimated by the Organisation for Economic Co- operation and Development to be in the range of five to eight percent of procurement value. When combined with the greater procurement coordination and management control information facilitated by e-procurement, costs have been reported to fall by up to 20 percent. The increase in competition from e- procurement may yield substantial savings through the provision of greater visibility and ease of access to the government market. E-procurement implementation depends on the approach to the development of the technology used and the business process reforms initiated to accompany it. Complex technology and high-risk approaches used in some countries are paralleled with low-risk approaches; incremental development of back office technology- driven functions and tools that have been gradually extended into a fully functional e-procurement market place. The degree of management involvement affects design; some governments have delegated the e-procurement project management role to the software developer, who may have been more prone to dismissing unjustifiable or unsustainable costs. Major sources of concern have been digital signatures being abandoned or corrupted, business models based on limited market information, and poorly understood outsourcing options and risks. Where technology departments have sole leadership, solutions may not correspond to the needs of procurement officials, 145 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT such that these officials have retained paper processes for most procurement activities. Support from MDBs to surmount such obstacles has been weak, based on anecdotal information from MDB partners and IEG interviews. MDBs including the Bank have developed an accreditation model for e-procurement systems to certify whether the system is suitable for Bank projects. However, this accreditation provides very limited help or guidance to countries. The Bank has sometimes provided consultants to countries, raising expectations in relation to e-procurement, but follow-up on how to go about it has been limited, sometimes leading to suboptimal outcomes. There is substantial scope for more support to arrive at better matched solutions. Sources: Aberdeen Group 2008; AfDB, IDB, World Bank, and Curtin University of Technology 2007. BANK INVOLVEMENT IN THE DEVELOPMENT OF E-PROCUREMENT Bank clients have reiterated the need for the Bank to develop or expand e- procurement initiatives. The World Bank Institute (WBI) has delivered structured learning in e-procurement to five countries in the East Asia and Pacific Region using the Global Development Learning Network. WBI has recently converted its e- procurement learning materials into an e-learning course. 157 As part of field visits to select case study countries, IEG sought information about the Bank’s role in e-procurement from country management, Bank procurement staff, and country client counterparts. Both Bank staff and country client counterparts were asked about the extent to which the Bank had focused on new procurement approaches such as e-procurement and whether it had been able to provide support to e-procurement. Bank country management from Azerbaijan, Tanzania, and Turkey indicated that the Bank is not seen as a leader in e-procurement. The Bank plays a minor role in Peru but is not seen as a leader. In Morocco the Bank is seen as a coleader, working alongside (but notably not coordinating well with) the European Commission. Findings are similar in Indonesia, where it was reported that there is no clear view as to the Bank’s leadership role in e-procurement within the development community. Country clients in Bangladesh, Morocco, and the Philippines had favorable opinions of bank support for innovation in e-procurement. Responses from Azerbaijan, Ethiopia, Peru, Senegal, Tanzania, and Turkey did not indicate that the Bank had provided cutting edge support. There were no responses from Indonesia or Mexico. 146 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT It is evident that other development partners have been extensively involved in e- procurement efforts in various field visit countries—in some cases their involvement has exceeded that of the Bank. The message from many of the field visit countries is that the Bank has not been a consistent leader in e-procurement development, and in some cases, Bank involvement has been limited to piecemeal support and ensuring that country e-procurement systems are appropriate for Bank-funded projects. Discussions with staff and clients suggest that to the extent that case study countries already had an existing e-procurement system, Bank involvement focused on certifying whether the systems were appropriate for use with Bank-funded projects. Bank contributions were noted as valuable in: • Bangladesh: The role of the Bank has been substantial and sustained beginning with technical assistance in 2004. Its Public Procurement Reform Project funded e-procurement development and capacity building. The e- procurement program has been especially intensive and has led to Bangladesh rolling out a nationwide system with associated training and business process reengineering; this program is expected to be expanded over the next one to two years. The Bank has also played a role in compliance checking during system development and assisting implementation. Additionally, and independent of project funding, the Bank provided advice regarding the e-signature legislation, which was fully taken on board and led to a re-drafting of the proposed law. • Morocco: The Bank assisted Morocco in the development of a public procurement portal in 2007; this has progressively developed in terms of capabilities and entities that use the system. 158 The portal is now used to publish offers, terms of reference, and results. The European Commission is also supporting Morocco in procurement, especially in training. Both the Bank and the European Union are reportedly acting in parallel and providing complementary assistance, without looking for synergies. • The Philippines: A fairly advanced e-procurement system, PhilGeps was initially supported by the Canadian International Development Agency. It then received additional support from the Bank as well as ADB. After a recent review of PhilGeps, the Bank advocated for the development and implementation of a “virtual store” and electronic reverse auctions. In 2011, the Philippines received global distance learning on e-procurement from WBI. In most countries (including Morocco and the Philippines), the Bank was one among a larger group of international players. It helped build on initiatives originally developed by governments on their own: 147 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT • India: Although not a case study country, reference to India is included due to its major internal efforts at e-procurement, usually at the state or procuring entity level, and with limited homogenization. The Bank has essentially been a follower, providing certification of certain systems for Bank use. • Indonesia: the Bank support included a development policy loan that contributed to procurement reform and capacity building—the latest development policy loan sets targets in the area of e-procurement function. The Bank also undertook an e-procurement assessment as part of a multidonor trust fund initiative. Other development partners have played a larger role in development procurement capacity in Indonesia—AusAID has been the most significant contributor and the Millennium Challenge Corporation is preparing projects and investments that include a technical assessment of the e-procurement system. Bank staff in Indonesia have limited involvement and knowledge of ongoing e-procurement work by other development partners. • Mexico: Mexico was considered an international leader in e-procurement when it began development of the CompraNet system in 1996; the system predates Bank involvement in e-procurement in Mexico. Though the Bank did not have a role in developing CompraNet, it did later provide advice on system improvements so that it was acceptable for use in Bank-financed projects. The Bank’s contribution to e-procurement in Mexico has no clear success record; the Bank has tried to influence the government to shift the focus of the program so that it is procurement driven rather than information technology driven, but has had limited success. • Peru: In Peru, an electronic system—SEACE—has been in place for many years. The Bank, along with the IDB, contributed to its development. SEACE is reportedly problematic due to internal issues, lack of qualified personnel, and an insistence on developing in-house solutions rather than acquiring state-of-the-art technology. Recent Bank support to Peru on e-procurement has been through technical assistance and training staff on Bank procurement, but there has not been an effort to improve the overall system. • Turkey: Development of Turkey’s e-procurement system has been linked to Support for Improvement in Governance and Management, with inputs from the European Union. 159 Bank staff visited Turkey in 2012 to evaluate the system’s compatibility with Bank procurement; so far there has been no substantial Bank participation other than this certification. Finally, case study countries also included some with very limited e-procurement capability. The Bank has in some cases offered support, though the level of engagement varies. • Azerbaijan: In 2005 the Bank attempted to develop an e-procurement training program using an Institutional Development Fund grant, but it did not fare well because of implementation problems that included government 148 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT reluctance about the participation of international consultants. However, the Bank is currently discussing an e-procurement collaboration with the government. The European Bank for Reconstruction and Development recently requested that the government modify its public procurement law in line with the UNCITRAL 2011 model, and the Bank participated in and presented a paper on Bank procurement during the discussions. The Bank is not perceived to be at the forefront of e-procurement in Azerbaijan. • Ethiopia: Ethiopia has a relatively limited e-procurement system that includes online access to bidding documents and manuals, created with the support of the Bank. The Bank has initiated discussions on e- procurement, which led to plans for a feasibility analysis of the current information technology infrastructure in the country. Further action on e-procurement is pending the results of the feasibility analysis. • Senegal: Perhaps the most engaged country, Senegal set up a public procurement website with the Bank’s assistance including procurement plans, tender invitations, requests for proposals, and contract awards. A study is ongoing to determine the feasibility of further e-procurement development, but there are no Bank projects supporting this development. • Tanzania: Tanzania also has very limited e-procurement, allowing for some online submission of procurement information. The Bank’s involvement in Tanzania included advising on a new procurement law, which provides for some e-procurement, feasibility studies, and a planned e-procurement pilot for 2016. Tanzania also received e-procurement training from WBI in 2009. However, questionnaire respondents indicated that the Bank has not adequately provided specific guidelines for e-procurement procedures, nor does it promote the use of e-procurement for Bank procurements, preferring paper methods instead. Use of Institutional Development Funds or Other Grants for E-Procurement Institutional Development Fund or other grants have been used quite frequently, though with mixed success, to support the development of e-procurement in some of the case study countries (see Chapter 2). A fund of $299,000 to strengthen e- government procurement was used in Azerbaijan in 2005 but did not fare well. In contrast, such grants in Morocco were directed to support public procurement reforms, human capacity building, and specifically developing an e-procurement system. A new grant aims to assist Moroccan authorities in disseminating the use of e tools in public procurement procedures. In the Philippines, grants were used to support development of e-procurement. In Indonesia, some of the e-procurement support came from a multidonor trust fund. Other funding instruments have also been used to support e-procurement. In Indonesia there has been e-procurement support through development policy 149 CHAPTER 4 ACHIEVING EFFICIENCY IN BANK PROCUREMENT lending. Bangladesh had a Technical Assistance Loan that contributed to e- procurement development. Ethiopia referred to an upcoming Specific Investment Loan that intended to include e-procurement, but the idea was abandoned, as it was determined that information technology capacity was too low. 150 Bibliography APEC (Asian Pacific Economic Cooperation). 1999. “Government Procurement Experts Group Non-Binding Principles on Government Procurement”. August Aberdeen Group. 2008. The E-Procurement 1999. Singapore: APEC. Benchmark Report: Driving Year-Over- ASIL (American Society of International Law). Year Superior Performance. Boston: 2013. Will the World Trade Aberdeen Group. Organization Finally Tackle Corruption ADB (Asian Development Bank). 2002. in Public Purchasing? The Revised “Understanding Public Procurement.” Agreement on Government Governance Brief, Asian Development Procurement. Bank. Manila: ADB. http//www.asil.org/insights130415.cf m. ———. 2007. Environmentally Responsible Procurement—A Reference Guide. Manila: Blue, Richard, and John Eriksson. 2011. ADB. “Evaluation of the Implementation of the Paris Declaration: United States AfDB. (African Development Bank). 2011. Government Synthesis Report”. “Evaluation of Paris Declaration Arlington, VA: The United States Implementation at the African Agency for International Development. Development Bank.” Operations Evaluation Department. Tunis: AfDB. Burger, Philippe, and I. Hawkesworth. 2011. How to Attain Value for Money: ———. 2012. “Determination of Financial Comparing PPP and Traditional Thresholds for Modes of Procurement of Infrastructure Public Procurement. OECD Goods, Works and Non-Consulting Journal on Budgeting, Volume 2011/1. Services”. African Development Bank Paris, France: OECD. Group, Procurement Policy and Fiduciary Services Department. January Casartelli, G., E. Wolfstetter. 2007. "Selection and 2012. Tunis: AfDB. Employment of Consultants: the Effectiveness of the World Bank's AfDB, IDB (Inter-American Development Bank), Policy" Working Paper. Washington, World Bank, and Curtin University of DC: World Bank. Technology. 2007. International Survey of Procurement. Perth. Commonwealth of Australia. 2008. Commonwealth Procurement Guidelines, Agaba, Edgar, and Nigel Shipman. 2007l “Public December 2008. Issued under Procurement Reform in Developing Regulation 7 of the Financial Countries: The Uganda Experience.” In Management and Accountability G. Pig and K.V. Thai (eds.) Advancing Regulations, 1997. Canberra: Public Procurement: Practices, Innovation, Commonwealth of Australia. and Knowledge Sharing (Boca Raton, FL: Pr. Academics Press), 373-91. Debevoise, E.W., and C. Yukins. 2010. “Assessing the World Bank’s Proposed Al-Azar, Rima. 2011. “A Tale of Two CDDs: Revision of Its Procurement Lessons Learned for Mitigating the Risk Guidelines,” The Government Contractor, of Fraud and Corruption in CDD 52, (21). Operations.” PowerPoint presentation, World Bank, Washington, DC. Dehn, Jan, Ritva Reinikka, and Jakob Svensson. 2003. “Survey Tools for Assessing Andrews, Matt. 2013. The Limits of Institutional Performance in Service Delivery.” In Reform in Development: Changing Rules Francois Bourguignon and Luiz Pereira for Realistic Solutions. Cambridge: da Silva (eds.), Evaluating the Poverty and Cambridge University Press 128-130. Distributional Impact of Economic Policies. 151 BIBLIOGRAPHY New York: Oxford University Press and ———. 1990. “A Review of Procurement in World Bank. Policy-Based Lending.” Report no. 8341, Washington, DC: World Bank. Dener Cem, Joanna. A Watkins and William. L. Dorotinsky. 2011. “Financial ———. 2002. India Country Assistance Evaluation. Management Information Systems—25 Washington, DC: World Bank Years of World Bank Experience on ———. 2005. The Effectiveness of World Bank What Works and What Doesn’t.” World Support for Community-Based and –Driven Bank, Washington, DC. Development: An OED Evaluation. De Silva, S. 2000. “Community-based Washington, DC: World Bank. Contracting; A Review of Stakeholder ———. 2006a. Debt Relief for the Poorest—An Experience.” Washington DC: World Evaluation Update of the HIPC Initiative. Bank. Washington, DC: World Bank. DfID (Deparment for International ———. 2006b.Engaging with Fragile States: An Development). 2011. “Approach to Evaluation of World Bank Support for Low- Value for Money.” London: Department Income Countries Under Stress. for International Development. Washington, DC: World Bank. European Union, World Bank, and Inter- ———. 2006c. Hazards of Nature, Risks to American Development Bank. 2009. Development – An IEG Evaluation of World Perú Informe de la gestión de las finanzas Bank Assistance for Natural Disasters. públicas Basado en el marco de referencia Washington, DC: World Bank. para la medición del desempeño (PEFA). Lima: European Union, World Bank and ———. 2006d. Small States: Making the Most of Inter-American Development Bank. Development Assistance – A Synthesis of World Bank Evaluation Findings. Gartner Group. 2012. “Survey Shows Why Washington, DC: World Bank. Projects Fail.” http://www.gartner.com/id=2034616. ———. 2007a. The Development Potential of Regional Programs – An Evaluation of GSDRC. 2010. Helpdesk Research Report: Public World Bank Support of Multicountry Financial Management and Frontline Operations. Washington, DC: World Service Bank. Delivery. http://www.gsdrc.org/docs/ open/HD653.pdf,. ———. 2007b. Water Management in Agriculture—Ten Years of World Bank IDB (Inter-American Development Bank). 2010. Assistance, 1994-2004. Washington, DC: “Guide for Acceptance of the Use of World Bank. Country Procurement Systems.” June 2010. Washington DC: IDB. ———. 2008a. Country Financial Accountability Assessments and Country Procurement IAEA (International Atomic Energy Agency). Assessment Reports: How Effective Are 2011. Agency for Financial Regulations World Bank Fiduciary Diagnostics? and Rules AM VI/1: Procurement of Washington, DC: World Bank. Goods and Services, February 16. Mimeo. ———. 2008b. Public Sector Reform: What Works IEG (Independent Evaluation Group). 1977. and Why? An IEG Evaluation of World “The Role and Use of Consultants in Bank Support. Washington, DC: World Bank Group Projects.” Report no. 1824, Bank. Washington, DC: World Bank. ———. 2008c. The World Bank in Indonesia 1999- ———. 1981. “An Interim Report on 2006--Country Assistance Evaluation. Procurement Issues in Bank-Financed Washington, DC: World Bank Projects.” Report no. 3557, Washington, DC: World Bank. 152 BIBLIOGRAPHY ———. 2009a. Bangladesh Country Assistance ———. 2011c. Trust Fund Support for Evaluation 2001-2008. Washington, DC: Development – An Evaluation of the World World Bank Bank’s Trust Fund Portfolio. Washington, DC: World Bank. ———. 2009b. Improving Effectiveness and Outcomes for the Poor in Health, Nutrition ———. 2011d. World Bank Country-Level and Population--An Evaluation of World Engagement on Governance and Bank Group Support since 1997. Anticorruption: An Evaluation of the 2007 Washington, DC: World Bank Strategy & Implementation Plan. Washington, DC: World Bank. ———. 2009c. Review of IDA Internal Controls: An Evaluation of Management’s Assessment ———. 2011e. World Bank Progress in and the IAD Review. Washington, DC: Harmonization and Alignment in Low- World Bank. Income Countries. Washington, DC: World Bank. ———. 2009d. The World Bank Group Guarantee Instruments 1990-2007 –An Independent ———. 2012a. The Matrix System at Work: An Evaluation. Washington, DC: World Evaluation of the World Bank’s Bank Organizational Effectiveness. Washington, DC: World Bank. ———. 2010a. Cambodia – An IEG Country Assistance Evaluation, 1999-2006. ———. 2012b. “The World Bank and Public Washington, DC: World Bank Procurement. An Independent Evaluation.” Approach Paper. ———. 2010b. Climate Change and the World Bank Washington, DC: World Bank. Group. Phase II: The Challenge of Low- Carbon Development. Washington, DC: ———. 2012c. The World Bank Group’s Response to World Bank. the Global Economic Crisis—Phase II. Washington, DC: World Bank. ———. 2010d. IDA Internal Controls: Evaluation of Management’s Remediation Program-- ———. 2013. “Comparison of the International Report on Management’s Implementation Instruments on Public Procurement.” and IAD’s Review of the Five-Point Action Background paper. Washington, DC: Plan. Washington, DC: World Bank. World Bank. ———. 2010d. Mozambique Country Program IMF (International Monetary Fund). 2012. Evaluation. Washington, DC: World Update on the Bank’s Business Bank. Modernization: Results, Openness, and ———. 2010e. Poverty Reduction Support Credits: Accountability Spring 2012.”IMF World An Evaluation of World Bank Support. Bank Development Committee. Washington, DC: World Bank. Kivistö, T., V.M. Virolainen, and E. Tella. 2003. ———. 2010f. The World Bank in Nigeria 1998- Consortia purchasing and logistics in 2007—Country Assistance Evaluation. Kuopio area (Finland)--Lessons learned Washington, DC: World Bank from a 4-year project. IRSPP conference, Budapest, and in KV Thai et al. (eds.), ———. 2011a. “Aid Effectiveness 2005-10: (2005): Benchmarking municipal public Progress in Implementing the Paris procurement activities in Finland. In Declaration.” World Bank, Washington, Challenges in Public Procurement: an DC. International Perspective. ———. 2011b. Capturing Technology for Korpela, J., and M. Tuominen. 1994. Development – An IEG Evaluation of World Benchmarking Logistics Performance with Bank Group Activities in Information and an Application of the Analytical Hierarchy Communication Technologies. Process. Lappeenranta, Finland: Washington, DC: World Bank. Lappeenranta University of Technology. 153 BIBLIOGRAPHY Kruger. K, 2009. “Ban on negotiations in tender ———. 2011c. Strengthening Country Procurement procedures – Undermining the best Systems, prepared for the 4th High Level value for money.” Journal of Public Forum on Aide Effectiveness, Procurement, 4. November 29 – December 1, 2011, Busan, Korea. OECD/DAC Working Mackie, Andrew, and Giovanni Caprio. 2011. Party on Aid Effectiveness Task Force Assessing the Impact of the FEFA on Procurement. Framework. Washington, DC: PEFA Secretariat. OECD-DAC and World Bank. 2004. “Mainstreaming and Strengthening Ministry of Foreign Affairs of Denmark. 2008. Public Procurement: A Strategic “Evaluation of the Implementation of Framework.” Draft report of the 3rd Joint the Paris Declaration: Phase One Round Table on Strengthening Synthesis Report.” Denmark: Ministry Procurement Capacities in Developing of Foreign Affairs. Countries. Washington, DC, and Paris. OECD (Organization for Economic Cooperation Pallas, Christopher L., and Jonathan Wood. and Development). 2003. Mainstreaming 2009. “The World Bank’s Use of the Procurement Function into the Public Expenditure Policy and Effectiveness Country Systems for Procurement: A Dialogue. Paris: OECD. Good Idea Gone Bad?” Development Policy Review 27 (2): 215-230. ———. 2005a. “Paris Declaration on Aid Effectiveness--Ownership, PEFA Secretariat. 2011. Independent Evaluation of the Public Expenditure and Financial Harmonization, Alignment, Results and Accountability Program. Washington, DC: Mutual Accountability.” Paris, France: PEFA. OECD. Pollit, C., and G. Bouckaert. 2004. Public ———. 2005b. “Strengthening Procurement Management Reform: A Comparative Capacities in Developing Countries.” Analysis. New York: Oxford University Volume 3 of the OECD-DAC Guidelines Press. and Reference Series, Harmonising Donor Practices for Effective Aid Delivery. Paris: Sanchez, A. 1997. The World Bank Procurement OECD. Function – Adjusting to Emerging Needs. Washington, DC. World Bank. ———. 2006a. “The Challenge of Capacity Retrieved Development: Working Towards Good from: http://intresources.worldbank.or Practice.” DAC Reference Document, g/INTPROCUREMENT/Resources/Pro Paris. curement-Strategy.pdf. ———. 2006b. “Methodology for Assessment of Singh, J., Dilip R. LImaye, Brian Henderson, and National Procurement Systems.” Paris: Xiaoyu Shi. 2012. “Public Procurement OECD. Of Energy Efficiency Services: Lessons ———. 2009. “Compendium of Country from International Experience.” Examples and Lessons Learned from Washington, DC: World Bank. Applying the Methodology for Tonkin, Christine. 2012. “Procurement Assessment of National Procurement Governance Reform at the International Systems: Sharing Experiences.” OECD Atomic Energy Agency.” Mimeo. Journal on Development 9(4). Thai, K.V., et al. 2005. “Benchmarking Municipal ———. 2011a. International Engagement in Fragile Public Procurement Activities in States: Can’t We Do Better? Paris: OECD. Finland.” In Challenges in Public ———. 2011b. A Practical Guide for Transforming Procurement: An International Perspective. Procurement Systems. Paris: OECD. Academic Press. 154 BIBLIOGRAPHY UNCITRAL (United Nations Commission on ———. 2007a. “Detailed Implementation International Trade Law) Secretariat. Review. India Health Sector. 2006-2007.” (1994, 2011) “UNCITRAL Model Law on INT, Washington, DC: World Bank. Public Procurement.” UNCITRAL, ———. 2007b. “Mandatory Provisions for NCB Vienna, Austria. (for Bank/IDA financed contracts).” Volcker, Paul, Gustavo Gaviria, John Githongo, Washington, DC: World Bank. Ben W. Heineman Jr., Walter Van ———. 2007c. “Operational Approaches and Gerven, and Sir John Vereker. 2007. Financing in Fragile States.” Independent Panel Review of the World Washington, DC. Bank Group Department of Institutional Integrity. Washington, DC: World Bank. ———. 2007d. “Strengthening World Bank Group Engagement on Governance and World Bank. 1997. “The World Bank Anti-Corruption.” Washington, DC: Procurement Function- Adjusting to World Bank. Emerging Needs.” SecM97-854, Washington, DC: World Bank. ———. 2008a. “Enhanced Support to Staff Working in Fragile and Conflict- ———. 2002a. “Fiduciary Management for Affected States.” Washington, DC: Community-Driven Development World Bank. Projects: A Reference Guide.” Washington, DC: World Bank. ———. 2008b. “Use of Country Procurement Systems in Bank-Supported Operations: ———. 2002b. Guidelines for Country Procurement Proposed Piloting Program.” R008- Assessment Reviews. Washington, DC: 0036/5. Washington, DC: World Bank. World Bank. ———. 2009a. “Guidance to Staff for Using the ———. 2003. “Standard Bidding Documents: Company Risk Profile Database Supply and Installation of Information (CRPD).” World Bank, Washington, DC. Systems.” Washington, DC. ———. 2009b. “Handling Suspected and ———. 2004. “Country Procurement Issues Alleged Fraud and Corruption in Paper—Rwanda.” Washington, DC: Procurement: OPCS and INT Protocol.” World Bank. Washington, DC: World Bank. ———. 2005a. “Africa Capacity Building Task ———.2009c. “Piloting Program in Use of Force Report: Forging Effective States Country Systems: First Progress and Engaged Societies.” Washington, Report.” Operations Policy and DC: World Bank. Country Services. Washington, DC: ———.2005b. “Expanding the Use of Country World Bank. Systems in Bank-Supported Operations: ———. 2010a. Financial Management Sector Issues and Proposals.” Operations Strategy FY11-13. 6, 9. Washington, DC: Policy and Country Services, Financial Management Sector Board. Washington, DC. ———. 2010b. “Guidance Note on the ———. 2005c. “Meeting the Challenge of Operational Risk Assessment Africa’s Development: A World Bank Framework (ORAF) – Risks to Group Action Plan.” Africa Region Achieving Results.” Operations Policy Department, Washington, DC: World and Country Services, World Bank, Bank. Washington, DC. ———. 2006. “Quality of Information and ———. 2010c. “Guidance Note: Procurement Communication Technology Arrangements Applicable to Public- Components in Bank Projects.” QAG Private Partnership Contracts Financed Assessment, Washington, DC: World Bnak. 155 BIBLIOGRAPHY under World Bank Projects.” Services under IBRD Loans and IDA Washington, DC: World Bank. Credits and Grants by World Bank Borrowers.” Washington, DC: World ———. 2010d. Monitoring Report 2010. PEFA Bank. Secretariat Report. World Bank, Washington, ———. 2011g. “Strengthening Country DC. http://pefa.org/sites/pefa.org/file Procurement Systems.” Prepared for the s/attachments/ENG-%20MR-10- 4th High Level Forum on Aide FinalReport-May192011.pdf. Effectiveness, November 29 – December 1, Busan, Korea. Washington, DC: ———. 2010e. “Piloting Program in the Use of World Bank. Country Procurement Systems—Second Progress Report.” Washington, DC: ———. 2011h. “Strengthening World Bank World Bank. Engagement on Governance and Anti- Corruption: Procurement Companion ———. 2010f. “Procurement Risk Assessment Piece.” Washington, DC: World Bank. and Management System User Guide.” OPSOR, May 2010, Updated July 2011, ———. 2011i. World Development Report: Conflict, November 2011, January 2012. Security, and Development. Washington, Washington, DC: World Bank. DC: World Bank. ———2010g. “Public Procurement of Energy ———. 2012a. “Analysis of World Bank Efficiency Services: Lessons from Completed Cases of Fraud and International Experience.” Washington, Corruption from the Perspective of DC: World Bank. Procurement.” Integrity Vice Presidency, Washington, DC: World ———. 2010h. “Republic of Moldova Country Bank. Procurement Assessment Report.” Washington, DC: World Bank. ———. 2012b. “Applying the Principles for Good International Engagement in ———. 2011a. “Community-Driven Fragile States and Situations to Development Projects: A Review of Strengthen and Transform Public Governance Issues.” OPCS Document, Procurement Systems.” Background World Bank, Washington, DC. note, OPSOR. Washington, DC: World ———. 2011b. Country Partnership Strategy Bank. Progress Report for the Republic of Moldova ———. 2012c. “The Bank's Procurement Policies for the Period FY09-FY13. Washington, and Procedures: Initiating Discussion DC: World Bank. and Approach Papers--Proposed ———. 2011c. “Diversity Benchmarking Study Review.” Report submitted to the Audit in Selected Countries.” OPSOR, Mimeo. Committee and to CODE, Washington, Washington, DC: World Bank. DC: World Bank. ———. 2011d. “Financial Management ———. 2012d. “Fiduciary Management for Information Systems—25 Years of Community-Driven Development World Bank Experience on What Works Projects: A Reference Guide.” and What Doesn’t.” Washington, DC: Washington, DC: World Bank. World Bank. ———. 2012e. Financial Management and ———. 2011e. “Green Procurement Policy Procurement in World Bank Operations: Framework in Selected Countries.” Annual Report for FY11. Washington, OPSOR, Mimeo. Washington, DC: DC: World Bank. World Bank. ———. 2012f. “Green Procurement in Selected ———. 2011f. “Guidelines: Procurement of Environmental Policy Framework: Goods, Works, and Non-Consulting Review of the World Bank‘s 156 BIBLIOGRAPHY Procurement Policies and Procedures.” ———. 2012p. “Update on the Bank’s Business Background Paper. Washington, DC: Modernization: Results, Openness, and World Bank. Accountability Spring 2012.” DC2012- 0005. Washington, DC: World Bank. ———. 2012g. “Guidance Note for Design and Management of Procurement ———. 2012q. “Virtual VP Review of the Responsibilities in Community-Driven Procurement Policy Review Proposal-- Development Projects.” Washington, An Approach Paper.” Washington, DC: DC: World Bank. World Bank. ———. 2012h. “Investment Lending Reform: ———.2013a. “Approach for Private Sector Modernizing and Consolidating Procurement in Projects financed by Operational Policies and Procedures.” IBRD/IDA, including Public Private OPCS, Washington, DC: World Bank. Partnerships.” Washington, DC: World Bank. ———. 2012i. “OPCPR Procurement Risk Assessment and Management System ———. 2013b. “Audit of the Bank’s Operational User Guide.” Washington, DC: World Framework for Using Investigation Bank. Results in Bank-Funded Projects.” Report No. IBRD FY12-16. Washington, ———. 2012j. “Procurement and PPP DC: World Bank. Transactions Guidance for MDB Public- Sector Engagements and Procurement ———. 2013c. “Procurement in World Bank Principals Applicable to Private Sector Investment Operations. Phase I: A Transactions—Guidance for MDBs.” Proposed New Framework.” OPCS. Washington, DC: World Bank. Washington, DC: World Bank. ———. 2012k. “Procurement Risk Assessment ———. 2013d. “Red Flags Learning Tool.” and Management System User Guide.” Integrity Vice Presidency. Washington, DC: World Bank. http://go.worldbank.org/D1K227ET81. ———. 2012l. “Project Appraisal Document on a ———. 2013e. “Situations of Urgent Need of Proposed Credit in the Amount of SDR Assistance or Capacity Constraints: 26.1 million ($40.0 million Equivalent to Simplified Procurement Procedures and the Republic of Moldova for the Guiding Principles.” Moldova Education Reform Project).” ———. 2013f. “Supporting Countries in Washington, DC: World Bank. Strengthening their Public Procurement ———.2012m. “Public Procurement of Energy Systems: Second Generation Reforms” Efficient Products: Lessons from (draft). Operations Risk Management Around the World.” Energy Sector Department. Washington, DC: World Management Program. Washington, Bank. DC: World Bank. ———.2013g. “Use of Country Procurement ———.2012n. “Review of the World Bank’s Systems.” Background Paper, Procurement Policies and Procedures: Washington, DC, World Bank. Achieving Social Diversity Through the World Bank Institute. 2012. “Public-Private use of Supplier Preferences in the Partnership Reference Guide, Version Procurement Process.” Background 1.0.” Washington, DC: World Bank. paper. Washington, DC: World Bank. World Trade Organization. 1996. “Agreement ———.2012o. “Review of the World Bank’s on Government Procurement (GPA).” Procurement Policies and Procedures, Geneva: World Trade Organization. The Use and Impact of the Bank’s Policy of Domestic Preferences.” Background Paper. Washington, DC: World Bank. 157 Endnotes Chapter 1 1 There are differing views on how and when in the bidding process to qualify firms. The European Union essentially prohibits qualifying firms as part of the bid evaluation process. It must be done solely in advance of the bidding, with no further consideration of qualifications during bid evaluation. In the United States, it is common practice to evaluate qualifications as the final step, after the bid evaluation is completed. In the Bank’s prequalification procedure the qualifications of the bidders are reviewed before the bidders are invited to bid. They are not reviewed as part of the evaluation process, but there is a process (informally used, a post-bid verification) where the borrower confirms that the prequalified applicants continue to meet the qualifications. In the Bank’s postqualification process again, the qualifications are not reviewed as part of bid evaluation. They are reviewed at the end after the lowest evaluated bid price is determined. 2 Some countries have raised another concern regarding bid submission—to permit the use of multiple bid boxes, which some countries maintain can make it easier for bidders to deposit offers in different locations. However, although the Bank could perhaps accommodate different locations for submitting bids, so long it was done in advance of bid opening, and which is not essentially different from submitting bids via courier (allowed), a problem arises when bids are opened at multiple locations. This makes a true public opening impossible. Electronic submission and online opening could eventually take care of this. 3Ideally, participants may need some form of qualification or expertise (possibly from training) and may also need to be accountable. Confidentiality during bid evaluation is important and needs to be protected. 4It is not clear how much the Bank could offer as a silent observer. However, if the Bank does advise the evaluation, questions may be raised as to its neutrality. 5In the Model Law itself, conditions of use for two-stage, request for proposal, and competitive negotiation procedures (which are lumped together), include (a) where it is difficult to formulate specifications, (b) research type contracts, (c) national security concerns, (d) failed tendering, (e) urgent need, and (f) catastrophic event, similar to urgent need. In summary, UNCITRAL seems to say that competitive negotiation is less desirable (higher risk of corruption) and should be used restrictively. However, it is useful (as are two-stage and request for proposal) when technical specifications are not know or difficult to develop. 6 Casartelli and Wolfstetter (2007) raised similar concerns. 7 This appears to be a reference to elapsed time intervals in PROCYS. 8Country ownership was recognized by the Bank as one of the factors critical to successful public procurement reforms (World Bank 2013f). 9 “Regarding the goal of reducing transaction costs, the Bank’s exercise of prior review, under which it verifies compliance with procurement rules, procedures, and policies, is the single most time-consuming element of the oversight process. Prior review would be eliminated in the case of use of country systems” (Pallas and Wood 2009). 10MAPS baseline indicators were used for the first stage of the use of country systems pilot at the Bank. MAPS compliance performance indicators were meant to be utilized as part of Stage III of the piloting program, although no country completed this. 11 Indicators 4(d) and 10(e), respectively. 12 Or, for the minority of countries that included sub-indicator 7(d) in their analysis, 130 out of a possible 165 points. 13 See paras 2.11-2.12 and 2.16-2.18 of the World Bank Guidelines. 159 ENDNOTES 14 Annex C of World Bank (2008b) contains the complete checklist for bidding documents. The checklist was based on sections of the Harmonized Master Procurement Documents, which were approved by the Heads of Procurement Group of the MDBs. 15 Although the overarching indicators and pillars are not traditionally given composite scores as part of the piloting program or MAPS exercises, IEG also calculated average scores received and average number of countries passing for each of these categories. 16 Fifty-seven percent as opposed to 17percent, 18percent, and 44percent for Pillars II, III, and IV, respectively 17 Note that the Turkey team declined to score several subindicators in Stage I, stating that they “need to be addressed at the level of the implementing agency for the pilot project” in Stage III. 18 Subindicator 7(d), “Clarity and transparency of rules for determining whether to engage international or national markets,” is excluded from this list, as this subindicator was only evaluated for three countries, Burkina Faso, Morocco, and Senegal, all of which were found satisfactory in this area. 19 Subindicator 1d. 20 State-owned enterprise participation was restricted for tenders held by that contracting entity, but not for tenders held by other procuring entities. 21 In all, 35 reports were reviewed in 26 countries. The Bank’s Country Procurement Assessment Report included a standard annex checklist for departures from Bank norms in country documents; however, not all reports incorporate this. Some issues listed in project loan/credit agreements are not spelled out in the annex. 22 Bank Guidelines for goods and works include use of country systems as a unique procurement method (although not used so far), although the consultant guidelines do not contain such provisions. 23These results are similar to those noted by Bank management, which find, in a review of 52 countries, that on average there are around 11 NCB provisions per country (World Bank 2013b). 24 The Bank requires at least 30 days or four weeks from the time of advertisement to the bidding deadline. 25 The Mandatory Provisions state only that bidding documents must be “acceptable to the Association.” The Bank Guidelines specify that bid documents may be the national language, but must “provide clear instructions on how bids should be submitted, how prices should be offered, and the place and time for submission of bids.” 26 Negotiation as a method of procurement (direct negotiation, or competitive negotiation) typically takes place at the time of contract award, following a competitive tender process. The Bank’s provisions for NCBs, however, provide that the contract must be awarded to the lowest priced technically responsive bid, with no negotiations permitted. 27 See Appendix A for a survey of areas requiring modification for NCB. 28 Often the status was not specified in the Stage II report and had to be judged by the IEG evaluator. 29 Danish Ministry of Foreign Affairs. 2008. The policies are those related to procurement of goods and works and selection and contracting of consultants, 30 which serve the same function as the Bank Guidelines. 31 “Advanced use” of country systems corresponds to full acceptance of a country’s procurement systems below thresholds established for ICB (as defined in IDB’s Guide for the Acceptance of the Use of Country Procurement Systems, approved by its Board of Executive directors in 2010). However these processes continue to be open to any eligible (national and foreign) competitor. 32 In addition to Turkey, Bhutan was another country that did complete Stage I and II of the use of country systems program, but then chose to follow an alternative path. Bhutan’s small size and the nature of its engagement with donors prompted a decision to first focus on the harmonization of donor and country policies before increasing use of the country system. 33In Azerbaijan, private suppliers appreciated “the good quality of tender specifications” of the Bank’s standard bidding documents. In Morocco, country clients recognized and praised the Bank’s contribution during the design and planning process (exchanges on specifications and evaluation criteria, establishment of procurement 160 ENDNOTES planning). In Tanzania, country clients reported that the requirement of having a procurement plan and Bank support in terms of elaborating comprehensive specifications and terms of reference have helped processing procurement in an efficient manner, meeting required quality and economy objectives. 34 The extent of current Bank involvement in budgetary processes is discussed in Chapter 5 of Volume 1 of this report. 35 Including the World Trade Organization Agreement on Government Procurement, the UNCITRAL model procurement law, and the European Union procurement Directives. 36 The guidelines differentiate between imported goods, domestically manufactured goods with 30 percent or more domestic value added, and other domestically manufactured goods. 37 Proposals for ecolabeling are included in the proposed revisions to the European Union’s public procurement directive (Appendix 7c). 38 This section draws on World Bank (2013b). 39 So far the Bank has not made an effort to follow the ADB in this regard. 40 Value for money is also sometimes elaborated to cover different procurement contexts, such as differences between contracting for works and obtaining professional services. 41 Others apply some value for money principles (such as life-cycle costing, for example) to varying degrees, often without calling it value for money. Some agencies use the term but without defining a distinct procurement process around it—for example, the United Nations procurement guide refers to value for money, but only in limited terms and makes no recommendations to operationalize it in procurement policies and practices. Chapter 2 42 “… ICB is the Bank's preferred approach to procurement. It is intended to offer an efficient and economic approach to procurement, given appropriate circumstances….” The appropriateness of accepted local procedures is recognized in the revised OMS 2.40 of April 1979. 43 IBRD and IDA CDDs were 28 percent of total investment loans approved from FY02 to FY11. The data used were extracted from the Bank’s internal database and the CDD database produced by the Social Development Department. 44 World Bank (2011e), section on Community Participation in Procurement (para. 3.19): “Where, in the interest of project sustainability, or to achieve certain specific social objectives of the project, it is desirable in selected project components to (a) call for the participation of local communities and/or nongovernmental organizations in civil works and the delivery of non-consulting services, or (b) increase the utilization of local know-how, goods, and materials, or (c) employ labor-intensive and other appropriate technologies, the procurement procedures, specifications, and contract packaging shall be suitably adapted to reflect these considerations, provided that these are acceptable to the Bank.” 45 According to the databases maintained by the CDD Anchor in the Social Development Department, of the 2,497 IBRD/IDA investment projects approved during FY02-11, more than a quarter (692) used the CDD approach, at least in part. Development policy operations and additional financing projects and trust-funded CDDs are excluded in the IEG sample. This left 2,044 projects, of which 552 were CDDs. The analysis is supplemented by data from the Bank’s internal database. 46Results are summarized in Appendix D. The CDD anchor further classifies CDD projects into four categories based on their focus/approach: (i) enabling environment for CDDs through policy and institutional reforms; (ii) decision making by participatory, elected local governments; (iii) community control and management of investment funds; and (iv) community control without management of investment funds. A CDD project can have components representing more than one category. For the purposes of this analysis, a project was assigned to the category with the largest allocation in the loan amount 47 In the CDD database, a CDD project can have four different objectives: improving local governance, improving service delivery, community empowerment or improving livelihood security. As most projects aim at more than one objective, for the purposes of this analysis, a project was assigned to the category of the primary objective. 48 Seven projects did not have ISR procurement ratings and were therefore dropped. 161 ENDNOTES 49 In addition, to benefit from synergies with other reviews under this evaluation, projects in “focus countries” and those with higher share of community control and management of investment funds component were given priority in the sample. Focus countries include: (a) field visit Countries (11): Azerbaijan; Bangladesh; Ethiopia; Indonesia; Mexico; Morocco; Peru; the Philippines; Senegal; Tanzania; and Turkey; (b) other focus countries (10): Albania; Bhutan; Colombia; Ghana; Honduras; La; Mozambique; Rwanda; Sierra Leone; and Vietnam; and (c) select countries where INT reviews of CDD projects have been undertaken (6): Cambodia; India, Indonesia; Kenya; the Philippines; and Vietnam. 50 In addition, six countries where INT reviews of CDD projects have been undertaken were also included: Cambodia, India, Indonesia, Kenya, the Philippines, and Vietnam, to enable a closer focus on problem identification. 51 In the Afghanistan National Solidarity project, for example, procurement issues related to procurement under the “district window” component directly managed by the project management unit. Similarly, in Central Africa Republic HIV/AIDS Project, procurement problems related to the centrally managed procurement of the treated bed nets. 52 Implementation of the Indonesia PNPM Rural Project suffered long delays because of difficulties in hiring supervising consultants and facilitators for project coordination and oversight and to provide technical assistance to the communities. 53In the Brazil Maranhao Project, for example, because of the remote location of the project, it was almost impossible to find three suppliers – a problem that the communities tried to overcome by fabricating (with good intentions) three separate bids, thereby opening themselves to charges of fraudulent behavior. 54For example, Bolivia: Second Participatory Rural Investment Project, Brazil: Amapa Sustainable Communities Project, Ethiopia: Pastoral Community Development Project. 55As discussed in country surveys described in Chapter 4, issues of misuse, and fraud and corruption are perceived to be pervasive, and Bank procurement processes within individual projects can only marginally address them. 56This analysis looked at FCS countries from FY06 to the present. In FY06 the Bank used the term low-income countries under stress to describe what is now considered FCS. The LICUS designation was given to those countries with a low CPIA score, and the list was divided into severe, core, and marginal categories. In FY08 a cutoff CPIA score of 3.2 was established and the LICUS list was revised and only included core and marginal categories. In FY09 the list was entitled “Fragile States List” and the cutoff score was revised to 3.0 for core fragile states and 3.2 for marginal. In FY11, per an agreement at the beginning of IDA 15 and along with other multilateral development banks, the World Bank has defined “fragile situations” as having either harmonized average CPIA rating of 3.2 or less or the presence of a United Nations and/or regional peacekeeping or peace building mission (for example, AU, European Union, OAS, NATO), with the exclusion of border monitoring operations, during the past three years. Most recently, in its new operational directives for investment lending (OP 10.0), expected to be in force from April 2013, the Bank has expanded its definition of fragile situations to include cases where the borrower/beneficiary is deemed by the Bank to experience capacity constraints because of fragility or specific vulnerabilities (including for small states). 57 Multidonor trust funds are typically recipient-executed trust funds. These findings are in line with the findings of the Scanteam (2010) report, which highlighted the increase in use of MDTFs as they become the preferred source of funds for operations in FCS, with IDA resources used less frequently. 58 The10 FCS countries were selected in a way as to include countries from different regions which have regularly appeared on the FCS country list. Countries selected were Afghanistan, Bosnia and Herzegovina, Burundi, Haiti, Lao PDR, Liberia, Nepal, Sierra Leone, Timor-Leste, and West Bank and Gaza. 59 The same countries were selected for the project level analysis as for the CAS review: Afghanistan, Bosnia and Herzegovina, Burundi, Haiti, Lao PDR, Liberia, Nepal, Sierra Leone, Timor-Leste, and West Bank and Gaza. 60 Emergency procedures are selectively applied, with country department clearance, to those operations that appear to merit them, and not to all operations in an FCS. There is no flag in the Bank internal database for OP 8.0 projects. IEG therefore used “emergency projects” as a proxy, under the assumption there is a higher likelihood of OP 8.0 being used in these projects. 162 ENDNOTES 61The Bank has, however, created some incentives designed to encourage and facilitate staff placement in FCS (see World Bank 2008a.) 62 As a result, frequently there was a loss of time with the use of ICB and international advertising, because of little external participation. 63 Excluded from the review is the procurement of off-the-shelf equipment and packaged software, such as printers, desktop computers, and similar devices. Also excluded are advisory projects that assist with policy or market development, such as telecommunications connectivity or e-governance. 64It was challenging to construct a sample, since information technology components are embedded within a range of sectors. IEG was able to develop a sample of projects based on the following criteria: (i) identified in the Bank project database as having ICT components; (ii) cited by the task team leader as ICT being critical to achievement of development outcomes; (iii) loan amount exceeding $50 million; and (iv) approved within the last 10 years, but in implementation for at least two years. The sample was adjusted to include additional projects recommended by procurement specialists See World Bank (2003). There are two versions of the documents to cover two-stage and single-stage bidding. 65 These documents were last updated in 2008. 66 The revisions are reflected in clauses 2.6, 2.20, and 3.6 of the guidelines. 67 Clause 2.6 of the guidelines. 68Clause 2.20 of the guidelines addresses the issue of technical specifications that might refer to specific brands or manufacturers. 69 Clause 3.6 of the guidelines, added through the 2011 revisions. Note that framework agreements could also be useful in many other areas of small and repeat purchases, for example, in education or health projects. IEG field surveys found that the Bank's current policy is somewhat limited (to use in emergencies or small works, and for a duration of no more than five years). On average, country clients perceived Bank advice in this respect as being modest. Several country visits confirmed that framework contracts are permissible under local law (Mexico, Peru, and Tanzania) but had not been acceptable to the Bank. In Bangladesh, the government has developed processes for using framework agreements under its own laws and uses them in its health sector. 70 The majority of respondents/interlocutors were project procurement staff, though some task team leaders also participated. 71 This contrasts with findings of the IEG ICT evaluation (IEG 2011b). 72 Prequalification is interpreted in different ways in different jurisdictions. The European Union Court of Justice, for example, required the separation of “selection criteria” and “award criteria,” with prequalification used in the former. In the United States, however, past performance is reviewed as an element of award criteria. 73 Procurement for the Bangladesh Central Bank Strengthening Project had to be restructured after an initial failure of bidding. What is reported here is how well the restructured procurement proceeded. It too encountered delays with one package, but overall was rated successful. 74 In one case, seven firms obtained bidding documents but only two bid. In this case, there did not seem to be an issue with the specification of goods to be procured thought there was some suggestion of possible collusive practices. 75 Similar views were expressed by Bank country staff in Turkey, who were especially supportive of competitive negotiation as a method of procurement. 76 This recommendation is in line with management’s response to IEG’s ICT evaluation. 77 This recommendation is in line with IEG’s ICT evaluation, which recommends looking accelerating clearances. 78Alsoraised in IEG’s ICT evaluation. In its response management offered to address the procurement capacity constraint through training and other means. To determine the World Bank PPP portfolio, IEG applied definitions used by the Bank’s Get PPP anchor, 79 which had undertaken a detailed scrutiny of the infrastructure PPP portfolio over the years FY10 and FY12. 163 ENDNOTES Under the guidance of Get PPP, and in line with an ongoing parallel IEG evaluation of PPP, IEG then constructed a selection of PPP projects for FY02-12 across all sectors. The selection was based on searches of PADs. 80 IEG selected the 46 sample PPP projects from a list of 73 projects based on data from FPD and Get PPP. Of the 73 projects, 27 related to creating an enabling environment and pipeline development and were not reviewed. The remaining 46, which involved Bank investment, were then reviewed. 81 Background Bank information can be found through the PPP in Infrastructure Resource Center: http://PPP.worldbank.org. This site provides a number of PPP resources and links to more, and also contains a database of PPP projects. 82IEG interviews with Bank procurement staff showed a positive reception to the new provisions in terms of constraints on PPP projects. However, there appears to be differing interpretations of the revisions, especially reference to “procedures acceptable to the Bank.” Some read this as a greater reliance on the principles stated in section I of the Guidelines (economy, efficiency, fairness, transparency). Others believe it invokes adherence to section II, the rules for ICB, which are much more prescriptive. Overall, though, the revisions are seen to bring the Guidelines closer in line with the principles of international good practice for PPP: compliance with local law; fairness, transparency, and competitive processes; and concession contracts that are reasonable in terms of price, quality, and risk allocation. 83An overall portfolio of 73 PPP projects was assembled by combining two PPP project databases, one from FPD and the other from Get PPP, comprising 55 Get PPP projects (FY10-12) and 18 FPD projects (FY01-12). The list of projects created under the guidance of and covered the period from the beginning of FY10 up to December 2011. 84The large number of projects from Africa Region may be because financing needs in this region are most acute, and hence there is a bigger role for the private sector. 85 One was a partial credit guarantee, and two were grant-funded under a carbon finance facility. the Bank provided a guarantee in two projects—the Botswana-Morupule B Generation Project 86 Additionally and the West Africa Economic and Monetary Union Capital Markets Development Project. 87 IFC’s equity investment was provided to the Indonesia Infrastructure Finance Facility. Chapter 3 88 Procurement risk involves both process risk (rules and procedures will not be followed) as well as outcome risk (procurement actions will not lead to optimal outcomes in economy). Present Bank practices emphasize process risk, which is intended to support underlying Bank procurement principles of transparency and competition and to help contain fraud and corruption. 89IAD Report on an Audit of the Management of Procurement Risks for Bank-Funded Projects July 2012. 90 On March 30, 2012, proposed revisions were adopted by the World Trade Organization to its Government Procurement Agreement that would explicitly require agreement parties to fight corruption in public procurement. If the proposed revisions are accepted, it will be the first time that a legal instrument of the World Trade Organization has directly addressed the issue of corruption in international trade regulation through the public procurement systems of member countries. See ASIL (2013). 91ICB contracts accounted for around half in terms of numbers of all prior review contracts in FY12 and FY11, based on IEG’s analysis of the Form 384 database. 92 Except in the Middle East and North Africa Region which has maxium thresholds per country and not per project. 93 See Appendix C on the Bank’s methods thresholds. Other MDBs/IFIs have oversight processes that have some differences compared to the Bank, although their guidelines are very similar. For example, at the IDB, all contract approvals are undertaken by field-based staff. Conversely, AfDB clears contracts through a single procurement unit in headquarters. Its limited field staff (of around four people) also report to Manila. With a much smaller procurement operation, much of the oversight is done by sector departments. ICB and prior review thresholds are generally the same, and the procurement unit is only involved with ICB contracts. 164 ENDNOTES Contemplated changes under discussion would put more oversight responsibility to the sectors, coupled with an accreditation program for sector staff. 94 In shopping, formal advertisement and invitations to receive bids are not required but rather several price quotations from at least three suppliers or contractors are compared to assure competitive prices. Other types of procurement methods for goods and works include: Limited International Bidding, Direct contracting, Contracting without competition (single source); and Force account. From 2011, framework agreements – long- term agreements that set out terms and conditions under which procurements are made throughout the term of the agreement—were added to the list. See World Bank (2011e). 95Prior review thresholds have been set as early as the 1980s. The first documented prior review thresholds that IEG located come from 1992. Prior review thresholds were set at the regional level (Sanchez 1997). 96IEG was able to find historical prior review and method thresholds for the following regions: Africa: 2004 (selected countries), 2005 (selected countries), 2010, 2012; East Asia and Pacific: 2006, 2011, 2012; Europe and Central Asia: 2006, 2007 (four revisions), 2010, 2011; Latin America and the Caribbean: 2007 (Colombia, Mexico, Brazil only), 2011; and South Asia: 2008, 2012. 97 BP11.0 was updated in January 2011 and revised again in April 2013. The 2011 BP 11.0 includes the following annexes: Annex A - General Responsibilities and Accountabilities for Procurement Work; Annex B - Decision Authority Matrix; Annex C - Maximum Prior Review Thresholds; Annex D - Mandatory Prior Review Thresholds for RPMs and the OPRC; Annex E - Handling of Procurement Complaints on Contracts Financed by the Bank. There are no changes in the cited thresholds. 98 The Bank’s Procurement Policies and Procedures: Initiating Discussion and Approach Papers – Proposed Review, February 29, 2012. 99 Examples were given of suppliers in the power sector and construction sectors that met all technical specifications and had the lowest price but where clients still perceived qualitative differences. 100 http://www.transparency.org/research/cpi/. In addition, two Kauffmann and Kraay Governance indicators were used for this analysis: Government Effectiveness and Control of Corruption. 101The overall CPIA score and two sub indicators were used for this analysis: sub indicator 13 Quality of Budget & Financial Management and sub indicator 16 Transparency, Accountability & Corruption in Public Sector. 102Additional results are presented in Appendix 5a for subgroups of countries and under alternative assumptions. Correlations presented here represent the most favorable scenarios; lower correlations are obtained under alternative scenarios. 103However, it has also been pointed out that not all completed Post-Procurement Reviews are uploaded into the Bank’s electronic Post-Procurement Review system, in the post-review module of P-RAMS, as required by the Bank’s Guidance Note to Bank Staff to Conduct Post-Procurement Reviews and IPRs (World Bank 2010c). Other issues have been raised, for example, the independence of such reports, and the extent of their 104 availability. Although Post-Procurement Reviews are ordered by Bank procurement staff, they are sometimes undertaken by the same persons, because of resource issues. Consultants used on Bank operations may also be used for Post-Procurement Review s, and they may sometimes undertake several series of Post- Procurement Reviews. Such practices could lead to conflict of interest. 105 FY 09 Annual Procurement Report, OPCS, World Bank. For example, the FY11 annual report makes almost no reference to the substantive findings of the Post- 106 Procurement Reviews, except to say “Post-review helps to strengthen client capacity through the actions taken to address identified issues.” 107 Bangladesh: Post-Procurement Review FY11 consolidated report. 108The sample consisted of 69 PADs from projects approved 2002 to June 30, 2012, based largely on the overall IEG country selection framework described in Chapter 1, drawing on projects in each region and in a combination of sectors. In all, there were 19 countries in the sample as follows: Azerbaijan, Bangladesh, Brazil, Cambodia, China, Ethiopia, Ghana, India, Indonesia, Lao PDR, Mexico, Morocco, Mozambique, the Philippines, 165 ENDNOTES Rwanda, Senegal, Tanzania, Turkey, and Vietnam. The sample was further divided between projects approved before and after the introduction of P-RAMs (in mid 2010). The sample contained 39 pre-P-RAMs projects and 30 projects approved post P-RAMs. 109 Project staff (including the task team leader, the project-relevant Bank procurement specialists, and representatives of the borrower and its implementing agency have a wide array of instruments with which to assess and manage these risks (summarized in Appendix 3a, Box 4) and the evidence shows that they exert significant flexibility within the overall procurement policy framework set by the Bank’s central guidelines. The procurement plan is essentially a team effort. 110These markers are only indicative empirical measures; risk items are not equal in themselves. Moreover, PADs with a greater number of identified risks were often also more highly articulated in their overall treatment of risk; those with fewer identified risks were often more cryptic in their treatment of risk issues, more vague about risk ratings, and less specific in their risk mitigation plans. 111 Disaggregating and classifying, the following categories emerge: Country Procurement Rules and Oversight (local law and public procurement agencies); Project Design issues (most generally decentralized procurement); and Implementation Capacity (comprised of procurement documentation and record keeping and staffing issues). There were also issues relating to the Procurement Preparation Phase (advertising, bidding documents); the Evaluation and Award Phase (including issues relating to the Bid Evaluation Committee; and complaints issues); Fraud and Corruption (collusion; fraudulent documents; contract packaging issues); and Contract Management (downstream risk of fraudulent documentation; quality of construction). Most came in projects approved after 2007, following implementation of the governance and anti-corruption 112 program, as well as publication of the Detailed Implementation Review reports completed by INT on the India Health Sector. 113Although this may reflect diminished numbers of CPARs in the latter period, other diagnostic work could also have been referred to in principle. 114The Procurement Risk Assessment Questionnaire contains a checklist of 11 risk factors, and 60 subquestions (Box 3.4). Once the procurement staff enters risk rating into the P-RAMS template for each risk factor, an overall procurement control risk rating (which P-RAMS labels “Project Risk”) is assigned to the project. The Mitigation Measure Action Plan describes risks and assigns their mitigation to a responsible entity with a planned due date and completion date. 115The questions in the assessment are not mandatory or exclusive, but are intended to aid the procurement staff in the risk assessment process by focusing on the types of generic risks that are likely to be faced when assessing any implementing agency in a given project. It was an explicit OPSOR intention to have only a “soft integration” of P-RAMS and ORAF (hard integration 116 would be too data heavy). 117There is significant variance in compliance rates between regions. East Asia and Pacific, Europe and Central Asia, and South Asia were above the Bank-wide average of 72 percent (80 percent, 81 percent and 72 percent respectively) whereas the Middle East and South Asia (61 percent) and Africa (58 percent) were below average in their completions. 118 Quantitative sections of the questionnaire gave P-RAMS an average score of 2.6, across respondents, that is, somewhere in between a modest and substantial risk mitigation tool. 119Risk Factor 4 (Staffing—the code of ethics for staff); Risk Factor 5 (Procurement Planning—collusion risk); and Risk Factor 8 (Evaluation and Award of Contract—unusual bid patterns); and Risk Factor 11(Procurement Oversight on anti-corruption agencies and their role in independent oversight). This refers to the following questions: Risk Factor 5 question 3; Risk Factor 8 question 3; and Risk Factor 10 120 question 6. 121For FY10, there were 80 instances of misprocurement, of 22,717 contracts awarded, and for FY11, there were 52 instances, of 17,713 contracts awarded. The Bank’s complaints database maintained by the procurement anchor is distinct from INT’s database on 122 complaints. The former can be triggered by any complaint related to the procurement process, and some may 166 ENDNOTES have (or may lead to fraud and corruption aspect. INT’s complaints refer to (suspected) cases of fraud and corruption. 123 According to management, prior review contracts account for around a fifth of the Bank’s total annual contracts, which amount to some 100,000 contracts per year. The most frequent causes for complaints in recent years have been: technical specification issues contained in the requests for proposals; alleged irregularities in either or both technical and financial evaluation of proposals; and awards of contract, often citing lack of qualification of winning vendors, sometimes accompanied by allegations of fraud and corruption. Cases of the latter may lead to INT referrals and entry into a separate INT database, which may lead to further investigation. The number of referrals that have resulted in INT mounting an investigation have been in a range of 90 –150 124 per year. 125INT is equipped to address fraud and corruption issues at two broad levels: investigation and prevention. Historically INT’s main role was to conduct investigations, which were initiated by request of the operating regions, or on its own account. The main vehicle for such investigations was the Detailed Implementation Review, which consisted of in depth analysis and data gathering on a large sample of contracts in several projects. They brought forth abundant information on fraud and corruption. Since discontinued, they have been replaced by individual contract investigations and final investigation reports, which are triggered by a combination of referrals from the operating regions, and/or from other sources, including from a hotline available to the public. 126Its 10 principal flags are complaints from bidder; multiple contracts below procurement threshold; unusual bid patterns; seemingly inflated agent fee; suspicious bidder; lowest bidder not selected; repeat awards to same contractor; changes in contract terms and value; multiple contract change orders; and poor quality of works and services. 127Undisclosed INT working paper provided to IEG: “Risks and Lessons relating to Implementation Identified through FY09-12 Final Investigation Reports and Internal Investigations.” 128 Under Bank rules, collusion is deemed to be a part of fraud and corruption, though in some jurisdictions (for example, the European Union or the United States) they are legally separate issues. 129 Many complaints did not lend themselves to easy classification. Those identified as being procurement related but whose details did not specify a particular procurement stage, were deemed to be “general procurement complaints.” Financial management complaints have been classified as such, and some which had to do with issues in implementation (for example, displacement of people as a result of project activities) have been classified accordingly. A large number could not be linked to any stage in the project cycle (corruption, bribery, nepotism, and so forth), and these have been classified as “other/to be determined.” 130There is also an anti-money-laundering list maintained by the Procurement anchor at the Bank that staff are required to consult. 131An analysis of the incidence of sanctions by Bank management observed that from 2007 to 2012, 157 contracts, concerning 54 projects in 33 countries had led to sanctions. There is no discernible pattern by category or method though there appears to be some association with the transport sector, small civil works, and countries with low CPIA indicators. 132 If OSD determines that there is sufficient evidence to support INT's accusations (that is, that it is “more likely than not” that sanctionable misconduct has occurred), Office of Suspension and Debarment issues the case to the firm and temporarily suspends the firm from eligibility for new Bank-financed contracts pending the final outcome of the sanctions proceedings. The firm has 30 days to submit an “explanation” to the Office, on the basis of which the case can be withdrawn or the recommended sanction revised, and 90 days to appeal the case to the Sanctions Board. If the firm does not appeal to the Sanctions Board, the case ends at the first tier, and the sanction recommended by Office of Suspension and Debarment (usually a form of debarment) is imposed, superseding the temporary suspension. To date, 60 percent of sanctions proceedings have been resolved at the first tier (that is, without an appeal from the firm). If the firm does appeal to the Sanctions Board, there is a de novo review of INT's case, with the Sanctions Board making a final determination as to whether there is sufficient evidence to support INT's accusations and, if so, what sanction is appropriate. 167 ENDNOTES 133The respondent may contest this action and the Evaluation and Suspension Officer can lift the temporary suspension. 134 According to INT, the Company Risk Profile Database will be enhanced in FY14, to include a more robust search engine along with being integrated into the electronic no-objection system. 135Task team leaders in the Roads Sector point to difficulties in getting good contractors who respect Environment, Health and Safety considerations. However, the Bank does not require ISO 18001, 14001, or other certificate. Requests were made for the Bank to develop a “performance database” to track both negative and positive information. 136 The Protocol for Handling Suspected and Alleged Fraud and Corruption in Procurement establishes a process for how to handle this “grey zone,” as does the Guidance Note for Staff for Using the Company Risk Profile Database. See World Bank (2009a, 2009b). Chapter 4 137 Including, ultimately, the use of computer-based forensics for review of bidding documents. 138Form 384 user guide from April 2009 mentions that its creation dates from 1998. Successive versions of the form and its progressive simplifications are discussed in Appendix 4b. http://intresources.worldbank.org/INTPROCUREMENT/Resources/384-Users_Guide.doc 139 In July 2008, the World Bank and the IDB signed a Memorandum of Understanding to share use of this tool. 140Available at www.iniciativasepa.org . SEPA was developed by the Bank, starting in 2006. In 2008, the World Bank and IDB signed a memorandum of understanding "to share use of this tool and thus improve governance of projects and contribute to the harmonization of processes contributing to achieve the Millennium Development Goals." 141Countries using SEPA (as of January 2013): Argentina, Armenia, Bosnia and Herzegovina, Colombia, Costa Rica, El Salvador, Guatemala, Honduras, Mexico, Moldova, Nicaragua, Panama, Peru, Dominican Republic, Uruguay. Belarus and Ukraine are also mentioned as users of SEPA on the SEPA website, however Bank management note that this is not the case. SEPA is being used for a single World Bank funded project in Chile. Work on a new version is ongoing with a main focus of SEPA 3.0 being the connectivity with other systems. 142Its effectiveness depends on the visibility of the SEPA platform to civil society and private firms. Users of the public website, in principle, can filter for countries, project, procurement method, and category to access the latest procurement plans of regional or thematic areas they are interested in including information on estimated and actual contract amounts, contract sign and completion dates, and details on bidders. 143Operations Procurement Electronic No Objection System (http://intranet.worldbank.org/WBSITE/INTRANET/OPERATIONS/INTPROCUREMENT/0,,contentMDK:2 3308187~menuPK:8955261~pagePK:60000209~piPK:60000211~theSitePK:278020,00.html). 144 Appendix B has details. Although IEG drew up a specific list of prior review contracts per country to construct a representative stratified sample, the response rate from country offices was highly variable in quantity and quality. Each regional office has its own format for maintaining contract data (including as may be expected, different currency units and time recording conventions, as well as its own unique reference number, which is not consistently linked to the Bank-wide Form 384. Clearly, this affects reliability of the results obtained and will have to be a caveat to their interpretation. 145 IEG also looked at a shorter interval, from the borrower’s first submission of draft bidding documents to the Bank’s Final No Objection to the Bid Evaluation Report of the contract. This interval is likely to be more meaningful for two reasons. First, Issue of Specific Notice in practice may occur before or after work on draft bidding documents begins. Second, ending with the Final No Objection better reflects steps within the procurement process itself, in contrast to country level factors that may affect contract signature. On average, 253 days were needed, that is, around 30 days more are needed for going to contract signature, which took 286 days. 146The business standards established in Bank Procedure BP 11.00 – Procurement (http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/EXTPOLICIES/EXTOPMANUAL/0,,contentM DK:20064756~menuPK:4564187~pagePK:64709096~piPK:64709108~theSitePK:502184,00.html) address 168 ENDNOTES specifically the response time needed by the procurement team; time taken by other parts of the Bank’s task team are not included. This analysis does not distinguish between different parts of the Bank’s project teams; results presented are for the overall time taken by the Bank. 147From the time the borrower issues the Specific Procurement Notice to signature of the contract. Further detailed breakdowns are in Appendix A. Based on the field visit sample in the IEG dataset. IDA countries are Bangladesh, Ethiopia, Senegal, and 148 Tanzania; IBRD countries are Azerbaijan, Indonesia, Mexico, Morocco, Peru, the Philippines, and Turkey. 149 Using quality- and cost-based selection. 150Though the mean time in the sample is 253 days, the standard deviation is as high as 160 around the mean value. 151 Similar results are obtained if a narrower time interval, only up to the final no-objection, is examined. 152One caveat to these findings is that linear regressions are based on the assumption of underlying normal distributions, which may not apply for some of the variables here. 153This is the case in one regression specification, which is focused on only the procurement-related steps of the interval. 154 2009 was a startup year and is therefore not used as a baseline. 155The database shows some errors in recording times. For example, in 62 cases (of 10,331) the average number of days that client take to respond has a negative values. However, these are not frequent. 156As discussed in Chapter 1 of this volume, proposed revisions to the European Union Procurement Directive go beyond this, to recommend the introduction of the “innovative partnership” procedure, which allows procuring agents to enter into partnership arrangements with suppliers to develop and subsequently supply new and innovative products, works, and services according to specified requirements that are performance oriented. 157From the WBI e-procurement online course: The objective of this learning course is to support MDB member countries to effectively adopt e-government procurement by providing tools that help place it within a broader country procurement reform. This course consists of a conceptual framework of e-government procurement aimed at helping participants develop a common understanding of fundamentals and components that can be used to supplement the existing MDB reform dialogue on procurement. 158 The Bank provided specific assistance in the following: development of a multi-criteria nomenclature aiming at facilitating research of calls for bids by types of services (works, supplies, and services); developing a monitoring system in view of alerting agencies on failures to comply with publishing requirements, developing multimedia guides for public buyers and bidders. 159 Revisions to European Union directives proposed in 2011 place substantial emphasis on e-procurement and propose to make its use mandatory, based on the belief that e-procurement is an important tool to assist small and medium-size enterprise development by making procurement opportunities more accessible and lower costs to bid. 169