83210 v4 The World Bank and Public Procurement—An Independent Evaluation Appendixes to Volume II: Achieving Development Effectiveness through Bank Procurement Contents ABBREVIATIONS ..................................................................................................................................................................... I APPENDIX A. BANK GUIDELINES AND PROCESSES ........................................................................................................ 3 APPENDIX B. CDD PROCUREMENT ................................................................................................................................... 31 PPP PROCUREMENT: COMPARISON OF THE BANK AND OTHER ORGANIZATIONS.................................................. 55 APPENDIX C. PROCUREMENT RISK CONCEPTS AND THRESHOLDS ........................................................................... 61 APPENDIX D. PROCUREMENT TRACKING SYSTEMS, TIME ANALYSIS, AND AFRICA REGION REVIEW ................. 89 BIBLIOGRAPHY .................................................................................................................................................................. 110 ENDNOTES .......................................................................................................................................................................... 119 ii Abbreviations ADB Asian Development Bank ISR Implementation status report AfDB African Development Bank MAPS Methodology for Assessing APEC Asian Pacific Economic Cooperation Procurement Systems CDD Community-driven development MDB Multilateral development bank CPAR Country procurement assessment MEAT Most economically advantageous report tender CPIA Country policy and institutional MRRD Ministry of Rural Rehabilitation and assessment Development DfID Department for International NCB National competitive bidding Development NEPE State Technical Unit FMIS Financial management information OPRC Operational Procurement Review system Committee HDI United Nations Human Development OPCS Operations Policy and Country Index Services IAD Internal Audit Vice Presidency PAD Project appraisal document IAEA The International Atomic Energy PPP Public-private partnership Agency P-RAMS Procurement risk assessment ICB International competitive bidding management system. process PROCYS Procurement Cycle Tracking System— ICR Implementation completion report Africa Region ICRR Implementation completion and RPM Regional procurement managers results reports SEPA Procurement plan execution system ICT Information technology SME Small and medium-sized enterprises IDA International Development Agency TOR Terms of reference IDB Inter-American Development Bank UNCITRAL United Nations Commission on IEG Independent Evaluation Group International Trade Law IFC International Finance Corporation WBI World Bank Institute INT Integrity Vice-Presidency WTO World Trade Organization ISDB Islamic Development Bank Appendix A. Bank Guidelines and Processes Overall Perceptions Table A.1. Bank Procurement Methods—Adequacy of Competition Bank Country proc Bank Country Private Civil Question Avg mgmt staff TTLs clients sector society Do the Bank's Procurement procedures ICB, NCB, and Consultants Selection Procedures in particular) lead to adequate competition in 2.9 2.9 3.2 2.8 3.3 2.9 2.5 terms of numbers of bidders, openness and fairness, geographical reach, etc.? Source: IEG questionnaire. Note: 4 = more than adequate; 3 = adequate; 2 = only somewhat adequate; 1 = inadequate; TTL = task team leader. Table A.2. Bank Procurement—Support to Integrity, Transparency, Cost, Quality, and Timelines Bank Country proc Bank Country Private Civil Question Avg mgmt staff TTLs clients sector society To what extent do the Bank's procurement requirements positively support the integrity, transparency, cost, quality, and timeliness of 3.1 2.6 3.6 3.0 3.5 3.0 2.8 delivering public sector projects? Source: IEG questionnaire. Note: 4 = more than adequate; 3 = adequate; 2 = only somewhat adequate; 1 = inadequate; TTL = task team leader. Table A.3. Bank Procurement—Capturing Bidder Interest Bank Country proc Bank Country Private Civil Question Avg mgmt staff TTLs clients sector society Are bidders s more likely to show interest if the project is under Bank procurement policy 2.8 3.6 3.0 3.2 3.2 3.2 and procedures, as opposed to a project 3.2 under national guidelines and processes? Source: IEG questionnaire. Note: 4 = more than adequate; 3 = adequate; 2 = only somewhat adequate; 1 = inadequate; TTL = task team leader. APPENDIX A BANK GUIDELINES AND PROCESSES Table A.4. Bank Procurement—Emphasis on Price Factors Question Bank proc Bank Country Private Avg staff TTLs clients sector To what extent does the Bank's oversight (no-objection) of prequalification/ short-listing, bidding documents/RFP, 2.9 2.9 2.7 2.9 3.0 bid/proposal evaluation, and contract award emphasize price? Source: IEG questionnaire. Note: 4 = more than adequate; 3 = adequate; 2 = only somewhat adequate; 1 = inadequate; TTL = task team leader. Table A.5. Bank Procurement—Flexibility in Response Bank proc Bank Country Question Avg staff TTLs clients Is the Bank able to respond quickly to special circumstances such as those requiring deviations from policies and procedures, or changes to 2.3 2.7 2.1 2.2 procurement provisions of the financing agreement, or changes to previously agreed arrangements? Source: IEG questionnaire. Note: 4 = more than adequate; 3 = adequate; 2 = only somewhat adequate; 1 = inadequate; TTL = task team leader. Table A.6. Bank Procurement—Incidence of Delay in Contract Award Bank Ctry Proc Bank Ctry Private Civil Question Avg mgmt Staff TTLs clients sector society How frequent is the incidence of delay in the award 2.9 3.2 2.5 2.9 2.7 2.9 3.2 of contracts? Source: IEG questionnaire. Note: 4 = more than adequate; 3 = adequate; 2 = only somewhat adequate; 1 = inadequate; TTL = task team leader. Methodology for Assessing Procurement Systems Indicators and Scores Table A.7. MAPS Indicators and Passing Scores Under the Piloting Program UCS required score Pillar I: Legislative and Regulatory Framework Indicator 1: Public procurement legislative and regulatory framework achieves the agreed standards and complies with applicable obligations. Sub-indicator 1(a): Scope of application and coverage of the legislative and regulatory framework. 3 Sub-indicator 1(b): Procurement Methods 2+ Sub-indicator 1(c): Advertising rules and time limits 3 Sub-indicator 1(d): Rules on Participation 3 Sub-indicator 1(e): Tender documentation and technical specifications 3 Sub-indicator 1(f): Tender evaluation and award criteria 3 Sub-indicator 1(g): Submission, receipt and opening of tenders 3 4 UCS required score Sub-indicator 1(h): Complaints 3 Indicator 2: Existence of Implementing Regulations and Documentation. Sub-indicator 2(a): Implementing regulation (sic) that provide defined processes and procedures not included 2 in higher-level legislation Sub-indicator 2(b): Model tender documents for goods, works, and services 2 Sub-indicator 2(c): Procedures for pre-qualification 2+ Sub-indicator 2(d): Procedures suitable for contracting for services or other requirements in which technical 2+ capacity is a key criterion. Sub-indicator 2(e): User's guide or manual for contracting entities 2 Sub-indicator 2(f): General Conditions of Contracts (GCC) for public sector contracts covering goods, works 3 and services consistent with national requirements and, when applicable, international requirements Pillar II: Institutional Framework and Management Capacity Indicator 3: The public procurement system is mainstreamed and well integrated into the public sector governance system. Sub-indicator 3(a): Procurement planning and associated expenditures are part of the budget formulation 2 process and contribute to multiyear planning. Sub-indicator 3(b): Budget law and financial procedures support timely procurement, contract execution, and 2 payment. Sub-indicator 3(c): No initiation of procurement actions without existing budget appropriations. 2 Sub-indicator 3(d): Systematic completion reports are prepared for certification of budget execution and for 2 reconciliation of delivery with budget programming. Indicator 4: The country has a functional normative/regulatory body. Sub-indicator 4(a): The status and basis for the normative/regulatory body is covered in the legislative and 3 regulatory framework. Sub-indicator 4(b): The body has a defined set of responsibilities including the following… 2 Sub-indicator 4(c): The body's organization, funding, staffing, and level of independence and authority (formal 2 power) to exercise its duties should be sufficient and consistent with the responsibilities. Sub-indicator 4(d): The responsibilities should also provide for separation and clarity so as to avoid conflict of 3 interest and direct involvement in the execution of procurement transactions. Indicator 5: Existence of institutional development capacity. Sub-indicator 5(a): The country has a system for collection and disseminating procurement information, 2 including tender invitations, requests for proposals, and contract award information. Sub-indicator 5(b): The country has systems and procedures for collecting and monitoring national 2 procurement statistics. Sub-indicator 5(c): A sustainable strategy and training capacity exists to provide training advice and 2 assistance to develop the capacity of government and private sector participation to understand the rules and regulations and how they should be implemented. Sub-indicator 5(d): Quality control standards are disseminated and used to evaluate staff performance and 2 address capacity development issues. 5 APPENDIX A BANK GUIDELINES AND PROCESSES UCS required score Pillar III: Procurement Operations and Market Practices Indicator 6: The country's procurement operations and practices are efficient. Sub-indicator 6(a): The level of procurement competence among government officials within the entity is 2 consistent with their procurement responsibilities. Sub-indicator 6(b): The procurement training and information programs for government officials and for 2 private sector participants are consistent with demand. Sub-indicator 6(c): There are established norms for the safekeeping of records and documents related to 2+ transactions and contract management. Sub-indicator 6(d): There are provisions for delegating authority to others who have the capacity to exercise 2 responsibilities. Indicator 7: Functionality of the public procurement market Subindicator 7(a): There are effective mechanisms for partnerships between the public and private sector. 2 Sub-indicator 7(b): Private sector institutions are well organized and able to facilitate access to the market. 2 Sub-indicator 7(c): There are no major systemic constraints (e.g. inadequate access to credit, contracting 2+ practices, etc.) inhibiting the private sector's capacity to access the procurement market. Sub-indicator 7(d) – Clarity and transparency of rules for determining whether to engage international or 2 national markets. *Only analyzed for 3 countries. Indicator 8: Existence of contract administration and dispute resolution provisions Sub-indicator 8(a): Procedures are clearly defined for undertaking contract administration responsibilities that 2 include inspection and acceptance procedures, quality control procedures, and methods to review and issue contract amendments in a timely manner. Sub-indicator 8(b): Contracts include dispute resolution procedures that provide for an efficient and fair 3 process to resolve disputes arising during the performance of the contract. Sub-indicator 8(c): Procedures exist to enforce the outcome of the dispute resolution process. 3 Pillar IV: Integrity and Transparency of the Public Procurement System Indicator 9: The country has effective control and audit systems. Sub-indicator 9(a): A legal framework, organization, policy, and procedures for internal and external control 2+ and audit of public procurement operations are in place to provide a functioning control framework. Sub-indicator 9(b): Enforcement and follow-up on findings and recommendations of the control framework 2+ provide an environment that fosters compliance. Sub-indicator 9(c): The internal control system provides timely information on compliance to enable 2+ management action. Sub-indicator 9(d): The internal control systems are sufficiently defined to allow performance audits to be 2 conducted. Sub-indicator 9(e): Auditors are sufficiently informed about procurement requirements and control systems to 2 conduct quality audits that contribute to compliance. Indicator 10: Efficiency of appeals mechanism. Sub-indicator 10(a): Decisions are deliberated on the basis of available information, and the final decision can 3 be reviewed and rules upon by a body (or authority) with enforcement capacity under the law. Sub-indicator 10(b): The complaint review system has the capacity to handle complaints efficiently and a 3 means to enforce the remedy imposed. Sub-indicator 10(c): The system operates in a fair manner, with outcomes of decisions balanced and justified 3 on the basis of available information. Sub-indicator 10(d): Decisions are published and made available to all interested parties and to the public. 2 Sub-indicator 10(e): The system ensures that the complaint review body has full authority and independence 3 for resolution of complaints. Indicator 11: Degree of access to information Sub-indicator 11(a): Information is published and distributed through available media with support from 2+ information technology when feasible. Indicator 12: The country has ethics and anticorruption measures in place. Sub-indicator 12(a): The legal and regulatory framework for procurement, including tender and contract 3 documents, includes provisions addressing corruption, fraud, conflict of interest, and unethical behavior and sets out (either directly or by reference to other laws) the actions that can be taken with regard to such behavior. Sub-indicator 12(b): The legal system defines responsibilities, accountabilities, and penalties for individuals 3 and firms found to have engaged in fraudulent or corrupt practices. Sub-indicator 12(c): Evidence of enforcement of rulings and penalties exists. 2+ 6 UCS required score Sub-indicator 12(d): Special measures exist to prevent and detect fraud and corruption in public procurement. 3 Sub-indicator 12(e): Stakeholders (private sector, civil society, and ultimate beneficiaries of procurement/end- 2 users) support the creation of a procurement market known for its integrity and ethical behaviors. Sub-indicator 12(f): The country should have in place a secure mechanism for reporting fraudulent, corrupt, or 3 unethical behavior. Sub-indicator 12(g): Existence of Codes of Conduct/Codes of Ethics for participants that are involved in 2 aspects of the public financial management systems that also provide for disclosure for those in decision making positions. 7 National Competitive Bidding Requirements and Compliance Table A.8. Bank NCB Requirements and Country Systems AZ BD ET ID MA PE PH SN Reports Countries Field Visit Field Visit Field Visit Field Visit/UCS Field Field Field Visit/UCS Field Reviewed Reviewed Visit Visit Visit/UCS /UCS 2012 2002 2005 2008 2005 2010 2008 2001 2004 2005 2012 2011 2004 2009 2012 35 26 CPAR 2009 LA CPAR LA LA LA LA LA CPAR LA LA LA LA LA LA CPAR Draft Bank Requirements for NCB There may be no regional / domestic preferences regarding the 18 15 1 1 1 1 1 1 1 1 1 1 sources of labor and material.1 Prospective bidders must be allowed at least 30 days for bid 19 14 1 1 1 1 1 1 1 1 1 preparation.2 Appropriate standard bidding and prequalification documents 15 14 1 1 1 1 1 must be used. Eligibility cannot be restricted based on nationality of bidder 17 13 1 1 1 1 1 1 1 1 and/or origin of goods.3 Award must be made to the lowest evaluated qualified and 15 13 1 1 1 1 1 1 responsive bidder. Minimum requirements must be explicitly stated in the 12 12 1 1 1 1 documents. Bidding opportunities must be advertised in the local press. 11 11 1 1 1 1 Bidders are not generally required to register with a local or 13 11 1 1 1 1 1 1 1 federal authority except under certain conditions.4 The procurement process cannot be cancelled, all bids rejected, 13 11 1 1 1 1 1 1 1 1 and/or rebidding conducted without approval. Public bid opening is required. 11 9 1 1 1 1 1 1 Bid evaluation criteria other than price may be allowed only if 8 8 1 quantified in monetary terms. Price negotiations may not be conducted with “winning” bidders 10 8 1 1 1 1 prior to contract signature.5 Parastatals may only be allowed to bid under certain conditions.6 9 7 1 1 1 Bids may not be rejected based only on a comparison with the procuring entity's estimate; invitations to bid shall not establish 11 7 1 1 1 1 1 1 1 minimums and maximums.7 Audits and inspection of records related to bid submission and 8 7 1 performance of the supplier are permitted. APPENDIX A BANK GUIDELINES AND PROCESSES Table A.8. Bank NCB Requirements and Country Systems AZ BD ET ID MA PE PH SN Reports Countries Field Visit Field Visit Field Visit Field Visit/UCS Field Field Field Visit/UCS Field Reviewed Reviewed Visit Visit Visit/UCS /UCS 2012 2002 2005 2008 2005 2010 2008 2001 2004 2005 2012 2011 2004 2009 2012 35 26 CPAR 2009 LA CPAR LA LA LA LA LA CPAR LA LA LA LA LA LA CPAR Draft “Two envelope” bid opening procedure is permitted for procurement of goods or works under specific conditions, 6 6 1 1 1 notably if domestic law precludes use of one envelope.8 Foreign firms' eligibility cannot be conditioned on joint ventures 7 5 1 1 1 1 with local firms. Qualification criteria shall be applied on a pass/fail basis. 5 5 1 Extension of the time period to prepare bids may only be 5 5 allowed under exceptional circumstances. An inflation clause is recommended for contracts over a year.9 7 5 1 1 1 Bidding documents are freely available.10 7 4 1 1 1 1 1 1 Bid security shall be in the form of a letter of credit or bank 5 4 1 1 1 1 guarantee from a reputable bank.11 Joint venture partners must be jointly and severally liable. 6 3 1 1 1 1 1 There may be no restrictions on the means of delivery of bids.12 3 3 Pre-qualification should be used only for large works projects. 5 3 1 1 1 1 Award must be published. 4 3 1 1 Bidding documents and contract shall include provisions on 3 3 sanctions for fraud and corruption. No preference may be given to suppliers or contractors based on region or locality of registration, small size, ethnic ownership, 4 2 1 1 1 1 etc. For contracts subject to prior review under the Loan Agreement, scope/conditions may not be modified during implementation 6 2 1 1 1 1 1 1 without Bank approval. Bidders shall not be eliminated on the basis of minor deviations. 4 2 1 1 1 Contractors/suppliers must be prequalified for large or 2 1 1 1 specialized contracts. An invitation to prequalify must be advertised for each 1 1 1 procurement involving large or complex potential contracts. There must be no set limitations to the number of firms who can 1 1 bid for a contract. Requests for clarification of bidding documents must be made in 1 1 writing, and response should be sent to all prospective bidders. 9 APPENDIX A BANK GUIDELINES AND PROCESSES Table A.8. Bank NCB Requirements and Country Systems AZ BD ET ID MA PE PH SN Reports Countries Field Visit Field Visit Field Visit Field Visit/UCS Field Field Field Visit/UCS Field Reviewed Reviewed Visit Visit Visit/UCS /UCS 2012 2002 2005 2008 2005 2010 2008 2001 2004 2005 2012 2011 2004 2009 2012 35 26 CPAR 2009 LA CPAR LA LA LA LA LA CPAR LA LA LA LA LA LA CPAR Draft The experience requirement for works contracts shall be (i) at least one previous contract at 80% of cost of contract being 2 1 1 1 procured and (b) an average annual turnover of 100% of contract being procured. Bidders shall be given at least twenty-eight (28) days from the receipt of notification of 1 1 award to submit performance securities. Bids shall not be disclosed excepted as needed for evaluation without bidder's written authorization until the award of the 1 1 contract. Automatic rebidding may be required if too few bids are 4 1 1 1 1 received.13 10 APPENDIX A BANK GUIDELINES AND PROCESSES TZ TR BT BR BF MK MU PA PL GH CO LA VN MZ SL AL BA EG Reports Countries Field Field UCS UCS UCS UCS UCS UCS UCS UCS Desk Desk Desk Desk Desk Desk Other Other Reviewe Reviewed Visit Visit/ Sampl Sample Sample Sample Sample Sample d UCS e 2003 2009 2008 2012 2010 2011 2012 2009 2007 2007 2009 2011 2010 2002 2010 2012 2012 2002 2002 35 26 CPAR LA LA LA LA LA LA LA LA LA LA LA LA CPAR LA CPAR LA CPAR CPAR excerpt Bank Requirements for NCB There may be no regional / domestic preferences regarding the sources 18 15 1 1 1 1 1 1 1 1 of labor and material.1 Prospective bidders must be allowed at least 30 days for bid 19 14 1 1 1 1 1 1 1 1 1 1 preparation.2 Appropriate standard bidding and prequalification documents must be 15 14 1 1 1 1 1 1 1 1 1 1 used. Eligibility cannot be restricted based on nationality of bidder and/or origin 17 13 1 1 1 1 1 1 1 1 1 of goods.3 Award must be made to the lowest evaluated qualified and responsive 15 13 1 1 1 1 1 1 1 1 1 bidder. Minimum requirements must be explicitly stated in the documents. 12 12 1 1 1 1 1 1 1 1 Bidding opportunities must be advertised in the local press. 11 11 1 1 1 1 1 1 1 Bidders are not generally required to register with a local or federal 13 11 1 1 1 1 1 1 authority except under certain conditions.4 The procurement process cannot be cancelled, all bids rejected, and/or 13 11 1 1 1 1 1 rebidding conducted without approval. Public bid opening is required. 11 9 1 1 1 1 1 Bid evaluation criteria other than price may be allowed only if quantified 8 8 1 1 1 1 1 1 1 in monetary terms. Price negotiations may not be conducted with “winning” bidders prior to 10 8 1 1 1 1 1 1 contract signature.5 Parastatals may only be allowed to bid under certain conditions.6 9 7 1 1 1 1 1 1 Bids may not be rejected based only on a comparison with the procuring entity's estimate; invitations to bid shall not establish minimums and 11 7 1 1 1 1 maximums.7 Audits and inspection of records related to bid submission and 8 7 1 1 1 1 1 1 1 performance of the supplier are permitted. “Two envelope” bid opening procedure is permitted for procurement of goods or works under specific conditions, notably if domestic law 6 6 1 1 1 precludes use of one envelope.8 Foreign firms' eligibility cannot be conditioned on join ventures with local 7 5 1 1 1 firms. Qualification criteria shall be applied on a pass/fail basis. 5 5 1 1 1 1 Extension of the time period to prepare bids may only be allowed under 5 5 1 1 1 1 1 exceptional circumstances. An inflation clause is recommended for contracts over a year.9 7 5 1 1 1 1 Bidding documents are freely available.10 7 4 1 11 APPENDIX A BANK GUIDELINES AND PROCESSES TZ TR BT BR BF MK MU PA PL GH CO LA VN MZ SL AL BA EG Reports Countries Field Field UCS UCS UCS UCS UCS UCS UCS UCS Desk Desk Desk Desk Desk Desk Other Other Reviewe Reviewed Visit Visit/ Sampl Sample Sample Sample Sample Sample d UCS e 2003 2009 2008 2012 2010 2011 2012 2009 2007 2007 2009 2011 2010 2002 2010 2012 2012 2002 2002 35 26 CPAR LA LA LA LA LA LA LA LA LA LA LA LA CPAR LA CPAR LA CPAR CPAR excerpt Bid security shall be in the form of a letter of credit or bank guarantee 5 4 1 from a reputable bank.11 Joint venture partners must be jointly and severally liable. 6 3 1 There may be no restrictions on the means of delivery of bids.12 3 3 1 1 1 Pre-qualification should be used only for large works projects. 5 3 1 Award must be published. 4 3 1 1 Bidding documents and contract shall include provisions on sanctions for 3 3 1 1 1 fraud and corruption. No preference may be given to suppliers or contractors based on region 4 2 or locality of registration, small size, ethnic ownership, etc. For contracts subject to prior review under the Loan Agreement, scope/conditions may not be modified during implementation without 6 2 Bank approval. Bidders shall not be eliminated on the basis of minor deviations. 4 2 1 Contractors/suppliers must be prequalified for large or specialized 2 1 contracts. An invitation to prequalify must be advertised for each procurement 1 1 involving large or complex potential contracts. There must be no set limitations to the number of firms who can bid for a 1 1 1 contract. Requests for clarification of bidding documents must be made in writing, 1 1 1 and response should be sent to all prospective bidders. The experience requirement for works contracts shall be (i) at least one previous contract at 80% of cost of contract being procured and (b) an 2 1 average annual turnover of 100% of contract being procured. Bidders shall be given at least twenty-eight (28) days from the receipt of notification of 1 1 1 award to submit performance securities. Bids shall not be disclosed excepted as needed for evaluation without 1 1 1 bidder's written authorization until the award of the contract. Automatic rebidding may be required if too few bids are received.13 4 1 1 Source: World Bank; data compiled from CPARs and Loan Agreements. 1 With the exception of unskilled labor, if available locally. 2 Except for commodities/small goods contracts. 3 Primary boycotts, in which the government maintains a policy of not purchasing from a particular country, are permitted, however. 4 Registration may be required if (i) registration criteria, process and cost are reasonable/efficient and (ii) qualified foreign firms are not precluded from competing. Notwithstanding, the Bank stated that a registration requirement should be discouraged. 5 There is one exception: where the bid price is substantially above market or budget levels and then only if negotiations are carried out to try to reach a satisfactory contract through reduction in scope and/or reallocation of risk and responsibility which can be reflected in a reduction in Contract Price. (See Guidelines para 2.60) 12 APPENDIX A BANK GUIDELINES AND PROCESSES 6 Participation by parastatals is permitted providing they (i) are financially autonomous, (ii) operate under commercial law, and (iii) are independent from borrower and its purchasing/contracting authority. 7 Except where the bid prices exceed the available budget. The practices or rejection of bids outside a range or “bracket” of bid values is also known as "price bracketing." 8 Two envelope procedure is also permitted where there are adequate safeguards against retaining second envelope unopened are incorporated in the two envelope procedures and effective bid protest mechanisms are already in place for the due processing of bid complaints. (All technical envelopes are opened first and, after review, price envelopes of all or only qualified/responsive bids are opened in the second round.) 9 Recommended for works contracts of one year or more in duration when domestic inflation rate is high. 10 In some cases the Bank has specified there shall be no fee; in other countries fees have been permitted. 11 It may sometimes also be a certified check. In Macedonia, bid security was to "follow the generally acceptable practice used in the local market." 12 A method of delivery may be specified in the case where bidders have to submit physical sample, however. 13 Provided that (1) all responsible bidders are allowed to bid, (2) the process is efficient and (3) no serious delays result. 13 Stage II Equivalence Analysis Table A.9. Stage II Equivalence Analysis—Average Scores Average score World Bank Guidelines Para. Policy Requirements Piloting Program Requirements (out of 3) Eligibility (paragraphs 1.6-1.8) Conditions for participation to be limited Those provisions of the Guidelines 1.7 to those that are essential to fulfil the would be made applicable through contract requirements. the Legal Agreement Exceptions indicated in para 1.8. (LA). Two-stage Bidding (paragraph Applies to large and complex, or Bank review of complex, high value, 2.4 2.6) turnkey, contracts that potentially attract and non-standardized bidding the participation of foreign bidders. processes as part of its assessment Turnkey contract of the country systems. (paragraph 2.22) Procedure may follow two stages: 1) non-priced technical proposals; 2) Possibility to use Bank SBDs or issuance of BDs to bidders, and harmonized MPDs for specific bids / submission of technical priced contracts to be identified in the proposals. Procurement Plan for the pilot pro- ject. Modifications as may be needed by the country institutional and legal framework to allow for the country policies and procedures to apply. Notification and Advertisement General Procurement Notice and The executing agency of each pilot 2.4 (paragraphs 2.7-2.8) Specific Procurement Notices for ICB project will be required to publish in contracts to be published on UN UN Development Business, and on Development Business online and Dg an electronic free access portal, all market. contracts identified as likely to attract foreign competition. Such contracts to be flagged in the Procurement Plan. Publication of all mandatory documents at the pilot project level on a fully functional and freely accessible website of the executing agency. Examination, evaluation and Bids to be evaluated and awarded on To be part of the bidding documents 2.5 comparison of bids; Post- the basis of the criteria stated in the accepted for use under pilot projects qualification (paragraphs 2.48- bidding documents and quantified using when international competition is 2.54 and 2.58) cost to the maximum extent. Post- anticipated (see checklist below). qualification applies when there is no pre-qualification. Alternative Conditions of Contract documents must clearly define General Conditions of Contracts 2.6 Contract (paragraphs 2.37 and the procedure to submit alternative bids, (GCC) for contracts covering goods, 2.38) and the scope of work, rights and works and services to be consistent obligations of the parties, and provide with international requirements when for fair and clear allocation of risk. international competition is sought. To be part of the bidding documents accepted for use under pilot projects when international competition is anticipated (see checklist below). Domestic preference for goods Defines the Bank's procedures for use Use of discriminatory domestic 2.6 (paragraphs 2.55-2.56) of domestic preference. preferences (e.g. by nationality) will not be allowed in the bidding documents accepted for use under pilot projects when international APPENDIX A BANK GUIDELINES AND PROCESSES competition is anticipated (see checklist below). Bid validity, Award of Contract Award to the lowest evaluated bidder Assessment of legal framework to 2.6 which meets the requirements of the ensure that the national law covers bidding documents including technical bid validity, contract award, and capacity and financial resources. rejection of bids requirements, and evaluation of compliance at the pilot project executing agency level, and monitoring during pilot project implementation. Bidding Documents Bidding documents must provide Existence of national sample bidding 2.7 (paragraphs 2.11-2.12, and information sufficient for a prospective documents not required, but 2.16-2.18) bidder to prepare a bid. Use of Bank mandatory use of standard bidding standard bidding documents is documents acceptable to the Bank mandatory for ICB - this requirement will when international competition is have to be modified for the pilots. sought under pilot projects. Specific provisions to be validated vis-à-vis the checklist below, and to be stated in the LA. Publication (paragraphs 2.57, Publication of awards is required within The executing agency for each pilot 2.7 2.59-2.60, and 2.61-2.64) two weeks. Rejection of all bids to be project would be required to have (a) justified. a functioning electronic system for retaining information on its procurement processes and managing invoices and certificates payments; and (b) a fully functional website that is freely accessible to all stakeholders, including bidders and civil society, for posting information regarding its organization, procurement regulations, SBDs, procurement opportunities, information related to the award of contracts, and procurement plans. Pricing (paragraphs 2.21 and Bid must be invited on the basis of Mandatory use of INCOTERMS for 2.7 2.23) specified INCOTERMS comparison purposes, or fully equivalent provisions. Prequalification(paragraphs Use of prequalification for large complex The Procurement Plan will be used to 2.8 2.9-2.10) requirements limited to capability and identify bidding processes that would resources to perform the particular benefit from prequalification. contract with regard to experience, past Advertisement as per requirements performance, personnel, facilities and above. Prequalification requirements financial position. to be assessed by comparison with the harmonized prequalification MPD. Joint ventures(paragraph 1.10) Mandatory association between firms is Those provisions of the Guidelines 2.8 not acceptable. would be made applicable through the LA. Validity of bids and bid security Bidding documents must state bid Review of clauses in the bidding 2.8 (paragraphs 2.13-2.14) validity period. Bid security not documents accepted for use under mandatory pilot projects when international competition is anticipated. Language (paragraph 2.15) English, French, or Spanish, in addition National language can be used 2.8 to the national language of the country. provided that it is used internationally and that a translation in English, French or Spanish is made available to foreign bidders for contracts that have been identified for international bidding in the Procurement Plan. Advertisement and bidding 15 APPENDIX A BANK GUIDELINES AND PROCESSES documents should be issued in an international language (English, French or Spanish) in addition to the national language, and contracts with foreign entities should be signed in the international language the bidders used for their bids. Price Adjustment (paragraphs Bidding documents must specify if price Requirement for price adjustment as 2.8 2.24-2.25) adjustment applies and price adjustment a contractual term to be included in must be based on a prescribed formula. the bidding documents accepted for use under pilot projects when international competition is anticipated. Currency and Payments Bids documents must state currency National currency acceptable 2.8 (paragraphs 2.28-2.36) provisions and the conversion provided it is freely and fully mechanism. Payment provisions must convertible and country does not also be stated in the bidding documents. apply restriction or control on foreign exchange. Contract terms to allow payment in foreign currency directly and/or as percentage of contract price at predefined exchange rates (in no more than 3 currencies) expressed in the bidding documents accepted for use under pilot projects when international competition is anticipated. Performance Security, Format and provisions to be included in (to be assessed as part of the 2.8 Liquidated Damages Force the bidding documents. assessment of the country legal and Majeure (paragraphs institutional framework). 2.39/2.40, 2.41 and 2.42) Applicable law and settlement The conditions of contract shall provide National law to apply. 2.8 of disputes (paragraph 2.43) for settlement of disputes and state the applicable law. Bid opening procedures Bids to be opened in public at the time Assessment of legal framework to 2.8 (paragraph 2.45) and place designated in the bidding ensure that the national law covers documents. Rejection of late bids. this requirement, and evaluation of compliance at the pilot project executing agency level, and monitoring during pilot project implementation. Rejection of bids 2.8 Time for bid preparation Time allowed for the preparation and Requirement to be stipulated for each 2.9 (paragraph 2.44) submission of bids shall be determined executing agency implementing pilot on the basis of the magnitude and projects. To be closely monitored as complexity of the requirement. one key compliance indicator (see Stage III). Confidentiality (paragraph After bid opening, evaluation and National law to apply. 2.9 2.47) recommendations and other information shall not be disclosed until after publication of the award. Source: IEG analysis of UCS pilot data. Note: On a scale of 0–3. 16 Applying the Concept of Value for Money EXAMPLES FROM SIX PUBLIC PROCUREMENT JURISDICTIONS The United Kingdom, the European Union, and Australia have taken a long-term and systematic approach to integrate value for money concepts into their public procurement practices. 1 Additionally, the Asia Pacific Economic Cooperation Forum (APEC) has endorsed value for money concepts since 1999. The International Atomic Energy Agency is one United Nations agency that has adopted value for money principles. The United Kingdom Guidance toward the approach taken by the United Kingdom has been provided for the most part by the U.K. Office of Government Commerce and the National Audit Office. 2 Their documents emphasize the following: Focusing on cost reduction, notably both the transaction overhead costs of conducting the procurement, and the cost of the goods, works, or services acquired. Getting better prices suggests negotiating better deals (price/quality), and when possible aggregating demand to capture scale economies. Improving management of the project, the contract, and the acquired assets. Balancing competition with risk. Competition is encouraged to obtain value, but risks need to be taken into account and where competition is weak benchmarking and theoretical cost models are suggested to verify that the purchasing agent is obtaining favorable offers. Taking the whole length of the contract into account (including decommissioning). Accounting for wider societal and environmental costs of the contract. Developing bid evaluation criteria that are relevant to the project—published by the purchaser and widely available. Recommending the use, by purchasing agents, of electronic commerce technology and new procurement practices. The U.K. Department for International Development (DfID) has also adopted a value for money approach (DfID 2011). It concurs with much of the above, emphasizing the importance of having a sound procurement strategy, investing up front in well-defined bid criteria and good terms of reference, and negotiating favorable contract terms. It adds that purchasing agents should develop relationships with suppliers. This last point is a feature of discussions in the United Kingdom over the need for long-term collaboration with suppliers, to save costs and 17 ensure continuity of supply and consistency of quality (albeit sometimes at the expense of small and medium-sized enterprises (SMEs) and of competition more generally). The United States—“Best Value” Procurement Traditionally, best value has been used in the United States to describe the trade-off process, and since 1997 the term has come to encompass a variety of techniques that can be used in conducting competitive procurement. 3 Today, the Federal Acquisition Regulation describes the best value continuum as any one or a combination of selection approaches that are used to obtain best value or value for money. There are two types of procurement techniques for obtaining best value: The tradeoff process is a competitive negotiation process where the purchaser conducts a tradeoff analysis of both price and non-price factors, and awards the contract to the bidder proposing the combination of factors that offers the best overall value. The contract may be award to other than the lowest priced, or the highest technically rated bid. The lowest price technically acceptable process describes a distinct variation on the competitive negotiation process, where the non-price factors are stated criteria of bid responsiveness. The contract is awarded to the bidder offering the technically acceptable proposal with the lowest price, with “technically acceptable” defined as complying with the technical criteria, including any non-price factors. The lowest price technically acceptable process is more closely aligned with standard bidding techniques, where value criteria are stated in advance, as is typical of Bank procedures. Best value is defined when planning the procurement—the non-price factors are evaluated for conformance with the technical criteria expressed in the solicitation, typically without gradations of scores for higher levels of achievement. In contrast, under the tradeoff process, best value is decided during evaluation, through the trade-off analysis of the different bids. Overall, best value procedures differ fundamentally from traditional sealed bidding procedures in which bidders offer the same or similar service or product in response to detailed solicitations, and bids differ only on price. In best value procurement, bids differ in elements beyond price. The purchaser may issue an award to a higher priced but technically superior offer or to a lower priced but less technically qualified proposal, depending on its view of which proposal presents the best overall value. Best value procurement has its critics, whose primary concern is the perception of fairness and the role of a procurement system in encouraging inclusiveness and transparency in government contracting. Concerns center on three issues: 18 The extent and consequence of discretion in ascribing and evaluating best value criteria, and in communicating with competitors in the conduct of negotiated procurement. Past performance is arguably the most significant noncost/nontechnical criteria in American best value procurement. Concerns are that it prejudices unjustly firms that do not have a past history of government contracting and can ultimately become little more than a tool for incumbency. There is a general concern that non-stated evaluative criteria appear to be subjective, and may preclude open and fair competition. The Federal Acquisition Regulation note that the procuring agency can obtain best value by various means, but explicitly provide only a few alternate considerations to price. If a requirement is not well defined and there is a performance risk to consider, the agency may properly rely more heavily on technical or past performance considerations. However, there are open questions as to whether the procuring agency might adopt evaluative factors for administrative ease. The flexibilities and innovations of best value procurement continue to attract advocates and detractors alike. Debate continues over the loosening of regulations regarding communication, the incorporation of less-traditional considerations, and the devolution of discretion entailed. The choice to place best value criteria above traditional price/responsiveness metrics requires a developed and supporting legal framework related to ethics, fraud and corruption, influence peddling, and similar issues. Where problems arise, mechanisms must be in place to address them. The European Union Value for money principles are incorporated in the European Union’s Most Economically Advantageous Tender (MEAT) system, which enables the contracting authority to use criteria that reflect qualitative, technical and sustainable aspects of a tender, in addition to price, when reaching an award decision. 4 Application of MEAT is a choice of the procuring agency—according to the directive, public contracts in the European Union must be awarded either solely on the basis of the lowest price, or under MEAT criteria. 5 But where MEAT is used, it may involve the use of various criteria such as quality, technical merit, aesthetic and functional characteristics, environmental characteristics, running costs, cost- effectiveness, after-sales service and technical assistance, delivery date and delivery period, in addition to price. Additional features of MEAT include: The contract notice must indicate whether MEAT criteria will be used to award the contract and what the criteria are. 19 The bid assessment and award criteria should not confer unrestricted freedom of choice on the contracting authority and should be made public. Bidding and contract award criteria should ensure effective competition. Whereas MEAT procedures offer procuring agents considerable latitude to define requirements, the directive maintains principles of open and competitive tendering, and fairness through clear specifications and objective assessment. The directive states: ….In order to guarantee equal treatment [of bidders], the criteria for the award of the contract should enable tenders to be compared and assessed objectively. If these conditions are fulfilled, economic and qualitative criteria for the award of the contract, such as meeting environmental requirements, may enable the contracting authority to meet the needs of the public concerned, as expressed in the specifications of the contract. Under the same conditions, a contracting authority may use criteria aiming to meet social requirements, in response in particular to the needs—defined in the specifications of the contract—of particularly disadvantaged groups of people to which those receiving/using the works, supplies or services which are the object of the contract belong. As of 2012, thresholds for application of the European Union directive for central government and subcentral contracting authorities were €5,000,000 for Works (including concession contracts); €130,000 or €200,000 for services, depending on the type of service; and €130,000 (€200,000, for certain defense contracts) for supply. Australia Australia has made value for money the core principle underpinning Australian government procurement. 6 Its procurement guidelines elaborate value for money concepts as: • Life-cycle analysis of all relevant (direct and indirect) costs and benefits on a fair and common basis. • Nondiscriminatory and competitive procurement processes based on bidding requirements and evaluation criteria that are logical, clear, comprehensive, and relevant. • Efficient, effective, and ethical use of resources. • Accountable and transparent decision making. The guidelines clearly state that cost is not the only determining factor in assessing value for money. In addition to life-cycle analysis, value for money assessment could include fitness for purpose, supplier performance history, and relative risk of competing proposals. 20 The Australian guidelines specifically address business development of SMEs through public procurement, saying that that government is committed to sourcing at least 10 percent of procurement to Australian and New Zealand firms with fewer than 200 employees. This is accomplished, first, by ensuring fair treatment (nondiscrimination) of SMEs, and second, by considering in the procurement process the value for money benefits of doing business with SMEs. This falls short of requiring an explicit preference for SMEs, but implies that requirements may be tailored to fit SME capabilities, and that evaluations may consider benefits of awarding contracts to SMEs. Asia-Pacific Economic Cooperation Forum In 1999, APEC’s Government Procurement Experts Group identified the principles of public procurement as: “transparency, value for money, open and effective competition, fair dealing, accountability and due process, and non-discrimination.” 7 Value for money is integral to the APEC procurement principles, which say: …The test of the best available value for money is a comparison of relevant benefits and costs on a whole of life basis…The lowest priced compliant offer does not necessarily represent best value for money. Other aspects of APEC’s value for money approach include: • The procurement function can deliver value for money, in taxpayer terms, through improvements in procurement processes and management. • Objective and functional requirements, which focus on what is to be achieved rather than how it is to be done, can encourage innovative solutions that may improve the value for money outcome. • Costs and benefits are to be assessed on a life-cycle basis, possibly including sensitivity analysis and calculations of discounted cash flows. • Besides price and fitness-for-purpose, other factors that may be taken into account include performance, quality, reliability, delivery, inventory costs, running costs, warranties and after-sale support, and disposal. • Competition is encouraged. Procurement methods should be chosen based on the best value for money outcome, especially in situations of limited competition. • Purchasing agents should be knowledgeable about the market in which they operate, assessing risks and avoiding unnecessary costs. • It is important to establish competence, viability, and capability of prospective suppliers. 21 • Where allowed, negotiation should be used to improve value for money. With respect to value for money, APEC covers many of the same areas as the United Kingdom and European Union. APEC does not make specific reference to environmental and social factors, but these are inferred as justifiable considerations. Its principles address ideas of negotiation, and alternative methods to address non- competitive markets, but are vague on details and on how the principles should be operationalized. The International Atomic Energy Agency The International Atomic Energy Agency (IAEA), a United Nations agency, offers a different perspective from the above organizations. While the approaches described above, like those of the Bank, are meant to provide guidance to others, IAEA principles and procedures are developed to guide the Agency in its own procurement activities. Accordingly, it is able to prescribe a system that fits with its internal technical and management capacity. Value for money is central to IAEA principles and processes, as stated in its regulations (IAEA 2011). The introduction of these concepts is the result of a concerted effort to change the Agency’s procurement from a legalistic or rules based approach, to an outcomes or strategic management approach (Tonkin 2012). IAEA emphasizes several areas where value for money can be achieved, through: • Integration of procurement strategies and planning with program delivery. • Identification of risks (market risks, or continuity of supply risks, for example) and development of mitigation strategies within the procurement strategy and plans. • Exploiting innovations that improve program delivery, through strengthened agency expertise, and by tapping supplier expertise. • Optimizing cost over the life of the acquisition, trading off quality and cost factors where appropriate. • Managing implementation to ensure that what was contracted is delivered, and to exploit opportunities to enhance value during implementation, especially in longer-term service contracts (by taking advantage of new technologies, for example). An interesting feature of IAEA’s practices is to separate high value, complex procurement from low-value straightforward purchasing. The former is assigned to procurement specialist teams, whereas the latter is considered low risk and is assigned to contracting officers. This lowers the administrative overhead cost of 22 managing the procurement process. The IAEA also provides more latitude to procuring agents than other organizations. Although, like others, it sees open and competitive tendering as a value for money principle, it provides a range of selection methods other than lowest price (or lowest life cycle cost). These include best technical proposal (similar to the Bank’s fixed budget selection method for consulting services), and competitive negotiation options (similar to UNCITRAL procedure). BROADER CONCEPTS OF VALUE FOR MONEY Non-Price Factors and Preferences The Australian provisions to use public procurement to strategically develop its SME sector are one example. The Independent Evaluation Group (IEG) found support for preferences in its 11-country survey, and notes that it is a common feature of many national procurement systems. Apart from the Bank’s domestic preference policy, seen by IEG as ineffective, the Bank does not allow preferences as it contradicts the principles of fairness and competition. To the extent to which the Bank wishes to retain the principle and practice of giving domestic preferences, discussed earlier in the present chapter, IEG recommends that the Bank explore other mechanisms to achieve this, in view of its present limited framework. Investing in procurement planning and developing clear and appropriate evaluation criteria The importance of strong front-end work is emphasized by the United Kingdom and IAEA. The objectives appear to be twofold: (i) to assure transparency, and (ii) to best align the stated requirements of the procurement with the true needs of the project (fit for purpose). The Bank Guidelines mandate procurement planning, seen to be an integral part of project design. IEG notes that although the Bank has traditionally supported up-front procurement planning, such support has been found to be below optimal, especially in complex projects. Management proposes to make stronger use of procurement planning as a tool in the future. 8 Reducing Cost through Competition and Negotiation Competition is generally accepted, within value for money principles, as an effective way to lower cost of acquisition. However, value for money also envisages instances of poor competition, and suggests that good negotiation can contribute to better outcomes in terms of cost and quality. 9 Bank policy and practice is very much oriented toward competition. Negotiation to improve outcomes, post contract award, is limited to acquiring professional services, following consultant recruitment procedures. Yet it is unlikely that the Bank could recommend 23 negotiation in all but restricted circumstances, as it would run counter to other principles of transparency and fairness. The Bank offers only limited solutions to situations of poor competition and other aspects of addressing market risk. Direct negotiation is an option, but it is an exception and little guidance is offered on how best to conduct the process or to determine if good value is achieved. 10 Limited competition, such as is found in information and communications technology (ICT) procurement, or sole sourcing (to incumbent public-private partnership [PPP] concessions, for example) are also not well accommodated in Bank practice. It is suggested that the Bank explore greater use of less-competitive methods, such as competitive negotiation as provided for by UNCITRAL and European Union, and as reported here is practiced in IAEA. 11 Strengthening Relationships with Suppliers Building “appropriate” relationships with suppliers is a feature of the U.K. value for money principles, although admitting that it limits competition and could be unfair to some classes of firms. In a similar vein, IAEA principles recommend that purchasing agents understand supplier capabilities and practices, and use prequalification and staged bidding to better assess bidder capability. Bank policies and practice encourage prequalification and multistage bidding, and would not discourage procuring agents from knowing their markets. However, Bank practice suggests assurance of an “arms-length” relationship between purchaser and bidder. Many Bank procedures are written to ensure limited interaction (detailed procedures of communications during bidding, for example), partly to avoid the possibility of collusion and influence. Maintaining its fraud and corruption mitigation agenda could be difficult if the Bank were to go in this direction. There is a concern that the traditional focus of public procurement on regulations and procedures discourages the exploitation of value-adding procurement methods thereby reducing the potential for innovation. Proposed revisions to the European Union Procurement Directive recommends introduction of the “innovative partnership” procedure (see the next session). This new method of procurement would allow procuring agents to enter into partnership arrangements with suppliers to develop, and subsequently supply new and innovative products, works and services according to specified requirements that are performance oriented. The developmental benefits of such supplier relationships are obvious, and mechanisms to support such arrangements within the Bank’s principles of fair and open competition, are recommended. 24 Reducing Transaction Costs The United Kingdom and IAEA both note that reducing the administrative overhead cost of procurement offers value for money benefits. IAEA has established lower cost processes for low-value, low-risk procurement. The Bank’s shopping procedure, might be seen as a similar approach. Besides, as noted in previous chapters, Bank practices in terms of more “proportionality” of resource use, relative to the objective of procurement, could be further improved through a series of measures; improved guidelines and processes for complex procurement (Chapter 2); more tolerance of risk, and the use of risk-based rather than value thresholds, frugal use of higher level clearance (Chapter 3); and more attention to elapsed time in contract processing (Chapter 4). All these measures would strengthen the application of value for money principles. Supporting Contract Implementation IAEA and the United Kingdom cite attention to managing contracts, as an important value for money factor—ensuring that what was contracted is delivered, is as important as the selection process. Furthermore, IAEA notes opportunities to adjust contracts during implementation, to reap gains of new technology for example, especially important for long-term service contracts. Regarding the capturing of value for money benefits through adjustment to contracts during implementation, this is to some extent covered in the Bank’s works contracts (closely modeled on the International Federation of Consulting Engineers), which give authority to the engineer to improve the implementation of works through variations. IEG has also noted the need to strengthen long-term service contracts, as these especially would almost certainly require ongoing renegotiation to obtain optimum outputs. IEG discusses the broader issue of contract implementation earlier in this chapter, and has pointed towards client requests for support. Greater attention in Bank practices to oversee and assist contract implementation is needed, though balance will have to be achieved between the role of different players within the Bank (especially task team leaders and procurement staff), as well as outside the Bank, for example, through national audit offices, country based citizens’ groups and the like. To summarize, the principle of value for money in procurement is being increasingly incorporated into the procurement approach of a number of public jurisdictions, with varying degrees of operational instruction and varying degrees of emphasis in terms of the aspects of value for money that they each seek to emphasize. Many aspects of these principles are already implicit in Bank procurement, and some are frequently used. Stronger direction could be given to staff and borrowers to fortify the adoption of such practices. A number of the 25 efficiency increasing (and hence transaction cost reducing) measures proposed in the present report would expand value for money, as would better management of risk, and the embracing of new procurement practices in frontier areas of Bank business. In such areas, more attention to specific procurement modalities would be required to achieve value for money. However, areas remain, where for reasons of open access, or transparency, the Bank may prefer to maintain its present systems and practices. Bank requirements do not accommodate the inclusion of social objectives through procurement (using procurement to create employment, perhaps targeting youth or minorities, for example). The difficulty for the Bank is to balance objectives of domestic laws and policies that target social and domestic industrial strategies, with its obligations to ensure fair treatment of international bidders. 26 Proposal to Update the European Union Public Procurement Directive A proposal to update the current European Union procurement directive (2004/18/EC) was issued on 20 December 2011; the “Proposal for a Directive of the European Parliament and of the Council on Public Procurement.” 12 The extent to which the proposal will be implemented, or the timing is not known, although there is some expectation that reforms maybe introduced later this year. 13 CAUTIOUS APPROACH The proposal is cautious in its ambitions. It says that it endeavors to retain continuity with the current directive, and not to depart from concepts that have been developed over the years, have evolved through case law, and are well known to practitioners. This is in line with the Bank approach, which will retain the Guidelines and keep much of the existing system intact. The proposal notes obligations of member states to comply with World Trade Organization (WTO) government procurement agreement requirements, although done so with respect to WTO’s proposed lighter regime for contracting authorities below the central government level. Thus it offers simplified procedures, yet invokes alignment with international best practice. This is something for the Bank to consider in light of its move towards country systems—use of country systems can simplify things for borrowers, but continued compliance with international norms is important. TOOL BOX A “toolbox” approach is proposed, but in fact does not change much. The options of open and restrictive tendering are retained, as are the negotiation procedures (competitive negotiation and competitive dialog—although with simplified procedures and broadened application beyond “complex” procurement). The one new procedure that is introduced is the “innovative partnership” procedure (e- procurement procedures are also emphasized as noted below). This allows procuring agents to enter into partnership arrangements with suppliers to develop, and subsequently supply, new and innovative products, works, and services according to specified requirements that are performance oriented. The toolbox approach is similar to the direction the Bank is suggesting – to transform guidelines and standard bidding documents from mandatory, to optional guidance documents, for example. The Bank does not use the negotiation procedures of the European Union (and others), although IEG recommends this. If adopted, the new “innovative partnership” is perhaps something worth exploring 27 by the Bank, especially for borrowers who would follow European Union directives. The objective is to use public procurement strategically to improve innovation (and competitiveness). If properly managed, for example by using open and competitive bidding, it offers developmental benefits and could be done without violating Bank principles. EMPHASIS ON E-PROCUREMENT Several measures are proposed to promote and mandate e-procurement: • Mandatory electronic notices • Mandatory electronic documents • Switch to electronic communications, including electronic bid submissions, within two years • Acceptance of dynamic purchasing systems, electronic auctions, electronic catalogues. This goes beyond the Bank’s current position on e-procurement, or what the Bank proposes in terms of promoting greater use. The Bank allows all of these features, but does not mandate them--given the Bank’s diverse client base, it is unlikely that it could (with some exceptions, such as the current mandatory publication of notices on the United Nations Development Business website, for example). MODERNIZATION/SIMPLIFICATION In addition to e-procurement, further suggestions to modernize, streamline, simplify, include: • Shortened time limits for bidding • Exclusion of firms from bidding for nonperformance • Provisions to modify contracts during implementation. These are relatively minor items, but the Bank might consider similar measures. The Bank’s mandatory bidding periods were criticized by clients in IEG’s country case studies, especially for international competitive bidding (ICB). With modern communications and more of the bidding process going on-line, shortened periods could be considered where it can be demonstrated not to compromise competition. Modification of contracts, especially long-term service contracts, has been noted earlier in this chapter. 28 VALUE FOR MONEY As reported earlier in this appendix, value for money is a feature of the existing European Union directive. It is proposed to expand this in several respects: • More explicit instruction to apply lif- cycle costing, noting that external costs are to be considered (such as environmental costs) where they can be monetized and verified. A common European Union methodology for life-cycle costing is to be used, when developed. • Factors directly linked to production processes may be included in technical specifications and technical award criteria. • Labels may be used in technical specification related to environmental factors (eco label) and other social factors (child labor, for example). • Firms may be sanctioned for violating social, labor and environmental laws. Value for money is a key element of the Bank’s proposed procurement reform, and as commented, the Bank’s system already incorporates many of the measures noted above (life-cycle costing, environmental specifications, and so forth). But the Bank should also consider developing methodologies around these approaches. SUPPORT TO SMES To address its objective of supporting local industry, the Bank could also consider the European Union suggestions to support SMEs. Several suggestions are made to promote SMEs: removing barriers to SMEs (reducing turn-over requirements, for example), simplifying bidding requirements (for example, allowing self-declarations as prima-facie evidence), “packaging” procurement to make packages more attractive to SMEs, and direct payment to subcontractors. KNOWLEDGE CENTERS The European Union proposal observes that there is a lack of capacity in many contracting authorities, and that this limits effective procurement. The proposal obliges member states to provide support structures offering legal and economic advice, guidance, training and assistance in preparing and conducting procurement procedures. The Bank has similarly identified borrower capacity building as important. It might similarly consider the establishment of a knowledge center that borrowers can tap into on an as needs basis. 29 Appendix B. CDD Procurement Summary of Previous Reports Commensurate with the increasing importance of community-driven development (CDD) projects in the Bank’s portfolio, procurement under CDD projects has been the subject of many reviews by the Networks and by IEG, and they yield mixed evidence. An early review, with positive findings, was undertaken by the central Social Protection Team unit. 14 It found that contracting by community groups through local shopping was “more efficient and transparent” and got lower prices than central government bidding. The review also noted several “good practice” procedures, including project designs that reflect skills and needs of communities, use of very simple contracting procedures, building-in incentives for efficient procurement, provision of active support to communities on prices and quality to help them in negotiating with suppliers, and ensuring transparency of procurement through proactive information sharing and disclosures. 15 A later field-based review by the South Asia Region of 84 CDD subprojects in India also concluded, positively, that better value for money was achieved in works implemented by communities without involving contractors, with savings in costs and time in most of the analyzed cases. 16 More interestingly from the procurement perspective, the review found that “compared to the conventional procurement cycle, the subprojects rated very highly in terms of procurement planning and scheduling, developing specifications, awarding decisions, and disclosure and contract management while scoring poorly in terms of market search for bidders, tendering, tender opening and evaluation.” The review concluded that success could be ascribed to the practice of “relationship-based procurement” wherein the communities established a mutually beneficial and accountable commercial relationship between the purchaser and the supplier. The report concluded that there was scope for further adaptation of the CDD procurement processes with focus on delivery of outcomes and results rather than on oversight and control of contracting for inputs. Yet other reviews by IEG, the Internal Audit Vice Presidency (IAD), and the Integrity Vice Presidency (INT) have flagged persistent concerns about the adequacy of fiduciary oversight of CDD projects. A comprehensive review of the CDD portfolio by IEG in 2005 noted that almost a quarter of surveyed staff expressed concerns about their ability to adequately monitor compliance with agreed fiduciary 31 APPENDIX B CDD PROCUREMENT processes in CDD projects (IEG 2005). In a similar vein, a 2008 IAD audit reported that the Bank did not conduct required procurement post-reviews for five out of 11 projects that required such reviews--impeding the Bank’s ability to detect contracts that did not comply with the agreed procurement procedures. And an INT investigation of two CDD projects in Kenya in 2011 was significantly critical in its findings: it pointed towards large scale misappropriations and leakages caused by failure to comply with agreed procedures during implementation (Al-Azar 2011; World Bank 2011a). Responding to INT’s concerns, Operations Policy and Country Services (OPCS) undertook a major review in 2012 of governance issues related to CDD projects (World Bank 2011a). Its review concluded that a well-designed CDD project can strengthen governance at the local level, since it incorporates the three key principles of good governance: transparency, accountability and participation. However, complementary and interdependent pieces have to come together to make CDD projects work as intended. Fraud and corruption in the procurement process (during contract implementation), for example, short-changing of project quality because of collusive practices, were seen as important risks. The review recommended special attention to risk assessment, especially the risks of “political capture” during project design. It also argued for field-based, intensive, collaborative supervision by the Bank, front-loaded during the first two years. The review concluded that with adequate care in design and supervision, CDD projects can be successful even in cases where governance is poor and corruption is rampant. 32 Portfolio Analysis CDD broadly defined is an approach that gives control over planning decisions and investment resources to community groups and local governments. CDD programs operate on the principles of local empowerment, participatory governance, demand- responsiveness, administrative autonomy, greater downward accountability, and enhanced local capacity. The CDD Anchor in the Social Development Department has been tracking the amount of Bank (IBRD/IDA) lending going toward CDD approaches since 2000 using the following CDD typology developed by a cross- regional/cross-sectoral group of CDD practitioners. This is the universe of projects for the evaluation period. This review uses the Social Development group definition of CDD and used their database of CDD projects. A few CDDs (3 percent of total) are embedded in development policy operations. Since the review will assess the functioning of World Bank procurement systems in its operations, we will focus on Investment Lending, hence on CDDs delivered in that form. The universe of CDDs is then 692 projects. It should be noted that projects that have some CDD components are also included in the CDD database, per the Social Development group definition and classification. The social development database includes 732 projects classified as CDDs, 711 (97 percent) are IBRD/IDA financed and the rest are financed by Global Environment Facility and other trust funds that do not consistently record implementation status report (ISR) ratings. Only IBRD/IDA financed operations are included in this analysis. As a first attempt to assess the effects of World Bank procurement systems on the development effectiveness of CDD projects, we will use two sets of data stemming from ISRs and from IEG project evaluations (Implementation Completion Report Reviews—ICRRs). From ISRs we will extract procurement ratings and outcome ratings. For simplicity we will use ratings from the last ISR, although ratings from all ISRs are also available and could be used to track changes in projects' procurement performance. It should be noted that ISRs are only done for main projects and not for subsequent additional financing projects. For this reason, the number of CDD projects with ISR ratings is smaller, as seen in Table B.1. In tables where ISR ratings are shown, CDD total will be 549 projects because of the existence of a few unrated projects. ICRRs do not have procurement ratings, but outcome ratings and risk to development outcome ratings can be extracted, either to confirm ISR outcome assessment and its sustainability. Only a subset of the CDDs reviews are closed (47 percent, see Table B.2) and even fewer have been reviewed by IEG as of today (21 33 APPENDIX B CDD PROCUREMENT percent). For this reason the tables with IEG ratings will show a CDD total of 143 projects. RDO totals are lower because this rating was introduced in the middle of our evaluation period (around FY08). Table B.1. World Bank Project Approvals (all projects—CDD and non-CDD), FY02-11 IBRD/IDA Trust funds Total IL Dev Pol IBRD/IDA and others Main Additional Total Operation total project financing CDD 552 140 692 19 711 21 732 Non-CDD 1,492 313 1,805 563 2,368 1,256 3,624 Total 2,044 453 2,497 582 3,079 1,277 4,356 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Table B.2. World Bank Project Approvals (investment lending projects), FY02-11 CDD Non- Total Main Project Additional Total CDD CDD financing Active 265 58 323 1052 1,375 Closed 287 82 369 753 1,122 Closed o/w have IEG 153 - 153 348 501 Evaluation Total 552 140 692 1,805 2,497 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Includes investment lending projects financed by IBRD/IDA and excludes those financed by Trust Funds and others. 34 APPENDIX B CDD PROCUREMENT Table B.3. IEG Outcome Ratings and Risk to Development Outcome Ratings for Investment Lending Projects, by Fiscal Year Outcome ratings Risk to development outcome ratings Approval FY MS+ S+ Total % Sat % Sat Moderate or Total % Moderate or (MS+) (S+) Lower lower CDD FY02 38 21 50 76 42 26 44 59 FY03 32 7 48 67 15 20 47 43 FY04 21 14 33 64 42 12 33 36 FY05 9 2 15 60 13 11 15 73 FY06 3 2 4 75 50 3 4 75 FY07 1 1 3 33 33 2 3 67 FY08-11 Total CDD 104 47 153 68 31 74 146 51 Non-CDD FY02 72 38 97 74 39 46 79 58 FY03 65 30 83 78 36 56 81 69 FY04 49 19 69 71 28 48 67 72 FY05 36 15 55 65 27 32 55 58 FY06 19 8 26 73 31 14 26 54 FY07 8 2 12 67 17 5 12 42 FY08 2 2 4 50 50 3 4 75 FY09 1 2 50 0 1 2 50 FY10-11 Total Non- 252 114 348 72 33 205 326 63 CDD CDD and Non-CDD FY02 110 59 147 75 40 72 123 59 FY03 97 37 131 74 28 76 128 59 FY04 70 33 102 69 32 60 100 60 FY05 45 17 70 64 24 43 70 61 FY06 22 10 30 73 33 17 30 57 FY07 9 3 15 60 20 7 15 47 FY08 2 2 4 50 50 3 4 75 FY09 1 0 2 50 0 1 2 50 FY10-11 Total 356 161 501 71 32 279 472 59 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Totals refer only to those projects with available or valid ratings. 35 APPENDIX B CDD PROCUREMENT Table B.4. ISR Procurement Ratings for Investment Lending Projects, by Fiscal Year Approval FY HS S MS MU U HU Total % Sat % Sat (MS+) (S+) CDD FY02 48 12 3 3 66 91 73 FY03 44 19 5 4 72 88 61 FY04 1 45 18 4 1 69 93 67 FY05 29 27 6 1 63 89 46 FY06 33 22 4 59 93 56 FY07 1 22 22 7 52 87 44 FY08 20 27 4 51 92 39 FY09 1 21 17 9 2 50 78 44 FY10 12 20 5 37 86 32 FY11 16 12 2 30 93 53 Total 3 290 196 49 11 0 549 89 53 Non-CDD FY02 1 80 35 4 4 1 125 93 65 FY03 3 75 34 8 3 1 124 90 63 FY04 2 88 31 16 4 141 86 64 FY05 1 80 67 6 4 1 159 93 51 FY06 1 93 62 10 2 168 93 56 FY07 81 49 10 5 1 146 89 55 FY08 74 63 16 4 157 87 47 FY09 61 61 8 3 133 92 46 FY10 87 69 11 2 169 92 51 FY11 2 117 53 13 3 188 91 63 Total 10 836 524 102 34 4 1510 91 56 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Totals refer only to those projects with available or valid ratings. Latest ISR ratings were used. Additional financing projects don't have ISR ratings. HS = highly satisfactory; S = satisfactory; MS = moderately satisfactory; MU = moderately unsatisfactory; U = unsatisfactory; HU = highly unsatisfactory. 36 APPENDIX B CDD PROCUREMENT Table B.5. ISR Procurement, ISR Outcome Ratings, and IEG Outcome Ratings ISR Procurement IEG outcome ratings ISR outcome ratings Ratings MS+ MU- Total MS+ MU- Total CDD MS+ 447 36 483 100 36 136 MU- 33 27 60 4 13 17 Total 480 63 543 104 49 153 % of Total MS+ 93% 7% 100% 74% 26% 100% MU- 55% 45% 100% 24% 76% 100% Total 88% 12% 100% 68% 32% 100% Non-CDD MS+ 1,193 145 1,338 240 71 311 MU- 53 84 137 10 24 34 Total 1,246 229 1,475 250 95 345 % of Total MS+ 89% 11% 100% 77% 23% 100% MU- 39% 61% 100% 29% 71% 100% Total 84% 16% 100% 72% 28% 100% CDD & Non-CDD MS+ 1,640 181 1,821 340 107 447 MU- 86 111 197 14 37 51 Total 1,726 292 2,018 354 144 498 % of Total MS+ 90% 10% 100% 76% 24% 100% MU- 44% 56% 100% 27% 73% 100% Total 86% 14% 100% 71% 29% 100% Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Latest ISR ratings were used. Data only includes those projects with available or valid ISR and ICRR ratings. Includes only investment lending projects. ISR = Implementation Status Report. Ratings: MS+ = moderately satisfactory or better; MU– = moderately unsatisfactory or worse. 37 APPENDIX B CDD PROCUREMENT Table B.6. ISR Procurement and IEG Risk to Development Outcome Ratings Risk to DO ratings Procurement ratings S+ MS+ MU- Total CDD Moderate or lower 55 70 4 74 Total 98 129 17 146 % Moderate or lower 56 54 24 51 Non-CDD Moderate or lower 136 191 14 205 Total 202 294 31 325 % Moderate or lower 67 65 45 63 CDD and Non-CDD Moderate or lower 191 261 18 279 Total 300 423 48 471 % Moderate or lower 64 62 38 59 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. Data only includes those projects with available or valid ISR and ICRR ratings. Includes only investment lending projects. DO = development outcome; S+ = satisfactory or better; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse. Table B.7. CDD Projects: ISR Procurement and IEG Outcome Ratings by Region Region IEG outcome ratings ISR procurement ratings MS+ MU- Total % MS+ MS+ MU- Total % MS+ AFR 37 19 56 66 177 23 200 89 EAP 5 2 7 71 61 5 66 92 ECA 22 8 30 73 63 6 69 91 LCR 18 9 27 67 79 13 92 86 MNA 8 4 12 67 29 5 34 85 SAR 14 7 21 67 80 8 88 91 Total 104 49 153 68 489 60 549 89 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. ISR = Implementation Status Report. Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. Table B.8. CDD Projects: ISR Procurement and IEG Outcome Ratings by Sector Board Sector Board IEG outcome ratings ISR procurement ratings MS+ MU- Total % MS+ MS+ MU- Total % MS+ Agriculture and Rural 30 12 42 71 165 16 181 91 Development Education 17 8 25 68 45 8 53 85 Health, Nutrition and 17 12 29 59 62 7 69 90 Population Social Development 13 4 17 76 35 2 37 95 Social Protection 12 7 19 63 48 6 54 89 Urban Development 4 4 100 41 3 44 93 Water 7 4 11 64 33 7 40 83 Other 15 5 20 75 60 11 71 85 Total 115 52 167 69 489 60 549 89 Note: As of November 8, 2012. ISR = Implementation Status Report. S+ = satisfactory or better; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse. 38 APPENDIX B CDD PROCUREMENT Table B.9. CDD Projects: ISR Procurement and IEG Outcome Ratings by Loan Size Loan size IEG outcome ratings ISR procurement ratings ($ millions) MS+ MU- Total % MS+ MS+ MU- Total % MS+ < 20 33 12 45 73 99 16 115 86 20≤ L <50 39 21 60 65 183 19 202 91 50≤ L <100 23 10 33 70 101 15 116 87 ≥100 20 9 29 69 106 10 116 91 Total 115 52 167 69 489 60 549 89 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. ISR = Implementation Status Report. S+ = satisfactory or better; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse. Table B.10. CDD Projects: ISR Procurement and IEG Outcome Ratings by Fragile State Status and CPIA Rating Rating Fragile state status CPIA ratings FCS Not FCS Total ≤3 >3 Total IEG outcome ratings MS+ 22 93 115 13 100 113 MU- 9 43 52 6 45 51 Total 31 136 167 19 145 164 % MS+ 71 68 69 68 69 69 ISR procurement ratings MS+ 73 416 489 46 437 483 MU- 16 44 60 14 45 59 Total 89 460 549 60 482 542 % MS+ 82 90 89 77 91 89 Source: IEG analysis and World Bank data. Note: As of November 8, 2012. The harmonized list of fragile situations for FY13 was used. Overall CPIA ratings for 2010 were used. CPIA = Country Policy and Institutional Assessment; FCS = fragile and conflict-affected situation; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse. . Table B.11. CDD Projects: ISR Procurement and IEG Outcome Ratings by CDD Concentration CDD IEG outcome ratings ISR procurement ratings Concentration MS+ Total % MS+ MS+ Total % MS+ ≤ 25% 32 44 73 157 177 89 25% < R ≤ 50% 20 33 61 85 100 85 50% < R < 90% 39 54 72 193 215 90 ≥ 90% 7 12 58 53 57 93 Total 98 143 69 488 549 89 Source: Social Development group database of CDD projects. Note: As September 24, 2012. CDD concentration is measured as a share of total IBRD/IDA lending amount. CDD = community-driven development; ISR = Implementation Status Report; MS+ = moderately satisfactory or better; MU- = moderately unsatisfactory or worse. 39 APPENDIX B CDD PROCUREMENT 40 APPENDIX B CDD PROCUREMENT Table B.12. CDD Projects: ISR Procurement and IEG Outcome Ratings by CDD component CDD component IEG Outcome Ratings ISR Procurement Ratings MS+ Total % MS+ MS+ Total % MS+ Decision making by participatory 28 45 62% 161 178 90% elected local governments Community Control and Management 52 74 70% 196 224 88% of Investment Funds Community Control without direct 53 75 71% 240 268 90% Management of Invest. Funds Enabling environment 53 86 62% 224 253 89% Total CDD Projects 104 153 68% 478 540 89% Source: Social Development group database of CDD projects. Note: As of September 24, 2012. The categorization is not mutually exclusive. Each CDD category shows the number of projects that had a non-zero amount assigned to that component. Total CDD amounts counts the number of projects that had an IEG outcome rating or an ISR procurement rating, respectively. CDD = community-driven development’ ISR = Implementation Status Report; MS+ = moderately satisfactory or better. Table B.13. CDD Projects: Procurement and Development Outcome by Main Project Objective Main project objective Procurement ratings Outcome ratings MU- Total % MU- MU- Total % MU- Service delivery 16 138 12 11 36 31 Empowerment 4 68 6 8 29 28 Improving local governance 6 43 14 4 16 25 Livelihood security prioritization 11 111 10 7 25 28 Total CDD projects 33 281 12 24 81 30 Source: IEG analysis and World Bank data. Note: As of September 24, 2012. Includes investment lending only. To identify the main project objective, only those projects that rated each area with 1 were considered. Procurement ratings are those of the last ISR. Development outcome ratings are those of IEG's ICRRs. CDD = community-driven development; MU- = moderately unsatisfactory or worse. 41 Project Reviews Table B.14. Sample of Projects Reviewed Approval Country Project ID Project name Amount FY ($ millions) 2002 Central African P073525 Multisectoral HIV/AIDS Project 17 2002 Ethiopia P050383 Food Security Project 85 2002 Jordan P076961 Horticultural Exports Promotion and Technology Transfer 5 Project 2002 Morocco P073531 Support for the Social Development Agency Project 5 2002 Senegal P074059 HIV/AIDS Prevention & Control Project 30 2002 Sierra Leone P073883 HIV/AIDS Response Project 15 2002 Tanzania P047762 Rural Water Supply and Sanitation Project 26 2003 Indonesia P059931 Water Resources & Irrigation Sector Management Program 70 2003 Kenya P078058 Arid Lands Resource Management Project Phase Two 60 2003 Rwanda P071374 Multi-Sectoral HIV/AIDS Project 31 2003 Tanzania P059073 Dar es Salaam Water Supply and Sanitation Project 62 2004 Angola P081558 3rd Social Action Fund (FAS III) 55 2004 Azerbaijan P076234 Rural Investment Project (AZRIP) 15 2004 Bangladesh P086791 Bangladesh - Reaching Out of School Children Project 51 2004 Brazil P080830 Maranhao Integrated Program: Rural Poverty Reduction 30 Project 2004 Honduras P083244 Nuestras Raices Program 15 2005 Brazil P076924 Amapa Sustainable Communities 5 2005 Congo, P086874 Democratic Republic of Congo Emergency Social Action 60 Democrat Project 2005 Guinea-Bissau P083453 Coastal and Biodiversity Management Project 3 2005 Indonesia P078070 Support for Poor and Disadvantaged Areas Project 104 2005 Peru P082588 Agricultural Research and Extension APL Phase 2 25 2006 Indonesia P085375 Third Water Supply and Sanitation for Low Income 138 Communities Project 2007 Afghanistan P0102288 Afghanistan: Emergency National Solidarity Project II 120 2007 Kenya P074106 Western Kenya CDD and Flood Mitigation Project 86 2008 Bolivia P101298 Second Participatory Rural Investment 20 2008 Ethiopia P108932 Pastoral Community Development Project II 80 2008 Haiti P106699 Haiti - Urban Community Driven Development Project / 16 PRODEPUR 2009 Brazil P110614 Sergipe State Integrated Project: Rural Poverty 21 2009 Ethiopia P106855 General Education Quality Improvement Project - APL 1 50 (GEQIP) 2010 India P101650 Andhra Pradesh Rural Water Supply and Sanitation 150 2010 Indonesia P115052 Third National Program for Community Empowerment in 785 Rural Areas (PNPM-Rural) 2011 India P102329 Rajasthan Rural Livelihoods Project (RRLP) 163 2011 Mozambique P107598 MZ PROIRRI Sustainable Irrigation Development 70 2011 Nicaragua P121779 Nicaragua Social Protection 20 EXAMPLES OF REVIEWS UNDERTAKEN FOR EACH CDD PROJECT (Available for all 34 projects from IEG) Sample Project 1: Afghanistan - Emergency National Solidarity Project - 43 Extracts from Project Appraisal Document Project Objective: “Building on the current National Solidarity Project, the proposed Grant will finance expansion to new areas to lay the foundations for strengthening of community level governance, and to support community-managed subprojects comprising reconstruction and development that improve access of rural communities to social and productive infrastructure and services.” Highlighted Components: The proposed IDA grant will finance about 23 percent of the total projected cost of the National Solidarity Program in the next three years for the following components: • Block grants for communities ($94.3 million) to carry out subprojects involving reconstruction and development activities through a facilitated participatory planning process. • Community Facilitation and Subproject Preparation ($9.7 million) to support local communities through: (i) facilitation exercises to establish inclusive community institutions, and identify local development needs and priorities; and (ii) assistance for preparing subproject proposals. • Community Development Council Capacity Development ($14.4 million) to assist Community Development Councils in carrying out subprojects. Procurement Arrangements: A large majority of the works to be procured under this project would include community subprojects. These subprojects are implemented by the communities using block grant provisions made by the project. Each community will get a block grant of $200 per family with an upper ceiling of limit of $60,000 per community. Therefore, the estimated value of most of the community subprojects would be less than $60,000 per project. The community shall carry out procurement following paragraph 3.17 of Bank Guidelines. The overall procurement risk of the National Solidarity Project is moderate. The project includes a large number of community level subprojects that are widely spread over the country and will be procured by the communities. IDA will be unable to carry out procurement reviews of a reasonable sample of subprojects as would be the case in a normal investment project. Regarding the mitigation plan, procurement administration is overseen by an international procurement specialist who is also building in house capacity at the Ministry of Rural Rehabilitation and Development (MRRD) level. IDA’S procurement team will continue to supervise the procurement administration closely and carry out post-procurement reviews periodically. The National Solidarity Project operational manual also provides step 44 by step procedures to be followed by communities for community procurement. The procurement procedures at the community level have been further simplified. The project will continue to provide training on community procurement to FP staff and Community Development Councils. Nature of Procurement Problems – Extracted from ISR, ICRs, ICRRs, and So Forth ISR 7: Anomalies in the MRRD Incremental Operating Cost Expenditure. The review of MRRD Incremental Operating Cost expenditure finds that civil servants resigned from government service and joined as consultants. This is against Article 15 of the Regulation of Personal Affairs of Civil Servants. ICR • Lack of familiarity by MRRD staff of Bank Procurement Guidelines and procedures contributed to delays. • There was one major contract with the oversight consultant that was amended five times. The amendments resulted in (i) time extensions from an initial period of 12 months to 42 months, and (ii) price modifications from an initial contract price of $5.3 million to $32.5 million. • Under the District School Window, 18 contracts were awarded to construct 8 to 16 classrooms for 58 schools in seven provinces. These contracts could not be completed during the project period and were extended to the second project. There were several contractual modifications to increase the quantities due to lack of diligence in the calculation of bills of quantities by MRRD’s engineering staff working on the District School Construction and Rehabilitation Window. • Careful engineering planning combined with more effective liaison with MoE and other donors could have contributed to avoid such incidents. IEG Comments - Based on the project documents, it appears that the community implemented subprojects had a smooth implementation reflecting extensive efforts at capacity building and technical assistance for the communities. No indication from the project documents of wastage or misuse of resources used for community-managed procurement. This would suggest that the Bank guidelines bearing on the CDD procurement were not an issue in this case. Transparency of the procedures at the local level as well as informal systems 45 of accountability among communities were key to assuring satisfactory use of project funds by the communities. - Procurement problems almost exclusively related to non-CDD components (for example, hiring of consultants and contracting of classroom construction under the district window) requiring procurement by a central agency. Design of CDD projects requires adequate attention to capacity for ancillary procurement needed to complement the procurement at the community level. - The project scope and coverage experienced major changes during implementation necessitating numerous ad hoc amendments to the procurement plans and change orders to previous awarded contracts. Up to a point this was unavoidable in a program of this nature but perhaps the task team could have done a better job in anticipating some of the changes as well as through more timely adjustments of the procurement plans. - Given the country context, procurement risk rating of “moderate” may have been too optimistic. Sample Project 2: Brazil—Maranhão Integrated Program: Rural Poverty Reduction Project Extracts from Project Appraisal Document • Project objective: “The proposed project will help Maranhão to achieve the goal of reducing poverty by increasing its [Human Development Index (HDI)] from 0.647 to 0.700 by the year 2007, through …(c) financing demand- driven community investments for income-generation, health and sanitation, education, culture, environmental management and others impacting the HDI and environmental sustainability …” • Highlighted Components: Community Subprojects ($36 million or 90 percent of total project cost) will support community matching grants for approximately 1,200 small-scale socioeconomic infrastructure, education, health, culture, productive, environmental and other investments aimed at raising the United Nations HDI. All 216 municipalities in Maranhão (other than Sao Luis) will be covered by this component, because of the importance of encouraging cross-sectoral integration throughout the state, getting all municipalities to focus on the HDIs, and strengthening local and municipal institutional arrangements. However, about 60 percent of total component cost will be concentrated in the 80 municipalities with the lowest HDIs. 46 Procurement Arrangements: In early 2003, an independent procurement review was conducted of the predecessor project and concluded that the procurement procedures followed under the project complied with the loan agreement. The State Technical Unit (NEPE) procurement staff was well-versed in Bank procurement policies and procedures. It was anticipated that all procurement financed by the project would be carried out by the beneficiary community associations. Procurement of goods and works under subprojects costing less than $50,000 implemented in remote areas would be carried out mainly through direct contracting and community participation. This procurement method is appropriate because most subprojects: (i) would be small and/or implemented in scattered or remote areas and therefore it will be difficult to obtain competitive proposals; (ii) can be managed directly by rural communities, who will also contribute directly to the work through the donation of unskilled labor and local materials; (iii) will be selected on the basis of willingness of the beneficiary communities to contribute to and physically supervise works execution; and (iv) would provide means by which communities could play an active role in the local development process. Additional procurement of goods estimated to cost less than $100,000 equivalent, up to an aggregate amount of $1.5 million, would follow national shopping procedures. Contracts estimated to cost more than $100,000 equivalent would be awarded on the basis of NCB procedures satisfactory to the Bank. Works contracts (other than contracts costing less than $50,000 in scattered or remote areas) estimated to cost less than $100,000 equivalent, up to an aggregate amount of $1.0 million, would be procured under lump sum, fixed price contracts awarded on the basis of quotations obtained from a minimum of three qualified local contractors in response to a written invitation. The invitation shall include a detailed description of the works, including basic specifications, the required completion date, a basic form of agreement acceptable to the Bank, and relevant drawings, where applicable. The award would be made to the contractor who offers the lowest price quotation for the required work, and who has the experience and resources to complete the contract successfully. The standard bidding documents for NCB agreed between the Bank and the federal government of Brazil would be used. No ICB is anticipated for any goods or works under the project. 47 Nature of Procurement Problems ISR 12: This ISR provides an update on recent developments related to the findings of the Procurement Post Review (100 percent coverage) carried out in August 2008. The task team leader has reviewed the PAD/LA for consistencies and reported to the SM accordingly. Following the conclusions of the Post-Procurement Review (evidence of systematic misapplication of Bank procurement guidelines, and red flags of possible fraud/corruption), several measures were taken: (i) INT was promptly notified of the Post-Procurement Review findings, (ii) the Borrower was notified (at the highest level) and given a chance to explain findings (letter attached), and (iii) the Borrower was asked not to approve further community subprojects until a corrective action plan was agreed to with the Bank. The borrower's reply to the findings (from October 17) did not significantly change the conclusions of the Post-Procurement Review. Consequently, the Bank will declare misprocurement. Given the above situation, project implementation has basically stalled. ICR: The Bank’s Post-Procurement Review revealed prima facie evidence that bidding quotations did not meet Bank requirements. The Mid-Term Review found overly-complex procurement procedures and lack of understanding of procurement rules and processes by associations. NEPE was found not to be providing adequate supervision and guidance to communities on problems to be avoided in procurement, and there was a lack of rigorous analysis by NEPE of problematic subproject documentation in order to make corrections. The Mid-Term Review recommended that NEPE (i) prepare procurement procedures designed to facilitate associations’ involvement, including model documents; (ii) send technicians to the subprojects to review their status; and (iii) expand the use of the MIS as a procurement control and supervision instrument. NEPE was also tasked with updating the Operational Manual, which at the time of the Mid-Term Review had no specific section on procurement. NEPE’s response to these recommendations was slow and incomplete and was in any case overtaken by issues related to the imminent closing of the project on December 31, 2008. Since procurement issues were first flagged only in April 2008, by then it was too late to take meaningful corrective actions. Concurrent Bank missions on financial management and regular field supervision at the time noted that (i) these procurement irregularities were not related to financial management practices, and (ii) the investment subprojects identified in April 2008 physically had been completed to the satisfaction of the beneficiaries. In other words, the problems identified were related specifically to the misapplication of procurement procedures. 48 The type of problem found in Maranhão (systematic misapplication of Bank procurement guidelines and confusion about how Brazilian Law 8666 on procurement must be applied under the project) has not been identified in the other CDD projects under implementation in Northeast Brazil. Nevertheless, one important lesson from this experience is that the application of Bank-approved procurement procedures for CDD projects requires considerable initial and periodic training of project implementation agencies which, in turn, need to transmit this training to beneficiary community associations. In the particular case of Brazil, this requires more pro-active engagement on the part of the Bank to periodically clarify to implementing agencies when and how Bank-approved procedures, as specified in the loan agreement and operational manual, should be used by communities, and how these may or may not relate to Brazilian Law 8666 on procurement. IEG Comments - Built on good experience with an earlier project. - Problems arose with compliance with procedure. No evidence of systematic fraud. Subprojects completed to the satisfaction of beneficiaries. - Importance of post reviews to ensure compliance. - Need for clear guidance to implementing agencies when conflict between Bank and government procedures. 49 ICT Procurement: Summary of Quality Assurance Group, IEG, and Bank Reports IEG reviewed evaluative and other lessons that emerge from earlier reviews of ICT that mention procurement, including a Quality Assurance Group assessment, an IEG evaluation, a Bank study on Financial Management Information Systems (FMIS), and the Bank’s ICT Strategy. Three of these were completed in the past two years, indicating that the effectiveness with which the Bank approaches ICT investment is an important topic in the Bank. The Quality Assurance Group assessment reviewed 23 projects with a combined procurement of ICT components valued at $680 million. 17 Its procurement discussion focuses primarily on resource and skills issues, and procurement planning. With respect to the latter, a key finding is that the special procurement requirements of project ICT components are not properly recognized. This is treated as a project design issue. The report concludes that the root of the problem is weakness in the procurement strategy, as less than one fourth (23 percent) of projects are rated as satisfactory or better in this respect. Today, the Quality Assurance Group assessment is somewhat dated (it covers projects that were under implementation eight or more years ago) but it points toward a broad spectrum of skills and resource issues, and weak procurement planning, that are constraints in achieving project outcomes. The IEG ICT evaluation covered the Bank’s work in the ICT sector in several broad areas (IEG 2011b). A part of the evaluation dealt with ICT applications such as the use of ICT for delivery of public services. Procurement was assessed for its contribution to their effective delivery. The evaluation offered the following observations and recommendations regarding procurement. • Procurement is a major constraint to implementing ICT projects. Procurement delays, attributable in part to risk-averse procurement specialists, slow down project implementation, in some cases stalling projects for years. • The Bank’s procurement rules are designed for infrastructure projects; for example, separating contracting from professional (consulting) services. They are cumbersome and not suited for a rapidly evolving sector like ICT. The Bank needs to better adapt its procurement rules to meet the needs of ICT projects. • The Bank lacks necessary ICT procurement-related expertise, leading to inconsistencies of approach. 51 • Change management is inherent in projects that have a significant ICT component (modernization of a tax system, for example). This further complicates specifying system requirements and managing implementation. This is a common theme of this study, reported by task team leaders (Indonesia) with direct experience, as evidenced in IEG’s field visits. The Bank’s recent ICT strategy followed the IEG ICT evaluation and reflects many of its findings and recommendations. In particular it notes a lack of ICT-related procurement expertise at the Bank, inconsistency of approach, and a procurement regime derived from infrastructure projects, not well suited for ICT procurement. Additional points raised by the strategy include: • A possibly higher incidence of corruption in information technology procurement than in other project components, based on a study in the Europe and Central Asia Region. • A need to build ICT-related procurement skills within the Bank, but also to build capacity in client agencies, particularly in areas related to developing technical specifications and bidding documents. • Acknowledgement of recent improvements and agreed areas for future change. The former include the 2011 revisions to the guidelines, which provide for two-stage bidding, framework contracts and continuation of engagement with a brand name supplier. Proposed future actions include the update of the Bank’s ICT standard bidding document, revisions to procurement specialist training materials, and allocation of large complex ICT to the most experienced procurement specialists. The FMIS study reviewed 55 closed and 32 active FMIS treasury-related projects covering the period 1984 and 2010 (World Bank 2011d). All include large ICT systems, implementation of which was a primary focus of the study. The study pointed toward 18–24 months of typical delay caused by lengthy procurement processes, stretching in rare cases to over two years. 18 This contributed to major delays and dissatisfaction on the part of the borrower. It also pointed towards technical capacity constraints on the Bank’s project teams and the project implementing unit staff of the borrower. The study recommends ensuring ICT expertise on the Bank project team and building capacity of the project implementing unit, with the latter focused on building organic capacity. The study also pointed to the need for appropriate procurement planning and recommends combining procurement into a smaller manageable number of packages, typically 52 three or fewer. Finally, it alluded, in one of its case study countries (Turkey) where procurement using country procurement systems was completed in six months. 19 Box B.1. FMIS Packages The Bank’s FMIS study reported from its case studies that, in most instances, three variants describe the design of procurement packages and procurement methods. Option 1 (Georgia, Moldova, and Tajikistan): One single-responsibility contract package using two- stage ICB was bid covering the implementation of all ICT software, hardware and network components. Option 2 (Albania, Azerbaijan, and Russian Federation): Two linked contracts were bid simultaneously involving (1) two-stage ICB procurement of a contract for the development of application software, including demonstration, and (2) single-stage ICB bidding of a contract for the installation of all hardware and network equipment. To ensure compatibility, implementation of the second contract depends on the inputs to be provided by the software developer under the first contract. The second contract was delayed due to this linkage. Option 3 (Kyrgyz Republic and Ukraine): Two separate independent contracts were bid simultaneously involving (i) two-stage ICB bidding of a contract for the development of applications software and installation of central hardware, including demonstration, and (ii) single-stage ICB bidding for the installation of standard field hardware, engineering support systems and network equipment. The study concluded with suggestions to improve the design of FMIS procurement: It is advisable to have two separate independent packages (Option 3). The study found that this reduced the complexity of ICT implementation and resulted in faster implementation. Another possibility is to use Option 1 for a well-defined and relatively small-scale implementation of the FMIS system. Bidders should be required to comply with all ‘mandatory’ requirements listed in the bidding documents. This ensures that minimum technical requirements are met. Technical evaluation/merit points should be considered only when there is a need to define and evaluate ‘desirable’ requirements that will add value to the minimum mandatory requirements—where there is quantifiable value expected from offers that exceed the minimum requirements. Where the technical requirements are well defined, the weight of price should be kept as high as possible (80 percent or more) to benefit from price competition, while ensuring the delivery of good quality solutions based on mandatory requirements. The weight of technical evaluation for desirable features can be up to 20 percent, if there is a need to consider some optional high-value technical requirements in addition to mandatory requirements. IEG agrees that the maximum level of 20 percent is reasonable. A higher level leads to more subjectivity in the evaluation and is also an indicator that technical requirements are not well understood. Source: IEG 2008a. 53 PPP Procurement: Comparison of the Bank and Other Organizations Over the past few years, multilateral development banks (MDBs) have been working to elaborate procurement requirements for PPP projects through their Heads of Procurement. There is an established working group to study PPP-related procurement matters that provides guidance on the matter of “acceptable practices,” and harmonizes approaches among the MDBs. Outputs are meant to guide the different MDBs as their policies develop. They include: • Procurement & PPP Transactions Guidance for MDB Public-Sector Engagements (2012). This guide aligns closely with the Bank Guidelines and the Guidance Note, although with less detail (for example, in treatment of bidding procedures). It elaborates the role of the MDBs in providing advisory services and guarantees. • Procurement Principles Applicable to Private Sector Transactions—Guidance for MDBs (2012). As with the guide described above, it reflects existing Bank practice, but expands into other areas. For example, it emphasizes the importance of due diligence, recognizing that private sector projects are often not subject to the same oversight as public sector projects (they might, for example fall outside the jurisdiction of a national procurement law). Typical due diligence requirements and documentation are detailed in attached checklists. The guide also addresses the need for arms-length arrangements (between a client and its affiliates, who might also act as contractors) and provides guidance on guarantees and unsolicited proposals. The approach of other MDBs is documented in the heads of procurement PPP working group guidance notes described above. As with the Bank, MDB policies are typically spelled out in the procurement guidelines of the organization, and supplemented by interpretation notes and guides. With the exception of the Black Sea Trade and Development Bank, requirements of the other MDBs are very similar to the Bank’s. 20 When it comes to on-lending arrangements, where the beneficiary is a PPP party, MDBs leave procurement to the intermediary to oversee, saying that procurement must follow well-established methods and commercial practices “acceptable” to the organization, or similar wording. For selection of concessions, all MDBs specify open and competitive tendering, and all provide for the exception to open tendering by requiring competitive tendering for downstream procurement. There are a few variations between the organizations: 55 • The European Bank for Reconstruction and Development requires a formal competitive process but defines it in a separate concession policy, rather than in its guidelines. • The African Development Bank (AfDB), Inter-American Development Bank (IDB), and Islamic Development Bank (ISDB) all specify ICB for selection of the concession, as did the Bank prior to the 2011 revision to its guidelines. The European Investment Bank (EIB) specifies “formal international” tender. • Except for the Bank, ADB, and ISDB, remaining MDBs employ a market comparison to contract terms, saying that costs must be in line with the market and the project budget, and that contract terms should be fair and reasonable. • EIB invokes compliance to local legislation. It also requires that European Union treaty conditions be met where a concession is not awarded through formal international tendering. These conditions include adequate advertising, international competition, and a fair and nondiscriminatory process. There is very little variation in the PPP procurement policies of the MDBs, as reflected in their procurement guidelines. Furthermore, through the heads of procurement working group, the MDBs have worked to harmonize such things as interpretations of “acceptability,” further aligning their practices. The policy similarities are unsurprising. Over the past 10 years there has been a push to align procurement guidelines. Differences that exist (AfDB, IDB, and ISDB specifying ICB for selection of concessions, for example) probably reflect one organization getting ahead of the other, rather than any fundamental disagreement. With policies so similar, it is differences in application and interpretation of those policies that most likely differentiate approaches to PPP procurement. In contrast to the Bank, the International Finance Corporation (IFC) does not have documented requirements regarding procurement in its PPP investments. IFC reviews a concession agreement (and key contracts in a typical transaction) as part of its overall assessment of credit, development and reputational risks in the transaction, rather than as part of a self-standing procurement exercise. IFC takes a commercial approach, assessing if the PPP arrangement was the result of an open and competitive process, prices are reasonable, the project complies with local laws, the project is financially viable, and the parties to the project are reputable, competent, and financially sound. This reflects the commercial orientation of IFC. Unlike the World Bank Group, which separates private and public sector lending operations between two different organizations (IFC and IBRD), most other MDBs have an in-house private sector lending window. Their operations include 56 departments dedicated to private sector lending, yet they manage to accommodate this private sector lending function within the terms of procurement policies that are almost identical to the Bank’s. Within these organizations, procurement policies, including PPP-specific provisions tend to be developed by public sector procurement specialists (MDB representatives on the heads of procurement PPP working group are all public sector procurement people, except for some participation of IFC). It is understandable then that the PPP-related procurement policies of other MDBs, like the Bank, address public accountability concerns and are not fully private-sector oriented, as are the IFC’s. Box B.2. Public Accountability frames Government Regulations for PPP Transactions Like the MDBs, governments, and those who may course financing through them with a sovereign guarantee, are constrained by their public accountability. A guide produced by the World Bank Institute gives examples of PPP procurement requirements for six countries (Brazil, Chile, Egypt, Mexico, the Philippines, and South Africa) and the European Union. These examples describe the legal requirements typically contained in a PPP or similar law. Also described are the methods used in the different countries for establishing qualifications of clients, bidding processes to be followed, conditions of negotiations, and basis of awarding concessions (such as price and technical capability). Refer to Table 3.5.1 of the WBI guide for a detailed description of the different PPP procurement procedures. Sources: World Bank Institute, IEG. Of the different country approaches described in the World Bank Institute (WBI) PPP reference guide countries following the European Union procedures have the most options (World Bank 2012j). The options allow open or restricted bidding (restricted to a minimum number of prequalified bidders), single or multistage bidding (where full proposals are invited at one time, or different aspects of the proposal are invited in sequence), and different approaches for negotiation, ranging from no negotiations at all to simultaneous negotiations with all bidders throughout the bid and evaluation process. In all cases, European Union regulations allow for contracts to be awarded to the best financial proposal (technically compliant) or the best combination of technical and financial scores. The following variations of country policies as compared to the Bank’s are noted: • All countries, including those following European Union procedures, use a competitive tendering process to arrive at a PPP concession contract. All permit prequalification as an option, except Philippines, which states it is the “norm” and South Africa, which mandates it (and further limits the number of prequalified bidders to between three and four). Prequalification restricts participation in the bidding to prequalified firms. This is not at odds with Bank policies; the Bank Guidelines mandate competitive bidding and its 57 guidance encourages prequalification. However, the prequalification process itself must be advertised and opened to all eligible bidders, and may not limit the number of bidders as does South Africa. • All countries require a single-stage bidding process, except Egypt and those using the European Union competitive dialog procedures. South Africa allows the addition of a second stage to bidding where the single stage process does not produce a clear preferred bid (“best and final offer” approach). All of these arrangements are fully in line with the Bank’s guidance, which provides for either single- or two-stage bidding, at the option of the government. • Negotiation, to arrive at a concession contract, is an area where there appears to be considerable divergence. The policies of three countries (Brazil, Chile, and Mexico) are silent on the topic. The Philippines and South Africa do not allow negotiation, apart from exceptions (where there is only one bid, for example), consistent with Bank policy. Egypt and the European Union provide for a “competitive negotiation” procedure, as an option. The Bank’s guidance note (and also the MDB guidance) does not accept competitive negotiation, where contract award is based on negotiations with several bidders. It is therefore at odds with the policies of Egypt and any country following European Union procedures. It is not clear if the other countries, whose policies are silent on the matter, accept the practice or not. • Except for Brazil and the Philippines, all countries, including those applying European Union policies, have two options for deciding who is awarded the concession contract. One option is to award on the basis of best financial offer among technically compliant offers (“lowest evaluated bid” in Bank terminology). The second is to use a point scoring system that combines financial and technical proposals to come up with an overall best offer. Brazil and Philippines do not allow the second option. The Bank, in its policies and guidance, accepts both methods, as does the guidance from heads of procurement. The WBI document notes that South Africa applies a preference for black-owned bidders. Similar preferences (for domestic bidders, for example) are not uncommon--the Bank allows some in its public procurement policies and would likely accommodate certain country-specific preference schemes when assessing “acceptability” of the procurement system. The WBI guide addresses the matter of unsolicited proposals, proposals from a private sector entity to government to enter into a PPP. These are seen positively, as a way to encourage the private sector to promote its ideas and expertise to aid the delivery of public services. Seven countries were surveyed (Chile, Indonesia, Italy, Korea, the Philippines, South Africa, and the U.S. state of Virginia). All accept 58 unsolicited proposals, but then conduct a competitive bidding if the proposal is worthwhile. To incentivize such proposals, the proponent is given an advantage during evaluation, compensated for its development costs if it loses the bidding, or given an opportunity to match a competing winning bid. The heads of procurement group guidance recommends this approach. The Bank’s guidance is vague, saying that awarding a contract to an unsolicited proposal may be considered as an exception under the guideline provisions for direct contracting, or that the Bank may require a competitive process, possibly “adding necessary flexibility.” 59 Appendix C. Procurement Risk Concepts and Thresholds Risk Concepts Box C.1. Procurement Risk in Relation to the Bank’s Principles of Procurement—Process and Outcome Risk Transparency in procurement process sees to the first injunction in the Bank’s Articles: that loan funds be used for the purposes intended. Sometimes less emphasized (even though measures to explicitly address fraud and corruption in procurement have become more visible) is the need for transparency in contract management and execution, so transparency in outcome is also needed. Transparency (widespread advertising of tenders) also serves the equity principle by ensuring tender information is broadly disseminated. The economy principle has been intended to embody a prime purpose of procurement, namely to secure for the Borrower optimal value for money in the use of Bank loan funds, as the outcome of a competitive tender, against the risk that many factors may work to impede that optimal outcome. While the achievement of an optimal outcome may depend on an effective procurement process, the optimal economic result itself is an outcome concept. As increasingly recognized today, “economy” on its own may not be adequate, especially if narrowly interpreted, for a best value outcome. The efficiency principle relates exclusively to the effectiveness of the procurement process (in pursuit of the economic outcome) and addresses the risk that the procedural requirements of procurement policy may be such as to be overly cumbersome, costly and slow to an extent that could slow project implementation. To that extent highly inefficient procurement processes or systems may be a factor which impacts project outcomes, but the efficiency principle relates exclusively to process. Too seldom emphasized in Bank discourse is the fact that there is a key risk-efficiency trade-off embodied in the balancing of efficiency and economy, which is dealt with in more detail in the main text. The equity principle requires that, in the execution of its procurement policies, the Bank ensure that suppliers from all eligible countries be given equal knowledge and an equal opportunity to bid on the tenders for contracts under projects being financed by the Bank. Clearly, the equity principle cannot be extended to outcomes, since there can be only one winner per bid. To that extent, the equity principle relates also only to process. The local development principle historically supported the purpose of World Bank lending to bring development to developing countries markets. (As data from procurement outcomes in recent years show an increasing degree to which Part II country suppliers have won ICB tenders on Bank projects —now in excess of 60 percent — suggest that the risk that favoring local suppliers may conflict with the economy principle have probably declined. Procurement Principle Process Outcome Risk Addressed Transparency X X Loans not used for purposes intended Economy X Procurement gives sub-optimal value for money, Efficiency X PR process too cumbersome, slow, delays outcome and project implementation Equity X Uneven playing field between eligible bidders Local development Local suppliers, contractors shut out by international - Goods, contractors, consultants X bidders - Local procurement systems Local PR systems not compatible with Bank policy X 61 62 Box C.2. Procurement Risk Management—Different Levels in the Bank Procurement risk is dealt with at different levels in the Bank, including the entity level, country and sector levels, and, most importantly, at the level of individual project transactions. The extent to which procurement risk is treated as a separate set of risks, as against being integrated into more general operational risk, varies both across levels and within them. At the level of the Bank as a whole (entity level): There are three key functions at the entity level which view procurement risk in real time, as part of ongoing operations: • Integrated Risk Management Report: This report, which is the Bank’s most aggregated report on risks facing the institution as a whole is based on a survey of Bank-unit “appetites and tolerances” for 41 identified risk items. One of these items is “FM, Procurement and Disbursement risk,” which views procurement risk mainly from the perspective of how it may impact the Bank’s overall disbursement rate. The Chief Risk Officer unit is currently engaged in a brainstorming pilot analysis which is attempting to seek answers to a number of procurement risk related questions, with a view to establishing the appetites and tolerances for this risk factor. Some key questions being explored include: How is procurement risk managed? Beyond the taxonomy of risk management tools, how much procurement risk should the Bank take on? Does the Bank follow a risk-return model in procurement where higher risks maybe tolerated where returns to development (measured either by IRR, progress towards PDOs, or IEG outcomes) are higher? How should procurement risk be flagged? Has the bar for procurement protection been set too high or too low? It is anticipated that this process could be completed by end FY13, but this will depend on senior management (VPU/EVPU) sign-off, and could also be potentially impacted by the arrival of a new Bank CFO in March. • The Chief Procurement Policy Officer: This is the Bank’s chief authority on procurement policy, and in that capacity is the apex of the Bank’s central and operational (Regional and Network) based procurement specialists who administer the function throughout the Bank, including a significant contingent of decentralized specialist staff located in the Bank’s field offices in many countries. This team represents the core of the Bank’s procurement implementation capability, it upholds procurement policy, it advises operational staff at the level of project transactions and their procurement requirements, it acts to settle complaints and disputes, and it offers a training component to both Bank staff and staff within borrowing countries involved in implementing Bank projects. In that sense this officer commands a team that is the backbone of the Bank’s capability in managing procurement risk at all levels of the institution. He or she also serves as an ultimate arbiter on procurement policy matters—often involving real time disputes on specific procurement contracts—where these cannot be resolved at a unit or regional level. • The Operations Procurement Review Committee (OPRC): The OPRC plays a key oversight role by reviewing and clearing the highest-value and most complex contracts financed under Bank-supported operations. All contracts above a defined threshold level must be reviewed by the OPCR, and some complex or sensitive contracts may be reviewed, regardless of their value. The thresholds for mandatory review were increased in January 2009 (unchanged since 1992) to (i) $50 million or more for civil works; (ii) $30 million or more for goods; (iii) $15 million or more for consultancy services; (iv) $5 million or more for all direct contracting and single-source contracts with firms; and, (v) $1 million or more for single-source contracts for individual consultants. 63 Source: IEG interviews with Bank staff. Box C.3. Procurement Risk Management Relative to Other Operational Risks in Bank Projects An important question in managing procurement risk at the level of individual projects is how this risk is viewed in comparison with other operational risks facing Bank projects. The Bank addresses this issue by distinguishing four categories of risk that affect projects, two of which (Country Risk and Sector Risk) are external to the projects, while the remaining two (Inherent Risk and Control Risk) refer specifically to factors arising from the project itself. Risks are rated at four levels: high, substantial, moderate, and low. The categories are as follows: External to the Project • Country Risk: Country risk relates to the general standing of the country in terms of business environment, operational risk, strength of institutions, stability of the economy, status of procurement reform, record on governance and general risk of fraud and corruption. Country risk assessments, including relevance to procurement (for example, as linked to CPIA results, and/or governance and fraud and corruption risk) should be contained in the Country Assistance Strategy for each borrowing country. • Sector Risk: This relates to risks generic to specific sectors (including those derived from strength of institutions, investment types and commercial risks) that should be reflected in Sector Strategy Notes compiled by the networks. Specific to the Project • Inherent Risk (Project Risk): For a given project the relevant elements of Country, Sector and Project risk, including the overall business environment and procurement capacity of the country, design of the project, its technical complexity, its operational and/or commercial objectives, market conditions, technology and financial risks, all combine to form the Inherent Risk of the project. These are independent of specific procurement risk factors, which are treated separately (see below) but, once identified, procurement risk should be an integral part of overall project risk. • Control Risk: Control risk is related to the internal control structure and the capacity of the implementing agency for the project to competently handle three aspects of procurement execution: Governance, Capacity in policies and procedures (knowledge of Bank Guidelines and systems), and Oversight of procurement implementation. In brief, Control Risk reflects the capability of the implementing agency to manage procurement process risk. Part of this capability involves the capacity to develop a dynamic Procurement Risk Mitigation Plan containing specific measures to address key procurement risk factors, and to monitor their application over time. In standard Bank practice, all Project Appraisal Documents (PADs) contain a procurement annex (written by the procurement specialist, but drawn from material produced by the borrower) built around the two key dimensions of control risk: An implementing agency capacity assessment; and the quality and effectiveness of the Procurement Risk Mitigation Plan. • Procurement Risk: Procurement risk is the risk deriving from the specific nature of the supply and service contracts that have to be procured to implement the various components of the project, as guide by the Bank’s or other procedural guidelines, with due regard to the four overriding principles contained in Box C.1. (As described below, there is a need to clarify this usage within the risk taxonomy currently used by the Bank) • Residual Risk: The Residual Risk is the level of procurement risk remaining in the project (in the judgment of the procurement specialist) after implementation of the procurement risk mitigation plan. Sources: OP/BP11.00, the P-RAMS template; OPCS Reports, for example, Use of Country Procurement Systems Proposed Piloting 64 Program. OPCS May 20, 2008 Annex D. Box C.4. Managing Procurement Risk in Projects—Summary of Tools and Procedural Devices Available to Procurement Staff to Manage Procurement Risk • Project Design: Project design is mainly driven by project • Timing of Procurement: In large and/or complex content and objectives, but quite often project components are procurements task team leaders/procurement chosen because their procurement features set them apart specialists may elect to conduct advanced from other components. Many projects also have project procurement (sometimes with retroactive financing) in implementation components designed to finance the order to ensure procurement does not delay project enhancement of implementation capacity of the implementation. Frequently pre-appointment of implementing agency. Projects with highly decentralized procurement staff to project implementing/managing objectives often have procurement delivery as part of the unit may be a condition of project approval or project design. effectiveness. • Procurement Plan: Written by the Borrower, under Bank • Frequency of Supervision: In higher risk projects supervision, and mandatory for every project, this lays out procurement specialist may call for more frequent the whole procurement strategy and content and highlights than annual supervision. the risks to be addressed. It is intended to be updated at least annually, and in that sense is both a planning and a dynamic • Use of website: In pursuit of the Transparency monitoring tool. principle many projects now use web sites as a means to display procurement tenders, awards and • Selection of Implementing Agency: The Borrower may outcomes. E-procurement is being encouraged but is propose but the Bank must agree on who will be the not yet widespread. implementing agency. The choice is most often between a Ministerial unit or agency, and project • Financing of Incremental Operating costs: As a implementation/management unit (project means to ensure that project implementation is implementing/managing unit) required by the Bank if it properly resourced projects are often designed to deems the Ministry unit to be lacking in capacity. include the salary and other operating costs of implementing agencies (excluding civil servant • Staffing: It is very common for measures to be taken to salaries where these units are staffed by government enhance the skills and experience of the implementing agency staff). staff (whether in the project implementing unit or another) by either or both training and hiring of consultants to the agency • Addressing Fraud and Corruption: The requirement as a means of enhancing capacity to implement procurement. to address ethical behavior and fraud and corruption risk in projects was introduced into the Procurement • Procurement Method: Within the broad thresholds set Guidelines in 1996, but was not widely practiced until Bank-wide by OPCSOR (and within that framework a decade later, following the governance and anti- separately by the Regional RPM) project staff are free to corruption program and IDA internal controls review. define which procurement method will be used for which Various tools are available: project components, based on their judgment of risk and efficiency in achieving an economic outcome. ICB + Prior o Ethics Pledge: Some projects require an Review (and/or NCB) remains the default procurement ethical conduct pledge from project method for all larger contracts (both goods and works), but counterparts. newer forms of lending (particularly in the social sectors) o GAAP/IS: Recent IL reform and the have increased the share of other methods. governance and anti-corruption have introduced Implementation Support (IS) and • Use of Pre-/postqualification: In large, complex projects Governance and Accountability Action Plan to with substantial technology dimensions task team focus and resource implementation and leaders/procurement specialists may choose to require pre- explicitly address transparency and fraud and qualification of bidders, to reduce know-how risk in corruption risk. procurement. o Civil Society Oversight: Increasing use is being made in project design of a role for CS in • Thresholds for Procurement Method and Review: oversight, often involving fraud and Maximum Prior Review thresholds are set institutionally (in corruption risk and using the project website, Annex C of BP 11.00), for goods, works and services, with among other means. different levels for each risk category of the implementing o Establish Complaints process is a agency. Regional Project task team leaders/procurement common means of providing access to red specialists can set lower thresholds where they believe risks flags and indicators of fraud and corruption are greater. Thresholds are a key tool for risk management at risk. the project level. PS may also increase the sample of contracts 65 to be post-reviewed (generally 20 percent); they may call for • New Tools –P-RAMS/ORAF: Mandatory since July Post-Procurement Reviews annually or twice a year, and may 2010 the P-RAMS provides a detailed checklist in the call for Independent Procurement Reviews. implementing agency capacity assessment of questions around 11 risk factors, and a Mitigation Measure Action Plan aimed at dynamically assessing all forms of procurement risk, including fraud and corruption risks and intends to inform the ORAF and the PAD. Source: IEG review. 66 Thresholds Analysis – “Methods” Thresholds SETTING METHOD THRESHOLDS TO OPTIMIZE COMPETITION The Implementing Agencies of borrowing countries can use a variety of procurement methods in World Bank funded projects. The method selected depends on a number of factors including the type and value of good, work or service procured and the potential interest of domestic and foreign bidders. 21 The Bank sets monetary threshold above which certain methods have to be used. In the case of goods and civil works these thresholds indicate, among others, above which contract value ICB shall be used. 22 Contracts below this value are most prominently procured through national competitive bidding (NCB). Monetary procurement method thresholds are set for each borrowing country by the regional procurement managers (RPMs), drawing on analyses of procurement risk undertaken as part of Country Procurement Assessment Reviews (CPARs) or other means, and including foreign bidders’ interest in participation in domestic markets; conditions, size and depth of the market and capacity of the local industry. 23 ICB thresholds therefore show a significant degree of variation across countries and within regions. More sophisticated clients and middle-income countries tend to have higher thresholds. The maximum ICB threshold used by the Bank is $25 million, for civil works contracts in Brazil. In East Asia and the Pacific ICB thresholds for civil works vary from $100,000 in Kiribati to $20 million in China within the same region. Some country-specific thresholds are set with a monetary range for risk categories low to high, as in India, where ICB is used for works contracts starting from $10 to 20 million. One feature distinguishing ICB and NCB lies in the requirements for the publication of Specific Procurement Notices. Paragraph 3.4 of the Bank’s procurement guidelines stipulates that borrowers may limit advertising to the local press for NCB processes. Foreign firms cannot be barred from competing for NCB contracts; the different treatment of advertising for NCB and ICB reflects the position that foreign firms would not normally bid for NCB contracts given their value and nature. 24 With its procurement guidelines the Bank aims to allow borrowing countries to buy high quality goods and services as economically as possible and expresses its position that this objective is best achieved through transparent, formal competitive bidding. For the procurement of equipment and civil works, ICB is the procurement method the World Bank encourages its borrowers to use in the majority of cases. 25 67 The setting of ICB threshold is critical given the expected benefits in competition and economy in the tender process. There is, however, a possible tradeoff with processing time and efficiency. ICB procurement tends to take longer than contracts awarded with NCB procedures, as IEG’s analysis of processing times in procurement in Chapter 4 suggests. ANALYSIS OF DISTRIBUTION OF ICB CONTRACTS IEG has attempted an analysis similar to the one undertaken by the AfDB (as described in Chapter 3). Using contract data from its budget tracking system, AfDB has shown that in works contracts foreign bidders entered bids on contracts for mostly those contracts in the top quintile (20 percent) of contract value. The AfDB concluded, accordingly, that to set the ceiling contract for ICB at any level lower than 80 percent of highest value would involve no increase in competitive bidding but would involve significantly more review work. To analyze the trends in methods used for procuring civil works in World Bank- funded projects, IEG used recent years of the Bank’s Form 384 database on prior reviewed contracts financed by the Bank. 26 Between 2007 and 2012 the Bank financed 3,618 prior reviewed civil works contracts procured through ICB (out of 12,572 Civil Works contracts total). This gives an average of around 50 prior reviewed civil works contracts procured through ICB per month. On the other hand 9,453 goods contracts were procured through ICB during the same period, an average of around 130 contracts per month. IEG undertook an aggregate analysis for procurement of civil works contracts only, but suggests a closer look at cluster of countries with similar parameters to determine the impact of current method thresholds. Based on their value civil works contracts procured through ICB between 2007 and 2012 were divided into quintiles and disaggregated into national and foreign contractors. Participation of foreign firms, in terms of numbers of contracts, is relatively low in the first four quintiles, ranging from 15.5 percent in the first quintile to 24.2 percent in the fourth quintile (Table C.1). These results vary only minimally when looking at the distribution by total value in those quintiles, though they are slightly higher than what is found in AfDB’s analysis; that is, more foreign firms win contracts in the lower quintiles of World Bank funded projects. Table C.1. Distribution of Civil Works Contracts Procured through ICB (2007–12) By number Quintiles Value of National contractors % Foreign contractors (regional % contracts ($m) (nos) and nonregional) (nos) 68 1 0 - 0.6 612 84.5 112 15.5 2 0.6 - 1.5 574 79.3 150 20.7 3 1.5 - 4.1 570 78.8 153 21.2 4 4.1 - 11.0 549 75.8 175 24.2 5 >11.0 436 60.3 287 39.7 Upper Percentiles 90th percentile >23.3 189 52.4 172 47.7 95th percentile >41.1 75 41.7 105 58.3 By value Quintiles Value of National Contractors % Foreign Contractors (Regional % Average Contract Contracts ($ (total $ millions) and Non-Regional) (total $ Value ($ millions) millions) millions) 1 0 - 0.56 169.0 85.3 29.1 14.7 0.3 2 0.56 - 1.54 569.0 78.9 152.0 21.1 1.0 3 1.54 - 4.11 1,500.0 78.7 406.0 21.3 2.6 4 4.11 - 11.00 3,710.0 75.7 1,190.0 24.3 6.8 5 >11.00 13,300.0 50.6 13,000.0 49.4 36.3 Upper Percentiles 90th percentile >23.30 9,460.0 46.0 11,100.0 54.0 56.9 95th percentile >41.10 5,990.0 39.9 9,010.0 60.1 83.3 Source: IEG analysis. The findings of IEG’s analysis suggest that about 80 percent of prior reviewed ICB contracts for civil works in the lowest four quintiles, with a maximum value of $ 11 million, were won by national firms. Although the results of IEG’s analysis are broadly similar to findings of the AfDB, there are some differences: in the highest quintile of contract sizes (with values greater than $ 11 million) in World Bank projects more contracts tend to be won by national contractors than in the case of the AfDB. In the case of the World Bank the majority of these high value contracts were won by national firms, with foreign firms winning only 40 percent. 27 The picture changes for the World Bank in the highest five percent of contracts (with values greater than $41 million), where more contracts were awarded to foreign firms. It has to be noted that the AfDB paper assumes a uniform level of ICB threshold for all countries while the World Bank uses country-specific thresholds that vary significantly as described above. Also, in practice, at the World Bank, the ICB procurement method may frequently be used even below the threshold set for the country; indeed the analysis suggests that average contract values are relatively low for the first two quintiles where 1,448 contracts with an average value below one million were procured through ICB (see also section on prior threshold analysis earlier in this chapter). In FY12 half of the civil works contracts procured through ICB used this procurement method for contract values below the threshold set on the country level. 69 If the upper limit of the fourth quintile was to be considered as the ICB threshold for civil works for the contracts in this aggregate analysis, the number of contracts with mandatory ICB use would have reduced significantly. This would represent an improvement in efficiency in terms of procurement processing time. In a previous section IEG analyzed a sample of contracts and extracted average processing times for procurement. The results found suggest savings of 93 days on average per contract when moving from ICB to NCB. However, given the fact that they are established on the country level, analysis of method thresholds for cluster of countries with similar characteristics is key. It would be important to quantify the trade-off between procurement efficiency and risk especially for groups of countries with lower ICB thresholds as explanatory factors might not be uncovered with an aggregate analysis. A caveat to the above analysis is the known limitations, in terms of quality and reliability of Form 384 data. Once the Bank’s proposed electronic no objection goes fully into effect, such limitations will pose fewer problems. 70 Thresholds Analysis – “Prior Review” and “Clearance” DATA SOURCES Data Sources – Bank-Wide Prior Review and Clearance Thresholds Data for the Bank-wide prior review and clearance threshold analysis were compiled from archived and current OP/BP 11.0 files and corresponding memoranda on the Bank’s intranet. There is no central repository for this information and it may be incomplete. Prior review threshold information from 1997 by clearance level were the earliest available to IEG (Sanchez 1997). Reference to Bank-wide maximum prior review thresholds are found in 2008 and 2009. 28 Clearance thresholds were revised in 2002. 29 Further revisions of some clearance thresholds are found in the 2008 prior review threshold updates. 30 Mandatory review thresholds for RPMs and the OPRC were further updated in 2009, as were regional and country level revision of prior review thresholds. Some regions, for example, Africa, adopt further levels of delegation for clearance (with corresponding thresholds) between procurement specialists, senior procurement specialists and hub coordinators. Data Sources—Regional Prior Review and Procurement Method Thresholds Regional and country level information on previous years’ prior review and method thresholds is not maintained on a uniform basis across all regions and is readily available only for a few regions. Historical threshold information was readily available for the East Asia and Pacific Region (from 2006), the Europe and Central Asia Region (from 2006), the South Asia Region (from 2008), and for Africa (from 2004). Historical threshold information was not readily available for the Latin America region or for the Middle East North Africa Region. Current regional threshold information was taken from the extranet, and as this is not up to date, it was supplemented by internal sources. 31 Data Sources - Project Level Prior Review Thresholds and Project Level Risk Project Appraisal Documents and procurement plans were scanned for project level prior review thresholds for goods and works, project risk, including residual project risk, procurement risk and residual procurement risk, on a sample basis for 59 projects, based on overall country selectivity criteria. IEG also drew upon additional project level prior review data in the Middle East and North Africa dashboard and in the procurement plan execution system (SEPA), adding eight more projects to the sample. IEG used the residual procurement risk rating when available, or, in its 71 absence, the procurement risk rating. Threshold data sources available to and used by IEG are summarized below: Table C.2. Prior Review and Procurement Method Thresholds Available to IEG Region Prior Review Threshold Information and Source Procurement Method Threshold Information and Source AFR Selected countries – 2004/2005 (RPM); 2012 (RPM) 2010 (external procurement website) 2012 (RPM) EAP 2006, 2011(EAP RPO website); 2012 (external 2006, 2011(EAP RPO website); 2012 (external procurement website) procurement website) ECA 2006, 2007 (four revisions in 2007), 2010, 2011 (ECA RPO 2006, 2007 (four revisions in 2007), 2010, 2011 (ECA RPO website); website); 2011 (external procurement website) 2011 (external procurement website) LCR Brazil, Mexico, Colombia – 2007 (RPM) Brazil, Mexico, Colombia – 2007 (email from LCR RPM), 2012 (external procurement website) MNA 2011 (external procurement website) 2011 (external procurement website) SAR 2008 (SAR RPO website); 2012 (external procurement 2008 (SAR RPO website); 2012 (external procurement website) website) Source: IEG. Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Africa. Data Sources—Governance Indicators Governance indicators used in the analysis include Transparency International’s Corruption Perceptions Index, the Kauffmann and Kraay Governance Indicators, and the World Bank’s County Policy and Institutional Assessment (CPIA). The Index scores provide a score, ranking a country on how corrupt its public sector is perceived to be on a scale of 1 to 10 (with 10 indicating no corruption). 32 Two Kauffmann and Kraay Governance indicators were used for this analysis: Government Effectiveness and Control of Corruption. The overall CPIA score and two sub indicators were used for this analysis: subindicator 13 Quality of Budget and Financial Management and subindicator 16 Transparency, Accountability and Corruption in Public Sector. 33 METHODOLOGY Sample Construction – Correlations with Governance Indicators Two samples were constructed, one from 2008 and one from 2011, for the comparison of Bank thresholds with external measures of country risk/country governance. The nonstandardized nature of the setting of thresholds across regions, and gaps in historically available information that varied considerably across regions are caveats to the sample construction. 72 Box C.5. Threshold Analysis – Sample Construction 2011 Sample: Europe and Cental Asia: Data from thresholds effective as of 2011 were used. Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement methods (ICB/NCB/Shopping). East Asia and Pacific: Data from thresholds effective as of 2011 were used. Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement methods (ICB/NCB/Shopping). South Asia: Data from thresholds effective as of February 2012 were used. Prior review thresholds are provided by procurement type (goods/works/consultants). Previous thresholds dated from 2008. Latin America and the Caribbean: Current (2012) thresholds were used. It was assumed that prior review thresholds were the same as NCB/ICB method thresholds in the absence of separate information. Middle East and North Africa: Current prior review thresholds were used (2012). Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement method (ICB/NCB/Shopping). Africs: 2010 procurement method (ICB/NCB) thresholds were used as 2011 prior review thresholds. 2008 Sample: Africa: RPM guidelines from 2005 outlining prior review thresholds for projects with three levels of risk were used for the 2008 sample. East Asia and Pacific: Data from thresholds effective as of 2006 were used. Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement methods (ICB/NCB/Shopping). Europe and Central Asia: Data from thresholds effective as of 2007 were used. Prior review thresholds are provided by procurement type (goods/works/consultants) and by procurement methods (ICB/NCB/Shopping). Latin America and the Caribbean: Threshold information for Brazil, Colombia, and Mexico. Other countries in the region were not included in the 2008 sample because data were not available. Middle East and North Africa: Historical threshold data unavailable –not included in the 2008 sample. South Asia: Data from thresholds effective as of 2008 were used. Prior review thresholds are provided by procurement type (goods/works/consultants). Sample Construction Issues: Diverse practices across Bank regions in terms of how prior review thresholds are set led to the construction of subsamples by IEG to maintain homogeneity. In the 2011 sample, thresholds from the Middle East and North Africa are divided into four categories by risk level of implementing agency (mirroring the format for Bank wide prior review thresholds established in 2009). Africa countries included in the 2008 sample had three risk levels (high, medium, and low risk), which also mirrors the format of Bank wide prior review thresholds from 2008. Some countries from South Asia used a range for their prior review threshold (for example, the 2008 prior review threshold for goods is “300,000 to 500,000”). IEG divided the thresholds into equivalent risk categories – for example, the 300,000 threshold was interpreted as the threshold for a high risk implementing agency and the 73 500,000 threshold was interpreted as a low risk implementing agency. For Europe and Central Asia and East Asia and Pacific Regions: Some thresholds were also set in terms of the first few projects reviewed (for example, Turkey 2011 prior review threshold for NCB goods is “First 2 & >500k”). In such cases the numerical threshold was considered the effective threshold and the “First 2” prior review requirement is not reflected in the analysis and correlations. Source: IEG analysis. Correlations between country level prior review thresholds and governance indicators were undertaken in clusters, reflecting different practices in setting thresholds across regions. For each coefficient, the vector of country level risk thresholds was correlated with each vector of country level governance indicators. A subgroup of countries was constructed to include those that did not differentiate thresholds by risk level of implementing agency. These countries were pooled with the overall sample, using three sets of assumptions, in the 2008 subsample, namely, that risk thresholds in these countries corresponded, in turn, to the high, medium or low risk implementing agency thresholds. Similarly, for the 2011 subsample, four series were constructed to accommodate high, substantial, moderate and low risk implementing agencies. Samples were also divided by category and method: ICB goods and ICB works, NCB goods, NCB works, to ensure consistent comparisons. RESULTS – ALL COUNTRIES • There are no strong correlations. The majority of the correlation coefficients are positive but weak/moderate in strength. There are a number of negative correlations, all very small in magnitude. • Correlations for NCB goods and works are lower than for ICB goods and works. • In most cases when comparing the correlation coefficients across the different levels of risk, the strongest coefficient is on the high risk thresholds, with lower coefficients on the substantial, moderate or low risk coefficient. 74 Table C.3. Correlations between Country Thresholds and Indices of Country Governance CPIA Transparency Subindicator 13. CPIA Subindicator International - Kauffmann and Quality of Budget 16: Transpar. Corruption Kraay - Kauffmann and and Financial Account.and Perceptions Regulatory Kraay - Control CPIA - Overall Management Corrup.in Pub. Sec Index Quality of Corruption 2008 ICB goods H (68) 0.51 (68) 0.45 (68) 0.32 (67) 0.46 (68) 0.57 (68) 0.25 M (66) 0.41 (66) 0.33 (66) 0.34 (65) 0.44 (66) 0.47 (66) 0.29 L (68) 0.13 (68) 0.09 (68) 0.19 (67) 0.28 (68) 0.18 (68) 0.22 ICB works H (68) 0.46 (68) 0.41 (68) 0.18 (67) 0.30 (68) 0.43 (68) 0.16 M (66) 0.47 (66) 0.42 (66) 0.22 (65) 0.30 (66) 0.42 (66) 0.17 L (68) 0.44 (68) 0.38 (68) 0.19 (67) 0.30 (68) 0.40 (68) 0.18 2011 ICB goods H (117) 0.33 (117) 0.32 (117) 0.20 (123) 0.12 (123) 0.28 (123) 0.15 S (110) 0.30 (110) 0.29 (110) 0.20 (116) 0.11 (116) 0.25 (116) 0.14 M (110) 0.16 (110) 0.14 (110) 0.12 (116) 0.04 (116) 0.13 (116) 0.05 L (117) 0.03 (117) 0.03 (117) 0.09 (123) 0.00 (123) 0.05 (123) -0.01 ICB works H (116) 0.41 (116) 0.36 (116) 0.24 (120) 0.24 (122) 0.40 (122) 0.20 S (109) 0.35 (109) 0.30 (109) 0.23 (113) 0.23 (115) 0.35 (115) 0.21 M (109) 0.29 (109) 0.36 (109) 0.20 (113) 0.18 (115) 0.29 (115) 0.13 L (116) 0.22 (116) 0.32 (116) 0.18 (120) 0.13 (122) 0.23 (122) 0.08 2008 NCB goods H (51) 0.39 (51) 0.39 (51) 0.25 (51) 0.27 (48) 0.45 (48) 0.14 M (49) 0.08 (49) 0.01 (49) 0.18 (49) 0.18 (46) 0.19 (46) 0.26 L (51) -0.11 (51) -0.19 (51) 0.06 (51) 0.04 (48) -0.02 (48) 0.20 NCB works H (54) 0.39 (54) 0.36 (54) 0.00 (54) 0.11 (54) 0.35 (54) -0.04 M (52) 0.38 (52) 0.36 (52) 0.01 (52) 0.14 (52) 0.34 (52) 0.04 L (54) 0.30 (54) 0.26 (54) 0.02 (54) 0.15 (54) 0.25 (54) 0.10 2011 NCB goods H (32) 0.29 (32) 0.09 (32) 0.26 (37) 0.14 (37) 0.44 (37) 0.29 S (28) 0.07 (28) -0.04 (28) 0.12 (33) 0.16 (33) 0.14 (33) 0.06 M (29) -0.1 (29) -0.14 (29) -0.02 (34) -0.08 (34) -0.09 (34) -0.09 L (32) -0.11 (32) -0.14 (32) -0.05 (37) -0.09 (37) -0.12 (37) -0.10 NCB works H (33) 0.38 (33) 0.26 (33) 0.27 (38) 0.21 (38) 0.51 (38) 0.35 S (28) 0.13 (28) 0.06 (28) 0.14 (35) 0.07 (35) 0.22 (35) 0.12 M (29) 0.03 (29) -0.02 (29) 0.06 (35) 0.00 (35) 0.08 (35) 0.03 L (33) 0.00 (33) -0.07 (33) 0.01 (38) -0.03 (38) 0.00 (38) -0.04 Source: IEG analysis. Note: All countries for which risk level data are available. For those countries and years which did not differentiate country risk by implementing agency, successive correlations have been undertaken assuming, in turn, that such risk levels correspond to each level on the three- (2008) or four- (2011) point scale for risk used in other countries. 75 RESULTS – SUBSAMPLE OF COUNTRIES (AFRICA AND LATIN AMERICA AND THE CARIBBEAN) IEG also analyzed results separately for the group of countries, in two regions especially, for which country threshold data do not differentiate thresholds by risk level of implementing agency (that is, there is only one threshold by procurement method/type for the country). Table C.4. Correlations between Country Thresholds and Governance Indices – Single Risk Threshold Countries CPIA CPIA – Overall subindicator 13. CPIA Quality of Subindicator Budget and 16: Transpar., Transparency Kauffmann Financial Account. and International - and Kraay - Kauffmann and Management Corrup. in Pub. Corruption Regulatory Kraay - Control Sec Perceptions Index Quality of Corruption 2008 ICB goods (43) 0.56 (43) 0.51 (43) 0.44 (43) 0.52 (43) 0.61 (43) 0.34 ICB works (43) 0.50 (43) 0.46 (43) 0.27 (42) 0.33 (43) 0.44 (43) 0.23 2011 ICB goods (102) 0.34 (102) 0.33 (102) 0.21 (105) 0.12 (105) 0.28 (105) 0.17 ICB works (101) 0.41 (101) 0.36 (101) 0.26 (103) 0.26 (104) 0.41 (104) 0.22 2008 NCB goods (26) 0.46 (26) 0.42 (26) 0.39 (26) 0.36 (25) 0.52 (25) 0.26 NCB works (29) 0.45 (29) 0.44 (29) 0.02 (29) 0.19 (29) 0.40 (29) 0.10 2011 NCB goods (20) 0.45 (20) 0.20 (20) 0.41 (22) 0.29 (22) 0.73 (22) 0.52 NCB works (22) 0.63 (22) 0.50 (22) 0.46 (24) 0.41 (24) 0.88 (24) 0.64 Source: IEG analysis. Note: These countries do not differentiate thresholds by risk level of implementing agency (ie there is only one threshold for the country), at the country level. Most consist of countries in the Latin America and the Caribbean and Africa Regions. • Although correlations for this subset of countries was generally higher than for all countries taken together, only a few are over 0.5. • As in the overall group, correlations for ICB goods and works are stronger overall in 2008 than in 2011. For NCB goods and works there is variance – for some indicators the correlation is higher in 2008, for others it is higher in 2011. • The indicator that has the strongest correlations by year, procurement category, and method is the Kauffmann and Kraay Regulatory Quality Indicator, and may best reflect the implicit criteria in terms of county governance/capacity that RPMs use in determining prior review thresholds. These data are displayed in Figure C.1 for the years 2008 and for one governance indicator, the CPIA score. These offer more granular country level insights and show 76 that while the highest prior review thresholds do tend to be found in countries that have relatively high CPIA scores, there are many thresholds that are very low among countries that have high CPIA scores, notably Armenia, Bhutan, Georgia, Macedonia, and Thailand from the 2008 graph and Chile, Georgia, Macedonia, Peru, Thailand, and Uruguay in the 2011 graph. Graphs based on other procurement methods and governance indicators show a similar pattern. Figure C.1. CPIA Scores and Correlation with Prior Review Thresholds (ICB Goods, 2008) Source: IEG compilation. A caveat to the analysis is that conclusions must take into account the quality of the underlying data, which is heterogeneous by region and not prepared on the same basis. Moreover, the dates at which revisions have occurred appear to be ad hoc, and have not been uniform across regions. In some regions, prior review threshold are also the method threshold (Latin America and the Caribbean, Africa, and some countries in the Middle East and North Africa), and in others this is not the case but relates to the challenges of interpreting threshold data. Sample sizes are small because of the absence of better historical prior review threshold information. 77 Figure C.2. CPIA Scores and Correlation with Prior Review Thresholds (ICB goods, 2011) 78 Post-Procurement Reviews Table C.5. Post Procurement Reviews: Project Sample Contract Value Country Project ID Project Title PPR FY ($) Africa Tanzania P055120 Transport Sector Support 2011/12 64, 458 Tanzania P111155 Zanzibar Urban Service Project 2011/12 694,848 Tanzania P111556 East Africa Public Health Laboratory 2011/12 109,705 Tanzania P114866 Secondary Education Development 2011/12 306,052 Ethiopia P101556 Electricity Access Additional Financing 2012 East Asia and Pacific Indonesia 7866 ID Empowerment of Urban Areas 2010 Lao PDR P120909 Upland Food Security Improvement 2012 989,981 Philippines P066076 Judicial Reform Support 2011 41,800 Europe and Central Asia Turkey 7539-TU Fourth Export Finance Intermediary Ln. 2012 12,227,940 Ukraine P095203 Second Export Development Project 2011 4,682,626 Azerbaijan P100582 Real Estate Registration Project 2011 Azerbaijan P083126 Real Estate Cadastre and Registration 2012 Middle East and North Africa Morocco P110833 Rural Roads II 2011 Morocco 7351 MOR Rural Potable Water and Sanitation 2011 Djibouti Morocco Consolidated Strategic Action Plans to Implement Tunisia Follow-up Activities Addressing Findings of PPRs and 2009 Yemen, Rep. of IPRs South Asia Bangladesh Consolidated PPR/IPR Summary contained in FY2011 Report covering 34 projects and 174 contracts 2011 9.7m India Consolidated Report on PPRs covering 24 projects and 1596 contracts 2012 113.84m Note: In some cases (Bangladesh, India, and Yemen) consolidated reports were compiled by the region, in others there were no consolidated reports and the analysis is based on individual PPR completions. PPR = Post-Procurement Review. 79 APPENDIX 3D POST PROCUREMENT REVIEWS Table C.6. Instances of Noncompliance in PPR/IPR Reports on Yemen (FY09–10) Coverage: Projects PPR IPR FY 09: FY09: Public Sector Governance (4 contracts), Health sector (17 contracts) 1 Project in each of the following Water and Sanitation (11 contracts), Transport (4 sectors contracts), FY10: Urban Development (2 contracts), Social Protection (11), Agriculture and Rural Development Rural Development sector ( 2 projects: 8 contracts) sector (2 projects; 20 contracts) FY10: Public Sector Governance sector (22 Education (2 contracts), contracts) Transport (5 contracts), Environment (8 contracts), Private Sector (2 contracts) Social Protection (16 contracts), Water and Sanitation sector (2 projects; 16 contracts) Total Number of contracts: 89 Total Number of contracts: 59 SUMMARY OF PROBLEMATIC FINDINGS A: Relating to the Borrower/Implementing Agency PPRs Inaccurate cost estimates (referential sources of unit prices not reliable); Recurrent overwriting of award decisions by the country’s High Tender Board (HTB) after the Bank NOs were granted; Weak procurement and contract administration capacity in project implementing units (exacerbated by high turnover of procurement staff). IPRs (i) Generalized average procurement capacity and poor (xvii) A case of non-compliance with GL’s requirement to specify procurement filing systems and contract management that in many liquidated damages for delayed completion/delivery; (xviii) a case of award cases led to extensive time overruns; made for a lower-priced option of the lowest evaluated substantially (ii) insufficiently comprehensive ToRs, Schedule of Requirements responsive bidder, not in accordance with the BER and the award and technical specifications; recommendation accorded with the Bank; (iii) unauthorized restriction in bidding documents for arithmetical (xix)A case where items re-advertised after cancellation due to change in errors exceeding 3% of bid price resulted in disqualification of requirements on the bids’ submission deadline, differed from those in the lowest priced/substantially responsive bidders; corresponding bidding documents; a case where currency of payment (iv) unjustified use of the SSS method and unauthorized use of specified in the contract was inconsistent with the contract price currency CQS and SSS in cases where the estimated contract value specified by the successful bidder in its bid; exceeded the thresholds established in the LA for the use of these (xx) A case where procured items were long stored in a government methods; location instead of distributed to the intended recipients; (v) lack of procurement-related documentation on file (e.g., proof of (xxi) A case where, during physical inspection, procured items could not advertisement in local newspapers, financial proposals and basis be found at the intended destination (and deficiencies noted with regard to for calculation of consultants’ fees and reimbursable expenses, the location of other items procured which caused them to rust; copies of performance securities when their furnishing was required (xxiii) In the particular case of one project implementing unit, prolonged in the contracts; lack of procurement staff and new management unaware of prior (vi) changes in the substance of bid (i.e., performance security, procurement activity; (xxii) poor to non-existent “after delivery” control of delivery period, payment terms, etc.) occurred after the bids’ receipt procured assets; (xxiii) expired Bank guarantees not returned to bidders; deadline; (xxiv) Required geographical spread of short-listed firms was not (vii) unguaranteed advance payments over 10%; complied; (viii) amount of performance security not increased as required by (xxv) Notification of award predated the Bank’s NO; the performance security provision consequently with an increase of (xxvi) Unauthorized suppression of provision on qualification of the bidder the contract price for additional works; as required alleging objective to develop local industry and due to low (ix) performance security improperly obtained in Yemeni Rials average cost of sub-projects; instead of the currency of the contract (US$) or a freely convertible (xxvii) A case where, though the bidding documents were accorded NO currency; by the Bank, payment terms were unacceptably set differently for “goods (x) required publication of evaluation results and contract award supplied from the purchaser’s country” from those for “goods supplied from was not made; (xi) fraud and corruption and audit clauses in abroad” (allegedly to avoid the use of LC in the case of local suppliers by bidding/contract documents were not in compliance with the agreed increasing slightly the advance payment); legal provisions; (xxviii) Obligations not originally included in the RFQ (that is, furnishing of (xii) modifications for aggregate increase by more than 15% of the performance security in the amount of 10% of the contract price), were original contract amount made without seeking the Bank’s NO; imposed to successful suppliers at the time the POs/contracts were 80 (xiii) physical inspection revealed that payments were made against issued; non-executed items of works and others with poor workmanship or (xxix) Opening of proposals occurred a week after the deadline for receipt not in accordance with the technical specifications; of proposals; (xiv) non-compliance with advertisement requirements and (xxx) Required prior reviews by the Bank not sought by Project minimum 30-day period allowed for preparation/ submission of bids; Management Unit staff also implementing another Bank-financed project (xv) a case of unauthorized use of the “shopping” method for works alleging not being aware that project-specific provisions applied; exceeding the threshold established in the LA; (xxxi) Details relating to “liquidated damages clause” not properly specified (xvi) a case where requirement to reduce the performance (four kinds of deficiencies found); delayed delivery of goods or completion guarantee after delivery/acceptance of goods was not complied and of works or services; 2% of the contract price was improperly retained as cash deposit for (xxxii) A case where liquidated damages were not deducted from the the duration of the warranty period; contract price payable on account of penalties despite a liquidated damages provision in the contract; (xxxiii) A case where changes to the scope of works to reduce the contract amount were made after the Bank had accorded NO; and (xxxiv) A case where extension of bid validity was requested from a limited number of bidders instead of from all bidders as required in the GLs. Source: Strategic Action Plan to Implement Follow-Up Activities Addressing Findings of FY09-FY10 PPRs and IPRs in the Middle East and North Africa Region. 81 Analysis of P-RAMS DATA DESCRIPTION The data provided for analysis covers the period since the implementation of Procurement Risk Assessment Management System (P-RAMS) in 2010 and ending in FY12 and consists of 794 eligible projects, of which 607 had at least one P-RAMS. Including multiple P-RAMS sequences, the number of eligible and actual P-RAMS rises to 917. COMPLIANCE AND COMPLETION For the purpose of this analysis a completed P-RAMS is one that provides information about both project and residual risk. There were 61 projects that listed their P-RAMS status as “in progress” but did not provide any information about project risk; therefore these observations were not counted as completed. Compliance rates were calculated by project, looking at the ratio of projects with at least one P-RAMS completed against the number of eligible projects (investment lending projects that were in the Project Concept Note phase on or after July 19, 2010). RISK MANAGEMENT AND DYNAMIC RISK MANAGEMENT Overall procurement risk is determined after the analysis of procurement risk by taking into account the 11 risk factors contained in the Procurement Risk Assessment Questionnaire—as well as other possible project risk factors (political climate, capacity, and so forth). After project risk is identified, the person completing the P-RAMS designs a risk mitigation plan with steps to address the risks defined in the previous stage, and enters each mitigation measure into the Mitigation Measures Action Plan in the template. The risk that may remain after the risk mitigation measures are carried out is called residual risk. First and Last P-RAMS: “Bookend Analysis” To analyze the risk management component of the P-RAMS, the sample of P-RAMS was tabulated based on initial project procurement risk. Of the 607 P-RAMS completed, 66 percent were categorized as having either high or substantial project procurement risk. Only 7 percent of P-RAMS exercises reported low initial project procurement risk. Risk was examined at the start and end of the available P-RAMs sequences for the project, comparing project risk to residual risk. For projects with 82 two or more P-RAMS exercises the project risk for the first P-RAMS was compared to the residual risk for the last P-RAMS, that is, using a “bookends” approach. Table C.7. Residual Risk by Project Risk for All Projects with P-RAMS Three Four or Five One Sequence Two Sequences Sequences (N = Sequences PROJECT RISK (N = 393) (N = 65) 13) (N = 11) ASSESSED RISK : High Freq. % Freq. % Freq. % Freq. % 134 21 2 4 RESIDUAL RISK High 18 13% 5 24% 0 0% 0 0% Substantial 91 68% 14 67% 2 100% 4 100% Moderate 25 19% 2 10% 0 0% 0 0% Low 0 0% 0 0% 0 0% 0 0% ASSESSED RISK: Substantial Freq. % Freq. % Freq. % Freq. % 124 23 7 3 RESIDUAL RISK High 1 1% 0 0% 0 0% 0 0% Substantial 34 27% 5 22% 3 43% 3 100% Moderate 86 69% 18 78% 4 57% 0 0% Low 3 2% 0 0% 0 0% 0 0% ASSESSED RISK: Moderate Freq. % Freq. % Freq. % Freq. % 116 16 3 2 RESIDUAL RISK High 0 0% 0 0% 0 0% 0 0% Substantial 1 1% 1 6% 0 0% 0 0% Moderate 87 75% 9 56% 2 67% 2 100% Low 28 24% 6 38% 1 33% 0 0% ASSESSED RISK: Low Freq. % Freq. % Freq. % Freq. % 19 5 1 2 RESIDUAL RISK High 0 0% 0 0% 0 0% 0 0% Substantial 0 0% 1 20% 0 0% 0 0% Moderate 1 5% 0 0% 0 0% 0 0% Low 18 95% 4 80% 1 100% 2 100% Source: IEG calculations from P-RAMS database provided by OPSOR. Note: These figures are for the “bookends” of risk, the initial risk rating for the first P-RAMS and the residual risk rating for the last P- RAMS. (n = 482). The main findings are: • The exercise tends to show greater difference in risk levels (as measured by the difference in initial and residual risk) for projects with high and substantial initial project procurement risk. Seventy-eight percent of projects (249 out of 318) with an initial assessed risk of high or substantial had a difference in residual risk rating of at least one risk category. 83 • Most projects with moderate risk tended to not see a difference in project residual risk, although 25 percent of projects (35 out of 137) with an initial risk rating of moderate had a residual risk rating of low. • Projects with a low initial risk rating remained with low residual risk rating 93 percent of the time (though two saw an increased residual risk rating). • The data does not show an increased decline in risk over multiple P-RAMS. For example, zero projects with four or five P-RAMS that had an initial risk level of high saw a reduction to moderate or low residual risk. This finding is replicated for projects that have an assessed risk level of substantial as well. Examination of Each P-RAMs in a Series: “Sequence Analysis” The last unexpected finding raises some questions as to the value added of the dynamic P-RAMS exercise. A closer examination was therefore undertaken looking at the detailed sequencing of risk from the first P-RAMS to the last P-RAMS for each given project where two or more sequences existed. Results are indicated in Table C.8. Table C.8. P-RAMS Risk Ratings for a Sample of Projects with Two or More P-RAMS 1st P-RAMS 2nd P-RAMS 3rd P-RAMS 4th P-RAMS ProjectID ProjectRisk ResidualRisk ProjectRisk ResidualRisk ProjectRisk ResidualRisk ProjectRisk ResidualRisk P094360 High Substantial High Substantial High Substantial High Substantial P123923 High Substantial High Substantial High Substantial Substantial Moderate P100530 Substantial Moderate Substantial Moderate Substantial Moderate Substantial Moderate P120349 Moderate Moderate High High Moderate Moderate Moderate Moderate P122319 Low Low Low Low Low Low Low Low P125689 High Substantial High Substantial High Substantial P123201 Substantial Moderate Substantial Substantial Substantial Substantial P126537 Moderate Low Moderate Low Substantial Moderate P122492 High Substantial High Substantial P122008 Moderate Moderate Moderate Moderate Source: IEG tabulation from P-RAMS database provided by OPSOR. Note: Selected by IEG to show the variation in patterns between project and residual risk for projects with multiple sequences. • Only one project showed the expected pattern of consistently reduced residual risk with additional P-RAMS. • The most common pattern seen was the same project risk rating through every P-RAMS in the sequence as well as the same residual risk rating through every P-RAMS. • The data do not show an increase in reduction of risk over multiple P-RAMS. For example, zero projects with four or five P-RAMS that had an initial risk level of high or substantial saw a reduction to moderate or low residual risk. 84 Of the 65 projects that had two P-RAMS, 21 showed no difference between initial project procurement risk and residual risk in any project phase. Four of these 21 projects with no change in risk were categorized as low risk projects; it can be assumed that the risk mitigation plan could not have changed the residual risk rating. Of the remaining 44 projects that did see some change in risk level between initial project procurement risk and residual risk, 39 projects had identical P-RAMS risk ratings in both sequences with the residual risk always being noted as lower than the project risk (for example, an initial risk rating of high and residual rating of substantial, for each project phase). At first glance it seems counterintuitive that a project would have a residual risk rating of moderate in one sequence followed by a project risk rating of substantial (more than moderate) in the next sequence. One possible explanation for this is that the risk mitigation plan from the previous phase was not completely implemented and the risk was not reduced as planned. This could also reflect too frequent changes in P-RAMS sequences. Another possibility is that changes in risk might be expected to occur over the course of implementation, and projects with P-RAMS have not been under implementation for any length of time, due to the newness of the tool (see Table C.9, which shows that only a few projects are under supervision). A third possibility is the inertia in terms of changing entries, and perhaps limited scrutiny of P-RAMS sequences. These potential explanations for the two P-RAMS sequence projects could apply to the three-P-RAMS projects, too. Table C.9. Sequence and Phase Tabulations Sequence 1 2 3 4 5 Total Phase Freq % Freq % Freq % Freq % Freq % Freq % Project Concept Note 215 40 0 0 0 0 0 0 0 0 215 32 Quality Enhancement 67 12 42 47 0 0 0 0 0 0 109 16 Review Decision Meeting 97 18 20 22 13 52 0 0 0 0 130 19 Negotiations 60 11 17 19 8 32 10 100 0 0 95 14 Supervision 104 19 10 11 4 16 0 0 1 1 119 18 Total 543 81 89 13 25 4 10 1 1 0 668 Source: IEG calculations from P-RAMS database One issue with analyzing the “spread” of risk rating data in the P-RAMS is that all of the measures are subjective, assigned by the procurement specialist or task team leader on the project. A complementary analysis looking at whether the risk mitigation measures were carried out and effective would be useful in addition to examining subjective assessments of risk provided in the P-RAMS. Table C.9 shows the distribution of P-RAMS through the project cycle, as of FY12. 85 CONCLUSION Overall it remains unclear whether the P-RAMS tool is performing as was intended. Though the program has yet to achieve the desired 100 percent compliance, it seems that goal will be achieved in the very near future. An analysis of the most important element of the P-RAMS – the risk measurement and mitigation shows mixed results. If residual risk over the cycle of the project is expected to decline, the analysis shows it has yet to occur. If however, risk is basically static and the goal of the P-RAMS is to mitigate risks where possible then the exercise is more successful (see the “bookend” versus “sequence” analysis in preceding pages). The usefulness of P- RAMS ultimately will depend on how much its findings are used for risk analysis. Further complementary qualitative analysis is required before a more complete conclusion can be drawn. Box C.6. INT Guidance to Staff on Procurement Risk INT’s draft paper on potential red flag issues, mostly alluding to procurement, offers a number of suggestions to operations staff: 86  Establish an integrity due diligence checklist for implementing agencies: National procurement systems could be strengthened by adding integrity checklists as part of Bid Evaluation Reports. OPCS could set standards, and the regions could offer training to local implementing agency staff in operating the checklists.  Define Bank Due Diligence in Prior Reviews: Especially as management is now suggesting that transaction-level fraud and corruption due diligence be de-emphasized, it becomes important that the general principles and what to look for in the prior review of the most risky transactions be more clearly defined in a revised OP/BP11.00.  Incorporate a Red Flag Checklist into Post-Procurement Reviews: The suggestion is to amend the 2002 Office Memorandum which establishes the basis for handling integrity risks in Post- Procurement Reviews to change the basis for the review from contract size to contract risk rating; provide guidance on detection of high-risk transactions (for example, single bidder, close to threshold amounts, existence of procurement complaint); attaching a red flag checklist; ensuring that where consultants are used to conduct Post-Procurement Reviews they are trained in red flag methodology.  Disclose Past Performance in PADs and ICRs: The suggestion is that particularly in the case of successor projects implemented by the same project implementing unit, the procurement performance of contracted companies be reflected in the ICRs of one project and the PAD of the ensuing project. A scorecard could be developed containing such items as: actual cost compared to cost estimates at bidding; percentage of bids with only one or two bidders (not clear if this refers to the bidders or the project implementing unit); number of transactions with overruns above 6 months; number transactions with procurement complaints.  Avoid Conflicts of Interest in Recruiting and Deploying STCs: Project staff should avoid giving unchecked procurement discretion to STCs who may or may not be in a potential COI situation. . 87 Appendix D. Procurement Tracking Systems, Time Analysis, and Africa Region Review Systems BANK-WIDE INFORMATION – EARLIER VERSIONS OF FORM 384 Earlier versions of the Form 384 template indicate a stronger emphasis on collecting procurement process data, including a number of dates regarding the bidding process as well as contract co-financing information. Before 2001 the Operations Procurement – Form 384 selection screen provided data entry tabs such as Activity Plan/Monitor, Project Plan/Monitor, Evaluation Report, Contract Evaluation, Form 384 and Project Information. A big part of the data populating the form was captured from information entered via these tabs earlier in the procurement process. The actual Form 384 data entry was divided into two separate sections for goods and works and for consultant services, each providing sections for entry of basic data, supplier/firm data and currency data. A number of contract dates were collected such as bid opening/proposal submission, evaluation no objection, contract received, contract signature and Form 384 date. In addition, fields for delivery completion target/actual dates, completion percentage, exchange rate and co- financing information were provided. Users were asked to enter the country codes of the next three lowest bidders/firms considered for contract award in addition to supplier/firm data of the winning bid. The 2001 revision of Form 384 came embedded in a general overhaul of the procurement information system aimed at reducing the amount of data entry in the system. Operations Procurement – Form 384 selections screens were trimmed down to Project Information, Form 384, Post Review and Activity Planning, with some of the tabs capturing extended information to pick up data entered elsewhere prior to the system revision. Cofinancing data were moved to their own tab and the following fields removed: delivery completion target and actual dates, completion percentage, completion as of date and one of three country codes to identify next lowest bidders. While aiming for simplification of data entry, the incentive for procurement data collection seemed to have remained a priority in 2001 as additional fields were opened for data entry: implementing agency, procurement method, borrower bid reference number, bid opening date, domestic preference indicator, award affected by preference indicator, bid evaluation report no objection date, prequalification of bidders required indicator, fixed price contract indicator, two stage bidding indicator, bid number, bidding description and contract description. In the 89 consultant services section no fields were removed from data entry but new fields were added such as the check box for misprocurement and allow for contract signature date before no objection date. Overall it seems that while a few key variables have been dropped from bank wide data collection in the 2001 revision, other indicators have been added reflecting a possible shift in specific monitoring needs but not a general reduction of data collection. This seems to have changed significantly with the 2009 revision when Form 384 became a single data entry screen, delinked from procurement bids/activities information and simplified with a broad stroke including the removal of all but key data entry fields necessary for basic procurement reporting. Form 384 in its latest version is divided into four sections: basic data, contract price, supplier firm/individual details, and contract comments. No objection date and contract signature date are the only contract dates collected in addition to basic information like procurement method and sector allocation. It is no longer necessary to enter the actual currency payment terms with the contract information. While the form no longer requires bidding activity other than the winning firm or individual, the most noticeable element added is the integrated anti-money laundering check which must be executed in order to complete Form 384. On entry of data the new Form 384 provides a list of suppliers that have already been cleared for the respective project and, if the supplier is not listed, the anti- money laundering check process is activated. Discussions with Bank’s procurement personal suggest that the 2009 revisions and simplifications of the Form 384 were in line with a general move away from extensive control on the contract accounting side within the Bank. However, recent Bank efforts indicate an increased interest to put in place an integrated bank-wide procurement information system and invest in more comprehensive data collection. REGIONAL PROCUREMENT TRACKING SYSTEMS – NATURE OF REPORTS PRODUCED Latin America and the Caribbean: SEPA—System Reports and Borrower Interaction 34 As a management information system SEPA allows users to create reports, approve procurement plans and inform the parties concerned about the status of execution thereof. The system provides a platform for interaction between the task team leader and the implementing agency, while at the same time providing an audit trail of the process. The borrower enters and updates the procurement plan according to the loan or grant agreement, the Bank is then prompted by a system-generated notification to furnish its no-objection or provide comments. 35 Once approved, the procurement plan is automatically updated on the SEPA website. 90 SEPA helps both the Bank and its borrowers to ensure that detailed procurement plans are prepared, and to identify variations between planned and actual procurement activities. Upload of information into SEPA usually starts with the first procurement plan of a project that covers the first 18 months. 36 According to the Bank’s guidelines, procurement plans are to be updated by the Borrower every year. Contracts previously awarded and those to be procured in the following twelve months are required to be included (World Bank 2011e). SEPA staff indicated that most executing agencies update their procurement plan in SEPA on average once per month. Procurement plans, updates and modifications are subject to the Bank’s prior review and no objection before they are posted and have to be published on the Bank’s external website. Procurement plans approved in SEPA are automatically published on the SEPA public website. They have to, however, be manually migrated to show up in the public documents on the Bank’s external website (http://www.worldbank.org), which SEPA staff do on a regular basis. Data in SEPA are populated by the borrower. Some date fields, such as estimated dates for procurement steps, and estimated amounts, are mandatory for procurement plan approval while others, including actual dates and amounts, hinge on the readiness of the borrower to enter additional information. Actual dates of contract execution do not require new approval by the Bank and can be entered by the borrower at any time. Africa: The Procurement Cycle Tracking System—PROCYS The principal function of PROCYS has been that of a management information system that measures responsiveness of participants in the procurement process, and monitors aggregate trends in countries and sectors to identify bottlenecks. Unlike SEPA it channels communications including their attachments through the system recording dates automatically. It is now partially integrated with other Bank systems, in the sense that the filing of no objection notices and related documents into the Bank system is also automated. It generates several managerial reports, including monthly no-objections (by country/by stage/average time taken); quarterly reports on RPM/OPRC No-Objections, status reports on all RPM/OPRC cases, and reports on average time taken per project. Duration of process time is tracked starting from initiation, i.e. when the first request arrives by email to PROCYS, to approval, that is, issuance of no objection. Box D.1. PROCYS Workflow The process is initiated when the borrower sends a request for procurement review to PROCYS via email; the Bank task team receives an email notification with a link to the request in the system. The task team leader/team reviews and where necessary submits it to the appropriate level of 91 procurement specialist. The comments or no objection are then communicated to the borrower by the task team. Both Bank staff and clients can retrieve project information via the web-based system. PROCYS stores communications with a date and time stamp. However, the unique transaction request number cannot be linked to contract identifiers used in other Bank systems tracking contract data, such as the Bank-wide Form-384. At the time a no-objection is issued, for any stage of the procurement process, all communications including attachments are filed under the project number. Source: IEG. Middle East and North Africa: The Procurement Portfolio Dashboard The procurement portfolio dashboard is available to Bank staff for download from the region’s intranet site in the form of Excel spreadsheets, available from October 2008. Database macros allow users to produce management reports filtered for information for the entire region, for a selected country department, country, or procurement staff. 37 A core element of the dashboard is information on project-level procurement activities planned and executed by procurement specialists, such as tracking compliance with scheduled procurement related activities, including procurement post reviews, procurement plan updates, P-RAMS status and complaints. Project-level data further include committed, disbursed, and cancelled project amounts, procurement method and prior review thresholds, procurement risk ratings, and task team leader and procurement specialist names. Data related to specific contracts or procurement stages are not collected. A main element of the dashboard is information on project level procurement activities planned and executed by procurement specialists, such as tracking compliance with scheduled procurement related activities, including procurement post reviews, procurement plan updates, P-RAMS status, and complaints. Project- level data further include committed, disbursed and cancelled project amounts, procurement method and prior review thresholds, procurement risk ratings, task team leader and procurement specialist names, among others. Data related to specific contracts or procurement stages are not collected. However, the summary reports on the dashboard are not presented in a way easily accessible for interested users, as they lack explanatory information such as indication of the time period covered and general legend. SUMMARY The preceding brief summary illustrates that at present from the management information system point of view it is not possible for central procurement staff to track procurement processes Bank-wide, in terms of parameters relating to efficiency of implementation. Only Form 384 is uniform Bank-wide, but it provides a very 92 limited amount of information, on a subset of contracts, and Bank and borrower contributions are not separated. Moreover its use as information of contract level disbursement has been reduced. Form 384 does not provide a platform that is accessible to borrowers or any persons external to the Bank, and its internal use is not management friendly as it does not process any automated reports. Other regional systems each have their aims and objectives, and reflecting these, advantages and disadvantages. Although the procurement anchor is aware of these difficulties, it has apparently been difficult to develop a system that is integrated with the Bank’s other platforms. Table D.1. Comparison of Procurement Monitoring Systems by Region LCR / ECA: Procurement AFRICA: Procurement MENA: Procurement Bank-wide: Form 384 Plan Execution System Cycle Tracking System Portfolio Dashboard SEPA PROCYS Database as clearinghouse Borrower enters and updates for communications between TTL or certified project Updated monthly by Unit of Entry Procurement Plan for the the borrower and the bank on team member staff in headquarters project procurement processes, initiated via email by borrower Centralize information based on information Individual contracts subject Monitor Procurement Plan Individual contracts subject to in Bank systems and to prior review execution prior review information on projects by team Approve Procurement Plan Contract award information Track response times of through the system; provide on contracts above the prior Clients, Task Team, public access to basic review threshold Procurement Specialist information GENERAL PROJECT LEVEL DATA Country yes yes yes yes Loan / Credit No yes yes yes Project ID yes yes yes yes Project Status (Pipeline yes yes /Active / Closed) Loan Amount/ Credit/ Grants Total Committed AND Local counterpart amount Total committed and (IBRD + Grants) recorded; including origin Grants currency and exchange rate Disbursed in FY and Disbursed Project Total Amount Total undisbursed Undisb. balance Cancelled Total cancelled Value of PPR Aggregated Contract Value Contracts Reviewed Loan Effective Date PCD Review Date Project Original AND Actual Appraisal Date Key Project Dates Closing Date Approval, Effective Project actual closing date and Closing Date 93 LCR / ECA: Procurement AFRICA: Procurement MENA: Procurement Bank-wide: Form 384 Plan Execution System Cycle Tracking System Portfolio Dashboard SEPA PROCYS Estimated time per P-RAMS Date procurement stage at least one and up to three TTL TTL name per project TTL name per Project TTL indicated Primary and Procurement Specialist up to three PS indicated PS name per project Secondary PS Agreement Type (IBRD, yes IDA, other) Lending Instrument (DPL, yes P4R, IL, other) Major Sector and Sector General sector code per Sector Information Major Sector and Sub-Sector Board project Number of Borrower, Executing Agency, Implementing Agencies Implementing Sub-Executing Agency Agencies PROCUREMENT PROJECT LEVEL DATA Amounts and Location Project Level Prior Review Threshold amounts set by the of Prior Review Threshold Bank AND by the Borrower Threshold (Proc Plan or Legal Agreement) Amounts and Location Project Level Procurement Threshold amounts for use of Review Threshold Method Threshold with each procurement method (Proc Plan or Legal Agreement) Number for Procurement Plan entered and Procurement Plans updated in SEPA by executing status current, agency; upon approval outdated and not Procurement Plan Status automatic publication on SEPA required, date of external website; historic current and due date procurement plans available next Procurement Plan Proposed ISR Rating, Last ISR Rating from Project Procurement Risk SAP, Procurement Rating Risk Rating, # Projects at Risk #P-RAMS in progress, archived and required; Project P-RAMS Status Share of projects per risk categories H, S, M, L and “no info” Procurement Flag yes all contracts in Procurement only contracts marked as all contracts in Contracts covered only prior review contracts Plan prior review Procurement Plan PPR completion actual vs planned by country and by person, Project Level Procurement indicates if PPR required for PPR/IPR action plan Post Reviews (PPR) project implementation, (scheduled / completed) Number of Post Reviews and Supervision Missions 94 LCR / ECA: Procurement AFRICA: Procurement MENA: Procurement Bank-wide: Form 384 Plan Execution System Cycle Tracking System Portfolio Dashboard SEPA PROCYS PROCUREMENT CONTRACT LEVEL DATA Contract Description yes yes yes yes Contract ID assigned by each contract transaction is Contract ID SAP and Borrower country Contract Reference Number assigned a request number reference Estimated contract value and Total contract amount and Individual Contract Value actual contract amount and Estimated contract value amount used payments Procurement Method (ICB, NCB; for consulting QCBS, yes yes yes QBS,FBS,LCS,CQS, SSS) Procurement Category (Goods, Civil Works, Cons. yes yes yes Services) Clearance Level (PS, RPM, yes OPRC) Bidders Nr of bids received and Awarded: name, country of firms considered, country of nationality, bid amount/total nationality of winning bid, amount paid; Shortlisted: bidder eligibility status name, country of nationality & (Anti-Money Laundering, amount offer list of debarred firms, etc.) Key Dates and Elapsed Borrower issue of specific Contract award date Date of first action Times procurement notice Borrower first submission to Signature of contract Bank of draft bid (preQ) Date of most recent action documents* Borrower delay (Average time Bank final no objection to draft Form-384 sent date taken by borrower beyond bid (PreQ) documents* normal processing time) Task team time (average time Borrower issue of Bid (PreQ) taken by task team to review documents* steps ) Procurement time (average Bid Opening* time taken by PS/RPM to review steps) Borrower submission to Bank Average Response Time for of Bid evaluation report* No Objection Bank No-Objection to Bid Number of Stages Given No Evaluation Report* Objection idle time (duration between today's date - date of last Signature of contract* action, usually defined if pending with borrower between two stages) Contract Completion* Total time client *stages vary by procurement Total time with TTL / PS / method and category Client Estimated and actual contract Average time with TTL / PS / dates can be entered Client Total days of process between # iterations (Number Bid Opening and Signing of interactions between TTL and Contract borrower / TTL and PS/RPM) 95 LCR / ECA: Procurement AFRICA: Procurement MENA: Procurement Bank-wide: Form 384 Plan Execution System Cycle Tracking System Portfolio Dashboard SEPA PROCYS Management type reports Procurement Plan monitoring Reports for the entire Average Response Time for on contract level for selected region, by country, by No Objected Cases project CMU, by person (PS) Number and amount of ex post Number and Amounts contracts initiated in specific Number of stages given No of supervision, lending time period by category for Objection closed and total selected project projects Report of Management Processes with elapsed times Response times by sector for Post Review Schedule between stages on contract selected country (Planned and Actual) level for selected country Portfolio Review Report with aggregate amounts for Cumulative PPR Support Effort Hours per case processing stage by project, Progress country and province Complaints: Number Number of contracts by of minimum, category, method and stage Monthly reports on No- maximum, average (expected, in process, Objections (by country/by days pending for completed, canceled) for stage/average time taken) status pending and selected country and project closed Quarterly reports on RPM/OPRC No-Objections General Procurement Plan report indicating total number PROCYS also provides an of projects and procurement external website for clients for plans by country for selected real-time status of all their project status (in preparation, requests sent to the Bank active, closed) and year Public reports Publication of Procurement Plans on SEPA public website Aggregated Reports of contract values by procurement method/category for selected values (country, project, sector, province) 96 Procurement Efficiency at the World Bank – Elapsed Time Analysis DATA COLLECTION AND SAMPLE CHARACTERISTICS Information on processing times of procurement contracts in the Bank is not available in any comprehensive centralized format that can be used by Bank management to track the procurement process for all regions or contracts. Data are highly decentralized and typically maintained by individual country procurement offices. Although procurement checklists are required to be maintained, which would serve as summary sheets, and could deliver dates on processing steps, these are often not available. IEG therefore conducted its own data collection for this analysis, linked to its field visits, and based on a template tracing critical dates in the “no-objection” process. 38 Detailed contract processing data at the country/contract level are in some cases maintained by the procurement coordinator’s offices and sometimes by sector units. Each regional office has its own format in terms of specific fields of information tracked, date conventions and, clearly, currency units. To construct a representative stratified sample IEG suggested a specific list of prior review contracts to each field visit country office, by procurement method and category, by size, and across a sample of fiscal years from 2007 to the present. However, information received is concentrated in 2010–12 with the year 2011 representing 32 percent of all observations in the sample. The response rate from country offices was highly variable in quantity and quality, ranging from 107 contracts received from Azerbaijan to four from Tanzania. Often, not all date fields were filled. Of 502 contracts received, only 201 provided the requested information on all date fields from Issue of the Specific Procurement Notice to Contract Signature. 39 By far the most frequently reported dates were the no objection to the bid evaluation report and the contract signature which are also provided to the Bank-wide Form 384. In regions where a central data repository for the procurement process has not been established, staff in the field indicated that dates for procurement steps prior to the no objection to the bid evaluation report have to be extracted from procurement documents and communications with country clients. It appears that information on all steps in the procurement process is not readily available in all countries. Clearly this affected the scope and quality of analysis that could be undertaken, as well as reliability of results obtained and will have to be a caveat to their interpretation. The request for contract sample data sent to the field visit countries was separated by key processing dates for Goods and Works (ICB, ICB with preQ; NCB, NCB with Pre-Q), Goods and Works (ICB/NCB Two Stage) and large values of consultant 97 services (quality- and cost-based selection, quality-based selection, Fixed Budget, least cost selection, consultant qualifications section, single source selection). For purposes of this analysis the respective dates for each procurement step were merged across ICB/NCB, ICB/NCB two-stage and consultant services selection allowing for global data set as well as sub set analysis. Due to a low response rate for information on contracts involving ICB/NCB two-stage and pre-qualification (evaluation reports), data provided for these groups was integrated with the overall ICB/NCB data. For all contracts dates for final Bid Evaluation Reports were used to construct time intervals. 40 Contract values were converted to dollars using monthly average exchange rates. 41 The fiscal year of the contract was determined based on the date of the Bank’s No Objection to the Bid Evaluation Report. MEASURES OF AVERAGE ELAPSED TIMES IEG’s analysis focuses primarily on two overall elapsed times: borrower issue of specific procurement notice to contract signature and first submission to the Bank of draft bid (preQ) documents to the Bank’s no objection to the bid evaluation report. IEG also attempts to selectively review two intermediate steps in the procurement process, to the extent that data permit: first submission to the Bank of draft bid (preQ) documents to the Bank’s final no objection to the draft bid (PreQ) documents and borrower submission to bank of bid evaluation report to the Bank’s no objection to the bid evaluation report. Because of data limitations, results for different intervals may refer to different subsets in the data set. SELECT FINDINGS – AVERAGE TIME The average days between the borrower’s first submission of the draft bidding documents to the Bank’s final no objection to bid evaluation report of the contract was 253 days. However, the variable displayed a high dispersion from the average, with a standard deviation of 160. Though 50 percent of the contracts included in this analysis go from the submission of the bidding documents to the No Objection to the bid evaluation report in less than 208 days, IEG’s analysis indicates the existence of contracts with significant longer duration, for two contracts even more than 900 days, driving up average processing times. Figure D.1 shows the distribution of data points for borrower issue of specific procurement notice to contract signature and first submission to the Bank of draft bid (preQ) documents to the Bank’s no objection to the bid evaluation report compared to the same intervals without those contracts that make for the highest five percent of elapsed times. 98 Figure D.1. Elapsed Time Distributions of Procurement Intervals All Observations 0 500 1,000 1,500 2,000 Excluding upper 5 percent 0 200 400 600 800 Submission BD to Bank-No Objection BER Issue Specific Procurement Notice-Signature Contract Source: IEG analysis of sample contract data. Notes: Two intervals are examined in each box: (1) Borrower Issue of Specific Procurement Notice to Contract Signature and (2) First Submission to the Bank of Draft Bid (preQ) Documents to the Bank’s No Objection to the Bid Evaluation Report The vertical line in each box denotes the median; horizontal lines denote “adjacent values.” The offset of the median towards the left and the ‘long tail’ of the distribution to the right describe the skewed distribution. Dots indicate outliers. More than a quarter of the processing time from submission of bidding documents until contract award is due to the preparation of the bid evaluation report after bid opening. In the case of consultant services more than 40 percent of processing time is dedicated to this step in the procurement process, likely because for the quality- and cost-based selection procurement method two documents need to be prepared by the borrower; an evaluation of technical proposals and, after the subsequent opening of financial proposals, a combined evaluation report. Both reports require a no objection by the Bank. This remains true even after excluding those contracts that bring in the highest five percent of elapsed times per category. Results in terms of processing times by category, method, and sector are given in Figure D.2. 99 Figure D.2. Average Elapsed Times in Procurement Steps from Submission of Bidding Documents by Borrower to Contract Signature by Category, Method, and Sector (days) Source: IEG analysis of sample contract data. SELECT FINDINGS: VARIATION IN ELAPSED TIMES Comparing the arithmetic mean and the mode of some of the distributions, the former is typically higher, due to a proportion of contracts that are more time consuming. A key finding is that the procurement process from the time the borrower issues the specific procurement notice to signature of the contract takes more time, on average, for procurement method quality- and cost-based selection used for the selection of consultant services, as compared to ICB and NCB. On average, the processing times to procure consultant services are more than 100 100 percent greater than for contracts awarded using NCB. And differences are not merely that of the average time taken, but also in the number of outliers. Figure D. 3. Average Elapsed Days in Procurement Steps from Submission of Bidding Documents by Borrower to Contract Signature, by Method (days) Source: IEG analysis of sample contract data. Note: With controls for outliers. The distribution of elapsed times for the intermediate procurement step from submission of draft bid documents to the bank’s no objection to the final bid evaluation report shows the existence of extreme values for all procurement methods, with especially high values for ICB and quality- and cost-based selection contracts (Figure D.3). Consultant services and ICB contracts differ only by a few days in average processing time taken for this interval. The contracts that contribute the upper five percent of processing time for consultant services selected by the 101 quality- and cost-based selection method take more than three times longer than the bottom 50 percent of contracts. REGRESSION ANALYSIS AND RESULTS Dependent Variable: Elapsed Time Interval (1) Issue of the Specific Procurement Notice to Contract Signature IEG undertook a simple regression analysis to obtain possible explanations for factors affecting elapsed times in procurement, considering alternative dependent variables in the regression specification. A first interval analyzed is the time taken from the Issue of the Specific Procurement Notice to Contract Signature. Explanatory variables included contract attributes: (contract value, procurement method, category of good or service being procured and major sector); control variables included country specific variables (gross domestic product, poverty rate). Results indicate a statistically significant positive relationship between contract value and processing time, robust to the introduction of controls (Table D.2). Coefficients denote marginal effects, and results suggest (specification 5) that for every $10 million increase in contract value, the number of days for processing increases by 14.6, independent of all other contract attributes and across all countries and regional offices. Results also indicate significantly lower elapsed times for NCB contracts compared to ICB. Dependent Variable: Elapsed Time interval (2) – Submission of Draft Bid Documents to No Objection to the Bid Evaluation Report Data on this time interval were available for 213 contracts. 42 The positive relationship between contract value and elapsed time remains statistically significant, and robust to the addition of controls (Table D.3). Marginal effects are broadly similar - the regression coefficient in Specification (4) implies that for every $10m increase in contract value, the number of days for Bank No Objection increases by 10.6 days, independent of all other contract attributes and across all countries and regional offices. The coefficient for method—NCB—as compared to other methods, is also significant, and the coefficient implies that it takes the World Bank, on average, 65 days less to issue no objection for NCB method contracts as compared to ICB. The second specification also adds country governance as a control variable, in the form of the CPIA. Results indicate that country level governance factors have a powerful influence on elapsed time, with lower elapsed times for countries with higher CPIA scores. Table D.2. Analysis of Processing Time from Issuing of Notice to Contract Signature (1) (2) (3) (4) (5) Variables SPN to Contract SPN to Contract SPN to Contract SPN to Contract SPN to Contract 102 Signature Signature Signature Signature Signature Contract 1.42e-06*** 1.55e-06*** 1.52e-06*** 1.47e-06*** 1.46e-06*** Value (1.12e-07) (1.34e-07) (1.41e-07) (9.90e-08) (1.01e-07) Social Sector 3.442 -18.05 -27.32 (51.06) (51.58) (52.15) Infrastructure 95.80** 65.00 64.89 Sector (44.10) (47.22) (52.47) Economic -65.66 -46.46 -49.65 Sector (46.38) (45.66) (47.34) Civil Works -8.500 52.32 48.90 Contracts (56.46) (61.05) (66.33) Consultant 135.1** 112.9* 70.51 Contracts (63.38) (66.28) (75.10) Goods 14.62 37.19 34.09 Contracts (59.56) (60.09) (63.01) NCB Method -112.1*** -113.5*** (39.27) (39.52) Cons. Method 57.45 93.17 (63.32) (66.99) GDP 0.0121* (0.00626) Average -0.725 Poverty Rate (4.488) Constant 243.9*** 198.1*** 185.5*** 196.4*** 158.0* (13.67) (31.30) (60.12) (59.08) (91.97) Observations 272 272 272 272 272 R-squared 0.074 0.191 0.305 0.342 0.346 Source: IEG analysis of sample contract data. Note: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. N = 272. All country and regional controls have been included in all specifications. GDP = gross domestic product; SPN = specific procurement notice. One caveat to the interpretation of these results is that the distribution of residuals in the regressions exhibits deviation from the normal distribution, possibly due to some high leverage data points in the 'long tail' discussed in previous sections. Although the non-normality of residuals does not affect the consistency of an estimate, it may affect its statistical significance. Table D.3. Analysis of Processing Time from Submission of draft Bid Documents to Bank’s No Objection to Final Bid Evaluation Report (1) (2) (3) (4) (5) Variables Processing Time for Processing Time Processing Time Processing Time for Processing Time for Bank No Objection for Bank No for Bank No Bank No Objection Bank No Objection Objection Objection NCB Method -68.19*** -47.92** -57.44** -79.32*** -64.61* (23.05) (21.92) (76.47) (26.42) (27.69) QCBS 28.41* 70.70*** 76.47*** 1.35 21.90 Method (24.00) (21.40) (22.36) (33.38) (34.87) 103 Contract 9.73e-07*** 1.16e-06*** 1.12e-06*** 1.00e-06*** 1.06e-06*** Value (2.34e-07) (2.13e-07) (1.96e-07) (1.75e-07) (1.78e-07) Social Sector -59.92 -43.55 -36.42 (39.03) (40.75) (41.27) Infrastructure -42.50 -36.87 -21.86 Sector (34.65) (36.00) (35.25) Economic -27.70 -44.30 -58.11 Sector (39.97) (42.73) (44.20) Civil Works -69.02* -51.14 Contracts (35.41) (35.93) Goods -99.35*** -76.20* Contracts (37.35) (40.35) GDP Level 0.0352*** (0.00962) Average -1.816 Poverty Rate (2.819) CPIA -335.33*** score (112.82) Constant 234.1*** 286.7*** 292.6*** 366.3*** 1,533*** (13.25) (40.59) (42.34) (54.00) (410.6) Observations 213 213 213 213 213 R-squared 0.153 0.382 0.392 0.406 0.445 Source: IEG analysis of sample contract data. Note: Specifications (2) to (5) include country and regional level controls. Robust standard errors in parentheses, *** p<0.01, ** p<0.05, * p<0.1. GDP = gross domestic product. N = 213. 104 Efficiency of Procurement Management in the Africa Region Further insights into procurement efficiency are afforded by IEG’s analysis of datasets made available by the Africa Procurement unit. Generated by PROCYS, the system tracks each step in a procurement process that involves interaction between the key players—borrowers, Bank task team leaders and Bank procurement staff—to monitor the responsiveness and efficiency of each party in the process. 43 ELAPSED TIMES IN THE PROCUREMENT PROCESS Each step in a procurement process that involves an exchange between participants in the procurement process is an entry in PROCYS. In 2012, 3,043 requests were logged; a number that has increased over time. This likely reflects the fact that use of PROCYS in terms of country coverage has increased, especially since 2010. 44 On average, there are two to three iterations both between client and task team leader, and task team leader and procurement specialist. In terms of numbers of days of elapsed time, in each iteration, the client takes the most time with an average over the 2009–12 period of almost 11 days; while the task team leader and procurement specialist take around 8 days each. There is evidence of increased efficiency in all parties; the number of days per iteration has shrunk considerably for clients, from 14 in 2009 to 5 days in 2012; from 10 to 5 days for task team leaders and from 8 to 7 days by the procurement specialist. Table D.4. Africa: Numbers of Iterations and Average Response Times, 2009–12 Year Number of Average number of iterations Average number of days taken per iteration records Client - TTL TTL – Procurement Client TTL Procurement specialist specialist 2009 Q1 360 2 2 16 10 9 Q2 404 2 2 14 11 11 Q3 522 2 2 11 8 5 Q4 672 3 2 17 10 6 2010 Q1 655 3 2 12 8 9 Q2 606 3 2 10 10 8 Q3 626 3 3 12 8 6 Q4 653 4 3 13 10 9 2011 Q1 671 4 3 13 8 9 Q2 719 3 3 11 9 6 Q3 661 3 3 15 10 10 Q4 726 3 2 9 8 6 2012 Q1 723 3 2 8 7 8 Q2 898 3 2 6 6 7 Q3 772 2 2 5 5 6 Q4 647 2 1 1 3 5 105 APPENDIX 4C EFFICIENCY OF PROCUREMENT MANAGEMENT IN THE AFRICA REGION Avg/Total 10,315 3 2 10.3 8.0 7.5 Source: PROCYS database. Note: TTL = task team leader. Aside from average elapsed time, IEG reviewed the distribution of elapsed time. Since the average numbers of days taken by client, task team leader, and procurement specialist was considerably greater than the median of their distributions, it suggests the presence of outliers that take much longer times per iteration. 45 For instance, the average number of days taken by procurement specialist per iteration is seven, but half the records (the median) take less than three days and at the 75th percentile, it requires eight days to process a contract. Table D.5. Africa: Response Times for Selected Procurement Categories by Clearance Level, 2009–12 Average numbers of days Procurement Category Clearance Level per iteration PS RPM OPRC (Number of records) Goods, works, and nonconsulting services 2690 221 40 Consulting services 4333 250 4 Client Goods, works, and nonconsulting services 11 16 8 Consulting services 10 13 10 TTL Goods, works, and nonconsulting services 7 7 13 Consulting services 7 10 5 Procurement specialist Goods, works, and nonconsulting services 7 8 16 Consulting services 6 8 25 Source: Africa procurement office PROCYS database. Note: OPRC =Operational Procurement Review Committee; PS = procurement specialist; RPM = regional procurement manager; TTL = task team leader. Clearance at the OPRC level experienced the largest delays in terms of time taken by Bank staff (task team leader and procurement specialist) with 27 days per iteration. However, clients do not take longer to respond to OPRC, with just 8 days per iteration, in contrast to 16 days on their iterations at the RPM level. There are also clear differences in the numbers of client and task team leader iterations, depending on the clearance level, with an average of five iterations for requests at the OPRC level, four at the RPM level, and three at the procurement specialist level. To estimate overall Bank performance (as separated from the client), IEG aggregated the time taken by the task team leader and the procurement specialist. Best and worst performers are indicated in Table D.6. Table D.6. Africa: Procurement—Best and Worst Performers, 2009–12 Country Number Average number of Average number Average number Average of days taken by client of days taken by of days taken by number of days records (per iteration) TTL (per procurement taken by TTL iteration) specialist (per +PS (per iteration) iteration) The longest numbers of days Guinea-Bissau 30 9 32 7 39 Burkina Faso 220 19 13 23 36 106 APPENDIX 4C EFFICIENCY OF PROCUREMENT MANAGEMENT IN THE AFRICA REGION Tanzania 878 14 9 12 21 The least number of days Mauritius 17 15 2 3 5 Gabon 44 9 3 4 7 Mozambique 776 8 6 2 7 Source: Africa procurement office PROCYS database. Note: PS = procurement specialist; TTL = task team leader. During the 2009–12 period, there are no significant differences in response times between procurement categories for any of the three sets of players in the process. In terms of specific products procured, clients spend the most time on “plant design, supply and installation” with 18 days on average, based on 93 requests recorded from 2009 to 2012. For task team leaders, it takes the most time—16 days—to process “Retroactive No-Objections,” based on 23 cases during the period of analysis. For procurement specialists, “output and performance-based road contracts” took the most time to process: 51 days; followed by “information systems,” with 15 days. It should be noted that the products procured did not seem uniformly defined. For example, in some cases procurement of consulting services was indicated to be related to firms or individuals, in other cases it was not specified. In terms of procurement method, the time taken by clients to respond to ICB-related requests was 15 days on average; compared to 8 days for NCB and 12 for quality- and cost-based selection requests. Response times for these methods are similar for task team leaders and procurement specialists; however, two-stage ICB takes 3 more days, per iteration, for task team leaders to process than for procurement specialists. Along with shopping, two-stage ICB takes the most time to process (11 days) for task team leaders than any other procurement method. For procurement specialists, the methods that take the most time are NCB, with 10 days; followed by ICB and quality- and cost-based selection that take 9 days. 46 The procurement methods that entail the shortest processing times are “force account” and “commercial practices.” Although there are few of these cases in our sample, 20 records for “force account” and only 2 for “commercial practices,” the times that clients, task team leaders and procurements specialists take for request processing are lower than the rest. They vary in between zero and four days. BORROWER DELAYS AND IDLE TIME As is clear from the table below, borrower delays in between steps in the procurement process are even greater than the actual time taken by all parties to process the different requests. And delays appear to vary by quarter. In 2012, for example, borrower delays in between steps averaged 44 days; while the total time taken to process all requests was 27 days. The 27 days refer to the average time per iteration taken by the borrower (15 days), task team leader (6 days) and by 107 APPENDIX 4C EFFICIENCY OF PROCUREMENT MANAGEMENT IN THE AFRICA REGION procurement staff (6 days). Borrower delays vary by quarter but they are always above Bank response times, on average, by more than three times. Table D.7. Africa: Borrower Delays and Bank Response, 2010–12 Borrower Response times (days) within iterations delay (days) Borrower Task team Procurement Bank Total between time time staff time response response Year-Quarter iterations time time 2010 Q1 18.3 9.2 3.8 5.1 8.9 18.1 Q2 38.1 16.1 5.5 4.6 10.1 26.2 Q3 49.6 19.3 9.3 5.9 15.2 34.6 Q4 38.6 12.4 5.7 7.3 13.0 25.4 2011 Q1 53.5 12.8 10.6 6.4 17.0 29.8 Q2 30.9 11.8 6.6 6.5 13.1 25.0 Q3 39.8 14.1 5.8 5.4 11.2 25.2 Q4 39.1 14.8 6.5 5.8 12.3 27.1 2012 Q1 38.1 15.7 6.9 6.1 13.0 28.7 Q2 63.0 17.5 6.9 7.2 14.1 31.6 Q3 38.1 15.1 6.6 6.0 12.6 27.6 Q4 38.4 11.4 4.8 5.8 10.6 22.0 Source: Africa procurement office PROCYS database. Note: Delays recorded in PROCYS are defined as the average time taken by the borrower beyond normal processing time (70 days for consultants and 90 days for goods, works, and services. Three clearance levels were indicated in the database: procurement specialist, RPM, and OPRC. Borrower delay was similar at the RPM and OPRC levels but much lower with clearance at the procurement specialist level. Table D.8. Africa: Borrower Delays by Clearance Level, 2010–12 Clearance level Number of records Borrower delay (days) PS 1,026 40 RPM 68 69 OPRC 4 65 Not available 494 40 Total 1,592 42 Source: Africa procurement office PROCYS database. Note: OPRC = Operational Procurement Review Committee; PS = procurement specialist; RPM = regional procurement manager. According to the most recent report from the Africa procurement unit, the countries with the largest borrower delays are Chad (66 days), Cameroon (63 days), and Ethiopia (62 days). 47 The largest Bank response times per iteration correspond to South Africa (21 days), Burkina Faso (19 days), and Central African Republic (16 days). 108 APPENDIX 4C EFFICIENCY OF PROCUREMENT MANAGEMENT IN THE AFRICA REGION Figure D.4. Procurement Processing Delays and Bank Response by Country Botswana Comoros Sao Tome and Principe Sierra Leone Central Africa Gabon Cape Verde Mauritania Procurement Swaziland Staff Congo Gambia Task Team South Africa Lesotho Central African Republic Malawi Borrower Delay Niger Sudan Madagascar Mali Togo Nigeria Mozambique Burundi Burkina Faso DRC Liberia Angola Benin Africa Zambia Republic of Congo Eastern Africa Rwanda Tanzania Kenya Uganda Senegal Ivory Coast Ghana Ethiopia Cameroon Chad 0 10 20 30 40 50 60 Source: Africa procurement office database. Note: Delays recorded in PROCYS are defined as the average time taken by the borrower beyond normal processing time (70 days for consultants and 90 days for goods, works, and services). 109 Bibliography for Realistic Solutions. Cambridge: Cambridge University Press 128-130. APEC (Asian Pacific Economic Cooperation). 1999. “Government Procurement Aberdeen Group. 2008. The E-Procurement Experts Group Non-Binding Benchmark Report: Driving Year-Over- Principles on Government Year Superior Performance. Boston: Procurement”. August 1999. Aberdeen Group. Singapore: APEC. ADB (Asian Development Bank). 2002. ASIL (American Society of International Law). “Understanding Public Procurement.” 2013. Will the World Trade Governance Brief, Asian Development Organization Finally Tackle Bank. Manila: ADB. Corruption in Public Purchasing? The ———. 2007. Environmentally Responsible Revised Agreement on Government Procurement—A Reference Guide. Procurement. Manila: ADB. http//www.asil.org/insights130415.c fm. AfDB. (African Development Bank). 2011. “Evaluation of Paris Declaration Blue, Richard, and John Eriksson. 2011. Implementation at the African “Evaluation of the Implementation of Development Bank.” Operations the Paris Declaration: United States Evaluation Department. Tunis: AfDB. Government Synthesis Report”. Arlington, VA: The United States ———. 2012. “Determination of Financial Agency for International Thresholds for Modes of Procurement Development. of Goods, Works and Non-Consulting Services”. African Development Bank Burger, Philippe, and I. Hawkesworth. 2011. Group, Procurement Policy and How to Attain Value for Money: Fiduciary Services Department. Comparing PPP and Traditional January 2012. Tunis: AfDB. Infrastructure Public Procurement. OECD Journal on Budgeting, Volume AfDB, IDB (Inter-American Development 2011/1. Paris, France: OECD. Bank), World Bank, and Curtin University of Technology. 2007. Casartelli, G., E. Wolfstetter. 2007. "Selection International Survey of Procurement. and Employment of Consultants: the Perth. Effectiveness of the World Bank's Policy" Working Paper. Washington, Agaba, Edgar, and Nigel Shipman. 2007l DC: World Bank. “Public Procurement Reform in Developing Countries: The Uganda Commonwealth of Australia. 2008. Experience.” In G. Pig and K.V. Thai Commonwealth Procurement Guidelines, (eds.) Advancing Public Procurement: December 2008. Issued under Practices, Innovation, and Knowledge Regulation 7 of the Financial Sharing (Boca Raton, FL: Pr. Management and Accountability Academics Press), 373-91. Regulations, 1997. Canberra: Commonwealth of Australia. Al-Azar, Rima. 2011. “A Tale of Two CDDs: Lessons Learned for Mitigating the Debevoise, E.W., and C. Yukins. 2010. Risk of Fraud and Corruption in CDD “Assessing the World Bank’s Operations.” PowerPoint Proposed Revision of Its Procurement presentation, World Bank, Guidelines,” The Government Washington, DC. Contractor, 52, (21). Andrews, Matt. 2013. The Limits of Institutional Dehn, Jan, Ritva Reinikka, and Jakob Reform in Development: Changing Rules Svensson. 2003. “Survey Tools for 110 Assessing Performance in Service Procurement of Goods and Services, Delivery.” In Francois Bourguignon February 16. Mimeo. and Luiz Pereira da Silva (eds.), IEG (Independent Evaluation Group). 1977. Evaluating the Poverty and “The Role and Use of Consultants in Distributional Impact of Economic Bank Group Projects.” Report no. Policies. New York: Oxford University 1824, Washington, DC: World Bank. Press and World Bank. ———. 1981. “An Interim Report on Dener Cem, Joanna. A Watkins and William. Procurement Issues in Bank-Financed L. Dorotinsky. 2011. “Financial Projects.” Report no. 3557, Management Information Systems— Washington, DC: World Bank. 25 Years of World Bank Experience on What Works and What Doesn’t.” ———. 1990. “A Review of Procurement in World Bank, Washington, DC. Policy-Based Lending.” Report no. 8341, Washington, DC: World Bank. De Silva, S. 2000. “Community-based Contracting; A Review of Stakeholder ———. 2002. India Country Assistance Experience.” Washington DC: World Evaluation. Washington, DC: World Bank. Bank DfID (Deparment for International ———. 2005. The Effectiveness of World Bank Development). 2011. “Approach to Support for Community-Based and – Value for Money.” London: Driven Development: An OED Department for International Evaluation. Washington, DC: World Development. Bank. European Union, World Bank, and Inter- ———. 2006a. Debt Relief for the Poorest—An American Development Bank. 2009. Evaluation Update of the HIPC Initiative. Perú Informe de la gestión de las finanzas Washington, DC: World Bank. públicas Basado en el marco de referencia ———. 2006b.Engaging with Fragile States: An para la medición del desempeño (PEFA). Evaluation of World Bank Support for Lima: European Union, World Bank Low-Income Countries Under Stress. and Inter-American Development Washington, DC: World Bank. Bank. ———. 2006c. Hazards of Nature, Risks to Gartner Group. 2012. “Survey Shows Why Development – An IEG Evaluation of Projects Fail.” World Bank Assistance for Natural http://www.gartner.com/id=2034616 Disasters. Washington, DC: World . Bank. GSDRC. 2010. Helpdesk Research Report: ———. 2006d. Small States: Making the Most of Public Financial Management and Development Assistance – A Synthesis of Frontline Service Delivery. World Bank Evaluation Findings. http://www.gsdrc.org/docs/open/ Washington, DC: World Bank. HD653.pdf,. ———. 2007a. The Development Potential of IDB (Inter-American Development Bank). Regional Programs – An Evaluation of 2010. “Guide for Acceptance of the World Bank Support of Multicountry Use of Country Procurement Operations. Washington, DC: World Systems.” June 2010. Washington DC: Bank. IDB. ———. 2007b. Water Management in IAEA (International Atomic Energy Agency). Agriculture—Ten Years of World Bank 2011. Agency for Financial Assistance, 1994-2004. Washington, Regulations and Rules AM VI/1: DC: World Bank. 111 APPENDIX 4C EFFICIENCY OF PROCUREMENT MANAGEMENT IN THE AFRICA REGION ———. 2008a. Country Financial Accountability Support. Washington, DC: World Assessments and Country Procurement Bank. Assessment Reports: How Effective Are ———. 2010f. The World Bank in Nigeria 1998- World Bank Fiduciary Diagnostics? 2007—Country Assistance Evaluation. Washington, DC: World Bank. Washington, DC: World Bank ———. 2008b. Public Sector Reform: What Works ———. 2011a. “Aid Effectiveness 2005-10: and Why? An IEG Evaluation of World Progress in Implementing the Paris Bank Support. Washington, DC: World Declaration.” World Bank, Bank. Washington, DC. ———. 2008c. The World Bank in Indonesia 1999-2006--Country Assistance ———. 2011b. Capturing Technology for Development – An IEG Evaluation of Evaluation. Washington, DC: World World Bank Group Activities in Bank Information and Communication ———. 2009a. Bangladesh Country Assistance Technologies. Washington, DC: World Evaluation 2001-2008. Washington, Bank. DC: World Bank ———. 2011c. Trust Fund Support for ———. 2009b. Improving Effectiveness and Development – An Evaluation of the Outcomes for the Poor in Health, World Bank’s Trust Fund Portfolio. Nutrition and Population--An Evaluation Washington, DC: World Bank. of World Bank Group Support since 1997. ———. 2011d. World Bank Country-Level Washington, DC: World Bank Engagement on Governance and ———. 2009c. Review of IDA Internal Controls: Anticorruption: An Evaluation of the An Evaluation of Management’s 2007 Strategy & Implementation Plan. Assessment and the IAD Review. Washington, DC: World Bank. Washington, DC: World Bank. ———. 2011e. World Bank Progress in ———. 2009d. The World Bank Group Guarantee Harmonization and Alignment in Low- Instruments 1990-2007 –An Independent Income Countries. Washington, DC: Evaluation. Washington, DC: World World Bank. Bank ———. 2012a. The Matrix System at Work: An ———. 2010a. Cambodia – An IEG Country Evaluation of the World Bank’s Assistance Evaluation, 1999-2006. Organizational Effectiveness. Washington, DC: World Bank Washington, DC: World Bank. ———. 2010b. Climate Change and the World ———. 2012b. “The World Bank and Public Bank Group. Phase II: The Challenge of Procurement. An Independent Low-Carbon Development. Washington, Evaluation.” Approach Paper. DC: World Bank. Washington, DC: World Bank. ———. 2010d. IDA Internal Controls: Evaluation ———. 2012c. The World Bank Group’s Response of Management’s Remediation Program-- to the Global Economic Crisis—Phase II. Report on Management’s Implementation Washington, DC: World Bank. and IAD’s Review of the Five-Point ———. 2013. “Comparison of the Action Plan. Washington, DC: World International Instruments on Public Bank. Procurement.” Background paper. ———. 2010d. Mozambique Country Program Washington, DC: World Bank. Evaluation. Washington, DC: World IMF (International Monetary Fund). 2012. Bank. Update on the Bank’s Business ———. 2010e. Poverty Reduction Support Modernization: Results, Openness, Credits: An Evaluation of World Bank 112 and Accountability Spring 2012.”IMF ———. 2006a. “The Challenge of Capacity World Bank Development Committee. Development: Working Towards Good Practice.” DAC Reference Kivistö, T., V.M. Virolainen, and E. Tella. 2003. Document, Paris. Consortia purchasing and logistics in Kuopio area (Finland)--Lessons ———. 2006b. “Methodology for Assessment learned from a 4-year project. IRSPP of National Procurement Systems.” conference, Budapest, and in KV Thai Paris: OECD. et al. (eds.), (2005): Benchmarking ———. 2009. “Compendium of Country municipal public procurement Examples and Lessons Learned from activities in Finland. In Challenges in Applying the Methodology for Public Procurement: an International Assessment of National Procurement Perspective. Systems: Sharing Experiences.” OECD Korpela, J., and M. Tuominen. 1994. Journal on Development 9(4). Benchmarking Logistics Performance with ———. 2011a. International Engagement in an Application of the Analytical Fragile States: Can’t We Do Better? Hierarchy Process. Lappeenranta, Paris: OECD. Finland: Lappeenranta University of Technology. ———. 2011b. A Practical Guide for Transforming Procurement Systems. Kruger. K, 2009. “Ban on negotiations in Paris: OECD. tender procedures – Undermining the best value for money.” Journal of ———. 2011c. Strengthening Country Public Procurement, 4. Procurement Systems, prepared for the 4th High Level Forum on Aide Mackie, Andrew, and Giovanni Caprio. 2011. Effectiveness, November 29 – Assessing the Impact of the FEFA December 1, 2011, Busan, Korea. Framework. Washington, DC: PEFA OECD/DAC Working Party on Aid Secretariat. Effectiveness Task Force on Ministry of Foreign Affairs of Denmark. 2008. Procurement. “Evaluation of the Implementation of OECD-DAC and World Bank. 2004. the Paris Declaration: Phase One “Mainstreaming and Strengthening Synthesis Report.” Denmark: Ministry Public Procurement: A Strategic of Foreign Affairs. Framework.” Draft report of the 3rd OECD (Organization for Economic Joint Round Table on Strengthening Cooperation and Development). 2003. Procurement Capacities in Developing Mainstreaming the Procurement Countries. Washington, DC, and Function into the Public Expenditure Paris. Policy and Effectiveness Dialogue. Paris: OECD. Pallas, Christopher L., and Jonathan Wood. 2009. “The World Bank’s Use of ———. 2005a. “Paris Declaration on Aid Country Systems for Procurement: A Effectiveness--Ownership, Good Idea Gone Bad?” Development Harmonization, Alignment, Results Policy Review 27 (2): 215-230. and Mutual Accountability.” Paris, France: OECD. PEFA Secretariat. 2011. Independent Evaluation of the Public Expenditure and Financial ———. 2005b. “Strengthening Procurement Accountability Program. Washington, Capacities in Developing Countries.” DC: PEFA. Volume 3 of the OECD-DAC Pollit, C., and G. Bouckaert. 2004. Public Guidelines and Reference Series, Management Reform: A Comparative Harmonising Donor Practices for Effective Aid Delivery. Paris: OECD. 113 APPENDIX 4C EFFICIENCY OF PROCUREMENT MANAGEMENT IN THE AFRICA REGION Analysis. New York: Oxford ———. 2003. “Standard Bidding Documents: University Press. Supply and Installation of Information Systems.” Washington, DC. Sanchez, A. 1997. The World Bank Procurement Function – Adjusting to Emerging Needs. ———. 2004. “Country Procurement Issues Washington, DC. World Bank. Paper—Rwanda.” Washington, DC: Retrieved from: World Bank. http://intresources.worldbank.org/I ———. 2005a. “Africa Capacity Building Task NTPROCUREMENT/Resources/Proc Force Report: Forging Effective States urement-Strategy.pdf. and Engaged Societies.” Washington, Singh, J., Dilip R. LImaye, Brian Henderson, DC: World Bank. and Xiaoyu Shi. 2012. “Public ———.2005b. “Expanding the Use of Country Procurement Of Energy Efficiency Systems in Bank-Supported Services: Lessons from International Operations: Issues and Proposals.” Experience.” Washington, DC: World Operations Policy and Country Bank. Services, Washington, DC. Tonkin, Christine. 2012. “Procurement ———. 2005c. “Meeting the Challenge of Governance Reform at the Africa’s Development: A World Bank International Atomic Energy Agency.” Group Action Plan.” Africa Region Mimeo. Department, Washington, DC: World Thai, K.V., et al. 2005. “Benchmarking Bank. Municipal Public Procurement ———. 2006. “Quality of Information and Activities in Finland.” In Challenges in Communication Technology Public Procurement: An International Components in Bank Projects.” Perspective. Academic Press. Quality Assurance Group UNCITRAL (United Nations Commission on Assessment, Washington, DC: World International Trade Law) Secretariat. Bank. (1994, 2011) “UNCITRAL Model Law ———. 2007a. “Detailed Implementation on Public Procurement.” UNCITRAL, Review. India Health Sector. 2006- Vienna, Austria. 2007.” INT, Washington, DC: World Volcker, Paul, Gustavo Gaviria, John Bank. Githongo, Ben W. Heineman Jr., ———. 2007b. “Mandatory Provisions for Walter Van Gerven, and Sir John NCB (for Bank/IDA financed Vereker. 2007. Independent Panel contracts).” Washington, DC: World Review of the World Bank Group Bank. Department of Institutional Integrity. Washington, DC: World Bank. ———. 2007c. “Operational Approaches and Financing in Fragile States.” World Bank. 1997. “The World Bank Washington, DC. Procurement Function- Adjusting to Emerging Needs.” SecM97-854, ———. 2007d. “Strengthening World Bank Washington, DC: World Bank. Group Engagement on Governance and Anti-Corruption.” Washington, ———. 2002a. “Fiduciary Management for DC: World Bank. Community-Driven Development Projects: A Reference Guide.” ———. 2008a. “Enhanced Support to Staff Washington, DC: World Bank. Working in Fragile and Conflict- Affected States.” Washington, DC: ———. 2002b. Guidelines for Country World Bank. Procurement Assessment Reviews. Washington, DC: World Bank. 114 ———. 2008b. “Use of Country Procurement July 2011, November 2011, January Systems in Bank-Supported 2012. Washington, DC: World Bank. Operations: Proposed Piloting ———2010g. “Public Procurement of Energy Program.” R008-0036/5. Washington, Efficiency Services: Lessons from DC: World Bank. International Experience.” ———. 2009a. “Guidance to Staff for Using Washington, DC: World Bank. the Company Risk Profile Database ———. 2010h. “Republic of Moldova Country (CRPD).” World Bank, Washington, Procurement Assessment Report.” DC. Washington, DC: World Bank. ———. 2009b. “Handling Suspected and ———. 2011a. “Community-Driven Alleged Fraud and Corruption in Development Projects: A Review of Procurement: OPCS and INT Governance Issues.” OPCS Document, Protocol.” Washington, DC: World World Bank, Washington, DC. Bank. ———. 2011b. Country Partnership Strategy ———.2009c. “Piloting Program in Use of Progress Report for the Republic of Country Systems: First Progress Moldova for the Period FY09-FY13. Report.” Operations Policy and Washington, DC: World Bank. Country Services. Washington, DC: World Bank. ———. 2011c. “Diversity Benchmarking Study in Selected Countries.” OPSOR, ———. 2010a. Financial Management Sector Mimeo. Washington, DC: World Strategy FY11-13. 6, 9. Washington, Bank. DC: Financial Management Sector Board. ———. 2011d. “Financial Management Information Systems—25 Years of ———. 2010b. “Guidance Note on the World Bank Experience on What Operational Risk Assessment Works and What Doesn’t.” Framework (ORAF) – Risks to Washington, DC: World Bank. Achieving Results.” Operations Policy and Country Services, World Bank, ———. 2011e. “Green Procurement Policy Washington, DC. Framework in Selected Countries.” OPSOR, Mimeo. Washington, DC: ———. 2010c. “Guidance Note: Procurement World Bank. Arrangements Applicable to Public- Private Partnership Contracts ———. 2011f. “Guidelines: Procurement of Financed under World Bank Projects.” Goods, Works, and Non-Consulting Washington, DC: World Bank. Services under IBRD Loans and IDA Credits and Grants by World Bank ———. 2010d. Monitoring Report 2010. PEFA Borrowers.” Washington, DC: World Secretariat Report. World Bank, Bank. Washington, DC. http://pefa.org/sites/pefa.org/files/ ———. 2011g. “Strengthening Country attachments/ENG-%20MR-10- Procurement Systems.” Prepared for FinalReport-May192011.pdf. the 4th High Level Forum on Aide Effectiveness, November 29 – ———. 2010e. “Piloting Program in the Use of December 1, Busan, Korea. Country Procurement Systems— Washington, DC: World Bank. Second Progress Report.” Washington, DC: World Bank. ———. 2011h. “Strengthening World Bank Engagement on Governance and Anti- ———. 2010f. “Procurement Risk Assessment Corruption: Procurement Companion and Management System User Piece.” Washington, DC: World Bank. Guide.” OPSOR, May 2010, Updated 115 APPENDIX 4C EFFICIENCY OF PROCUREMENT MANAGEMENT IN THE AFRICA REGION ———. 2011i. World Development Report: User Guide.” Washington, DC: World Conflict, Security, and Development. Bank. Washington, DC: World Bank. ———. 2012j. “Procurement and PPP ———. 2012a. “Analysis of World Bank Transactions Guidance for MDB Completed Cases of Fraud and Public-Sector Engagements and Corruption from the Perspective of Procurement Principals Applicable to Procurement.” Integrity Vice Private Sector Transactions— Presidency, Washington, DC: World Guidance for MDBs.” Washington, Bank. DC: World Bank. ———. 2012b. “Applying the Principles for ———. 2012k. “Procurement Risk Assessment Good International Engagement in and Management System User Fragile States and Situations to Guide.” Washington, DC: World Strengthen and Transform Public Bank. Procurement Systems.” Background ———. 2012l. “Project Appraisal Document note, OPSOR. Washington, DC: World on a Proposed Credit in the Amount Bank. of SDR 26.1 million ($40.0 million ———. 2012c. “The Bank's Procurement Equivalent to the Republic of Policies and Procedures: Initiating Moldova for the Moldova Education Discussion and Approach Papers-- Reform Project).” Washington, DC: Proposed Review.” Report submitted World Bank. to the Audit Committee and to CODE, ———.2012m. “Public Procurement of Energy Washington, DC: World Bank. Efficient Products: Lessons from ———. 2012d. “Fiduciary Management for Around the World.” Energy Sector Community-Driven Development Management Program. Washington, Projects: A Reference Guide.” DC: World Bank. Washington, DC: World Bank. ———.2012n. “Review of the World Bank’s ———. 2012e. Financial Management and Procurement Policies and Procedures: Procurement in World Bank Operations: Achieving Social Diversity Through Annual Report for FY11. Washington, the use of Supplier Preferences in the DC: World Bank. Procurement Process.” Background paper. Washington, DC: World Bank. ———. 2012f. “Green Procurement in Selected Environmental Policy Framework: ———.2012o. “Review of the World Bank’s Review of the World Bank‘s Procurement Policies and Procedures, Procurement Policies and The Use and Impact of the Bank’s Procedures.” Background Paper. Policy of Domestic Preferences.” Washington, DC: World Bank. Background Paper. Washington, DC: World Bank. ———. 2012g. “Guidance Note for Design and Management of Procurement ———. 2012p. “Update on the Bank’s Business Responsibilities in Community- Modernization: Results, Openness, Driven Development Projects.” and Accountability Spring 2012.” Washington, DC: World Bank. DC2012-0005. Washington, DC: World Bank. ———. 2012h. “Investment Lending Reform: Modernizing and Consolidating ———. 2012q. “Virtual VP Review of the Operational Policies and Procedures.” Procurement Policy Review Proposal- OPCS, Washington, DC: World Bank. -An Approach Paper.” Washington, DC: World Bank. ———. 2012i. “OPCPR Procurement Risk Assessment and Management System 116 ———.2013a. “Approach for Private Sector Procurement in Projects financed by IBRD/IDA, including Public Private Partnerships.” Washington, DC: World Bank. ———. 2013b. “Audit of the Bank’s Operational Framework for Using Investigation Results in Bank-Funded Projects.” Report No. IBRD FY12-16. Washington, DC: World Bank. ———. 2013c. “Procurement in World Bank Investment Operations. Phase I: A Proposed New Framework.” OPCS. Washington, DC: World Bank. ———. 2013d. “Red Flags Learning Tool.” Integrity Vice Presidency. http://go.worldbank.org/D1K227ET8 1. ———. 2013e. “Situations of Urgent Need of Assistance or Capacity Constraints: Simplified Procurement Procedures and Guiding Principles.” ———. 2013f. “Supporting Countries in Strengthening their Public Procurement Systems: Second Generation Reforms” (draft). Operations Risk Management Department. Washington, DC: World Bank. ———.2013g. “Use of Country Procurement Systems.” Background Paper, Washington, DC, World Bank. World Bank Institute. 2012. “Public-Private Partnership Reference Guide, Version 1.0.” Washington, DC: World Bank. World Trade Organization. 1996. “Agreement on Government Procurement (GPA).” Geneva: World Trade Organization. . 117 Endnotes Appendix A 1 Others apply some value for money principles (such as life-cycle costing, for example) to varying degrees, often without calling it “value for money.” Some agencies use the term but without defining a distinct procurement process around it—for example, the United Nations procurement guide refers to value for moeny, but only in limited terms and makes no recommendations to operationalize VfM in procurement policies and practices. 2 Getting Value for Money from Procurement: How Auditors Can Help, Office of Government Commerce and National Audit Office, UK, 2001; Regularity, Propriety and Value for Money, Treasury Officer of Accounts, HM Treasury, UK, November 2004; Procurement Efficiency and Value for Money Measurement, Office of Government Commerce, 2005; and An Introduction to Public Procurement, Office of Government Commerce, 2008. 3“Best Value” was ascribed its modern regulatory definition by FAC 97-02, Sept. 30, 1997, as part of the rewrite of Federal Acquisition Regulation Part 15. 4Directive 2004/18/EC of the European Parliament and of the Council of March 31, 2004 on the Coordination of Procedures for the Award of Public Works Contracts, Public Supply Contracts and Public Service Contracts. The EU Directive says, “…Where the contracting authorities choose to award a contract to the most economically advantageous tender, they shall assess the tenders in order to determine which one offers the best value for money.” 5 MEAT must be used when competitive negotiation procedures are applied (Article 29). 6Commonwealth of Australia, Commonwealth Procurement Guidelines, December 2008. Issued under Regulation 7 of the Financial Management and Accountability Regulations 1997. 7 APEC Government Procurement Experts Group Non-Binding Principles on Government Procurement. August 1999. 8 Procurement in World Bank Investment Operations. Phase I: A Proposed New Framework. OPCS, March 29, 2013. 9 Although negotiation after contract award is not generally accepted good practice, with the possible exception of in the United Kingdom. 10 By using benchmarks, for example, as illustrated in a simple example in Chapter 4. 11 See Chapter 6 for a description of competitive negotiation. 12 The proposal is currently under legislative review, between the Council and the European parliament. In December 2012, the Council adopted a compromise text, which is now the basis for negotiations between council and parliament. There is some expectation, at the European Union, that the new directive may be adopted before the end of 2013. Implementation dates are unclear. 13Governance in public procurement is increasingly on the agenda of international organizations as well, including the World Trade Organization, as witnessed by proposed amendments to its Government Procurement Agreement (Chapter 5). Appendix B 14 The 2000 Bank document Community-based Contracting—A Review of Stakeholder Experience highlighted local shopping among the very simple contracting procedures. 15 Examples of incentives were, for example, savings reverting back to communities for financing additional facilities. 119 ENDNOTES 16 Although this conclusion may not have been surprising, since the sample had purposively focused on successfully completed subprojects. Construction cost-savings ranged from 12 percent to 56 percent compared to original estimates. At the same time, the quality and sustainability of the completed projects was found to be high. consultant. 18 A Mongolia case study noted that procurement of the ICT system went through three procurement processes over two years The study reports on three FMIS related projects in Mongolia (P051855, P077778, P098426), with the first project approved in June 1998. Two were active at the time of the report (2011). 19 The case study covers one Bank funded project (P035759), approved in 1995 and closed in 2002, and a subsequent Government-funded component. The cost of the Bank project was $78.5 million. In 1999, the Government and the Bank agreed to restructure the project, and the government initiated the Public Expenditure Management (PEM) component as a separate activity funded from the its own budget. Project funds were used to develop a customs information system and improvement of debt management system, together with related advisory support. The PEM component was implemented as a government led activity in parallel to the Bank project, with a cost of approximately $15.7 million. 20Unlike the other MDBs, the Black Sea Trade and Development Bank takes a very much hands off approach. It only requires procurement methods that “lead to sound selection of goods, works and services at fair market prices.” Appendix C 21Resource Guide: Procurement Methods http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:20109695~pa gePK:84269~piPK:60001558~theSitePK:84266~isCURL:Y,00.html. 22 Most countries also set monetary thresholds for the use of shopping procurement method. 23 World Bank Procurement Method Thresholds http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:21038090~isC URL:Y~menuPK:2926825~pagePK:84269~piPK:84286~theSitePK:84266,00.html. 24http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:21038037~is CURL:Y~menuPK:2926825~pagePK:84269~piPK:84286~theSitePK:84266,00.html. Bank guidelines do make publication of Specific Procurement Notices in UNDB and DG market electronic portal mandatory for ICB processes. 25 Procurement of goods, works and non-consulting services under IBRD Loans and IDA Credits and Grants by World Bank Borrowers, Paragraph 1.3, January 2011. 26 The field “amountUSE” of Form 384 was used as contract value. 27 When looking at the distribution by value a higher share is won by foreign firms, though still more than half of contracts in the highest quintile go to national firms (Table 1, “Distribution by Value”). 28 In 2008, three risk levels were defined by implementing agency (high, medium, or low). In 2009, Bank-wide maximum prior review thresholds were revised and categories expanded into high, substantial, moderate, or low. 292002BP 11.0, Annex A Decision Authority Matrix. This document specifies clearance thresholds for task team leaders/PS/PAS, RPA, Regional Vice Presidents, and OPRC. http://intranet.worldbank.org/WBSITE/INTRANET/OPSMANUAL/0,,contentMDK:22819821~menuPK:51457 764~pagePK:51458737~piPK:51458739~theSitePK:210385,00.html 30Email from OPCPR retrieved from: http://intranet.worldbank.org/WBSITE/INTRANET/INTCOUNTRIES/INTAFRICA/INTAFRPRO/0,,content MDK:21772310~pagePK:64168332~piPK:64168299~theSitePK:1938585,00.html, this specified, for example, task team leader/procurement specialist levels of justification for single source selection of consultants. 120 ENDNOTES 31http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/PROCUREMENT/0,,contentMDK:21038090~m enuPK:2926825~pagePK:84269~piPK:84286~theSitePK:84266~isCURL:Y~DIR_PATH:WBSITE/EXTERNAL/PR OJECTS/PROCUREMENT/,00.html. 32 http://www.transparency.org/research/cpi/ 33http://www.worldbank.org/ida/IRAI-2011.html. Bank data on CPIA scores for IBRD countries were also used. Appendix D 34 Sistema de Ejecucion de Planes de Adquisiciones in Spanish. 35SEPA does not handle procurement documents or attachments through the system. 36To upload a procurement plan the government enters project information and data on the related contracts. On the project level this includes procurement method and prior review thresholds, names of implementing agencies, responsible task team leaders and procurement specialists, project amounts and key project dates, among other things. On the contract level this includes contract description, estimated amount, the percentage of the contract financed by the local counterpart to the project, and estimated dates for all procurement stages related to the contract. Over the course of execution, actual amounts and actual dates are entered into the system, after the estimated dates. Dates are entered according to procurement method and review method. After the actual contract signature date is recorded, contracting data such as bidder’s names and their bids, winning bidder and contract value can be entered. 37In the form of Excel spreadsheets, available from October 2008. Database macros allow users to produce management reports filtered for information for the entire region, for a selected country department, country or procurement staff. 38 The data template is included in Appendix A, which contains the overall questionnaire template. 39 Information was requested on the procurement steps borrower issue of specific procurement notice, borrower first submission to Bank of draft bid (preQ) documents, Bank final no objection to draft bid (PreQ) documents, borrower issue of Bid (PreQ) documents, borrower bid (PreQ) opening date/ minutes, borrower submission to Bank of bid evaluation report, Bank no-objection to bid evaluation report, borrower submission to Bank of negotiated contract, Bank no-objection to negotiated contract, date of signature of contract. 40 For consultant services selected by quality and cost the dates of submission of / no objection to the negotiated contract were considered as the final bid evaluation report. For contracts flagged as ICB/NCB Two Stage the first stage Submission of / No Objection to Bid Evaluation Report was used, only one observation was received for the second stage. For contracts flagged as subject to Pre-Qualification the dates of Submission of / No Objection to the final Bid Evaluation Report were used. Where Bid Evaluation Report Submission / No Objection dates were not provided the “Award No Objection” date in the bank-wide Form 384 database was used if the contract information received from the field could be matched with Form 384. 41 International Monetary Fund, International Financial Statistics: http://elibrary-data.imf.org/. 42For the regression analysis of average processing times from submission of draft bid documents to no objection to final bid evaluation report only quality- and cost-based selection procurement is considered for consultant services contracts. 43Two databases were provided by the regional procurement management office, on user responsiveness and elapsed times. Data cover the period from 2009 to 2012; however, data for 2009 are incomplete and 2010 provides a first reasonably full dataset. 44Numbers of iterations for 2011 were 2,778; for 2010, 2,540; and for 2009, 1,970. While 2009 was a startup year, the system is now at “steady state.” Arguably it could also suggest an increase in the number of iterations. However, data in Table B.14 suggest that this may not be the case. 121 ENDNOTES 45 The database shows some errors in recording times. For example, in 62 cases (of 10,331) the average number of days that client take to respond has a negative values. However, these are not frequent. 46 For clients, limited international bidding takes 19 days to process. Other lengthier client response times correspond to ICB and two-stage ICB with 15 and 14 days, respectively. 47The latest monthly report produced by the Africa Procurement Unit on April 2, 2013, that includes no objections processed through July 1, 2011, and excludes 1 percent of outliers. 122