83210 v2 The World Bank and Public Procurement—An Independent Evaluation Appendixes to Volume I: Building Procurement Capacity and Systems Contents ABBREVIATIONS .................................................................................................................................. III APPENDIX A: BANK PROCUREMENT SYSTEMS .............................................................................. 1 Background—Country Program and Bank Procurement Interventions................................................................... 25 Interviews with Country Management—The Country Context ................................................................................ 28 Interviews; Key Bank Procurement Staff in the Country ......................................................................................... 32 Interviews; Field Office-based Bank Task Team Leaders....................................................................................... 48 Interviews: Country Clients ..................................................................................................................................... 53 Interviews: Private Sector and Civil Society ........................................................................................................... 62 Summary of Findings and Overall Assessment ...................................................................................................... 66 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT ........................................................ 68 Country Assistance Strategies ................................................................................................................................ 68 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA ................................................................ 82 World Bank AAA Support – Non-Lending Technical Assistance ............................................................................ 86 Procurement in Budget Management – PEFAs and Integrated Capacity ............................................................... 88 APPENDIX D: PROCUREMENT-RELATED LENDING OPERATIONS, 2002–11 ............................... 99 Sample Construction and Select Findings .............................................................................................................. 99 Investment Lending ............................................................................................................................................... 101 APPENDIX E: COLLABORATION ..................................................................................................... 105 PREM, Financial Management, and Procurement ................................................................................................ 105 WBI Support to Building Procurement Capacity ................................................................................................... 106 BIBLIOGRAPHY ................................................................................................................................. 111 ENDNOTES......................................................................................................................................... 118 ii Abbreviations AAA Analytical and advisory assistance PIU Project implementing unit CAS Country assistance strategy PPP Public-private partnership CDD Community driven development PREM Poverty Reduction and Economic CFAA Country financial accountability Management Vice assessment Presidency/Network CPAR Country procurement assessment RFP Request for proposal report RPM Regional procurement managers CPIA County policy and institutional SSS Single-source selection assessment WBI World Bank Institute CQS Consultant qualifications selection DfID Department for International Development DPL Development policy loan DPO Development policy operation ESW Economic and sector work ICB International competitive bidding process ICT Information and communications technology IDA International Development Agency IDF Institution development fund IEG Independent Evaluation Group IFC International Finance Corporation IL Investment Loan LCS Least-cost selection LEG Law, justice and development MAPS Methodology for Assessing Procurement Systems NCB National competitive bidding OECD Organization of Economic Cooperation and Development- Development OECD-DAC Development Assistance Committee, Organizational for Economic Cooperation and Development OPRC Operational Procurement Review Committee OPSOR Operations Risk Management, Operations Country Policy and Services Vice Presidency ORAF Operational Risk Assessment Framework P4R Program for results PEFA Public expenditure and financial accountability PER Public expenditure review iii APPENDIX A: BANK PROCUREMENT SYSTEMS Appendix A: Bank Procurement Systems PRIOR REVIEW AND ICB Figure A.1. Procurement Methods at the Bank: Prior Review and ICB (2000–11) (no. and $ billions) Source: World Bank data. Note: ICB = International Competitive Bidding process. It is assumed, in the absence of disaggregated contract level data, that all contracts subject to prior review are wholly financed by the Bank. 1 APPENDIX A: BANK PROCUREMENT SYSTEMS Table A.1. World Bank Lending, Prior Review and International Competitive Bidding Contracts: (goods , civil works, and services) (2000–11) Value of Prior ICB No. of All new IL new review ICB contract ICB commit- commit Prior No. of contracts contracts Prior s as % contracts ments ments review prior ICB No. of as % of as % of review ICB of prior as % of (ILs+DPL ($ contracts review contract ICB new new contracts contract review prior s) ($ billions ($ contract s ($ contrac commit- commit- as % of s as % contract review FY billions) ) billions) s (000) billions) ts (000) ments ments ILa of ILa s contracts 2000 15.3 10.2 8.2 8.5 5.2 2.8 54 34 81 51 63 33 2001 17.3 11.5 7.6 7.9 5.2 2.5 44 30 66 46 69 32 2002 19.5 9.7 7.6 8.1 5.0 2.6 39 25 78 51 65 32 2003 18.5 12.5 7.8 8.3 4.8 2.4 42 26 62 38 62 29 2004 20.1 13.9 8.9 8.4 5.1 2.4 44 26 64 37 58 29 2005 21.9 15.3 9.5 8.3 6.0 2.1 43 28 62 40 64 26 2006 23.6 16.3 7.9 7.6 5.2 1.8 34 22 49 32 65 24 2007 24.5 18.3 9.8 7.5 6.7 2.0 40 27 54 37 68 27 2008 24.7 18.1 11.9 7.6 8.0 2.0 48 32 66 44 67 26 2009 46.4 28.1 10.9 8.8 7.1 2.0 23 15 39 25 66 22 2010 58.5 35.5 13.3 11.1 8.8 2.1 23 15 37 25 67 19 2011 42.6 31.1 13.7 10.6 9.4 1.9 32 22 44 30 69 18 Total 332.9 220.3 117.0 102.8 76.6 26.7 35 23 53 35 65 26 Source: World Bank data. Note: ICB Contract data includes only contracts subject to World Bank prior review. ICB = International Competitive Bidding; IL = investment lending. a. The value of contracts entered in the Form 384 represents the total commitment against a contract and not the actual portion of the contract eligible for Bank disbursement. Although broad directional trends in the ratios of these contracts to IL are relevant, the Bank financed portion of the contracts will be lower for all years. Because of the absence of contract accounting at the Bank’s controllers, it is not possible to determine disbursement of the Bank- funded portion of an ICB contract. 2 APPENDIX A: BANK PROCUREMENT SYSTEMS Table A.2. World Bank Prior Review & ICB Contract Data, (goods, civil works and services) (2000–11) Value of No. of Prior ICB ICB All new No. of all review ICB contract contrac commit- IL new Prior prior No. of contracts contract Prior s as % ts as % ments commit- review reviewed ICB ICB as % of s as % review ICB of prior of prior (ILs+DPL ments contracts (G&CW) contracts (G&CW) new of new contract contract review review s) ($B ($ (G&CW) ($ contracts (G&CW) contract commit- commit- s as % s as % contract contrac FY billions billions) billions) (000) ($ billions) s (000) ments ments of ILa of ILa s ts 2000 15.3 10.2 6.6 5.0 5.2 2.8 43 34 65 51 79 56 2001 17.3 11.5 6.5 4.6 5.2 2.5 38 30 56 46 81 55 2002 19.5 9.7 6.4 5.0 5.0 2.6 33 25 66 51 77 51 2003 18.5 12.5 6.5 4.8 4.8 2.4 35 26 52 38 74 50 2004 20.1 13.9 7.3 4.7 5.1 2.4 36 26 52 37 70 51 2005 21.9 15.3 8.0 4.5 6.0 2.1 36 28 52 40 76 47 2006 23.6 16.3 6.7 3.8 5.2 1.8 28 22 41 32 77 48 2007 24.5 18.3 8.3 4.1 6.7 2.0 34 27 46 37 81 51 2008 24.7 18.1 10.3 3.9 8.0 2.0 42 32 57 44 77 51 2009 46.4 28.1 9.5 4.5 7.1 2.0 21 15 34 25 75 43 2010 58.5 35.5 11.4 5.0 8.8 2.1 19 15 32 25 78 42 2011 42.6 31.1 12.2 4.6 9.4 1.9 29 22 39 30 77 42 Total 332.9 220.3 99.8 54.5 76.6 26.7 30 23 45 35 77 49 Source: World Bank data. Note: ICB contract data includes only contracts subject to World Bank prior review. ICB = international competitive bidding; IL = investment lending. a. The value of contracts entered in the Form 384 represents the total commitment against a contract and not the actual portion of the contract eligible for Bank disbursement. Although broad directional trends in the ratios of these contracts to IL are relevant, the Bank financed portion of the contracts will be lower for all years. Because of the absence of contract accounting at the Bank’s controllers, it is not possible to determine disbursement of the Bank-funded portion of an ICB contract. 3 APPENDIX A: BANK PROCUREMENT SYSTEMS Table A.3. World Bank Lending: DPLs and Investment Lending (new commitments, 2002–11) Loan commitment values Number of loans ($ billions) Proportion of IL projects % Approval FY Dev policy Invest-ment Dev policy Investment By value By number lending lending lending lending of loans 2002 51 192 9.8 9.7 50 79 2003 50 209 6.0 12.5 67 81 2004 43 213 6.2 13.9 69 83 2005 56 236 6.6 15.3 70 81 2006 53 244 7.3 16.3 69 82 2007 57 261 6.3 18.3 74 82 2008 47 272 6.6 18.1 73 85 2009 71 256 18.4 28.1 60 78 2010 85 299 23.0 35.5 61 78 2011 69 315 11.6 31.1 73 82 Total 582 2497 101.7 198.7 66 81 Average excluding 53.25 242.75 7.5 16.9 68 82 FY09-10 Source: World Bank data. Note: IL = investment lending. Table A.4. World Bank Disbursements, Prior Review and International Competitive Bidding Contracts (goods, civil works, and services), 2000–11 Prior review ICB Prior review contrac contra Total ICB contracts contracts as ICB contracts Prior review ts ($ cts ($ disbursement IL as % of IL % of IL as % of total contracts as % billions billion s (ILs+DPLs) disbursement disbursement disbursement disbursement of total FY ) s) ($ billions) s ($ billions) sa sa s disbursements 2000 8.2 5.2 18.4 12.6 41 65 28 45 2001 7.6 5.2 17.1 11.4 46 67 31 44 2002 7.6 5.0 17.8 11.0 45 69 28 43 2003 7.8 4.8 18.9 10.6 45 74 25 41 2004 8.9 5.1 16.9 10.9 47 81 30 53 2005 9.5 6.0 18.6 12.3 49 77 33 51 2006 7.9 5.2 20.7 12.8 40 62 25 38 2007 9.8 6.7 19.2 12.9 52 76 35 51 2008 11.9 8.0 19.6 13.3 60 89 41 61 2009 10.9 7.1 27.6 16.7 43 65 26 39 2010 13.3 8.8 40.1 19.5 45 68 22 33 2011 13.7 9.4 32.1 19.6 48 70 29 43 Total 117.0 76.6 266.7 163.5 47 72 29 44 Source: World Bank data. Note: ICB Contract data include only contracts subject to World Bank prior review. DPL = development policy lending; ICB = international competitive bidding; IL = investment lending. a. The value of contracts entered in the Form 384 represents the total commitment against a contract and not the actual portion of the contract eligible for Bank disbursement. Although broad directional trends in the ratios of these contracts to IL are relevant, the Bank financed portion of the contracts will be lower for all years. Because of the absence of contract accounting at the Bank’s controllers, it is not possible to determine disbursement of the Bank-funded portion of an ICB contract. 4 APPENDIX A: BANK PROCUREMENT SYSTEMS REFERENCES TO PROCUREMENT ISSUES IN IEG EVALUATIONS Table A.5. Procurement Issues in IEG Evaluations Type of evaluation Corporate Sector CAE/CAS Dimension Procurement-related issues No. % No. % No. % Number of IEG evaluations reviewed in each category FY2002-11 30 56 54 Of which, number with evaluative Total FY02-FY11 12 23 29 findings on procurement FY 2002-06 3 11 13 FY 2007-11 9 16 16 Overall question Is procurement viewed as a problem in effective 10 83 21 91 24 83 development? Procurement recommendations Does the evaluation include a recommendation specific 4 33 5 22 7 24 to procurement? Analytic work and building country Was analytic work undertaken? 0 0 5 22 11 38 capacity Has analytic work helped to build borrower 1 8 2 9 7 24 capacity? Country capacity Did the evaluation identify legal/institutional 1 8 4 17 15 52 shortcomings in procurement? Does the evaluation identify lack of capacity within the 2 17 12 52 12 41 country? Bank capacity and organization Is there any discussion of inadequate capacity within the 5 42 8 35 9 31 Bank? Has decentralization of Bank staff helped procurement? 1 8 1 4 4 14 Donor harmonization Do current procurement processes impede or limit donor 7 58 2 9 5 17 harmonization? Use of country systems Is there progress toward the use of country systems, or 2 17 2 9 3 10 are there obstacles to their use? Effectiveness of Bank procurement Does the evaluation suggest that present systems in the 2 17 7 30 14 48 systems country have corruption problems? Does the evaluation suggest that present systems in 2 17 5 22 5 17 Bank procurement have corruption problems? Does the IEG evaluation suggest that procurement 1 8 8 35 13 45 Fiduciary issues reform is needed for reducing corruption? Efficiency Delaysa Does the IEG evaluation include any specific mention of 4 33 12 52 10 34 delays as a result of procurement? Cost/ Quality Is procurement identified as constraining quality or cost? 1 8 1 4 6 21 Sector-specific procurement Are there procurement issues specific to a particular 3 25 7 30 10 34 sector? Special topics in procurement Does the evaluation discuss procurement in CDD, PPP 1 8 4 17 0 0 or ICT areas? Private sector activity Does the evaluation suggest that procurement 0 0 3 13 3 10 reform is needed for higher participation of PS in World Bank bids? Source: IEG. Note: CAE = Country Assistance Evaluation; CAS = Country Assistance Strategy; CDD = community-driven development; ICT = information and communications technology; PPP = public-private partnership; PS = procurement specialist. a. Delays could be due to shortfalls on the side of either the Bank or the borrower, and they are not analyzed separately here. 5 APPENDIX A: BANK PROCUREMENT SYSTEMS COUNTRY SELECTIVITY FRAMEWORK FOR SAMPLE SELECTION Table A.6. Part 1. Field Visits and Desk Review Countries AAA products: CPARs, MAPS, and others AAA products: PERs and PEFAs Country Field visit Priority CPARS/CPIPs MAPS/UCS pilots PERs PEFAs PEFA series PEFA series NLTA and other 11 case desk review involving World led by World trust funds study countries Bank Bank countries (add 10) selected Azerbaijan 1 FY02; FY08 FY03 FY08 FY08 FY08 Bangladesh 1 FY02 FY02;FY09 FY06;FY11 FY06;FY11 FY06;FY11 Ethiopia 1 FY03; FY12 FY02;FY04;FY06;FY07;FY08;FY09;FY11 FY07;FY10 FY10 None Indonesia 1 FY01 OECD 07 FY03;FY05;FY06;FY07;FY10;FY11;FY12 FY07;FY11 FY07;FY11 FY07;FY11 FY08;FY11 Mexico 1 FY02; FY08 FY02;FY04;FY05;FY06;FY09 - Morocco 1 FY00; FY08 UCS Stage 1 & 2 FY02 FY09 FY09 FY09 Peru 1 FY01; FY05; FY08 OECD FY02;FY11 FY09 FY09 None Philippines 1 FY02; FY08; FY12 UCS Stage 1 & 2 FY09;FY10;FY11;FY12 FY10 FY10 FY10 FY08 Senegal 1 FY03 First OECD MAPs; UCS FY05;FY11 FY07;FY11 FY07 FY07 Stage 1 & 2 Tanzania 1 FY03 OECD FY03;FY04;FY06;FY08;FY09;FY10;FY11 FY06;FY09;FY10 FY06;FY09 FY06;FY09 FY06;FY07 Turkey 1 FY01 (FY08 was IFA) UCS Stage 1 FY07;FY09;FY12 FY09 FY09 FY09 Albania 1 FY01 OECD FY07 FY06;FY11 FY06;FY11 FY06;FY11 Bhutan 1 UCS Stage 1 FY10 FY10 FY10 FY07 Colombia 1 FY01; FY05 UCS Stage 1 FY04 FY09 FY09 FY09 Ghana 1 FY03 UCS Stage 1 FY04;FY05;FY06;FY08;FY09;FY11 FY05;FY06;FY10 FY06;FY10 FY06 FY06;FY07 Honduras 1 FY04 none FY07;FY10 FY06;FY09;FY11 FY06;FY09 FY06 Lao PDR 1 FY02 OECD FY02;FY11 FY06;FY10 FY10 FY10 Mozambique 1 FY02 none FY02;FY03;FY09;FY10 FY06;FY08;FY11 None None FY05 Rwanda 1 FY04 (CPIP) UCS Stage 1 & 2 FY03 FY08;FY10 FY08;FY10 FY08;FY10 FY11 Sierra Leone 1 FY04 (CPIP); FY12 OECD FY04;FY10 FY07;FY10 FY07;FY10 None Vietnam 1 FY02 OECD 6 APPENDIX A: BANK PROCUREMENT SYSTEMS Investment Lending and IDFs with Procurement Content Development Policy Lending with Procurement Content Country Field Visit 11 Priority IDF Grants IL (at least IL (with some Procurement content) DPL (with DPL (with some procurement prior Programmatic Case Study Desk 30% share of actions) DPL series Countries Review Procurement prior Selected Countries content) actions (add 10) >15%) Azerbaijan 1 FY02;FY05 FY08;FY09 FY02 Bangladesh 1 FY02;FY08 FY02;FY04;FY06;FY08;FY10;FY11 FY07;FY09 FY03;FY05;FY06;FY07;FY09 YES Ethiopia 1 FY03;FY04;FY06;FY08;FY09;FY10;FY11 FY04;FY05 Indonesia 1 FY03 FY05 FY08;FY09;FY10;FY11 Mexico 1 FY02;FY09 Morocco 1 FY05;FY09 FY03;FY06 Peru 1 FY05;FY06 FY03;FY04;FY05;FY09;FY10 Philippines 1 FY02;FY05;FY07;FY08 FY02;FY07 Senegal 1 FY07;FY10;FY11 FY06;FY07;FY08;FY09;FY10 Tanzania 1 FY06;FY10 FY02;FY05;FY06;FY09;FY11 Turkey 1 FY02 FY09 FY02 Albania 1 FY06 FY02;FY04;FY05;FY07 Bhutan 1 FY09 FY08 FY08 FY09 FY07;FY09 Colombia 1 FY08 FY09;FY10 FY03;FY05 FY03;FY04;FY05 More than 1 Ghana 1 FY03;FY10;FY11 FY05 FY03;FY05;FY06;FY07;FY08;FY09 YES Honduras 1 Lao PDR 1 FY03;FY05 FY02;FY07 FY02 Mozambique 1 FY03;FY10 FY06;FY10 FY06;FY07;FY08;FY09;FY10 More than 1 Rwanda 1 FY02;FY05 FY11 FY05;FY06;FY09;FY10;FY11 YES Sierra Leone 1 FY04;FY09;FY10 FY10;FY11 FY03;FY05;FY07;FY09;FY10;FY11 More than 1 Vietnam 1 FY03;FY05;FY10 FY03;FY04;FY11 FY10;FY11 FY04;FY09;FY10;FY11 More than 1 7 APPENDIX A: BANK PROCUREMENT SYSTEMS Prior Review Composition of Investment Lending Contracts Country Field Visit 11 Priority Percentage of Percentage of Percentage of Percentage of Investment Total Lending Ratio of Total Value FY12 Case Study Desk HNP Prior HNP Prior TWE Prior TWE Prior Lending Commitments Investment ($ millions) Countries Review Review Review Review Review Commitments FY12 ($ millions) Lending Selected Countries Contracts by Contracts by Contracts by Contracts by FY12 ($ millions) Commitments to (add 10) Number FY12 Value FY12 Number FY12 Value FY12 Total Commitments FY12 Azerbaijan 1 4 1 40 94 50.0 50.0 1.0 891.7 Bangladesh 1 17 12 8 70 866.3 866.3 1.0 609.5 Ethiopia 1 3 0 26 28 920.0 920.0 1.0 442.5 Indonesia 1 25 2 20 55 845.8 3245.8 0.3 110.3 Mexico 1 12 89 555.3 1457.5 0.4 53.9 Morocco 1 40 99 265.8 665.8 0.4 5.9 Peru 1 4 22 63 68 0.0 0.0 0.0 63.0 Philippines 1 10 2 12 38 325.0 825.0 0.4 78.4 Senegal 1 39 67 65.6 65.6 1.0 38.2 Tanzania 1 62 80 320.0 420.0 0.8 532.3 Turkey 1 59 5 3 56 500.0 1100.0 0.5 72.9 Albania 1 15 5 11 44 71.6 71.6 1.0 76.7 Bhutan 1 5.3 Colombia 1 3 39 360.0 660.0 0.5 18.3 Ghana 1 5 4 48 74 260.3 410.3 0.6 89.2 Honduras 1 36 92 18.2 104.2 0.2 59.1 Lao PDR 1 9 3 25 85 8.6 8.6 1.0 49.6 Mozambique 1 3 1 42 55 267.0 377.0 0.7 53.6 Rwanda 1 83 96 85.0 250.0 0.3 27.5 Sierra Leone 1 37 19 43.0 67.0 0.6 15.8 Vietnam 1 5 2 58 63 729.0 1149.0 0.6 702.9 8 APPENDIX A: BANK PROCUREMENT SYSTEMS Table A.6. Part 2: Other Bank Borrowers, by Region AAA Products: CPARs MAPS and Region/Country Others AAA Products: PERs and PEFAs PEFA series involving World PEFA series led NLTA and Other Trust CPARS/CPIPs MAPS / UCS Pilots NLTA and Other Trust Funds PEFAs Bank by World Bank Funds AFRICA Angola FY03 FY06 - FY06 Benin FY07 Botswana FY06 FY09 FY06 Burkina Faso FY05 UCS Stage 1 FY07;FY10 FY07 None Burundi FY04 FY09 Cameroon FY05 OECD FY08 Cape Verde FY04 FY08 Central African Republic FY08;FY10 Chad OECD FY09 Comoros FY08 Congo, Democratic Republic of FY04 FY08 Congo, Republic of FY07 FY07 FY06 FY07 Cote d'Ivoire FY04 FY08 Eritrea FY03 - Gabon FY06 Gambia, The FY05 - Guinea FY02 FY07 Guinea-Bissau FY06;FY09 FY06;FY09 FY06 Kenya FY06 OECD FY06 FY06;FY09 FY06 None FY06 Lesotho FY08 FY07 Liberia FY07;FY10 FY09 FY07;FY10 Madagascar FY03 FY06;FY08 FY06;FY08 FY08 Malawi FY04 FY05;FY06;FY08 FY06;FY08 None Mali FY04 FY08;FY11 FY08;FY11 FY08 Mauritania FY02 FY08 FY08 None Mauritius FY03; 90% UCS Stage 1 & 2 FY07;FY11 FY11 None Namibia FY08 Niger FY04 OECD FY08 Nigeria FY04;FY07 FY05 - FY05 9 APPENDIX A: BANK PROCUREMENT SYSTEMS AAA Products: CPARs MAPS and Region/Country Others AAA Products: PERs and PEFAs PEFA series involving World PEFA series led NLTA and Other Trust CPARS/CPIPs MAPS / UCS Pilots NLTA and Other Trust Funds PEFAs Bank by World Bank Funds AFRICA Sao Tome and Principe FY07 FY07;FY10 Seychelles FY08;FY10 South Africa FY02 FY08 South Sudan FY09 - FY09 Sudan FY11 - FY11 Togo FY04 FY09 Uganda FY04 OECD FY06;FY08;FY09 FY09 Zambia FY03 FY05;FY08 Zimbabwe - EAST ASIA AND PACIFIC Cambodia FY04 China FY03 FY05;FY07;FY08 FY05;FY07;FY08 Fiji Islands FY05 Kiribati Mekong Mongolia FY03 OECD FY09 FY09 Papua New Guinea FY04 FY05:FY09 FY05:FY09 FY05:FY09 Samoa FY05 FY06;FY10 Solomon Islands FY08 Thailand FY05 FY09 Timor-Leste FY03 FY11 FY07;FY10 None None FY11 Tonga FY03 FY07;FY10 Tuvalu FY07;FY11 Vanuatu FY06;FY09 EUROPE AND CENTRAL ASIA Armenia FY03;FY09 FY08 Belarus FY09 Bosnia and Herzegovina FY02;FY07 - Bulgaria FY04 - Croatia FY04 - 10 APPENDIX A: BANK PROCUREMENT SYSTEMS AAA Products: CPARs MAPS and Region/Country Others AAA Products: PERs and PEFAs PEFA series involving World PEFA series led NLTA and Other Trust CPARS/CPIPs MAPS / UCS Pilots NLTA and Other Trust Funds PEFAs Bank by World Bank Funds AFRICA Georgia FY07 FY08 FY08 FY08 Kazakhstan FY06 FY09 Kosovo FY04 FY07;FY09 Kyrgyz Republic FY03;FY07 FY06;FY09 Lithuania - Macedonia FY02;FY07 UCS Stage 1 & 2 FY07 Moldova FY03;FY10 FY06;FY08;FY11 FY06;FY08 FY08 Montenegro FY09 Poland UCS Stage 1 & 2 - Romania FY05 - Russia FY07 - Serbia FY02 FY07;FY10 FY07;FY10 FY07 Slovak Republic - Tajikistan FY03 FY07 FY07 FY07 Ukraine FY07 FY07;FY11 FY07;FY11 FY07;FY11 Uzbekistan FY02 - LATIN AMERICA AND THE CARIBBEAN Antigua and Barbuda FY10 Argentina FY02 FY04 - FY04 Belize FY09 Bolivia FY09 Brazil FY02;FY04 UCS Stage 1 & 2 FY09 Chile FY04 - Costa Rica FY05 OECD FY10 Dominica FY03 FY10 Dominican Republic FY04 FY07;FY09;FY10 FY07;FY09 None Ecuador FY05 - El Salvador FY04 FY09 Grenada FY06;FY10 Guatemala FY05 OECD FY10 Guyana FY08 - 11 APPENDIX A: BANK PROCUREMENT SYSTEMS AAA Products: CPARs MAPS and Region/Country Others AAA Products: PERs and PEFAs PEFA series involving World PEFA series led NLTA and Other Trust CPARS/CPIPs MAPS / UCS Pilots NLTA and Other Trust Funds PEFAs Bank by World Bank Funds AFRICA Haiti FY07 FY08 Jamaica FY05 FY07 Nicaragua FY03 - Panama FY06 OECD - Paraguay FY03;FY07 FY08;FY11 FY08;FY11 FY08 St. Kitts & Nevis FY07;FY09 St. Vincent and the Grenadines FY06 St. Lucia FY06;FY10 Trinidad and Tobago FY06;FY08 Uruguay FY06 - MIDDLE EAST AND NORTH AFRICA Algeria FY03 FY10 FY10 FY10 Djibouti FY04 - Egypt FY04 FY09 Iran FY05 - Iraq FY05 FY11 FY08 FY11 Jordan UCS Stage 1 FY07 Kuwait FY08;FY11 FY10 Lebanon FY06 FY06 FY11 FY06 Libya - Saudi Arabia FY09 - Syrian Arab Republic FY10 FY06 Tunisia FY04 FY10 West Bank and Gaza FY05;FY09 FY07 Yemen, Rep. of OECD FY08 SOUTH ASIA Afghanistan OECD FY10 FY05;FY08 FY05;FY08 FY05;FY08 FY10 India FY03 UCS Stage 1 FY09;FY10 FY10 FY09;FY10 Maldives FY09 Nepal FY01 FY08 FY08 FY08 12 APPENDIX A: BANK PROCUREMENT SYSTEMS AAA Products: CPARs MAPS and Region/Country Others AAA Products: PERs and PEFAs PEFA series involving World PEFA series led NLTA and Other Trust CPARS/CPIPs MAPS / UCS Pilots NLTA and Other Trust Funds PEFAs Bank by World Bank Funds AFRICA Pakistan FY10 FY09 FY09 FY09 FY10 Sri Lanka FY03; OECD - Investment Lending and IDFs with Procurement Content Development Policy Lending with Procurement Content Region/Country IDF Grants IL (at least 30% IL (with some Procurement DPL (with share of DPL (with some procurement prior actions) Programmatic Procurement content) content) prior actions >15%) DPL series AFRICA Angola FY03 Benin FY04 FY04;FY05;FY07;FY09 YES Botswana Burkina Faso FY05 FY10 FY03;FY04;FY08;FY09;FY10 YES Burundi FY03 FY04;FY09;FY11 FY07 FY03;FY07;FY10 Cameroon FY08 FY05;FY08 FY06 Cape Verde FY10 FY10 FY07;FY09;FY10 Central African Republic FY09 FY09;FY11 FY09;FY11 More than 1 Chad FY07 FY02;FY03;FY05 FY02;FY03;FY05 More than 1 Comoros FY11 Congo, Democratic Republic of FY02;FY07;FY08;FY11 Congo, Republic of FY02;FY07;FY10 FY05 Cote d'Ivoire FY08 FY02 FY02;FY09 Eritrea FY02 Gabon FY06 Gambia, The FY02;FY08;FY10 Guinea FY05 FY02;FY11 FY02;FY11 YES Guinea-Bissau FY08 FY10 FY10 YES Kenya FY02 FY02;FY06;FY10;FY11 Lesotho FY08 Liberia FY08;FY11 FY09 FY09 YES Madagascar FY04;FY08 FY07 FY05;FY06;FY07;FY08 YES 13 APPENDIX A: BANK PROCUREMENT SYSTEMS Investment Lending and IDFs with Procurement Content Development Policy Lending with Procurement Content Region/Country IDF Grants IL (at least 30% IL (with some Procurement DPL (with share of DPL (with some procurement prior actions) Programmatic Procurement content) content) prior actions >15%) DPL series Malawi FY03 FY10 Mali FY07;FY11 FY08;FY09 Mauritania FY04 FY07 Mauritius FY08 Namibia Niger FY02 FY10 FY11 FY02;FY04;FY09;FY11 YES Nigeria FY03;FY05;FY08 FY02;FY05;FY07;FY10 FY10;FY11 Sao Tome and Principe FY05 Seychelles FY10 FY10 South Africa South Sudan Sudan Togo FY08;FY09;FY10;FY11 Uganda FY04;FY06;FY08 FY08 FY03;FY04;FY05;FY06;FY07;FY08;FY11 YES Zambia FY06;FY11 FY11 Zimbabwe EAST ASIA AND PACIFIC Cambodia FY04 FY02;FY06 FY08 China FY02;FY05 FY06;FY09 Fiji Islands Kiribati Mekong Mongolia FY03 FY03;FY06;FY08;FY10 Papua New Guinea Samoa FY05 Solomon Islands Thailand FY04;FY07 FY04;FY10 Timor-Leste FY03;FY11 FY06 Tonga FY04;FY11 FY11 FY11 Tuvalu Vanuatu EUROPE AND CENTRAL ASIA 14 APPENDIX A: BANK PROCUREMENT SYSTEMS Investment Lending and IDFs with Procurement Content Development Policy Lending with Procurement Content Region/Country IDF Grants IL (at least 30% IL (with some Procurement DPL (with share of DPL (with some procurement prior actions) Programmatic Procurement content) content) prior actions >15%) DPL series Armenia FY03 FY04;FY10 Belarus Bosnia and Herzegovina FY02;FY04 Bulgaria Croatia FY10 FY10 Georgia FY06 FY06 FY04;FY06;FY07 Kazakhstan FY09 FY10 Kosovo FY05;FY10 Kyrgyz Republic FY03 FY03;FY09 FY03 Lithuania FY02 Macedonia FY04 FY04 FY04 YES Moldova FY04 FY05 FY07 Montenegro FY10 FY05 Poland Romania FY03 Russia Serbia FY02;FY04;FY07 FY10 FY10 Slovak Republic FY03 Tajikistan FY04 FY07;FY09 Ukraine FY03 FY03;FY08 FY09 FY02;FY04;FY09 YES Uzbekistan FY09 ALTIN AMERICA AND THE CARIBBEAN Antigua and Barbuda Argentina FY11 FY05;FY06;FY07;FY10;FY11 FY02 Belize Bolivia FY02 Brazil FY07;FY10 FY03;FY06;FY08;FY09;FY10 FY10 FY06;FY09;FY10;FY11 Chile FY02;FY08 Costa Rica FY11 Dominica FY07 Dominican Republic FY02 Ecuador FY02 FY03 El Salvador FY02 FY10 FY05;FY06;FY09 15 APPENDIX A: BANK PROCUREMENT SYSTEMS Investment Lending and IDFs with Procurement Content Development Policy Lending with Procurement Content Region/Country IDF Grants IL (at least 30% IL (with some Procurement DPL (with share of DPL (with some procurement prior actions) Programmatic Procurement content) content) prior actions >15%) DPL series Grenada FY06 Guatemala FY02;FY08 FY09 FY06;FY09;FY10 YES Guyana FY03 FY06 FY03;FY06 Haiti FY05 FY05;FY06 FY10;FY11 FY05;FY07;FY10;FY11 YES Jamaica FY06 FY02 Nicaragua FY04;FY11 Panama FY02;FY11 FY02;FY07;FY11 FY07 FY07;FY08;FY09 Paraguay FY05 FY04 St. Kitts & Nevis St. Vincent and the Grenadines FY04 St. Lucia Trinidad and Tobago Uruguay FY07 MIDDLE EAST AND NORTH AFRICA Algeria FY06 Djibouti FY05 Egypt Iran Iraq FY10 Jordan FY05 Kuwait Lebanon Libya Saudi Arabia Syrian Arab Republic Tunisia FY05 FY11 West Bank and Gaza Yemen, Rep of FY06;FY08;FY10;FY11 FY08 FY08 SOUTH ASIA Afghanistan FY02;FY05;FY07 FY02;FY05;FY07 FY09 FY05;FY06;FY07;FY09 India FY05;FY09 FY07;FY10;FY11 FY02;FY04;FY05;FY07;FY08 Maldives FY05 16 APPENDIX A: BANK PROCUREMENT SYSTEMS Investment Lending and IDFs with Procurement Content Development Policy Lending with Procurement Content Region/Country IDF Grants IL (at least 30% IL (with some Procurement DPL (with share of DPL (with some procurement prior actions) Programmatic Procurement content) content) prior actions >15%) DPL series Nepal FY02;FY08 FY03;FY05 FY04 Pakistan FY06 FY04;FY06;FY09;FY11 FY02;FY03;FY04;FY05;FY06;FY07;FY10 Sri Lanka FY04 FY08;FY10 17 APPENDIX A: BANK PROCUREMENT SYSTEMS Prior Review Composition of Investment Lending Contracts Region/Country Percentage of Percentage of HNP Percentage of Percentage of TWE Prior Investment Lending Total Lending Ratio of Investment Total Value HNP Prior Review Prior Review TWE Prior Review Contracts by Value Commitments FY12 Commitments Lending FY12 ($ Contracts by Contracts by Value Review FY12 ($ millions) FY12 ($ Commitments to Total millions) Number FY12 FY12 Contracts by millions) Commitments FY12 Number FY12 AFRICA Angola 0 0 26 93 70.5 Benin 1 39 3 13 56.0 86.0 0.7 12.4 Botswana 100 100 4.8 Burkina Faso 74 89 88.9 303.9 0.3 55.5 Burundi 16 7 35 51 15.0 50.0 0.3 18.8 Cameroon 6 18 24 10 162.0 162.0 1.0 20.3 Cape Verde 100 100 53.5 65.5 0.8 5.2 Central African Republic 2 2 7 2 17.0 17.0 1.0 38.1 Chad 27 62 25.0 25.0 1.0 6.0 Comoros 0.1 Congo, Democratic Republic of 16 18 42 36 182.2 Congo, Republic of 48 17 15.0 15.0 1.0 39.5 Cote d'Ivoire 4 3 20 46 150.0 300.0 0.5 41.5 Eritrea 0 0 0 0 0.0 Gabon 0.2 Gambia, The 0.0 6.0 0.0 3.9 Guinea 7 2 50 56 53.3 53.3 1.0 31.6 Guinea-Bissau 100 100 0.3 Kenya 43 13 35 84 711.9 711.9 1.0 566.1 Lesotho 2 2 12 93 23.0 23.0 1.0 17.0 Liberia 1 0 13 94 42.0 47.0 0.9 188.7 Madagascar 10 3 55 50 6.0 6.0 1.0 19.0 Malawi 4 2 20 43 235.0 235.0 1.0 33.5 Mali 39 16 53.0 53.0 1.0 17.7 Mauritania 53 76 24.6 Mauritius 100 100 0.0 35.0 0.0 0.2 Namibia 0 0 0 0 0.0 Niger 12 5 33% 47 50.0 115.0 0.4 30.6 Nigeria 13 8 50 69 1070.0 1145.0 0.9 358.6 18 APPENDIX A: BANK PROCUREMENT SYSTEMS Prior Review Composition of Investment Lending Contracts Region/Country Percentage of Percentage of HNP Percentage of Percentage of TWE Prior Investment Lending Total Lending Ratio of Investment Total Value HNP Prior Review Prior Review TWE Prior Review Contracts by Value Commitments FY12 Commitments Lending FY12 ($ Contracts by Contracts by Value Review FY12 ($ millions) FY12 ($ Commitments to Total millions) Number FY12 FY12 Contracts by millions) Commitments FY12 Number FY12 Sao Tome and Principe 20 71 0.0 4.2 0.0 0.6 Seychelles 0.5 South Africa 42 100 0.0 0.0 0.0 270.8 South Sudan 9 23 72 60 0.0 0.0 0.0 93.7 Sudan 4 11 2 2 5.5 Togo 14.0 28.0 0.5 11.1 Uganda 18 16 32 59 135.0 235.0 0.6 122.8 Zambia 15 18 51 78 50.0 80.0 0.6 59.9 Zimbabwe 71 96 1.7 EAST ASIA AND PACIFIC Cambodia 6 4 12 42 43.5 China 48 74 1260.0 1260.0 1.0 829.9 Fiji Islands 0.1 Kiribati 65 42 22.9 22.9 1.0 2.6 Mekong 26.0 26.0 1.0 0.2 Mongolia 45 60 15.0 15.0 1.0 14.7 Papua New Guinea 6 2 17.6 Samoa 46 90 23 8 8.0 8.0 1.0 6.2 Solomon Islands 26 36 0.0 2.0 0.0 1.9 Thailand 50 90 0.0 Timor-Leste 8 11 21 44 14.2 Tonga 88 51 27.2 36.2 0.8 5.9 Tuvalu 100 100 11.9 11.9 1.0 0.3 Vanuatu 0.2 EUROPE AND CENTRAL ASIA Armenia 11 12 23 58 44.0 124.0 0.4 54.7 Belarus 62 99 287.6 Bosnia and Herzegovina 28 18 2 11 160.0 160.0 1.0 19.3 Bulgaria 60 98 37.8 19 APPENDIX A: BANK PROCUREMENT SYSTEMS Prior Review Composition of Investment Lending Contracts Region/Country Percentage of Percentage of HNP Percentage of Percentage of TWE Prior Investment Lending Total Lending Ratio of Investment Total Value HNP Prior Review Prior Review TWE Prior Review Contracts by Value Commitments FY12 Commitments Lending FY12 ($ Contracts by Contracts by Value Review FY12 ($ millions) FY12 ($ Commitments to Total millions) Number FY12 FY12 Contracts by millions) Commitments FY12 Number FY12 Croatia 7 2 31 94 90.7 90.7 1.0 208.1 Georgia 11 1 25 50 173.0 213.0 0.8 135.1 Kazakhstan 9 1 3 93 1068.0 1068.0 1.0 129.4 Kosovo 1 0 8.9 Kyrgyz Republic 6 15 8 7 28.8 58.8 0.5 41.3 Lithuania 0 0 0 0 0.0 Macedonia 38 45 50.0 50.0 1.0 30.0 Moldova 18 24 8 26 29.2 29.2 1.0 28.7 Montenegro 18 13 44 53 16.0 101.0 0.2 14.4 Poland 27 3 0.0 991.4 0.0 12.3 Romania 32 14 0.0 1893.6 0.0 69.9 Russia 0.0 0.0 0.0 169.4 Serbia 7 1 17 80 213.0 Slovak Republic 0 0 0 0 0.0 0.0 0.0 0.0 Tajikistan 4 1 6 8 49.9 49.9 1.0 14.9 Ukraine 2 0 80 92 150.0 150.0 1.0 276.2 Uzbekistan 15 2 38 68 180.0 180.0 1.0 75.9 LATIN AMERICA AND THE CARIBBEAN Antigua and Barbuda 0 0 0 0 0.01 Argentina 21 27 16 47 192.6 Belize 80 16 1.74 Bolivia 19 4 6 36 79.00 79.00 1.00 9.60 Brazil 15 0 37 91 1658.0 3208.0 0.5 741.2 Chile 100 100 40.00 40.00 1.00 6.36 Costa Rica 13 39 1.1 Dominica 0 0 0 0 0.00 Dominican Republic 6 23 11 4 20.0 90.0 0.2 8.7 Ecuador 0.49 El Salvador 140.0 140.0 1.0 1.9 Grenada 5.00 5.00 1.00 1.15 20 APPENDIX A: BANK PROCUREMENT SYSTEMS Prior Review Composition of Investment Lending Contracts Region/Country Percentage of Percentage of HNP Percentage of Percentage of TWE Prior Investment Lending Total Lending Ratio of Investment Total Value HNP Prior Review Prior Review TWE Prior Review Contracts by Value Commitments FY12 Commitments Lending FY12 ($ Contracts by Contracts by Value Review FY12 ($ millions) FY12 ($ Commitments to Total millions) Number FY12 FY12 Contracts by millions) Commitments FY12 Number FY12 Guatemala 0 2 11.6 Guyana 0 0 0 0 0.00 Haiti 0 3 6 6 170.0 170.0 1.0 81.5 Jamaica 50 42 0.00 100.00 0.00 0.44 Nicaragua 34 4 29 71 65.00 65.00 1.00 37.91 Panama 50 23 0.0 66.0 0.0 13.1 Paraguay 65 85 0.00 100.00 0.00 0.72 St. Kitts & Nevis St. Vincent and the Grenadines 1.19 St. Lucia 1.39 Trinidad and Tobago 0 0 0 0 0.00 Uruguay 7 0 7 11 59.00 319.00 0.18 0.57 MIDDLE EAST AND NORTH AFRICA Algeria 0 0 0 0 0.0 Djibouti 6 20 6 6 19.2 19.2 1.0 5.9 Egypt 51 98 440.0 440.0 1.0 798.6 Iran 0 0 0 0 0.0 Iraq 6 3 36 63 45.4 Jordan 0.0 250.0 0.0 5.3 Kuwait 0 0 0 0 0.0 Lebanon 29 49 27.0 27.0 1.0 10.1 Libya 0 0 0 0 0.0 Saudi Arabia 0 0 0 0 0.0 0.0 0.0 0.0 Syrian Arab Republic Tunisia 31 18 50.0 50.0 1.0 0.9 West Bank and Gaza 17 5 11.7 Yemen, Rep. of 17 13 37 35 61.0 61.0 1.0 6.5 SOUTH ASIA Afghanistan 5 0 22 24 144.0 144.0 1.0 194.8 21 APPENDIX A: BANK PROCUREMENT SYSTEMS Prior Review Composition of Investment Lending Contracts Region/Country Percentage of Percentage of HNP Percentage of Percentage of TWE Prior Investment Lending Total Lending Ratio of Investment Total Value HNP Prior Review Prior Review TWE Prior Review Contracts by Value Commitments FY12 Commitments Lending FY12 ($ Contracts by Contracts by Value Review FY12 ($ millions) FY12 ($ Commitments to Total millions) Number FY12 FY12 Contracts by millions) Commitments FY12 Number FY12 India 11 7 31 76 3178.3 3178.3 1.0 921.2 Maldives 8 14 0.8 Nepal 34 33 51 62 83.0 143.0 0.6 113.9 Pakistan 22 55 1790.0 1790.0 1.0 412.9 Sri Lanka 5 5 19 82 324.0 324.0 1.0 30.5 22 APPENDIX A: BANK PROCUREMENT SYSTEMS COUNTRY CASE STUDY QUESTIONNAIRE Box A.1. Instructions for Filling out the Case Study Questionnaire  Interviews with different country stakeholders are intended to yield information, based on their perceptions, of the relevance and efficacy of (i) the Bank’s support to procurement capacity building and (ii) the development effectiveness of its own procurement modalities.  The questionnaire will be used in conjunction with documentary information on Bank support to the country procurement system (diagnostic work, technical assistance and lending) to be provided separately.  The final section of the questionnaire draws together information from both sources, to formulate an overall assessment of (i) and (ii) in the country concerned.  The entire case study is to be typed directly into this template in the fields indicated.  Text questions: Guide the discussion and provide an opportunity for the elaboration of ideas not included in the scored tables. Responses are to be provided in text form. Please insert below and expand as needed. Length: 1-2 paras to up to 2-3 pages or more, as needed.  Tables: Each section is followed by a table of scored questions. Scoring will be done by the consultant, following discussion with respondent. Put an X in the relevant box only.  If there are a group of respondents for a given section, the consultant will score the questions using her/ his judgment on the most prevalent view in the group. Wide score variation can be mentioned in the text response.  Scores on the four point scale are to be broadly interpreted as follows: High: 71% of the time or more; Medium: 51-70%; Low: 31-50% and Negligible: 1-30%.  Not Applicable or Not Relevant questions in scored tables: If a question is considered not applicable or not relevant, do not score it. If there are no X marks for a given question, it would be considered not applicable in the circumstance. Box A.2. Field Visits: Additional Guidance (Discussions with Board Members) (October 1 ,2012) Field visits: Additional Guidance (discussions with Board members) (Oct 1 2012) More focus on some aspects of the Bank's own procurement system, especially:  Value for money and the efficiency of present systems  Operation of the present system in terms of encouraging innovation and risk taking (especially, public-private partnership and information and communications technology [ICT])  Procurement capacity building - new areas and best practice—for example, e-procurement  Contract management - should this be under the purview of procurement? What are the issues in contract management that could be addressed through changes in the World Bank procurement process?  Differences in the views on Bank systems/ use of country systems by Ministry of Finance officials who have a portfolio perspective, and implementing units in individual ministries/implementing agencies - need to take each into account separately. The latter may wish to perpetuate Bank systems due to self interest.  Evidence on risk-reward trade-offs  Prior to the trip the consultant/staff should be informed about basic facts of how the Bank has intervened in the area of procurement capacity, as well as how well its own systems are working, based on documents made available to each team member.  During the field visit, (i) more facts as well as anecdotes, examples, data, will come to light and these should be described/woven into the text sections of the responses, together with any evidence. One key area is likely to be advisory work that is informal and quick response, 23 APPENDIX A: BANK PROCUREMENT SYSTEMS that we have found hard to trace. Other examples may be difficulties encountered in particular aspects of implementation of present procurement rules.  Second, the field visits present an opportunity to (ii) gather different person's perceptions on the Bank's effectiveness. Each perspective offers different insights and the scored sections are meant to represent the perceptions of each counterpart.  Afterwards - final write up. In section 7 of the Questionnaire, the consultant finally draws together both the facts and the perceptions (views of different stakeholders), to come up with a bottom line on both the key questions addressed in the Approach Paper - (i) effectiveness of Bank's capacity building and (ii) quality of Bank procurement systems with regard to the development effectiveness of its own lending. Box A.3. Guidance 1: Objectives of the Evaluation and Design of the Field Visit Public expenditure is a key policy tool, and good procurement practices are a major determinant of its effectiveness. Good national procurement practices are also essential to the success of the Bank’s newly-introduced lending instrument, the Program for Results. A key question for the planned evaluation is how effectively the Bank has helped build well-functioning public procurement systems. Bank policies and procedures for procurement under Bank-financed investment operations also impact on the development effectiveness of the Bank’s own lending. The Bank seeks to ensure that the funds it provides are used for the purpose intended, achieve their aim of supporting development as effectively and efficiently as possible, and ensure the integrity of Bank lending. A second question examined is how well the policies and procedures for public procurement under Bank-financed investment projects have fulfilled the aims of economy, efficiency and transparency. The present questionnaire is intended to be used for one of a set of around ten country case studies to be undertaken for the evaluation, to inform the questions raised above through interviews with key stakeholders. Each field visit is expected to include discussions with the following groups of stakeholders: (1) Bank staff (including Bank country management, the procurement coordinator and procurement staff for the concerned country and Bank task team leaders; (2) Procurement staff in the country (in Bank project implementation units, in Country Sector Ministries implementing Bank projects and in the country’s National Procurement Authority); (3) Private suppliers/ contractors including industry and trade representatives, and (4) Civil Society procurement review organizations. The questionnaire is therefore structured around these four pillars. In some cases, the same question will be posed to more than one stakeholder. This is intended to gauge differences in perception regarding the role of the Bank among different parties involved in the procurement process. The final section will triangulate the evidence from each set of stakeholders and summarize the findings of the evaluation in an overall assessment of the role of the Bank’s support to procurement and the effectiveness of its own procurement systems. Nature of Bank Interventions for Capacity Building Bank support to building country procurement capacity has taken multiple forms: First, advisory services to clients in the form of diagnoses of national procurement systems, using (1) the Country Procurement Assessment Report (CPAR); (2) the Methodology for Assessing Procurement Systems (MAPS), developed as a benchmarking exercise in conjunction with the OECD-DAC, self- standing Integrated Fiduciary Assessments, and Public Expenditure and Financial Accountability (PEFA) reports, that include a specific indicator on public procurement. Second, via its different forms of lending instruments, the Bank has supported the development of procurement capacity, including (1) through project implementation units established for the execution of Bank-financed projects; (2) technical assistance investment loans and (3) a series of Institutional 24 APPENDIX A: BANK PROCUREMENT SYSTEMS Development Fund (IDF) grants that focus in whole or part on the strengthening of borrower countries’ procurement systems. (4) Support for strengthening national procurement systems has also been offered through the Bank’s development policy operations (DPOs). DPOs are disbursed directly into countries’ budget systems, based on programs of agreed. Additional Themes based on CODE discussion of Approach Paper More focus on some aspects of the Bank's own procurement system, especially:  Value for money and the efficiency of present systems  Operation of the present system in terms of encouraging innovation and risk taking (especially, PPP and ICT)  Procurement capacity building - new areas and best practice? – for example, e-procurement  Contract management - should this be under the purview of procurement? What are the issues in contract management that could be addressed through changes in the WB procurement process?  Differences in the views on Bank systems/use of country systems by Ministry of Finance officials who have a portfolio perspective, and implementing units in individual ministries/implementing agencies - need to take each into account separately. The latter may wish to perpetuate Bank systems due to self interest.  Evidence on risk-reward trade-offs  Prior to the trip the consultant/staff should be informed about basic facts of how the Bank has intervened in the area of procurement capacity, as well as how well its own systems are working, based on documents made available to each team member.  During the field visit, (i) more facts as well as anecdotes, examples, data, will come to light and these should be described/woven into the text sections of the responses, together with any evidence. One key area is likely to be advisory work that is informal and quick response that we have found hard to trace. Other examples may be difficulties encountered in particular aspects of implementation of present procurement rules.  Second, the field visits present an opportunity to (ii) gather different person's perceptions on the Bank's effectiveness. Each perspective offers different insights and the scored sections are meant to represent the perceptions of each counterpart.  Afterwards—final write up. In section 7 of the questionnaire, the consultant finally draws together both the facts and the perceptions (views of different stakeholders), to come up with a bottom line on both the key questions addressed in the Approach Paper - (i) effectiveness of Bank's capacity building and (ii) quality of Bank procurement systems with regard to the development effectiveness of its own lending. Background—Country Program and Bank Procurement Interventions Table A.7. Key Persons, Definitions, and Abbreviations Guide: Persons to be Surveyed 1. Bank Country Management/ Operational Adviser BCM 2. Bank procurement staff / PIU staff BPS 3. Bank TTLs TTL 4. Country/ National Procurement Authority CPA 5. Country Sector Ministries implementing Bank projects CSM 6. Suppliers and industry or trade representatives STR 7. Civil Society procurement review organizations CSC 25 APPENDIX A: BANK PROCUREMENT SYSTEMS Key persons: Location and contact information Location and Telephone Email Country Director Senior Operations Advisor / Manager Head of WB Procurement – Country Senior Staff in country procurement (1) Senior Staff in country procurement (2) Administrative support to mission IEG control team member (SR/CP) Abbreviations and Explanations AAA Analytic and Advisory Activities (diagnostic reports, and so forth; see also ESW) CAS Country Assistance Strategy (Bank work program in a country for a three-year cycle) CASCR Country Assistance Strategy Completion Report (prepared by the region and reviewed by IEG end of CAS cycle) CDD Community Driven Development (Loans implemented frequently by NGOs and local communities) CMU Country Management Unit (This unit has the work plan and overall budget for Bank work on a country) CPAR Country Procurement Assessment Report (major WB diagnostic tool till around 2006 when it was rendered non- compulsory) CPIA Country Policy and Institutional Assessment (This is a score given by World Bank to all IDA countries, on the quality of the governance framework) CQS Consultants’ Qualifications Selection DPO Development Policy Operation (loan that disburses directly into the budget with no procurement specifics. Contrast IL and P4R- see below) FBS Fixed Budget Selection ICT Information and Communication Technologies (Areas of complex procurement) IDF Institutional Development Fund (Grants for technical assistance which were widely used for procurement) ESW Economic and Sector Work (see also AAA) FY Fiscal Year (in the World Bank: July 1 to June 30) ICB International Competitive Bidding IFA Integrated Fiduciary Assessment (Assessment that includes procurement and financial management) IL Investment Loan (Traditional loans that disburse against project elements, using the Bank procurement system. Currently around 70-80% of Bank lending; see also DPO and Program for Results.) INT Department of Institutional Integrity (World Bank department that investigates allegations of fraud and corruption) LCS Least Cost Selection LEGPrA Legal Procurement Adviser (Legal department staff that provide advice on procurement issues) MAPS Methodology for Assessment of National Procurement Systems. (Devised by OECD-DAC in conjunction with the World Bank and other MDBs as a standardized diagnostic tool for countries procurement systems. See: http://siteresources.worldbank.org/INTPROCUREMENT/Resources/OECE-Methodology.pdf) NCB National Competitive Bidding ORAF Operational Risk Assessment Framework. (devised by the World Bank as a tool for capturing overall project risk) P4R Program-for-Results (New World Bank lending instrument approved 2012; uses country procurement systems within a project) P-RAMS Procurement Risk Assessment & Management System (scorecard for assessing procurement risk. made mandatory from 2010. use greatly increased from 2012) PAS Procurement Accredited Staff (Accredited to clear / approve procurement transactions at the WB) PER Public Expenditure Review (Broad-based economists analysis on public expenditure; sometimes includes issues of procurement within budget management) PEFA Public Expenditure and Financial Accountability (Standardized reports on the economics of public expenditure; they include a brief section on procurement. See: http://web.worldbank.org/WBSITE/EXTERNAL/PEFA/0,,menuPK:7313471~pagePK:7313134~piPK:7313172~the SitePK:7327438,00.html) PPP Public-Private Partnership 26 APPENDIX A: BANK PROCUREMENT SYSTEMS QBS Quality-Based Selection QCBS Quality- and Cost-Based Selection RPM Regional Procurement Manager (there are eight; four World Bank regions have one each and two have two each) SSS Single Source Selection TTL Task Team Leader WPA Country Work Program Agreement (agreed between the CMU and various sector departments and ancillary units such as the procurement department in the region and so forth) Table A.8. Overview: World Bank Procurement Interventions in Country Summary Number / Name of Operations Year(s) (Board) Diagnostic, Analytic and Advisory work (CPAR, MAPS, IFA, PEFA) CPAR s MAPS Use of Country Systems Pilot Stage 1 Integrated Fiduciary Assessment PEFA 2007; 2010 Public Expenditure Reviews (that may have had reference to procurement) Non-Lending Technical Assistance Other Investment Lending, Policy Lending (DPOs); IDF and other Grants Title of Operation Type (IL, DPL, IDF, Year (Board) Amount % Procurement Other) ($m) Procurement Thresholds Short List National Countries NCB’$000 Works NCB ‘$000 Goods Consultants ‘$000 Example: Ethiopia 5,000 500 200 Table A.9. Overview: World Bank Lending Program, Key Projects, and Portfolio in Country New Commitments per FY Amount ($m) Investment IL in IL in Health, Value of prior Value of ICB (July 1 to June 30) lending Transport/ Nut., Pop./ Educ. review contracts contracts (%) Energy & (%) $m (Form 384) Mining (%) Key procurement Key projects FY06-present Year of approval Value US$m TTL (supervision phase) staff (Infrastructure: Energy / Transport) (Health / Education) CDD PPP ICT 27 APPENDIX A: BANK PROCUREMENT SYSTEMS Interviews with Country Management—The Country Context Box A.4. Guidance 2: Interviews with Senior Country Management Key respondents: Although country management (Country Director/Sr. Operations Adviser) is the main target for this section of the questionnaire, some questions are also relevant for Senior procurement staff working on the country, to see whether there is any difference in perception between these two groups. Some questions here are also repeated in the sections below for country procurement staff. Text responses in this section should focus on the responses from country management. Text responses in the following section should focus on responses from procurement staff. Any differences in response between country management and country procurement staff that may subsequently emerge are pointed out in the final summary section. Key Questions: Interviews with senior management in the relevant country are intended to focus on (i) the role of country strategy and the extent to which procurement capacity development is an objective; (ii) the role of the Bank in this context, compared to other partners in development; and (iii) the views of senior management with regard to how the procurement system of the Bank has functioned in terms of achieving the development effectiveness of its own lending. The order of the themes is reversed in the Guidance questions below to ease the introduction of the procurement theme, in the context of the lending program. Quantitative data requested: Work Program Agreements (that is, budget data for allocation out of country budgets to the procurement support function for country procurement strengthening as well as to operational procurement for the Bank’s own lending). Q2.1 Describe the overall pattern of Bank lending in the country in recent years, in terms of distribution by sector, and any trend moves toward policy-based lending in recent years. To what extent has the country lending program prominently included the following types of projects that call potentially for different approaches and methods of procurement: (i) large and complex infrastructure projects (Transport/Energy); (ii) decentralized projects in the areas of health and education, including those regarded as community-driven development (CDD) projects? Give examples. (iii) Have there been any projects in the following areas of potentially complex procurement: PPPs, projects with large ICT components? (iv) Are there any P4R projects under preparation? Response: Q2.2 How effective has Bank procurement s been in terms of enabling its lending to achieve project/development objectives, especially with reference to specific projects typical of (i) large value procurement in infrastructure/transport/energy sectors; (ii) decentralized projects such as CDDs with small value procurement and (iii) complex or innovative areas such as PPP, ICT ? Response: Q2.3 How far does current the Bank procurement system accommodate the overarching considerations (economy, efficiency, open competition, transparency and encourage the development of local markets)? Have there been any known or 28 APPENDIX A: BANK PROCUREMENT SYSTEMS specific issues or circumstances that have arisen in the context of Bank procurement that have made it difficult to take these considerations into account thus impairing the Bank fiduciary mandate? If there have been any Independent Procurement Reviews or audits in the past 5 years, have any issues been raised? Have there been any investigations in the country related to the Integrity Vice Presidency that have touched on procurement issues? Response: Q2.4 To what extent has the strengthening of the country procurement system been an element of Bank country strategy? Has procurement reform been an element of the present or previous country assistance strategies? If yes, has the evolution of progress achieved through the Country Assistance Strategy (CAS) been tracked over time? To what extent has country management provided support for advisory and non-lending technical assistance for procurement? Which Bank instruments to support PR capacity building (IDF Grants, Diagnostic work, DPLs, procurement reform support components in SILs) have been more effective? Has the Bank focused on new approaches/e-procurement? Response: Q2.5 Can the resource envelope for building country procurement capacity be identified separately from resources provided for support to procurement in project implementation? If yes, can you provide data on the overall resource transfer from the country work program agreement (WPA data) to procurement specialists for both support to project implementation and for assistance with developing the country’s own procurement capacity? Bank Budget Assigned for Bank Budget Assigned to Any supplementary sources for Total Country Procurement Associated with procurement capacity development procurement capacity development FY Bank Budget ($m) investment lending ($m) ($m) (Trust Funds, IDF Grants etc) ($m) FY06 FY07 FY08 FY09 FY10 FY11 FY13 Q2.6 To what extent has Bank support for improving the countries’ capacity for procurement been supplemented by other partners in development? How significant was the role of the Bank, relative to other partners? To what extent is the Bank looking at local/regional harmonization issues (for example, with the European Union)? Has the Bank been in the forefront in new areas eg e- procurement? If country procurement systems have improved over the past ten years to what extent can you attribute this to Bank intervention? Response: 29 APPENDIX A: BANK PROCUREMENT SYSTEMS How much has Bank support for country public procurement been integrated in the framework for overall public expenditure management? How effective have the Bank’s interventions been and how have they been received by the client? Table A.10. Interviews with Country Management Modest High (4) Substantial (3) /Low (2) Negligible (1) Part I - Supporting Country Procurement Capacity- Relevance and Effectiveness 2.1.1 To what extent has procurement capacity development been [ ] [ ] [ ] [ ] an objective of the CMU (Country Management Unit) (2005 to present)? 2.1.2 To what extent is improving country procurement capacity [ ] [ ] [ ] [ ] built in to the country work program? 2.1.3 To what extent have improvements in country procurement [ ] [ ] [ ] [ ] capacity been tracked over time in the CAS / CASCR exercises? 2.1.4 Have improvements in country procurement capacity through [ ] [ ] [ ] [ ] Bank efforts been effective? 2.1.5 To what extent has support for building country [ ] [ ] [ ] [ ] procurement capacity been provided by other partners in development? 2.1.6 If yes, how central was the role of the Bank compared to [ ] [ ] [ ] [ ] other development partners? 2.1.7 Have recent diagnostic tools for country procurement capacity [ ] [ ] [ ] [ ] been adequate, since mandatory updates of the CPAR ceased? (recent CPARs, MAPs, IFAs, PERs, PEFAs)? 2.1.8 To what extent does present information on country [ ] [ ] [ ] [ ] procurement capacity appear to be up to date, in terms of Bank project implementation capacity? 2.1.9 To what extent is present information on country [ ] [ ] [ ] [ ] procurement capacity up to date, in terms of overall country procurement capacity? 2.1.10 Has procurement been included as a component of other [ ] [ ] [ ] [ ] fiduciary assessment exercises? (PEFA, PER etc)? 2.1.11 To what extent does the country look at the process of integrating [ ] [ ] [ ] [ ] procurement reform with public expenditure management strengthening and the overall Budgeting, Planning and Audit process? 2.1.12 To what extent does the Bank’s country economics unit [ ] [ ] [ ] [ ] (PREM) look at public procurement issues in the context of public financial management / public expenditure reviews 2.1.13 To what extent has the reform of country procurement and [ ] [ ] [ ] [ ] financial management been integrated by the Bank? Part II. Procurement and the Development Effectiveness of Bank Projects Risk Management/ Efficiency 2.2.1 How has procurement risk been rated, on average, in recent Bank [ ] [ ] [ ] [ ] projects eg in P-RAMs? 2.2.2 How would you rate overall country risk eg as reflected in the CPIA? 2.2.3 To what extent has procurement risk been a concern in terms of [ ] [ ] [ ] [ ] Bank project execution? 2.2.4 To what extent could procurement risk be a concern in the overall [ ] [ ] [ ] [ ] public sector of the country? 2.2.5 Are the procurement risk assessments (P-RAMs) and capacity [ ] [ ] [ ] [ ] assessments that are done as part of project planning effectively integrated into the overall project risk assessment framework 30 APPENDIX A: BANK PROCUREMENT SYSTEMS Modest High (4) Substantial (3) /Low (2) Negligible (1) ORAF? 2.2.6 How effective is the P-RAMs / ORAF framework at identifying and [ ] [ ] [ ] [ ] reducing residual project procurement risk? 2.2.7 How effective are INT procedures and processes at flagging, [ ] [ ] [ ] [ ] identifying, and mitigating fraud and corruption risk in procurement? 2.2.8 Are present arrangement and thresholds s for prior and post review [ ] [ ] [ ] [ ] too stringent 4) or too lax 1), for helping to controlling risk? 2.2.9 Has concern about fiduciary risk detracted from efficiency in project [ ] [ ] [ ] [ ] implementation ( eg, by slowing procurement decision making among procurement staff)? 2.2.10 How effective are the Bank's internal clearance procedures TTLs, [ ] [ ] [ ] [ ] Procurement staff, RPMs, legal procurement staff and OPSOR, at controlling fiduciary risk? 2.2.11 Do the Bank’s internal clearance processes impact the efficiency of [ ] [ ] [ ] [ ] project execution? 4: high negative impact; 1: negligible impact). 2.2.12 How frequent is the incidence of delay in the award of contracts? [ ] [ ] [ ] [ ] 2.2.13 To what extent could delays be attributed to Bank processes for procurement administration? 2.2.14 To what extent could delays be attributed to limited borrower [ ] [ ] [ ] [ ] capacity for procurement? 2.2.15 To what extent could delays be attributed to the intrinsic complexity [ ] [ ] [ ] [ ] of the relevant projects? 2.2.16 Once a contract is awarded, is there still a need/ role for a [ ] [ ] [ ] [ ] procurement staff or accredited staff to oversee contract management / implementation? or, are TTL skills adequate)4=strong need; 1=negligible need Part III. Competition and Development Outcomes 2.2.17 Do the Bank's Procurement procedures ICB, NCB, and [ ] [ ] [ ] [ ] Consultants Selection Procedures in particular) lead to adequate competition in terms of numbers of bidders, openness and fairness, geographical reach, etc.? 4=more than adequate; 3=adequate, 2=only somewhat adequate1=inadequate) 2.2.18 Are the Bank’s oversight processes field staff for routine [ ] [ ] [ ] [ ] clearance, RPM for more complex projects and clearance at headquarters for the largest projects) appropriately balanced in terms of the level of review of procurement? 4=too much oversight; 3=high but not too high; 2=well balanced, 1=insufficient for the country) 2.2.19 Are bidders s more likely to show interest if the project is under [ ] [ ] [ ] [ ] Bank procurement policy and procedures as opposed to a project under national )? 2.2.20 Has the use of International Competitive Bidding (ICB) and [ ] [ ] [ ] [ ] consultant selection methods been appropriate in terms a balanced tradeoff between competition and quality and efficiency of project execution ? 4=too high; 3=high but not too high; 2=well balanced, 1=used too little) 2.2.21 To what extent do the Bank's procurement requirements [ ] [ ] [ ] [ ] positively support the integrity, transparency, economy, quality, and timeliness of delivering project outputs? 4=significant positive effect; 1=no effect or negligible) 2.2.22 To what extent do the Bank's procurement requirements [ ] [ ] [ ] [ ] adversely affect the economy, quality, and timeliness of delivering project outputs? 4=significant adverse effect; 1=no effect or negligible) 31 APPENDIX A: BANK PROCUREMENT SYSTEMS Modest High (4) Substantial (3) /Low (2) Negligible (1) 2.2.23 How do the Bank's procurement requirements affect development [ ] [ ] [ ] [ ] outcomes across the portfolio? 2.2.24 Is there a justification for continuing with the domestic preferences currently contained in the Bank’s ICB guidelines? Interviews; Key Bank Procurement Staff in the Country Box A.5. Guidance 3: Interviews with the Bank’s Senior Procurement Staff for the Country Key respondents: Respondents would include the most country-based procurement staff as well other key procurement team members who are field based. In cases where there is an overall procurement head assigned to a country who is Washington based, or in another country, a phone interview with such a person can be used to validate / qualify this section. Staff in Bank Project Implementation Units would also be included. Some questions in this section are also repeated in other sections. In such cases the questions are highlighted. Text responses in this section should focus on the responses from Procurement staff. Any differences in response between country procurement staff and other clusters (for example, Bank management, Bank task team leaders, country clients) that may subsequently emerge should be pointed out in the final summary section. Key Questions: Interviews with Bank procurement staff are intended to obtain more details on (i) capacity development for country procurement, whether through diagnostic and advisory work, through non-lending technical assistance/ grant based programs, or through lending – both investment lending and development policy operations. (ii) The interviews in this section will also focus on the extent to which Bank procurement supports the overall development effectiveness of Bank investment lending, including lending in areas that may pose particular challenges for Bank procurement. Finally (iii) the extent to which harmonization with procurement systems of other donors and with national procurement systems is an issue for project implementation is also explored. Quantitative data requested (details described below): Data on a sample of 50 recent prior review contracts, across a selection of the following sectors: infrastructure (transport and energy) as well as the social sectors (health and education), for ICB, NCB and consultants selection. Procurement Capacity Building through Diagnostic and Advisory Work Q3.1 Background Information. How many procurement staff are assigned to the country? How many are headquarters staff? At what levels? How are total procurement resources on the country divided between project reviews and capacity development? Is advisory work handled by field based staff or headquarters staff? How many projects are handled by each and how many contracts, subject to prior review are handled by each per month on average. What part of the total is referred to regional procurement management not at the country level (RPM offices) and 32 APPENDIX A: BANK PROCUREMENT SYSTEMS what part is referred to procurement anchor staff at the Bank’s headquarters (OPSOR or Operations Policy and Country Services). Response: Q3.2 Procurement Knowledge and Advisory work—Quantity and Quality of outputs. First, discussing the nature of the Bank’s economic and sector work (ESW) and analytical work on the countries’ procurement systems, what has been the most significant recent ESW undertaken—CPARs, MAPS? How up to date is the knowledge of the country procurement system as a result? Given that CPARs are no longer mandated, is there still adequate country knowledge of country procurement systems and capacity? What are your views on the extent to which the MAPs exercise has been able to maintain the required information? How systematic and how accessible is country procurement knowledge? Is this knowledge embedded primarily in ESW or primarily embedded in the context of projects? What has been the extent of client participation in these exercises and has it evolved over time? Response: Q3.3 Procurement Processes: Link of Procurement to Financial Management and the Budget Process. Have recent procurement diagnostics substantively discussed the integration of procurement into the Budgeting, Planning and Audit Process? Has this been included in recent Public Financial Management, PEFA or Public Expenditure Review reports? (i) Do they discuss whether procurement planning and associated expenditures are part of the budget formulation process and contribute to multiyear planning? (ii) Do they discuss whether the budget law and financial procedures support timely procurement, contract execution, and payment? (iii) Do they discuss whether procurement actions can only be initiated with existing budget appropriations? Response: Q3.4 Procurement Knowledge and Advisory work –Scope, Outputs and Results. Discuss, based on the detailed template below, the extent to which recent diagnostic and advisory work has included areas of (i) legal and regulatory reform; (ii) institutional strengthening/ market strengthening; and (iii) human capacity building. To what extent were the results envisaged in these programs achieved? Response: Q3.5 To what extent has the Bank been able to provide support to new modalities of procurement implementation, for example, through e-procurement, through advisory work on establishing framework agreements, and so forth? To what extent has the Bank discussed new procurement concepts/modalities, for example, value for money, competitive negotiation? 33 APPENDIX A: BANK PROCUREMENT SYSTEMS Q3.6 Procurement Diagnostics and the Use of Country Systems Pilot. Did the country participate or express a desire to participate in Bank's use of country systems pilot? If yes, to what extent did the country participate – Stage 1 MAPS diagnostics alone or also Stage 2: examination of equivalence of country and Bank systems? What are the views on the pilot experiences in the country or on the pilot program and its design? Were the criteria too lax or stringent, does the process measure the correct parameters? What was the value to the country of completing the pilot program? Is obtaining accreditation important/useful or would it rather continue to follow Bank policies for Bank-funded projects? In the opinion of the Bank procurement staff, is it possible to move towards a use of portions the country procurement system even if not a full use of the country system? What would be the main risks in doing so? Are there other suggestions for improvements to the process to achieve the objectives of the Paris accord for expanded use of country systems. Q3.7 Procurement Capacity Building – Quality of the Results Framework. To what extent was there an adequate results framework for Bank interventions to improve country procurement capacity? Was there a logical link between Bank inputs and expected outputs or achievements at the country level? Were results indicators appropriately chosen relative to the goals to be achieved? Were they quantifiable? Were baselines and targets specified? To what extent did the results framework target variables under Bank Group control? How did the results framework take account of exogenous factors—for example, collaboration with other IFIs/bilateral aid donors or the country’s own reform progress? Were the results that the various interventions aimed to achieve either excessively over- or underambitious? Does the operation have adequate provisions for monitoring and evaluation and is there a good chance this will be implemented? Q3.8 Outcomes. To what extent are the Bank interventions likely to achieve immediately expected results/ outputs)? To what extent are Bank reforms likely lead to better broader procurement outcomes? Are they likely to be sustainable? Table A.11. Interviews with Country Procurement Staff – Country Program and Diagnostic/Advisory Support Substantial Modest /Low High (4) (3) (2) Negligible (1) Supporting Country Procurement Capacity - Comparison with Management Perceptions 3.1.1 (2.1.1) To what extent has procurement capacity development been an [ ] [ ] [ ] [ ] objective of the CMU Country Management Unit) 2005 to present)? 3.1.2 (2.1.2)To what extent is improving country procurement capacity built in to [ ] [ ] [ ] [ ] the country work program? 3.1.3 (2.1.5)To what extent has support for building country procurement capacity [ ] [ ] [ ] [ ] been provided by other partners in development? 3.1.4 (2.1.6)If yes, how central was the role of the Bank compared to other [ ] [ ] [ ] [ ] development partners? 34 APPENDIX A: BANK PROCUREMENT SYSTEMS Substantial Modest /Low High (4) (3) (2) Negligible (1) 3.1.5 (2.1.8)To what extent does present information on country procurement [ ] [ ] [ ] [ ] capacity appear to be up to date, in terms of Bank project implementation capacity? 3.1.6 (2.1.9) To what extent is present information on country procurement [ ] [ ] [ ] [ ] capacity up to date, in terms of overall country procurement capacity? 3.1.7 (2.1.10) Has procurement been included as a component of other fiduciary [ ] [ ] [ ] [ ] assessment exercises? PEFA, PER etc)? 3.1.8 (2.1.11)To what extent does the country look at the process of integrating [ ] [ ] [ ] [ ] procurement reform with public expenditure management strengthening and the overall Budgeting, Planning and Audit process? 3.1.9 (2.1.12)To what extent does the Bank’s country economics unit PREM) look [ ] [ ] [ ] [ ] at public procurement issues in the context of public financial management / public expenditure reviews Quality and Achievements of Recent Diagnostic and Advisory Work on Procurement 3.1.10 (2.1.7)Have recent diagnostic tools for country procurement capacity been [ ] [ ] [ ] [ ] adequate, since mandatory updates of the CPAR ceased? recent CPARs, MAPs, IFAs, PERs, PEFAs)? 3.1.11 To what extent have more recent diagnostic tools, such as the MAPS been [ ] [ ] [ ] [ ] an improvement on earlier tools such as the CPAR? 3.1.12 To what extent has the MAPS exercise facilitated increased country [ ] [ ] [ ] [ ] ownership and improved dialogue with the government on procurement reform? 3.1.13 Were all pillars of the MAPS assessment covered? [ ] [ ] [ ] [ ] 3.1.14 Was the MAPS exercise limited to the Baseline Indicators or did also include [ ] [ ] [ ] [ ] Compliance and Performance indicators? 3.1.15 Is there any monitoring framework in place today with indicators for tracking [ ] [ ] [ ] [ ] improvements? 3.1.16 Did the government participate in preparation of procurement ESW/AAA? [ ] [ ] [ ] [ ] 3.1.17 Has client or stakeholder participation improved over time? [ ] [ ] [ ] [ ] 3.1.18 Is there a clear Action Plan, agreed with the country, for improving country [ ] [ ] [ ] [ ] procurement capacity over time, that is being implemented? Scope of Advisory Work – Legal Reform and Institution Building 3.1.19 Was legal reform, such as development of a procurement law or [ ] [ ] [ ] [ ] strengthening of commercial law, a part of the country procurement support program? 3.1.20 Was the Bank’s advice effective in having the new law or revisions [ ] [ ] [ ] [ ] promulgated? 3.1.21 Was the Bank effective in helping aligning the law with best practice? [ ] [ ] [ ] [ ] 3.1.22 How effective was the Bank’s efforts with respect to seeing the law applied in [ ] [ ] [ ] [ ] practice? 3.1.23 Did the Bank’s work help establish an arbitration process? [ ] [ ] [ ] [ ] Scope of Advisory Work – Institutional Strengthening 3.1.24 Was institutional strengthening a part of the capacity building program? [ ] [ ] [ ] [ ] 3.1.25 Did the Bank contribute to the establishment of a functioning central [ ] [ ] [ ] [ ] policy/regulatory body that oversees public procurement for the country? 3.1.26 Was the Bank able to have an effective complaints and dispute resolution [ ] [ ] [ ] [ ] function established? 3.1. 27 Did the Bank effectively assist with establishing a central portal ideally [ ] [ ] [ ] [ ] electronic) for dissemination of procurement information laws, procedures, opportunities, call for bids, contract awards, and policy announcements)? 3.1.28 How effective was the Bank in assisting the development of procurement [ ] [ ] [ ] [ ] tools standard bidding documents, standard requests for proposal, standard 35 APPENDIX A: BANK PROCUREMENT SYSTEMS Substantial Modest /Low High (4) (3) (2) Negligible (1) contract forms, guidance notes) 3.1.29 How effective was the Bank’s assistance at furthering the development of [ ] [ ] [ ] [ ] electronic procurement in the country? 3.1.30 Has the Bank been successful in helping establish a debarment process? [ ] [ ] [ ] [ ] 3.1.30b Has the Bank been helpful in terms of advice on new procurement [ ] [ ] [ ] [ ] modalities, such as framework agreements? 3.1.30c Has the Bank been able to provide guidance on new procurement [ ] [ ] [ ] [ ] modalities such as e-procurement? 3.1.30d Has the Bank discussed new procurement concepts such as value for [ ] [ ] [ ] [ ] money? 3.1.30d Has the Bank discussed new procurement concepts such competitive [ ] [ ] [ ] [ ] negotiation? Use of Country Systems Pilot 3.1.31 To what extent did the country express an interest in participating in the [ ] [ ] [ ] [ ] Bank’s Use of Country Systems pilot? 3.1.32 Did the country believe that the use of its own procurement policy and [ ] [ ] [ ] [ ] procedures for Bank projects would be an improvement on the use of the Bank’s system? 3.1.33 In the views of Bank procurement staff would it be possible to take [ ] [ ] [ ] [ ] incremental steps towards greater use of country systems even if the country system is not adopted in its entirety? Quality of the Results Framework and Monitoring and Evaluation - Procurement Diagnostic /AAA work 3.1.34 Overall, in the Bank’s interventions for building country procurement capacity, [ ] [ ] [ ] [ ] was there a logical and clear link between Bank inputs and expected outputs or achievements at the country level? 3.1.35 Were results indicators used, and were they appropriately chosen relative to [ ] [ ] [ ] [ ] the goals to be achieved? Were they quantifiable? 3.1.36 Were baselines and targets specified? [ ] [ ] [ ] [ ] 3.1.37 To what extent did the framework target variables under Bank Group control? [ ] [ ] [ ] [ ] 3.1.38 Did the results framework take account of exogenous factors – for example, [ ] [ ] [ ] [ ] collaboration with other bilateral aid donors / IFIs? 3.1.39 Were the results that the operation aimed to achieve over -ambitious, [ ] [ ] [ ] [ ] compared to the content of the operation? 3.1.40 To what extent is the operation likely to achieve its aimed results? [ ] [ ] [ ] [ ] 3.1.41 To what extent are its reforms likely to be sustainable? [ ] [ ] [ ] [ ] 3.1.42 To what extent is it likely to lead to the medium term strengthening of the [ ] [ ] [ ] [ ] procurement system? 3.1.43 Did the programs of intervention have adequate provisions for monitoring and [ ] [ ] [ ] [ ] evaluation monitoring and evaluation)? 3.1.44 Is there a good chance that monitoring and evaluation will be implemented? [ ] [ ] [ ] [ ] Procurement Capacity Building Through NLTA, IDF Grants, Investment or Development Policy Lending Q3.9 To what extent was there an overall program of professional capacity building for improved procurement discussed with the country/with other partners in development? Q3.10 To what extent was the building of human resources in procurement envisaged in Bank support? Did the Bank limit training to project implementation 36 APPENDIX A: BANK PROCUREMENT SYSTEMS unit staff for bank projects or were staff in government ministries, even if not related to Bank projects, involved? Was the national procurement office staff trained? Has a cadre of trained public procurement officials been established? Was the training sustainable and was the country able to continue the program after the end of the Bank intervention? Does the country retain the cadre of public procurement professionals or do they leave for other employment? Q3.11 The Bank has used different vehicles for providing support to the development of procurement capacity in its client countries, in addition to its diagnostic and advisory work. Specifically, these have included other forms of non- lending technical assistance such as grant funds including Institutional Development Funds (IDF grants); support through investment lending through loans wholly or partly oriented towards procurement reform. Also in the course of Bank normal investment lending, counterparts acquire valuable procurement skills. Finally, the Bank has also sometimes included elements of procurement reform as prior actions in its DPOs as a part of overall public sector management. Discuss the relative importance of these vehicles in the country, and their relative effectiveness at achieving sustained improvements in country procurement capacity – legal, institutional, market or human – and in achieving better procurement outcomes. Q3.12 Have IDF or other grants been used to support the development of new procurement modalities such as e-procurement, framework agreements, etc.? Has the Bank used its capacity building to discuss concepts such as value for money? Table A.12. Interviews with Bank Country Procurement Staff—Capacity Building High (4) Substan- Modest Negligible tial (3) /Low (2) (1) Planning and coordinating capacity development 3.2.1 To what extent have the Bank's capacity building initiatives been based on [ ] [ ] [ ] [ ] a diagnostic of needs? 3.2.2 Was the capacity building program developed and prioritized jointly [ ] [ ] [ ] [ ] between the Bank and the government? 3.2.3 Was there a preference in terms of vehicles for capacity building, either by [ ] [ ] [ ] [ ] the Bank or the government? 3.2.4 Did the Bank coordinate with other donors in design of capacity building [ ] [ ] [ ] [ ] support, and between different Bank interventions? Development of Human Resources in public procurement 3.2.5 Was HRD in procurement a part of the capacity building program? [ ] [ ] [ ] [ ] 3.2.6 Was the selection of individuals for training programs appropriate to the [ ] [ ] [ ] [ ] needs of developing procurement capacity in the country? 3.2.7 Was the Bank effective in mainstreaming procurement training into public [ ] [ ] [ ] [ ] sector capacity development ? 3.2.8 Did the Bank limit its training to PIU staff of Bank projects, or to [ ] [ ] [ ] [ ] counterparts for Bank projects? 3.2.9 Were HRD initiatives continued/sustained by the country following Bank / [ ] [ ] [ ] [ ] other donor support? 37 APPENDIX A: BANK PROCUREMENT SYSTEMS 3.2.10 Has a cadre of public procurement professionals been established? [ ] [ ] [ ] [ ] 3.2.11 Has the cadre of public procurement professionals trained been retained in [ ] [ ] [ ] [ ] public sector service? 3.2.12 Did the Bank's capacity building work assist with accreditation or [ ] [ ] [ ] [ ] certification process that has been sustained by the government? Procurement Capacity Building through Technical Assistance Loans/ other Investment Lending 3.2.13 Was procurement capacity building through specific targeted investment/ [ ] [ ] [ ] [ ] Technical assistance loans a significant vehicle for procurement capacity building in the country? 3.2.14 If not, did the country include significant components for procurement [ ] [ ] [ ] [ ] capacity building in other investment loans? Was procurement capacity building a significant objective of such loans? 3.2.15 Was the capacity building program coordinated with diagnostic work? [ ] [ ] [ ] [ ] 3.2.16 Were the right areas targeted? 3.2.17 Did project supervision offer an adequate mechanism for tracking / follow [ ] [ ] [ ] [ ] up? 3.2.18 Was there sufficient flexibility in the structure of the loan to adapt to needs [ ] [ ] [ ] [ ] over the course of program execution? 3.2.19 Overall, how suitable were these TA loans, or loan components in other [ ] [ ] [ ] [ ] loans, for the development of procurement capacity? Procurement Capacity Building through Bank Non-Lending Technical Assistance / IDF Grants 3.2.20 Was procurement capacity building through Non-lending Technical [ ] [ ] [ ] [ ] assistance programs or IDF Grant a significant vehicle for procurement capacity building in the country? 3.2.21 Was the funding sufficient for a program design which could have impact? [ ] [ ] [ ] [ ] 3.2.22 In the absence of supervision arrangements that apply to Bank lending, [ ] [ ] [ ] [ ] was there a suitable mechanism for follow up? 3.2.23 Was the NLTA/ IDF/ other grant coordinated with diagnostic work [ ] [ ] [ ] [ ] procurement diagnostics or PEFAs/ PERs)? 3.2.24 Was it coordinated with Bank lending for procurement reform via [ ] [ ] [ ] [ ] investment lending or DPLs)? 3.2.25 Overall, how suitable a vehicle were these NLTA/ IDF grants for the [ ] [ ] [ ] [ ] support of procurement capacity? Procurement Capacity building through Bank Investment Lending by Learning-By-Doing/ Transfer of Knowledge 3.2.26 To what extent did the Bank contribute to the support of developing [ ] [ ] [ ] [ ] procurement capacity through learning by doing or transfer of knowledge? 3.2.27 To what extent did learning by doing in procurement occur in sectors such [ ] [ ] [ ] [ ] as infrastructure eg. transport or energy), which had a large proportion of large value contracts / ICB? 3.2.28 To what extent did learning by doing in procurement occur in sectors such [ ] [ ] [ ] [ ] as health or education, which could have a large proportion of small value contracts / decentralized procurement administration? 3.2.29 What is the extent to which PIUs contribute to the development of human [ ] [ ] [ ] [ ] capacity in procurement? 3.2.30 Do executing agencies prefer to use PIUs, rather than to do procurement [ ] [ ] [ ] [ ] through existing government structures? 3.2.31 What is the extent to which project procurement implementation is [ ] [ ] [ ] [ ] undertaken directly by staff in government agencies? 3.2.32 Are other arrangements, such as use of procurement management [ ] [ ] [ ] [ ] contracts ever used? 3.2.33 How effective has the Bank been at ensuring the capability of PIUs? [ ] [ ] [ ] [ ] 3.2.34 How successful has the Bank been at reducing the use of PIUs in [ ] [ ] [ ] [ ] accordance with the objectives of the Paris declaration on aid effectiveness? 38 APPENDIX A: BANK PROCUREMENT SYSTEMS 3.2.35 To what extent do country / government staff who have gained experience [ ] [ ] [ ] [ ] under one Bank project then continue to work on the procurement of other Bank projects? 3.2.36 To what extent are country / government staff who have gained experience [ ] [ ] [ ] [ ] under Bank projects then able to transfer their skills to national procurement? 3.2.37 To what extent do the use of the Bank's Procurement Guidelines and [ ] [ ] [ ] [ ] Standard Documents help develop country procurement capacity? 3.2.38 Overall, how useful a vehicle is investment lending under the Bank’s [ ] [ ] [ ] [ ] current guidelines for the support of country procurement capacity? Procurement Capacity Building through Development Policy Lending 3.2.39 Were prior actions oriented towards procurement reform important in the [ ] [ ] [ ] [ ] country context? 3.2.40 Was there sufficient coordination between the Bank's Economic Policy [ ] [ ] [ ] [ ] staff and Procurement staff? 3.2.41 Were prior actions monitored? [ ] [ ] [ ] [ ] 3.2.42 In the case of programmatic DPOs, were triggers effectively used? [ ] [ ] [ ] [ ] 3.2.43 Were inputs provided from diagnostic work? [ ] [ ] [ ] [ ] 3.2.44 Overall, how useful a vehicle is investment lending under the Bank’s [ ] [ ] [ ] [ ] current guidelines for the support of country procurement capacity? Country Procurement Capacity – Achieving Collateral Objectives 3.2.45 How effectively do the Bank's procurement regulations and practices [ ] [ ] [ ] [ ] contribute to the development of local and regional suppliers, manufacturers and contractors? 3.2.46 How effectively do the Bank's procurement regulations and practices [ ] [ ] [ ] [ ] contribute to the government's policies regarding the environment green procurement)? 3.2.47 How effectively do the Bank's procurement regulations and practices [ ] [ ] [ ] [ ] contribute to the government's policies related to improving labor standards? 3.2.48 How effectively do the Bank's procurement regulations and practices [ ] [ ] [ ] [ ] contribute to the government's policies regarding social development of disadvantaged groups, indigenous groups, or others identified for special attention? 3.2.49 How effectively do the Bank's procurement regulations and practices [ ] [ ] [ ] [ ] contribute to the government's policies that specifically target economically depressed areas, groups, or individuals poverty reduction)? 3.2.50 Has the Bank been helpful in terms of advice on new procurement [ ] [ ] [ ] [ ] modalities, such as framework agreements? 3.2.51 Has the Bank been able to provide guidance on new procurement [ ] [ ] [ ] [ ] modalities such as e-procurement? Quality of the Results Framework and Monitoring and Evaluation – Lending and NLTA (see also Q 3.1.34 to 3.1.44 above) 3.2.52 Overall, in the Bank’s lending interventions for building country [ ] [ ] [ ] [ ] procurement capacity, was there a logical link between Bank inputs and expected outputs or achievements at the country level? 3.2.53 Were results indicators used, and were they appropriately chosen relative [ ] [ ] [ ] [ ] to the goals to be achieved? Were they quantifiable? 3.2.54 Were baselines and targets specified? [ ] [ ] [ ] [ ] 3.2.55 To what extent did the framework target variables under Bank Group [ ] [ ] [ ] [ ] control? 3.2.56 Did the results framework take account of exogenous factors – eg, [ ] [ ] [ ] [ ] collaboration with other bilateral aid donors / IFIs? 3.2.57 Were the results that the operation aimed to achieve over -ambitious, [ ] [ ] [ ] [ ] compared to the content of the operation? 39 APPENDIX A: BANK PROCUREMENT SYSTEMS 3.2.58 To what extent is the operation likely to achieve its aimed results? [ ] [ ] [ ] [ ] 3.2.59 To what extent are its reforms likely to be sustainable? [ ] [ ] [ ] [ ] 3.2.60 To what extent is it likely to lead to the medium term strengthening of the [ ] [ ] [ ] [ ] country procurement system? 3.2.61 Did the programs of intervention have adequate provisions for monitoring [ ] [ ] [ ] [ ] and evaluation)? 3.2.62 Is there a good chance that M&E will be implemented? [ ] [ ] [ ] [ ] Procurement and the Development Effectiveness of Bank Lending Q3.13 In terms of the Bank’s own lending program, how effective is the current Bank procurement system in providing support to the achievement of the program objectives? First, are the present levels of review, procurement thresholds, and so forth, adequate for ensuring that procurement processes achieve economy and efficiency in Bank lending? Do they help to control fraud and corruption and ensure that money is used for its intended purposes? Are the present risk management frameworks effective? Response: Q3.14 To what extend to the Bank’s procurement systems help to achieve fair competition among international and national suppliers for the Bank’s contracts? Is the level and the quality of outreach adequate? To what extent do capable and reputable candidates successfully bid for Bank projects? Response: Q3.15 To what extent does the overall Bank project execution system provide for appropriate supervision of contract management? Is this given adequate attention in the supervision process? To what extent are procurement skills needed in the contract management phase of project execution? Response: Q3.16 Discuss with reference to actual projects/components of projects in the country, the adequacy and appropriateness of the Bank’s procurement methods in those areas of Bank lending that may need more specialized procurement skills—for example, (i) community-driven development projects (CDDs), where procurement functions are sometimes entrusted to non-government personnel in a number of decentralized implementation units; (ii) in the area of public-private partnerships (PPP), especially in infrastructure, where the emphasis is on risk sharing between the government, the Bank, and private contractors; and (iii) projects intensive in information technology (ICT), where projects vary widely and where the Bank or government entity may not know all the options available. Do Bank procurement systems allow attention to concepts such as value for money? Response: Q3.17 Discuss the extent to which the harmonization of Bank procurement regulation and procedures with those of other bilateral aid donors or other MDBs has led to any difficulties in project lending, for example, in the areas of sector-wide lending (or approaches), or in terms of other projects with parallel or joint financing. Response: 40 APPENDIX A: BANK PROCUREMENT SYSTEMS Table A.13. Interviews with Bank Country Procurement Staff—Procurement and the Bank’s Development Effectiveness—Risk Management/Efficiency (see references to the same questions in Table A.12) High (4) Substantial Modest /Low Negligible (3) (2) (1) 3.3.1 (2.2.1) How has procurement risk been rated, on average, in recent [ ] [ ] [ ] [ ] Bank projects eg in P-RAMs) 3.3.2 (2.2.2) How would you rate overall country risk eg as reflected in the [ ] [ ] [ ] [ ] CPIA) 3.3.3 (2.2.3) To what extent has procurement been a fiduciary concern in [ ] [ ] [ ] [ ] terms of Bank project execution? 3.3.4 (2.2.4) To what extent could procurement be a fiduciary concern in the [ ] [ ] [ ] [ ] overall public sector of the country? 3.3.5 (2.2.5)Are the procurement risk assessments P-RAMs) and capacity [ ] [ ] [ ] [ ] assessments that are done as part of project planning effectively integrated into the overall project risk assessment framework ORAF)? 3.3.6 (2.2.6) How effective is the P-RAMs / ORAF framework at identifying [ ] [ ] [ ] [ ] project procurement risk? 3.3.7 (2.2.7)How effective are INT procedures and processes at flagging, [ ] [ ] [ ] [ ] identifying and mitigating fraud and corruption risk in procurement? 3.3.8 (2.2.8) Are present arrangements and thresholds for prior and post [ ] [ ] [ ] [ ] review too stringent 4) or too lax 1), for helping to controlling fiduciary risk? 3.3.9 (2.2.9) Has concern about fiduciary risk unduly impacted procurement [ ] [ ] [ ] [ ] decision making among procurement staff? 3.3.10 (2.2.10) How effective are the Bank's internal clearance procedures [ ] [ ] [ ] [ ] TTLs, Procurement staff, RPMs, legal procurement staff and OPSOR, at controlling fiduciary risk/ reducing misprocurement? 3.3.11 (2.2.11) Do the Bank’s internal clearance processes impact the efficiency [ ] [ ] [ ] [ ] of project execution? 4: high negative impact; 1: negligible impact). 3.3.12 (2.2.12) How frequent is the incidence of delay in the award of [ ] [ ] [ ] [ ] contracts? 3.3.13 (2.2.13) To what extent could delays be attributed to Bank processes for procurement administration? 3.3.14 (2.2.14) To what extent could delays be attributed to limited borrower [ ] [ ] [ ] [ ] capacity for procurement? 3.3.15 (2.2.15) To what extent could delays be attributed to the intrinsic [ ] [ ] [ ] [ ] complexity of the relevant projects? 3.3.16 (2.2.16) Once a contract is awarded, is there still a need/ role for a [ ] [ ] [ ] [ ] procurement staff or accredited staff to oversee contract management / implementation? or, are TTL skills adequate)4=strong need; 1=negligible need 3.3.17 To what extent are there problems encountered during contract [ ] [ ] [ ] [ ] implementation that could be traced back to flaws in setting up the procurement? 3.3.18 Is contract implementation under Bank financing more effective than that [ ] [ ] [ ] [ ] in projects financed by the government? Planning and oversight 3.3.19 In terms of project preparation, how effective is the Bank's procurement [ ] [ ] [ ] [ ] design / planning process at contributing to overall project outcomes 3.3.20 With respect to the Bank's requirement that the borrower prepare a [ ] [ ] [ ] [ ] procurement plan that is updated regularly throughout the project, to what extent does this requirement contribute to procurement outcomes? 3.3.21 Is the extent of use of International Competitive Bidding ICB) effective for [ ] [ ] [ ] [ ] 41 APPENDIX A: BANK PROCUREMENT SYSTEMS project outcomes?(4=too high, 3=appropriate, 2= somewhat low, 1=much too low) 3.3.22 To what extent does the requirement to use the Bank's standard bidding [ ] [ ] [ ] [ ] documents (SBD/standard request for proposal) for ICB, and where required, national SBD for NCB contribute to procurement outcomes? 3.3.23 To what extent does the Bank's oversight (no-objection) of [ ] [ ] [ ] [ ] prequalification/short/listing, bidding documents/ RFP, bid/proposal evaluation, and contract award help to appropriately weight factors such as:, bidder qualifications, and quality of bids and proposals, and contract fairness? 3.3.24 To what extent does the Bank's oversight appropriately weight bidding [ ] [ ] [ ] [ ] administrative requirements such as bid security association , powers of attorney, and authentication of document? 3.3.25 To what extent does the Bank's oversight (no-objection) of [ ] [ ] [ ] [ ] prequalification/ short-listing, bidding documents/RFP, bid/proposal evaluation, and contract award emphasize price? 3.3.26 Are the clearance levels of TTLs, PSs including PASs, RPMs, legal [ ] [ ] [ ] [ ] including Country Lawyer and/or LEGPrA) and OPRC appropriate (or could clearances take place at lower levels without affecting fiduciary considerations)? (4=too high, 3=appropriate, 2= somewhat low, 1=much too low) 3.3.27 Is Bank staff involved in reviewing and approving procurement actions [ ] [ ] [ ] [ ] competent for their assigned level? 3.3.28 Is the Bank able to respond quickly to special circumstances such as [ ] [ ] [ ] [ ] those requiring deviations from policies and procedures, or changes to procurement provisions of the financing agreement, or changes to previously agreed arrangements? Competition and Development Outcomes (see 2.2.17-2.2.23) 3.3.29 (2.2.17) Do the Bank's Procurement procedures ICB, NCB, in particular) [ ] [ ] [ ] [ ] lead to adequate competition in terms of numbers of bidders, openness to new bidders, geographical reach, etc.? (4=more than adequate; 3=adequate, 2=only somewhat adequate1=inadequate) 3.3.30 (2.2.18) Are the Bank’s oversight processes field staff for routine [ ] [ ] [ ] [ ] clearance, RPM for more complex projects and clearance at headquarters for the largest projects) appropriately balanced in terms of the level of review of procurement? (4=too much oversight; 3=high but not too high; 2=well balanced, 1=insufficient for the country) 3.3.31 (2.2.19) Are suppliers more likely to show interest if the project is under [ ] [ ] [ ] [ ] Bank procedures as opposed to a project under national procedures)? 3.3.32 (2.2.20) Has the use of International Competitive Bidding ICB) been [ ] [ ] [ ] [ ] appropriate in terms of the balance between achieving competition and improving quality and achieving ease of project execution ? (4=too high; 3=high but not too high; 2=well balanced, 1=used too little) 3.3.33 (2.2.21) To what extent do the Bank's procurement requirements [ ] [ ] [ ] [ ] positively support the integrity, transparency, cost, quality, and timeliness of delivering project outputs? (4=significant positive effect; 1=no effect or negligible) 3.3.34 (2.2.22) To what extent do the Bank's procurement requirements [ ] [ ] [ ] [ ] adversely affect the cost, quality, and timeliness of delivering project outputs? (4=significant adverse effect; 1=no effect or negligible) 3.3.35 (2.2.23) If there is sometimes a trade-off between achieving outcomes [ ] [ ] [ ] [ ] for a specific project and maintaining the integrity of a system across all of the Bank's portfolio, how do the Bank's procurement requirements affect development outcomes across the portfolio? 42 APPENDIX A: BANK PROCUREMENT SYSTEMS Table A.14. Procurement in Specialized Areas of Lending Community-Driven Development and other Decentralized Projects 3.4.1 To what extent have Community Driven Projects been important in the [ ] [ ] [ ] [ ] Bank’s lending program in the country, in the last 5 years? (name in Table 1.2) 3.4.2 To what extent have other decentralized development projects been [ ] [ ] [ ] [ ] important in the Bank’s lending program in the country, in the last 5 years? 3.4.3 To what extent do such projects use NCB procedures? [ ] [ ] [ ] [ ] 3.4.4 To what extent do such projects use ‘shopping’? [ ] [ ] [ ] [ ] 3.4.5 Has the training of decentralized project implementation staff in Bank [ ] [ ] [ ] [ ] procurement methods been difficult to achieve? 3.4.6 Has the training of decentralized project implementation persons in any [ ] [ ] [ ] [ ] sound procurement practicess been difficult to achieve? 3.4.7 To what extent are project difficulties encountered in areas such as financial [ ] [ ] [ ] [ ] management (account keeping, audit) rather than procurement? Public Private Partnership Projects in Infrastructure 3.4.8 To what extent have there been any significant PPP projects financed by [ ] [ ] [ ] [ ] the Bank in the past 5 years?(name in Table 1.2) 3.4.9 To what extent did the Bank get involved with the projects from the inception [ ] [ ] [ ] [ ] and design stage? 3.4.10 To what extent did the Bank enter later in the process as financier? 3.4.11 To what extent did the project face procurement issues in implementation [ ] [ ] [ ] [ ] (eg due to selection of partner not under Bank methods)? 3.4.12 How well did the Bank assist the client to understand its requirements and [ ] [ ] [ ] [ ] to develop a strategy for the procurement? 3.4.13 How useful are the Bank's various PPP resources manuals, information [ ] [ ] [ ] [ ] notes, web sites, etc) to prepare and implement a PPP project, especially in the area of procurement? 3.4.14 How appropriate and effective were the Bank's procurement procedures [ ] [ ] [ ] [ ] regarding award of concessions? 3.4.15 How appropriate and effective were the Bank's procurement procedures [ ] [ ] [ ] [ ] regarding procurement of goods, works and consulting services under the project? 3.4.16 How effective was the Bank's involvement during implementation of the PPP [ ] [ ] [ ] [ ] contract? 3.4.17 How well did the Bank coordinate its assistance with other financiers? [ ] [ ] [ ] [ ] 3.4.18 How adequate are the procurement provisions provided under Bank [ ] [ ] [ ] [ ] guidelines for PPPs ? Information and Communication Technology Projects 3.4.19 To what extent have there been any significantly complex ICT components [ ] [ ] [ ] [ ] in projects financed by the Bank in the past 5 years? (eg, involving hardware and customized software, professional services including service contracts and/or complex systems)? (name on Table 1.2) 3.4.20 How well did the Bank assist the client to understand its ICT requirements [ ] [ ] [ ] [ ] and to develop technical and bidding specifications suitable for procurement? 3.4.21 How appropriate and effective were the Bank's procurement procedures [ ] [ ] [ ] [ ] including standard bidding documents, guides and other procurement tools for the types of ICT systems procured? 3.4.22 To the extent that there were procurement problems, to what degree [ ] [ ] [ ] [ ] didthey relate to the procurement process itself (methods of selection/choice of bidding and contract documents/ administrative requirements)? 3.4.23 If there were problems in the procurement process, to what degree did [ ] [ ] [ ] [ ] 43 APPENDIX A: BANK PROCUREMENT SYSTEMS they relate to project technical specifications? 3.4.24 To what degree did the procurement problems relate to contract variations, [ ] [ ] [ ] [ ] contract administration, dispute resolution, contract payment issues? 3.4.25 To what extent were procurement problems related to project design, [ ] [ ] [ ] [ ] including design and planning of the procurement process? 3.4.26 To what extent were flexibilities available in Bank procurement guidance at [ ] [ ] [ ] [ ] the time made use of? 3.4.27 To what extent are there greater flexibilities in procurement for such [ ] [ ] [ ] [ ] projects today? 3.4.28 How effective was the Bank's supervisionduring ICT contract [ ] [ ] [ ] [ ] implementation? 3.4.29 Overall, how effective were the Bank's procurement methods and [ ] [ ] [ ] [ ] procedures for ICT at achieving outputs in terms of qualityand desired features, timely completion, cost, fair competition? 3.4.30b Would implementation of projects in this area benefit from new [ ] [ ] [ ] [ ] procurement modalities, such as framework agreements? 3.4.30c Would implementation of projects in this area benefit from new [ ] [ ] [ ] [ ] procurement modalities such as e-procurement? 3.4.30d Would implementation of projects in this area benefit from new [ ] [ ] [ ] [ ] procurement concepts such as value for money? Table A.15. Harmonization of Bank Procurement Practices with Other Donors and IFIs To what extent do differences between the Bank's procurement policies and those of other financing agencies (multilateral and 3.5.1 bilateral) adversely impact project implementation? [ ] [ ] [ ] [ ] Rate the extent to which such differences are problematic in the following [ ] [ ] [ ] [ ] specific areas: 3.5.2 (i) The design and planning process (rigor of defining [ ] [ ] [ ] [ ] requirements, costing, and scheduling procurement in the planning stages) 3.5.3 (ii) Requirements on establishing and staffing PIUs [ ] [ ] [ ] [ ] 3.5.4 (iii) Advertising requirements [ ] [ ] [ ] [ ] 3.5.5 (iv) Prequalification/short-listing requirements [ ] [ ] [ ] [ ] 3.5.6 (v) Procurement/selection methods and conditions of use [ ] [ ] [ ] [ ] 3.5.7 (vi) Preferences [ ] [ ] [ ] [ ] 3.5.8 (vii) Bidding processes, including use of standard bidding/RFP [ ] [ ] [ ] [ ] documents, procedures for issuing documents, and receipt of bids 3.5.9 (viii) Bid/proposal evaluation and factors for determining winning [ ] [ ] [ ] [ ] bid/proposal 3.5.10 (ix) Conditions of contract [ ] [ ] [ ] [ ] 3.5.11 (x) Oversight practices (review, no objection). [ ] [ ] [ ] [ ] 3.5.12 (xi) Complaints procedures [ ] [ ] [ ] [ ] 3.5.13 (xii) Debarment procedures [ ] [ ] [ ] [ ] 3.5.14 (xiii) Arbitration and mediation [ ] [ ] [ ] [ ] 3.5.15 (xiv) Other (specify) [ ] [ ] [ ] [ ] Request for Sample Contract Data Q3.18 For a sample of at least 50 recent contracts already awarded, please provide the following information, either by filling out the relevant table, or by providing the 44 APPENDIX A: BANK PROCUREMENT SYSTEMS raw data available. Please select the contracts to include around 30–35 from large value ICB contracts for goods, works and services. Please also include 10–15 contracts for consulting services. Remaining contracts (5–10) may include NCB subject to prior review. 45 APPENDIX A: BANK PROCUREMENT SYSTEMS Procurement Contract data – large value contracts Contract 1 Contract 2, etc Country Name Project No Project Name Year of Approval Contract Description Contract No Network Sector Board Major Sector Implementing Agency Contract Value estimated Contract Value awarded %Bank financed % financed by Borrower/ other sources Procurement category (goods / works / consulting services) Procurement method (ICB, ICB with preQ; ICB Two Stage; NCB, NCB with Pre-Q; NCB two Stage; for consulting: QBS, QCBS, FBS,LCS,CQS,SSS) Domestic preference allowed Misprocurement flag Price escalation flag Protest flag KEY PROCESSING DATES/ STAGES Goods and works (ICB, ICB with preQ; NCB, NCB with Pre-Q) – Borrower issue of specific procurement notice Borrower first submission to Bank of draft bid (preQ) documents Bank final no objection to draft bid (PreQ) documents No of iterations (No of times Borrower submitted draft bid docs to Bank) Borrower issue of Bid (PreQ) documents Borrower bid (PreQ) opening date/ Minutes Borrower submission to Bank of Pre-Q evaluation report (if applicable) Bank No-Objection to Pre-Q Evaluation Report (if applicable) Borrower submission to Bank of Bid evaluation report Bank No-Objection to Bid Evaluation Report Date of signature of contract Date of contract amendments Date of contract completion KEY PROCESSING DATES/ STAGES Goods and works (ICB/ NCB Two Stage) Borrower issue of specific procurement notice Borrower first submission to Bank of draft bid documents Bank final no objection to draft bid documents No of iterations (No of times Borrower submitted draft bid docs to Bank) Borrower issue of Stage 1 Technical Bid documents Borrower Stage 1 bid opening date/ Minutes Borrower submission to Bank of Stage 1 Eval Report / Minutes Bank No-Objection to Stage 1 Borrower issue of invitation to Stage 2 Bidding 46 APPENDIX A: BANK PROCUREMENT SYSTEMS Procurement Contract data – large value contracts Contract 1 Contract 2, etc Borrower Stage 2 bid opening date/ Minutes Borrower submission to Bank of Stage 2 Eval Report / Minutes Bank No-Objection to Final (Stage 2) Bid Evaluation Report Date of signature of contract Date of contract amendments Date of contract completion KEY PROCESSING DATES/ STAGES - Consultant services (large value) (QBS, QCBS, FBS,LCS,CQS,SSS) QBS, QCBS, FBS,LCS,CQS,SSS Borrower issue of specific procurement notice Borrower first submission to Bank of draft RFP Bank final no objection to draft RFP No of iterations (No of times Borrower submitted draft bid docs to Bank) Borrower issue of RFP Borrower Proposalsopening date/ Borrower submission to Bank of evaluation report Bank No-Objection to Evaluation Report Borrower submission to Bank of NecotiatedContrctNegotiatedContrat CCContract Bank No-Objection toNegotiated Contract Date of signature of contract Date of contract amendments Date of contract completion SUPPLEMENTARY INFORMATION IF AVAILABLE Number of bids received Bid Evaluation Committee Composition? (were details provided to Bank?) Bank comments on Committee composition (did Bank offer comments?) Bid Evaluation Committee meeting to define rating system?(If yes, Date) Bid Evaluation Committee meeting to define final scores? (if yes, Date) 47 APPENDIX A: BANK PROCUREMENT SYSTEMS Interviews; Field Office-based Bank Task Team Leaders Box A.6. Guidance 4: Task Team Leader Interviews Key Respondents: The key respondents in this section are Bank task team leaders, especially those working in the infrastructure (transport/energy) and social (health and education) sectors. Task team leaders of particular projects in the areas of PPP, ICT and CDD should also be interviewed. Task team leaders from other sectors may join as available. To the extent that there are field-based task team leaders of lending projects that may have included some components on procurement reform (for example, public expenditure reform DPOs) they would also be discussants for some sections. Key Questions: Interviews with task team leaders in the relevant country are intended to focus largely on how the procurement system of the Bank has functioned in terms of achieving the development effectiveness of its own lending, and to learn about obstacles/trade-offs that may be encountered. This will include questions on the extent to which procurement risk has been a factor in assessing and managing overall project risk. Some questions on building procurement capacity are also included. Repeated Questions: Several but not all questions here are deliberately repeated from previous sections, in order to capture the perspective of task team leaders, as distinct from other stakeholders in the Bank’s country procurement arrangements. If this is the case, the previous occurrences of the question are indicated. Text responses in this section should focus on the responses from task team leaders alone. Any significant differences in response between task team leaders and other respondents that may subsequently emerge may be pointed out in the final summary section. Q 4.1 To what extent do current Bank procurement policy and procedures for investment lending strike an appropriate balance in terms of safeguarding against fiduciary risk on the one hand, and enabling timely and efficient achievement of project outputs on the other? To what extent can delays and red tape be attributed to Bank procurement methods and procedures, as compared to borrower processes, or limits in terms of borrower capacity? Response: Q 4.2 Do Bank procurement methods achieve an adequate level of competition among bidders? Are new entrants qualified bidders and consultants encouraged to participate? Are bidders from other parts of the country/adjacent regions encouraged in NCB? Response: Q 4.3 To what extent does Bank procurement staff encourage or discourage fiduciary risk taking in the procurement process? Have you been adequately served by the procurement staff in their assessment of the part that procurement risk will play in overall project risk? Have procurement risk mitigation plans been realistic and effective in reducing residual risk? Is there an appropriate balance between the fiduciary objective and the (policy) considerations of economy, efficiency, quality, 48 APPENDIX A: BANK PROCUREMENT SYSTEMS open competition, and transparency? Is the level of attention to technical and administrative details appropriate? Response: Q 4.4 To what extent do new procurement arrangements such as P4R offer more effective new vehicles for handling procurement of certain projects? Response: Q 4.5 Have task teams encountered any difficulties in Bank procurement processes in the context of complex procurement (eg in the areas of large infrastructure, consulting services, PPP, ICT, and CDD project)? If so, elaborate. Are the newly introduced Guidance Notes in these areas helpful? To what extent is there a need for further changes in this area? Response: Q 4.6 In the case of co-financed projects, are Task Teams encountered any difficulties in project implementation due to differences in procurement methods and procedures of the Bank and other implementing agencies (bilateral or multilateral development bank) ? If so, elaborate. Response: Table A.16. Interviews with Bank TTLs—Procurement and Bank Development Effectiveness FOR TTLS: RISK MANAGEMENT/ EFFICIENCY (SEE REFERENCES TO THE SAME QUESTIONS IN TABLE A.12 ABOVE) Substan- Modest /Low Negligi High (4) tial (3) (2) ble (1) 4.1.1 (omitted) 4.1.2 (omitted) 4.1.3 (3.3.3; 2.2.3) To what extent has procurement been a fiduciary concern in [ ] [ ] [ ] [ ] the country in terms of Bank project execution? 4.1.4 (omitted) 4.1.5 (3.3.5; 2.2.5)Are the procurement risk assessments (P-RAMs) and [ ] [ ] [ ] [ ] capacity assessments that are done as part of project planning effectively integrated into the overall project risk assessment framework ORAF)? 4.1.6 (3.3.6; 2.2.6) How effective is the P-RAMs / ORAF framework at [ ] [ ] [ ] [ ] identifying project procurement risk? 4.1.7 (3.3.7; 2.2.7)How effective are INT procedures and processes at flagging, [ ] [ ] [ ] [ ] identifying and mitigating fraud and corruption risk in procurement? 4.1.8 (3.3.8; 2.2.8) Are present arrangements and thresholds for prior and post [ ] [ ] [ ] [ ] review too stringent 4) or too lax 1), for helping to controlling fiduciary risk? 4.1.9 (3.3.9; 2.2.9) Has concern about fiduciary risk unduly impacted [ ] [ ] [ ] [ ] procurement decision making among procurement staff? 4.1.9b To what extent is mis-procurement an issue in the country? Does this occur with a high level or frequency? 4.1.9c To what extent do projects have to be re-bid? Does this happen frequently? 4.1.10 (3.3.10; 2.2.10) How effective are the Bank's internal clearance [ ] [ ] [ ] [ ] procedures TTLs, Procurement staff, RPMs, legal procurement staff and OPSOR, at controlling fiduciary risk/ reducing misprocurement? 4.1.11 (3.3.11; 2.2.11) Do the Bank’s internal clearance processes impact the [ ] [ ] [ ] [ ] efficiency of project execution? 4: high negative impact; 1: negligible impact). 4.1.12 (3.3.12; 2.2.12) How frequent is the incidence of delay in the award of [ ] [ ] [ ] [ ] 49 APPENDIX A: BANK PROCUREMENT SYSTEMS contracts? 4.1.13 (3.3.13; 2.2.13) To what extent could delays be attributed to Bank [ ] [ ] [ ] [ ] processes for procurement administration? 4.1.14 (3.3.14; 2.2.14) To what extent could delays be attributed to limited [ ] [ ] [ ] [ ] borrower capacity for procurement? 4.1.15 (3.3.15; 2.2.15) To what extent could delays be attributed to the intrinsic [ ] [ ] [ ] [ ] complexity of the relevant projects? 4.1.16 (3.3.16; 2.2.16) Once a contract is awarded, is there still a need/ role for a [ ] [ ] [ ] [ ] procurement staff or accredited staff to oversee contract management / implementation? or, are TTL skills adequate) 4=strong need; 1=negligible need 4.1.17 (3.3.17, 3.3.17) To what extent are there problems encountered during [ ] [ ] [ ] [ ] contract implementation that could be traced back to flaws in setting up the procurement process? 4.1.18 (3.3.18) Is contract implementation under Bank financing more effective [ ] [ ] [ ] [ ] than that in projects financed by the government? FOR TTLS: PROJECT IMPLEMENTATION PLANNING AND OVERSIGHT – ROLE OF PROCUREMENT 4.1.19 (3.3.19 ) In terms of project preparation, how effective is the Bank's [ ] [ ] [ ] [ ] procurement design / planning process at contributing to overall project outcomes? 4.1.20 (3.3.20 ) With respect to the Bank's requirement that the borrower prepare [ ] [ ] [ ] [ ] a procurement plan that is updated regularly throughout the project, to what extent does this requirement contribute to procurement outcomes? 4.1.21 (3.3.21 ) Is the extent of use of International Competitive Bidding ICB) [ ] [ ] [ ] [ ] effective for project outcomes?(4=too high, 3=appropriate, 2= somewhat low, 1=much too low) 4.1.22 (3.3.22 ) To what extent does the requirement to use the Bank's standard [ ] [ ] [ ] [ ] bidding documents SBD) for ICB, and where required, national SBD for National Competitive Bidding NCB) contribute to procurement outcomes? 4.1.23 (3.3.23 ) To what extent does the Bank's oversight (no-objection) of [ ] [ ] [ ] [ ] prequalification, bidding documents, bid evaluation, and contract award help to appropriately weight factors such as:, bidder qualifications, and quality of goods/works? 4.1.24 (3.3.24) To what extent does the Bank's oversight appropriately weight [ ] [ ] [ ] [ ] bidding technicalities such as bid security, joint venture arrangements, power of attorney, and authentication of document? 4.1.25 (3.3.25 ) To what extent does the Bank's oversight (no-objection) of [ ] [ ] [ ] [ ] prequalification, bidding documents, bid evaluation, and contract award help to appropriately weight factors such as: evaluated price? 4.1.26 (3.3.26 ) Are the clearance levels of TTLs, PSs including PASs, RPMs, [ ] [ ] [ ] [ ] legal including Country Lawyer and/or LEGPrA) and OPRC appropriate? 4.1.27 (3.3.27 ) Is Bank staff involved in reviewing and approving procurement [ ] [ ] [ ] [ ] actions competent for their assigned level? 4.1.28 (3.3.28 ) Is the Bank able to respond quickly to special circumstances [ ] [ ] [ ] [ ] such as those requiring deviations from policies and procedures, or changes to procurement provisions of the financing agreement, or changes to previously agreed arrangements? FOR TTLS: COMPETITION AND DEVELOPMENT OUTCOMES (SEE 2.2.17-2.2.23) 4.1.29 (3.3.29; 2.2.17) Do the Bank's Procurement procedures ICB, NCB, in [ ] [ ] [ ] [ ] particular) lead to adequate competition in terms of numbers of bidders, openness to new bidders, geographical reach, etc.? 4=more than adequate; 3=adequate, 2=only somewhat adequate1=inadequate) 4.1.30 (3.3.30; 2.2.18) Are the Bank’s oversight processes field staff for routine [ ] [ ] [ ] [ ] clearance, RPM for more complex projects and clearance at headquarters for the largest projects) appropriately balanced in terms of the level of 50 APPENDIX A: BANK PROCUREMENT SYSTEMS review of procurement? 4=too much oversight; 3=high but not too high; 2=well balanced, 1=insufficient for the country) 4.1.31 (3.3.31; 2.2.19) Are suppliers more likely to show interest if the project is [ ] [ ] [ ] [ ] under Bank procedures as opposed to a project under national procedures)? 4.1.32 (3.3.32; 2.2..20) Has the use of International Competitive Bidding ICB) [ ] [ ] [ ] [ ] been appropriate in terms of the balance between achieving competition and improving quality and achieving ease of project execution ? 4=too high; 3=high but not too high; 2=well balanced, 1=used too little) 4.1.33 (3.3.33; 2.2..21) To what extent do the Bank's procurement requirements [ ] [ ] [ ] [ ] positively support the integrity, transparency, cost, quality, and timeliness of delivering project outputs? 4=significant positive effect; 1=no effect or negligible) 4.1.34 (3.3.34; 2.2..22) To what extent do the Bank's procurement requirements [ ] [ ] [ ] [ ] adversely affect the cost, quality, and timeliness of delivering project outputs? 4=significant adverse effect; 1=no effect or negligible) 4.1.35 (3.3.35; 2.2..23) If there is sometimes a trade-off between achieving [ ] [ ] [ ] [ ] outcomes for a specific project and maintaining the integrity of a system across all of the Bank's portfolio, how do the Bank's procurement requirements affect development outcomes across the portfolio? Table A.17. Procurement in Specialized Areas of Lending – Interviews with TTLs For TTLs - Community Driven Development and other Decentralized Projects Substan- Modest Negligible High (4) tial (3) /Low (2) (1) 4.2.1 (3.4.1 ) To what extent have Community Driven Projects been important in [ ] [ ] [ ] [ ] the Bank’s lending program in the country, in the last 5 years? (name in Table 1.2) 4.2.2 (3.4. 2) To what extent have other decentralized development projects [ ] [ ] [ ] [ ] been important in the Bank’s lending program in the country, in the last 5 years? 4.2.3 Omitted [ ] [ ] [ ] [ ] 4.2.4 Omitted [ ] [ ] [ ] [ ] 4.2.5 (3.4.5 ) Has the training of decentralized project implementation persons in [ ] [ ] [ ] [ ] Bank procurement methods been difficult to achieve? 4.2.6 (3.4.6 ) Has the training of decentralized project implementation persons in [ ] [ ] [ ] [ ] any sound procurement practices been difficult to achieve? 4.2.7 (3.4.7 ) To what extent are project difficulties encountered in areas such as [ ] [ ] [ ] [ ] financial management (account keeping, audit) rather than procurement? For TTLs - Public Private Partnership Projects in Infrastructure 4.2.8 (3.4.8 ) To what extent have there been any significant PPP projects [ ] [ ] [ ] [ ] financed by the Bank in the past 5 years?(name in Table 1.2) 4.2.9 (3.4.9 ) To what extent did the Bank get involved with the projects from the [ ] [ ] [ ] [ ] inception and design stage? 4.2.10 (3.4.10 ) To what extent did the Bank enter later in the process as financier? 4.2.11 (3.4.11 ) To what extent did the project face procurement issues in [ ] [ ] [ ] [ ] implementation (eg due to selection of partner not under Bank methods)? 4.2.12 (3.4.12 ) How well did the Bank assist the client to understand its [ ] [ ] [ ] [ ] requirements and to develop a strategy for the procurement? 4.2.13 (3.4.13 ) How useful are the Bank's various PPP resources manuals, [ ] [ ] [ ] [ ] information notes, web sites, etc) to develop and implement a PPP project, especially in the area of procurement? 51 APPENDIX A: BANK PROCUREMENT SYSTEMS Substan- Modest Negligible High (4) tial (3) /Low (2) (1) 4.2.14 (3.4.14 ) How appropriate and effective were the Bank's procurement [ ] [ ] [ ] [ ] procedures regarding award of concessions? 4.2.15 (3.4.15 ) How appropriate and effective were the Bank's procurement [ ] [ ] [ ] [ ] procedures regarding procurement of goods and works under the project? 4.2.16 Omitted [ ] [ ] [ ] [ ] 4.2.17 (3.4.17 ) How well did the Bank coordinate its assistance with other [ ] [ ] [ ] [ ] financiers? 4.2.18 (3.4.18 ) How adequate are the procurement flexibilities provided under Bank guidelines for such projects? For TTLs - Information and Technology Projects 4.2.19 (3.4.19 ) To what extent have there been any significant complex ICT [ ] [ ] [ ] [ ] components in projects financed by the Bank in the past 5 years? (eg, involving hardware and customized software, professional services including service contracts and/or complex systems)? (name on Table 1.2) 4.2.20 (3.4.20 ) How well did the Bank assist the client to understand its [ ] [ ] [ ] [ ] requirements and to develop specifications for the procurement? 4.2.21 (3.4.21 ) How appropriate and effective were the Bank's procurement [ ] [ ] [ ] [ ] procedures including standard bidding documents, guides and other procurement tools) for the types of information technology goods/systems/services procured? 4.2.22 (3.4.22 ) To the extent that there were procurement problems, to what [ ] [ ] [ ] [ ] degree do they relate to the procurement process itself (methods of selection/choice of documents/ conducting of the procurement)? 4.2.23 (3.4.23 ) If there were problems in the procurement process, to what degree did they relate to technical specifications, (requirements, form of contract, T&C of contract)? 4.2.24 (3.4.24 ) To what degree did the procurement problems relate to contract administration variations, dispute resolution, payment issues? 4.2.25 (3.4.25 ) To what extent were procurement problems related to project design, including design and planning of the procurement process? 4.2.26 (3.4.26 ) To what extent were flexibilities available in Bank procurement guidance at the time made use of? 4.2.27 (3.4.27 ) To what extent are there greater flexibilities in procurement for [ ] [ ] [ ] [ ] such projects today? 4.2.28 (3.4.28 ) How effective was the Bank's involvement during contract implementation? 4.2.29 (3.4.29 ) Overall, how appropriate and effective were the Bank's [ ] [ ] [ ] [ ] procurement procedures for ICT at achieving outputs in terms of cost, quality, desired features, timely completion? Table A.18. Harmonization of Bank Procurement Practices with Other Donors and IFIs—Interviews with TTLs (3.5.1 ) To what extent do differences between the Bank's procurement policies and those of other financing agencies (multilateral and bilateral) adversely impact project 4.5.1 implementation? [ ] [ ] [ ] [ ] 4.5.2 (3.5.2) Omitted [ ] [ ] [ ] [ ] 52 APPENDIX A: BANK PROCUREMENT SYSTEMS Interviews: Country Clients Box A.7. Guidance 5: Interviews with Country Procurement Counterparts and Public Procurement Officials Key Respondents: Two main groups of country clients are expected to be included: (i) officials from ministries implementing Bank projects (in the areas of infrastructure (transport and energy) and in social sectors (health and education). In addition (ii) interviews should include personnel from the National Procurement Office, who can comment on the overall status of public procurement in the country. Key Questions: Interviews with (i) officials from ministries implementing Bank projects, would focus largely on how the procurement system of the Bank has functioned in terms of helping achieving the development effectiveness of its own lending, and to learn about obstacles/ trade-offs that may be encountered. Some views on building procurement capacity may also be offered. Interviews with persons from the National Procurement Office would be expected to focus more on (ii) the building of national procurement capacity and support from the Bank in this regard. It would be necessary to give a brief introduction to the many different vehicles used by the Bank for supporting the development of procurement capacity: diagnostic and advisory work comprising the CPAR and MAPS exercises, other non-lending support through technical assistance or grant programs such as the IDF, and finally through Bank lending that focused on procurement reform in whole or in part, through investment lending as well as through DPLs. Repeated Questions: Several but not all questions here are deliberately repeated from previous sections, in order to capture the perspective of the country procurement officers, as distinct from Bank stakeholders. Some questions are omitted to avoid those that are internal to the Bank and to reduce the overall length. If questions have been repeated, the previous occurrences of the question are indicated to facilitate subsequent comparisons. Responses in this section should focus on the views of country officials and country procurement staff alone. Any significant differences in response that may emerge between country officials and procurement staff and other respondents may be pointed out in the final summary section of this questionnaire. (Note, questions marked in red are similar but not identical in text to those referenced earlier. Questions with an alphabetical suffix are new insertions in a sequence.) Q 5.1 For the implementation of Bank projects, to what extent do the Bank’s procurement processes provide safeguards and protections against (i) suboptimal procurement (poor value for money) and (ii) fraud and corruption in procurement? Are the Bank’s prior/post reviews helpful for the latter? To what extent do current Bank processes provide other benefits such as greater competition and greater transparency? Are bidders more likely to participate in Bank financed projects than those that are country financed? Is the Bank placing adequate emphasis on efficient procurement and best value for money? Q 5.2 To what extent is there a trade-off between safeguarding against risks and achieving efficiency in project execution (for example, in terms of elapsed time, number of persons needed for project implementation, and so forth). To what extent 53 APPENDIX A: BANK PROCUREMENT SYSTEMS do Bank processes give sufficient consideration to quality of procured items, fair and open competition, value for money, as well as to the participation of national bidders/consultants? Q 5.3 How efficient is Bank procurement staff in terms of responding to client needs and considerations? To what extent do they focus on issues most relevant to best procurement outcomes and best practices? How flexible are Bank methods and procedures ? Reply with examples. Q 5.4 To what extent can Bank procurement methods and procedures deal with certain areas of potentially complex procurement, eg in projects that are intensive in ICT or in PPPs? And how effective are Bank methods and procedures in decentralized projects, some of which are implemented by community leaders? Is there any evidence from project outcomes to suggest that the risks involved in such procurement methods have been excessive? Overall, is the procurement system adequate to meet the needs of complex and new forms lending? Q 5.5 To what extent is harmonization of procurement systems among donors or with regional trade/procurement rules (for example, the United Nations Commission on International Trade Law, European Union, and so forth) helpful for project implementation? To what extent has it occurred? Q 5.6 Did the country participate in the use of country systems pilot? If the Bank were to use country procurement systems, instead of its own system, how could this impact project implementation? Would there be equal or greater project development effectiveness? Would there be equal/greater consideration of (i) transparency/protection against fraud; (ii) competition and economy; (iii) implementation efficiency in terms of elapsed time and costs; (iv) encouragement to local industry? Table A.19. Interviews with Country Client Counterparts in Ministries Implementing Bank Projects: Procurement and the Effective Implementation of Bank Project Lending (For Country Clients in Implementing Ministries: Risk Management/Efficiency (see §§3.3-3.5)) Substan- Modest Negligible High (4) tial (3) /Low (2) (1) 5.1.1 (omitted) 5.1.2 (omitted) 5.1.3 (4.1.3; 3.3.3; 2.2.3) To what extent has procurement been a fiduciary [ ] [ ] [ ] [ ] concern in the country in terms of Bank project execution? 5.1.4 (omitted) 5.1.5 Omitted 5.1.6 Omitted 5.1.7 Omitted 5.1.8 (4.1.8; 3.3.8; 2.2.8) Are present arrangements and thresholds for prior [ ] [ ] [ ] [ ] 54 APPENDIX A: BANK PROCUREMENT SYSTEMS Substan- Modest Negligible High (4) tial (3) /Low (2) (1) and post review too stringent 4) or too lax 1), for helping to controlling fiduciary risk? 5.1.9 (4.1.9; 3.3.9; 2.2.9) Has concern about fiduciary risk unduly impacted [ ] [ ] [ ] [ ] procurement decision making among procurement staff? 5.1.9b To what extent is misprocurement an issue in the country? Does this occur with a high level or frequency? 5.1.9c To what extent do projects have to be re-bid? Does this happen frequently? 5.1.10 (4.1.10; 3.3.10; 2.2.10) How effective are the Bank's internal clearance [ ] [ ] [ ] [ ] procedures TTLs, Procurement staff, RPMs, legal procurement staff and OPSOR, at controlling fiduciary risk/ reducing misprocurement? 5.1.11 (4.1.11; 3.3.11; 2.2.11) Do the Bank’s internal clearance processes impact [ ] [ ] [ ] [ ] the efficiency of project execution? 4: high negative impact; 1: negligible impact). 5.1.12 (4.1.12; 3.3.12; 2.2.12) How frequent is the incidence of delay in the [ ] [ ] [ ] [ ] award of contracts? 5.1.13 (4.1.13; 3.3.13; 2.2.13) To what extent could delays be attributed to Bank processes for procurement administration? 5.1.14 (4.1.14; 3.3.14; 2.2.14) To what extent could delays be attributed to limited [ ] [ ] [ ] [ ] government capacity for procurement? 5.1.15 (4.1.15; 3.3.15; 2.2.15) To what extent could delays be attributed to the [ ] [ ] [ ] [ ] intrinsic complexity of the relevant projects? 5.1.16 Omitted 5.1.17 (3.3.17, 3.3.17) To what extent are there problems encountered during [ ] [ ] [ ] [ ] contract implementation that could be traced back to flaws in setting up the procurement process? 5.1.18 (4.1.18; 3.3.18) Is contract implementation under Bank financing more [ ] [ ] [ ] [ ] effective than that in projects financed by the government? For Country Clients in Implementing Ministries: Project Implementation Planning and Oversight – Role of Procurement 5.1.19 (4.1.19; 3.3.19 ) In terms of project preparation, how effective is the [ ] [ ] [ ] [ ] Bank's procurement design / planning process at contributing to overall project outcomes? 5.1.20 (4.1.20; 3.3.20 ) With respect to the Bank's requirement that the borrower [ ] [ ] [ ] [ ] prepare a procurement plan that is updated regularly throughout the project, to what extent does this requirement contribute to procurement outcomes? 5.1.21 (4.1.21; 3.3.21 ) Is the extent of use of International Competitive Bidding [ ] [ ] [ ] [ ] ICB) effective for project outcomes?(4=too high, 3=appropriate, 2= somewhat low, 1=much too low) 5.1.22 (4.1.22; 3.3.22 ) To what extent does the requirement to use the Bank's [ ] [ ] [ ] [ ] standard bidding documents SBD) for ICB, and where required, national SBD for National Competitive Bidding NCB) contribute to procurement outcomes? 5.1.23 (4.1.23; 3.3.23 ) To what extent does the Bank's oversight (no-objection) [ ] [ ] [ ] [ ] of prequalification, bidding documents, bid evaluation, and contract award help to appropriately weight factors such as:, bidder qualifications, and quality of goods/works? 5.1.24 (4.1.24; 3.3.24) To what extent does the Bank's oversight appropriately [ ] [ ] [ ] [ ] weight bidding technicalities such as bid security, joint venture arrangements, power of attorney, and authentication of document? 5.1.25 (4.1.25; 3.3.25 ) To what extent does the Bank's oversight (no-objection) [ ] [ ] [ ] [ ] of prequalification, bidding documents, bid evaluation, and contract award help to appropriately weight factors such as: evaluated price? 55 APPENDIX A: BANK PROCUREMENT SYSTEMS Substan- Modest Negligible High (4) tial (3) /Low (2) (1) 5.1.26 (4.1.26; 3.3.26 ) Are the clearance levels of TTLs, PSs including PASs,[ ] [ ] [ ] [ ] RPMs, legal including Country Lawyer and/or LEGPrA) and OPRC appropriate? 5.1.27 (4.1.27; 3.3.27 ) Is Bank staff involved in reviewing and approving [ ] [ ] [ ] [ ] procurement actions competent for their assigned level? 5.1.28 (4.1.28; 3.3.28 ) Is the Bank able to respond quickly to special [ ] [ ] [ ] [ ] circumstances such as those requiring deviations from policies and procedures, or changes to procurement provisions of the financing agreement, or changes to previously agreed arrangements? For Country Clients in Implementing Ministries: Competition and Development Outcomes 5.1.29 (4.1.29; 3.3.29; 2.2.17) Do the Bank's Procurement procedures ICB, NCB, [ ] [ ] [ ] [ ] in particular) lead to adequate competition in terms of numbers of bidders, openness to new bidders, geographical reach, etc.? 4=more than adequate; 3=adequate, 2=only somewhat adequate1=inadequate) 5.1.30 (4.1.30; 3.3.30; 2.2.18) Are the Bank’s oversight processes field staff for [ ] [ ] [ ] [ ] routine clearance, RPM for more complex projects and clearance at headquarters for the largest projects) appropriately balanced in terms of the level of review of procurement? 4=too much oversight; 3=high but not too high; 2=well balanced, 1=insufficient for the country) 5.1.31 4.1.31; 3.3.31; 2.2.19) Are suppliers more likely to show interest if the [ ] [ ] [ ] [ ] project is under Bank procedures as opposed to a project under national procedures)? 5.1.32 (4.1.32; 3.3.32; 2.2..20) Has the use of International Competitive Bidding [ ] [ ] [ ] [ ] ICB) been appropriate in terms of the balance between achieving competition and improving quality and achieving ease of project execution ? 4=too high; 3=high but not too high; 2=well balanced, 1=used too little) 5.1.33 (4.1.33; 3.3.33; 2.2..21) To what extent do the Bank's procurement [ ] [ ] [ ] [ ] requirements positively support the integrity, transparency, cost, quality, and timeliness of delivering project outputs? 4=significant positive effect; 1=no effect or negligible) 5.1.34 (4.1.34; 3.3.34; 2.2..22) To what extent do the Bank's procurement [ ] [ ] [ ] [ ] requirements adversely affect the cost, quality, and timeliness of delivering project outputs? 4=significant adverse effect; 1=no effect or negligible) 5.1.35 4.1.35; 3.3.35; 2.2..23) If there is sometimes a trade-off between [ ] [ ] [ ] [ ] achieving outcomes for a specific project and maintaining the integrity of a system across all of the Bank's portfolio, how do the Bank's procurement requirements affect development outcomes across the portfolio? For Country Clients in Implementing Ministries: Community Driven Development and other Decentralized Projects 5.1.1 Omitted 5.1.2 Omitted 5.1.3 Omitted 5.1.4 Omitted 5.1.5 (4.2.5; 3.4.5 ) Has the training of decentralized project implementation [ ] [ ] [ ] [ ] persons in Bank procurement methods been difficult to achieve? 5.1.6 (4.2.6; 3.4.6 ) Has the training of decentralized project implementation [ ] [ ] [ ] [ ] persons in any sound procurement practices been difficult to achieve? 5.1.7 (4.2.7; 3.4.7 ) To what extent are project difficulties encountered in areas [ ] [ ] [ ] [ ] such as financial management (account keeping, audit) rather than procurement? For Country Clients in Implementing Ministries: Public Private Partnership Projects in Infrastructure 5.1.8 Omitted [ ] [ ] [ ] [ ] 5.1.9 Omitted [ ] [ ] [ ] [ ] 56 APPENDIX A: BANK PROCUREMENT SYSTEMS Substan- Modest Negligible High (4) tial (3) /Low (2) (1) 5.1.10 Omitted 5.1.11 (4.2.11; 3.4.11 ) To what extent did the PPP project face procurement [ ] [ ] [ ] [ ] issues in implementation (eg due to selection of partner not under Bank methods)? 5.1.12 (4.2.12; 3.4.12 ) How well did the Bank assist the client to understand its [ ] [ ] [ ] [ ] requirements and to develop a strategy for the procurement? 5.1.13 Omitted [ ] [ ] [ ] [ ] 5.1.13b Overall, how appropriate and effective were the Bank's procurement methods and procedures for PPP projects? 5.1.14 Omitted [ ] [ ] [ ] [ ] 5.1.15 Omitted [ ] [ ] [ ] [ ] 5.1.16 Omitted [ ] [ ] [ ] [ ] 5.1.17 (4.2.17; 3.4.17 ) How well did the Bank coordinate its assistance with [ ] [ ] [ ] [ ] other financiers? 5.1.18 (4.2.18; 3.4.18 ) How adequate are the procurement flexibilities provided [ ] [ ] [ ] [ ] under Bank guidelines for such projects? For Country Clients in Implementing Ministries: Information and Technology Projects 5.1.19 (4.2.19; 3.4.19 ) To what extent have there been any significant complex [ ] [ ] [ ] [ ] ICT components in projects financed by the Bank in the past 5 years? (eg, involving hardware and customized software, professional services including service contracts and/or complex systems)? (name on Table 1.2) 5.1.20 (4.2.20; 3.4.20 ) How well did the Bank assist the client to understand its [ ] [ ] [ ] [ ] requirements and to develop specifications for the procurement? 5.1.21 (4.2.21; 3.4.21 ) How appropriate and effective were the Bank's [ ] [ ] [ ] [ ] procurement procedures including standard bidding documents, guides and other procurement tools) for the types of information technology goods/systems/services procured? 5.1.22 Omitted 5.1.23 Omitted 5.1.24 Omitted 5.1.25 Omitted 5.1.26 (4.2.26; 3.4.26 ) To what extent were flexibilities available in Bank [ ] [ ] [ ] [ ] procurement guidance at the time made use of? 5.1.27 (4.2.27; 3.4.27 ) To what extent are there greater flexibilities in [ ] [ ] [ ] [ ] procurement for such projects today? 5.1.28 Omitted 5.1.29 (4.2.29; 3.4.29 ) Overall, how appropriate and effective were the Bank's [ ] [ ] [ ] [ ] procurement procedures for ICT at achieving outputs in terms of cost, quality, desired features, timely completion? 5.1.30 (4.5.1; 3.5.1 ) To what extent do differences between the Bank's [ ] [ ] [ ] [ ] procurement policies and those of other financing agencies (multilateral and bilateral) adversely impact project implementation? 5.1.31 (4.5.2; 3.5.2) Omitted 5.1.32 Overall, is the Bank procurement system adequate to meet the challenges [ ] [ ] [ ] [ ] of new forms of Bank lending? 5.1.33 Overall, is there scope for simplification in Bank procurement methods [ ] [ ] [ ] [ ] and procedures processes? 5.1.34 Would procurement under Bank financed projects be easier if the national [ ] [ ] [ ] [ ] procurement system was used? 5.1.35 Would procurement under Bank financed projects be as effective in terms [ ] [ ] [ ] [ ] of competition, value for money, transparency and providing fiduciary 57 APPENDIX A: BANK PROCUREMENT SYSTEMS Substan- Modest Negligible High (4) tial (3) /Low (2) (1) safeguards, if national procurement systems were used? Use of Country Systems Pilot 5.1.36 (3.1.31)To what extent was the Bank’s Use of Country Systems pilot of [ ] [ ] [ ] [ ] interest to the country? 5.1.37 (3.1.32) Did the country believe that the use of its own procurement [ ] [ ] [ ] [ ] systems for Bank projects would be an improvement on the use of the Bank’s systems? 5.1.38 (3.1.33) In the views of Bank procurement staff would it be possible to [ ] [ ] [ ] [ ] take incremental steps towards greater use of country systems even if the country system is not adopted in its entirety? Q 5.7 From the perspective of country procurement officials especially those in a National Procurement office, how was the World Bank in terms of helping to build national procurement capacity? What were the most useful elements of such support? Eg, helping with legal and regulatory reform, developing procurement and procurement related institutions, and with the functioning of an efficient public procurement market, helping with professional and technical capacity development, and planning, prioritizing and implementing reform elements over a medium term time horizon? Q 5.8 To what extent have the elements of Bank support changed over time, to become better adapted to country conditions, and better focused on performance of the procurement system, as opposed to the adoption of regulatory and institutional changes? Q 5.9 To what extent has the Bank been able to provide cutting edge support in terms of new procurement modalities, such as e-procurement, or in the preparation of framework agreements? Has the Bank contributed to an awareness of the value of e-procurement e.g. for increasing transparency? Has the Bank encouraged the introduction of concepts such as value for money? Q 5.10 What were the most effective/important vehicles of Bank support for national procurement capacity development? For example, diagnostic work in the form of CPARs/MAPS exercises? Integrated financial management and procurement reform diagnostics such as PEFAs and Public Expenditure Reviews? Non-lending Technical Assistance or IDF grants? Support embedded in investment loan components? Or the “prior actions” and triggers of Development Policy loans? Discuss which would be the most effective forms of support going forward. Do they contribute towards integrated progress in procurement capacity building? 58 APPENDIX A: BANK PROCUREMENT SYSTEMS Table A.20. Interviews with Country Procurement Staff / National Procurement Office on Bank Support to Country Procurement Capacity Building (see sections 3.1 AND 3.2 above) Substan Modest Negligible High (4) -tial (3) /Low (2) (1) For National Procurement Office – Bank Support to Country Procurement Capacity 5.2.1 Omitted 5.2.2 (3.1.2; 2.1.2) To what extent is improving country procurement capacity [ ] [ ] [ ] [ ] built in to the country work program? 5.2.3 (3.1.3; 2.1.5) To what extent has support for building country procurement [ ] [ ] [ ] [ ] capacity been provided by other partners in development? 5.2.4 (3.1.4; 2.1.6)If yes, how central was the role of the Bank compared to other [ ] [ ] [ ] [ ] development partners? 5.2.4b Has the Bank been helpful in terms of advice on new procurement [ ] [ ] [ ] [ ] modalities, such as framework agreements? 5.2.4c Has the Bank been able to provide guidance on new procurement [ ] [ ] [ ] [ ] modalities such as e-procurement? 5.2.5 Omitted 5.2.6 (2.1.9) To what extent is present information on country procurement [ ] [ ] [ ] [ ] capacity up to date, in terms of overall country procurement capacity? 5.2.7 Omitted [ ] [ ] [ ] [ ] 5.2.7b How useful is it to the country to include procurement assessments as a [ ] [ ] [ ] [ ] component of other fiduciary assessment exercises? PEFA, PER, etc.)? 5.2.8 (2.1.11)To what extent does the country look at the process of integrating [ ] [ ] [ ] [ ] procurement reform with public expenditure management strengthening and the overall Budgeting, Planning and Audit process? 5.2.9 Omitted For National Procurement Office – Quality and Achievements of Recent Diagnostic and Advisory Work on Procurement 5.2.10 (2.1.7) Omitted [ ] [ ] [ ] [ ] 5.2.11 (3.1.11) To what extent have more recent diagnostic tools, such as the [ ] [ ] [ ] [ ] MAPS been an improvement on earlier tools such as the CPAR? 5.2.11b How helpful were the Bank’s CPARs in terms of identifying areas for procurement improvement? 5.2.12c To what extent has the MAPS exercise generated dialogue with the [ ] [ ] [ ] [ ] government on procurement reform? 5.2.13 Omitted [ ] [ ] [ ] [ ] 5.2.14 (3.1.14)Was the MAPS exercise limited to the Baseline Indicators or did [ ] [ ] [ ] [ ] also include Compliance and Performance indicators? 5.2.15 (3.1.15)Is there any monitoring framework in place today with indicators for [ ] [ ] [ ] [ ] tracking improvements? 5.2.16b (3.1.16) Did the government participate in preparation of procurement [ ] [ ] [ ] [ ] ESW/AAA 5.2.17 Omitted [ ] [ ] [ ] [ ] 5.2.18b (3.1.18) Is there a clear Action Plan for improving country procurement [ ] [ ] [ ] [ ] capacity over time? For National Procurement Office – Scope of Advisory Work – Legal Reform and Institution Building 5.2.18c Overall, has the Bank’s diagnostic and advisory work focused on the most important areas for strengthening procurement? 5.2.19 (3.1.19)Was legal reform, such as development of a procurement law or [ ] [ ] [ ] [ ] strengthening of commercial law, a part of the Bank’s country procurement support program? 5.2.20 Omitted [ ] [ ] [ ] [ ] 5.2.21 (3.1.21)If yes, was the Bank effective in aligning the law with best practice? [ ] [ ] [ ] [ ] 5.2.22 (3.1.22)How effective was the Bank’s efforts with respect to seeing the law [ ] [ ] [ ] [ ] 59 APPENDIX A: BANK PROCUREMENT SYSTEMS Substan Modest Negligible High (4) -tial (3) /Low (2) (1) applied in practice? 5.2.23 Omitted [ ] [ ] [ ] [ ] For National Procurement Office – Scope of Advisory Work – Institutional Strengthening 5.2.24 (3.1.24)Was institutional strengthening a part of the capacity building [ ] [ ] [ ] [ ] program? 5.2.25 (3.1.25)Did the Bank contribute to the establishment of a functioning [ ] [ ] [ ] [ ] central policy/regulatory body that oversees public procurement for the country? 5.2.26 (3.1.26)Was the Bank able to have an effective complaints and dispute [ ] [ ] [ ] [ ] resolution function established? 5.2.27 (3.1.27)Did the Bank effectively assist with a central portal ideally [ ] [ ] [ ] [ ] electronic) for dissemination of procurement information laws, procedures, opportunities, contract awards, and policy announcements)? 5.2.28 (3.1.28)How effective was the Bank in assisting the development of [ ] [ ] [ ] [ ] procurement tools standard bidding documents, proforma contracts, guides) 5.2.29 (3.1.29)How effective was the Bank’s assistance at furthering the [ ] [ ] [ ] [ ] development of electronic procurement in the country? 5.2.30 (3.1.30)Has the Bank been successful in helping establish a debarment [ ] [ ] [ ] [ ] process? For National Procurement Office – Planning and coordinating capacity development 5.2.31 (3.2.1) To what extent have the Bank's capacity building initiatives been [ ] [ ] [ ] [ ] based on a diagnostic of needs? 5.2.32 (3.2.2) Was the capacity building program developed and prioritized jointly [ ] [ ] [ ] [ ] between the Bank and the government? 5.2.33 (3.2.3) Omitted [ ] [ ] [ ] [ ] 5.2.34 (3.2.4) Did the Bank coordinate with other donors in design of capacity [ ] [ ] [ ] [ ] building support, and between different Bank interventions? For National Procurement Office – Development of Human Resources in public procurement 5.2.35 (3.2.5) Was HRD in procurement a part of the capacity building program? [ ] [ ] [ ] [ ] 5.2.36 (3.2.6) Omitted [ ] [ ] [ ] [ ] 5.2.37 (3.2.7) Was the Bank effective in mainstreaming procurement training into [ ] [ ] [ ] [ ] public sector capacity development ? 5.2.38 (3.2.8) Did the Bank limit its training to PIU staff of Bank projects, or to [ ] [ ] [ ] [ ] counterparts for Bank projects? 5.2.39 (3.2.9) Were HRD initiatives continued/sustained by the country following [ ] [ ] [ ] [ ] Bank / other donor support? 5.2.40 (3.2.10) Has a cadre of public procurement professionals been established [ ] [ ] [ ] [ ] and is it retained? 5.2.41 (3.2.11) Has the cadre of public procurement professionals trained been [ ] [ ] [ ] [ ] retained in public sector service? 5.2.42 (3.2.12) Did the Bank's capacity building work assist with accreditation or [ ] [ ] [ ] [ ] certification process that has been sustained by the government? For National Procurement Office – Procurement Capacity Building through Technical Assistance Loans/ other Investment Lending / NLTA/ IDFs 5.2.13a (3.2.13; 3.2.20) Was procurement capacity building through specific [ ] [ ] [ ] [ ] targeted investment/ Technical assistance loans , or through Non-lending Technical assistance programs or IDF Grant a significant vehicle for procurement capacity building in the country? 5.2.14 (3.2.14) Omitted [ ] [ ] [ ] [ ] 5.2.15 (3.2.15) Omitted [ ] [ ] [ ] [ ] 5.2.16 (3.2.16) Were the right areas targeted? 60 APPENDIX A: BANK PROCUREMENT SYSTEMS Substan Modest Negligible High (4) -tial (3) /Low (2) (1) 5.2.17 (3.2.17) Did the Bank adequately track / follow up? [ ] [ ] [ ] [ ] 5.2.18 [ ] (3.2.18) Was there sufficient flexibility in the structure of the loan to adapt [ ] [ ] [ ] to needs over the course of program execution? 5.2.19 (3.2.19)Overall, how suitable were these TA loans, loan components, [ ] [ ] [ ] [ ] NLTA or IDF, for the support of procurement capacity? 5.2.20 (3.2.20) (see 3.2.13 above) [ ] [ ] [ ] [ ] 5.2.21 Omitted [ ] [ ] [ ] [ ] 5.2.22 Omitted [ ] [ ] [ ] [ ] 5.2.23 Omitted [ ] [ ] [ ] [ ] 5.2.24 Omitted [ ] [ ] [ ] [ ] 5.2.25 Omitted [ ] [ ] [ ] [ ] For National Procurement Office – Procurement Capacity building through Bank Investment Lending by Learning-By-Doing/ Transfer of Knowledge 5.2.26 (3.2.26) To what extent did the Bank contribute to the support of [ ] [ ] [ ] [ ] developing procurement capacity through learning by doing or transfer of knowledge? 5.2.37 (3.2.27) To what extent did learning by doing in procurement occur through [ ] [ ] [ ] [ ] large value contracts / ICB? 5.2.28 (3.2.28) To what extent did learning by doing in procurement occur in [ ] [ ] [ ] [ ] sectors such as health or education, which could have a large proportion of small value contracts / decentralized procurement administration? 5.2.29a (3.2.29) What is the extent to which PIUs contribute to the development of [ ] [ ] [ ] [ ] human capacity in procurement? 5.2.30 Omitted [ ] [ ] [ ] [ ] 5.2.31 Omitted [ ] [ ] [ ] [ ] 5.2.32 Omitted [ ] [ ] [ ] [ ] 5.2.33 Omitted [ ] [ ] [ ] [ ] 5.2.34 Omitted [ ] [ ] [ ] [ ] 5.2.35 Omitted [ ] [ ] [ ] [ ] 5.2.36 To what extent are country / government staff who have gained experience [ ] [ ] [ ] [ ] under Bank projects then able to transfer their skills to national procurement? 5.2.37 Omitted [ ] [ ] [ ] [ ] 5.2.38 Overall, how useful a vehicle is investment lending under the Bank’s [ ] [ ] [ ] [ ] current guidelines for the support of country procurement capacity? For National Procurement Office – Procurement Capacity Building through Development Policy Lending 5.2.29 Were prior actions oriented towards procurement reform important in the [ ] [ ] [ ] [ ] country context? 5.2.30 Was there sufficient coordination between the Bank's Economic Policy [ ] [ ] [ ] [ ] staff and Procurement staff? 5.2.31 Omitted [ ] [ ] [ ] [ ] 5.2.32 Omitted [ ] [ ] [ ] [ ] 5.2.33 Omitted [ ] [ ] [ ] [ ] 5.2.34 Overall, how useful a vehicle were the Bank’s development policy loans for [ ] [ ] [ ] [ ] the support of country procurement capacity? 61 APPENDIX A: BANK PROCUREMENT SYSTEMS Interviews: Private Sector and Civil Society Box A.8. Guidance 6: Interviews with Private Sector Suppliers, and with Civil Society Organizations Key Respondents: Two additional groups of country clients are expected to be included in this section: (i) A selection of private sector suppliers, contractors and consultants to World Bank projects. Representatives of a chamber of commerce for a sector such as infrastructure (energy and transport) contractors or other representatives of social sectors could be included here. In addition (ii) interviews should include personnel from any private NGO or civil society organization that attempts to monitor procurement of any public sector projects. Key Questions: Interviews with (i), private sector suppliers, contractors and consultants would focus largely on positive and negative aspects of the procurement system of the Bank, from the suppliers’ point of view, and to learn about obstacles/ trade-offs that may be encountered. Interviews with (ii) private NGOs or civil society organizations that monitor procurement would discuss the extent to which such entities help enhance transparency in public procurement, and whether they have views on the efficacy of Bank procurement safeguards in this respect. Repeated Questions: Some questions here are deliberately repeated from previous sections, in order to capture the perspective of private sector suppliers as well as civil society organizations, as distinct from Bank stakeholders. If questions have been repeated, the previous occurrences of the question are indicated to facilitate subsequent comparisons. Responses in this section should focus on the views of private sector suppliers as well as civil society organizations alone. Any significant differences in response that may emerge between these groups and other respondents may be pointed out in the final summary section of this questionnaire. (Note, questions marked in red are similar but not identical in text to those referenced earlier. Questions with an alphabetical suffix are new insertions in a sequence). Q 6.1 From the point of view of prospective suppliers, contractors and consultants what is the view of the Bank’s procurement system? On the positive side, compared with other procurement systems (for example, the national procurement system) do they add value to projects, do they increase economy, efficiency, and transparency, do they enhance fair competition and national participation? Are there also negatives in terms of, for example, technical, financial and administrative requirements (bid bond, and so forth), too detailed proposal submission, lengthy process of approvals, and so forth? Please discuss. Q 6.2 On balance, is there a value to retaining the present Bank procurement system as it is? If not in what respects would you like to see it altered? Q 6.3 How well does the Bank procurement system perform compared with other systems for complex projects such as large infrastructure, ICTs, PPPs, and CDDs? Q 6.4 In your view, would it be preferable to move to the full use of country systems, under country (and not Bank) oversight and protections? 62 APPENDIX A: BANK PROCUREMENT SYSTEMS Table A.21. Interviews with Firms: Procurement and Effective Implementation of Projects Substantial Modest Negligible High (4) (3) /Low (2) (1) Interviews with Suppliers: Project Implementation Planning and Oversight – Role of Procurement 6.1.1 (5.1.33; 4.1.33; 3.3.33; 2.2..21) To what extent do the Bank's procurement [ ] [ ] [ ] [ ] requirements positively support the integrity, transparency, cost, quality, and timeliness of delivering project outputs? 4=significant positive effect; 1=no effect or negligible) 6.1.2 (5.1.34; 4.1.34; 3.3.34; 2.2..22) To what extent do the Bank's procurement [ ] [ ] [ ] [ ] requirements adversely affect the cost, quality, and timeliness of delivering project outputs? 4=significant adverse effect; 1=no effect or negligible) 6.1.3a (5.1.19, 4.1.19; 3.3.19 ) More specifically, how useful to private suppliers, [ ] [ ] [ ] [ ] compared to other public contracts, is the Bank’s process of advertizing via general and specific tener notices? 6.1.4 (5.1.20; 4.1.20; 3.3.20 ) How useful is the Bank's requirement that the [ ] [ ] [ ] [ ] borrower prepare a procurement plan that is updated regularly throughout the project? 6.1.5 (5.1.22; 4.1.22; 3.3.22 ) To what extent does the requirement to use the [ ] [ ] [ ] [ ] Bank's standard bidding documents SBD) for ICB, and where required, national SBD for National Competitive Bidding NCB) help suppliers? (discuss: comparison with national system) 6.1.6 (5.1.23; 4.1.23; 3.3.23 ) To what extent does the Bank's oversight (no- [ ] [ ] [ ] [ ] objection) of prequalification, bidding documents, bid evaluation, and contract award help to appropriately weight factors such as, bidder qualifications, and quality of goods/works? 6.1.7 (5.1.24; 4.1.24; 3.3.24) To what extent does the Bank's oversight [ ] [ ] [ ] [ ] appropriately weight bidding technicalities such as bid security, joint venture arrangements, power of attorney, and authentication of document? 6.1.8 (5.1.25; 4.1.25; 3.3.25 ) To what extent does the Bank's oversight (no- [ ] [ ] [ ] [ ] objection) of prequalification, bidding documents, bid evaluation, and contract award help to appropriately weight factors such as: evaluated price? Interviews with Suppliers: Competition and Development Outcomes 6.1.9 (5.1.29; 4.1.29; 3.3.29; 2.2.17) Do the Bank's Procurement procedures ICB, [ ] [ ] [ ] [ ] NCB, in particular) lead to adequate competition in terms of numbers of bidders, openness to new bidders, geographical reach, etc.? 4=more than adequate; 3=adequate, 2=only somewhat adequate1=inadequate) 6.1.10 (5.1.31; 4.1.31; 3.3.31; 2.2.19) Are suppliers more likely to show interest if [ ] [ ] [ ] [ ] the project is under Bank procedures as opposed to a project under national procedures)? 6.1.11 (5.32; 4.1.32; 3.3.32; 2.2.20) Has the use of International Competitive [ ] [ ] [ ] [ ] Bidding ICB) been appropriate in terms of the balance between achieving competition and improving quality and achieving ease of project execution ? 4=too high; 3=high but not too high; 2=well balanced, 1=used too little) Interviews with Suppliers: Risk Management/ Efficiency 6.1.12 (3.3.4; 2.2.4) To what extent could procurement be a fiduciary concern in [ ] [ ] [ ] [ ] the overall public sector of the country? 6.1.13 (5.1.3; 4.1.3; 3.3.3; 2.2.3) To what extent is there risk of fraud and [ ] [ ] [ ] [ ] corruption in Bank financed / managed projects? 6.1.14 (5.1.10; 4.1.10; 3.3.10; 2.2.10) How effective are the Bank's procedures, at [ ] [ ] [ ] [ ] controlling fiduciary risk/ reducing misprocurement? 6.1.15 (5.1.11; 4.1.11; 3.3.11; 2.2.11) Do the Bank’s internal clearance processes [ ] [ ] [ ] [ ] impact the efficiency of project execution? 4: high negative impact; 1: negligible impact). 6.1.16 (5.1.12; 4.1.12; 3.3.12; 2.2.12) How frequent is the incidence of delay in the [ ] [ ] [ ] [ ] 63 APPENDIX A: BANK PROCUREMENT SYSTEMS Substantial Modest Negligible High (4) (3) /Low (2) (1) award of contracts? 6.1.17 (4.1.13; 3.3.13; 2.2.13) To what extent could delays be attributed to Bank processes for procurement administration? 6.1.18 (5.1.14; 4.1.14; 3.3.14; 2.2.14) To what extent could delays be attributed to [ ] [ ] [ ] [ ] limited government capacity for procurement? 6.1.19 (5.1.15; 4.1.15; 3.3.15; 2.2.15) To what extent could delays be attributed to [ ] [ ] [ ] [ ] the intrinsic complexity of the relevant projects? 6.1.20 (5.1.18; 4.1.18; 3.3.18) Is contract implementation under Bank financing [ ] [ ] [ ] [ ] more effective than that in projects financed by the government? Q 6.5 To what extent is there effective external scrutiny by civil society organizations or by NGOs on public procurement in the country? What sectors of public procurement does it cover (for example, some/all sectors/regions, and so forth)? Q 6.6 Do such organizations perceive any differences in terms of value for money/project outcomes, and in the consideration for transparency, economy, efficiency, competition, participation of national firms, between Bank financed and other projects? Can this be attributed to the Bank’s present procurement system? Q 6.7 In the view of such civil service organizations, would it be preferable to move to the full use of country procurement system, under country (and not Bank) oversight and protections? Table A.22. Interviews with Civil Society Organizations: Bank Procurement and Effective Implementation of Projects Substan- Modest Negligible High (4) tial (3) /Low (2) (1) Interviews with CSOs: Project Implementation Planning and Oversight – Role of Procurement 6.2.1 (6.1.1; 5.1.33; 4.1.33; 3.3.33; 2.2..21) To what extent do the Bank's [ ] [ ] [ ] [ ] procurement requirements positively support the integrity, transparency, cost, quality, and timeliness of delivering public sector projects? 4=significant positive effect; 1=no effect or negligible) 6.2.2 (6.1.2; 5.1.34; 4.1.34; 3.3.34; 2.2..22) To what extent do the Bank's [ ] [ ] [ ] [ ] procurement requirements adversely affect the cost, quality, and timeliness of delivering public sector projects? 4=significant adverse effect; 1=no effect or negligible) 6.2.3 To what extent does the Bank's oversight (no-objection) of prequalification/short list, bidding documents/RFP, bid/proposal evaluation, and contract award help to appropriately weight factors affecting the execution of public project s? Interviews with CSOs: Competition and Development Outcomes 6.2.9 (6.1.9; 5.1.29; 4.1.29; 3.3.29; 2.2.17) Do the Bank's Procurement [ ] [ ] [ ] [ ] procedures ICB, NCB, in particular) lead to adequate competition in terms of numbers of bidders, openness to new bidders, geographical reach, etc.? 4=more than adequate; 3=adequate, 2=only somewhat adequate1=inadequate) 6.2.10 (6.1.10; 5.1.31; 4.1.31; 3.3.31; 2.2.19) Are suppliers more likely to show [ ] [ ] [ ] [ ] interest if the project is under Bank procedures as opposed to a project under national procedures)? 64 APPENDIX A: BANK PROCUREMENT SYSTEMS Substan- Modest Negligible High (4) tial (3) /Low (2) (1) 6.2.11 (6.1.11; 5.32; 4.1.32; 3.3.32; 2.2.20) Has the use of International [ ] [ ] [ ] [ ] Competitive Bidding ICB) been appropriate in terms of the balance between achieving competition and improving quality and achieving ease of project execution ? 4=too high; 3=high but not too high; 2=well balanced, 1=used too little) Interviews with CSOs: Risk Management/ Efficiency 6.2.12 (6.1.12; 3.3.4; 2.2.4) To what extent could procurement be a fiduciary [ ] [ ] [ ] [ ] concern in the overall public sector of the country? 6.2.13 (6.1.13; 5.1.3; 4.1.3; 3.3.3; 2.2.3) To what extent is there risk of fraud [ ] [ ] [ ] [ ] and corruption in Bank financed / managed projects? 6.2.14 (6.1.14; 5.1.10; 4.1.10; 3.3.10; 2.2.10) How effective are the Bank's [ ] [ ] [ ] [ ] procedures, at controlling fiduciary risk/ reducing misprocurement? 6.2.15 (6.1.15; 5.1.11; 4.1.11; 3.3.11; 2.2.11) Do the Bank’s internal clearance [ ] [ ] [ ] [ ] processes impact the efficiency of project execution? 4: high negative impact; 1: negligible impact). 6.2.16 (6.1.16; 5.1.12; 4.1.12; 3.3.12; 2.2.12) How frequent is the incidence of [ ] [ ] [ ] [ ] delay in the award of contracts? 6.2.17 (6.1.17; 4.1.13; 3.3.13; 2.2.13) To what extent could delays be attributed to Bank processes for procurement administration? 6.2.18 (6.1.18; 5.1.14; 4.1.14; 3.3.14; 2.2.14) To what extent could delays be [ ] [ ] [ ] [ ] attributed to limited government capacity for procurement? 6.2.19 (6.1.19; 5.1.15; 4.1.15; 3.3.15; 2.2.15) To what extent could delays be [ ] [ ] [ ] [ ] attributed to the intrinsic complexity of the relevant projects? 6.2.20 (6.1.20; 5.1.18; 4.1.18; 3.3.18) Is contract implementation under Bank [ ] [ ] [ ] [ ] financing more effective than that in projects financed by the government? 65 APPENDIX A: BANK PROCUREMENT SYSTEMS Summary of Findings and Overall Assessment Box A.9. Guidance 7: Summary of Findings (5-10 pages or more as needed) This section is intended to pull together the core findings of the perceptions in the previous sections, to combine it with the facts known before the mission and any other factual information that has come to light, and to come up with a ‘bottom line’ on the overall view of the Bank’s role in procurement in the country concerned. It provides an opportunity to sort and prioritize in terms of strategic importance. This section triangulates the findings of the previous sections with factual information, point out differences in perspectives that may have arisen between the various stakeholders encountered, and offers overall conclusions. Q 7.1 Provide an overview of the relevance and effectiveness of (i) the Bank’s interventions in the country from the point of view of developing country capacity. Comment on those vehicles (CPAR/MAPs, development policy loan (DPL), investment loan, PEFA, and so forth) that were more or less relevant or effective, those that were less so, and the evolution of their contributions over time. Discuss the strengths and weaknesses of the Bank’s interventions, in terms of the design, relevance, instrument and constraints that may have been faced by the Bank team, and areas in which design or performance could be strengthened. Comment on results achieved, in the short term, medium term and long term, and the likelihood of results achieved being sustained. (5-10 pages) • Overview – attention given to capacity building? Constraints? Role relative to donors? • Quality and contributions of diagnostic work (CPAR/MAPs/NLTA) to an overall capacity building strategy. • Contributions of procurement reform undertaken in the context of economic and public financial management reform (PEFAs, PERs) • Building procurement reform through Bank lending (DPL/IL)? • Grants (IDF/other). Role of WB in training and capacity building? • Relevance of focus of the Bank’s capacity building – has it helped embrace new areas such as e- procurement, framework agreements, and so forth? Q 7.2 Provide an overview of the relevance and effectiveness of (ii) the Bank’s current procurement methods and procedures in terms of aiding the development effectiveness of its own lending. Comment on the extent to which the present guidelines achieve an appropriate balance between safeguarding of fiduciary mandate and the consideration for economy and efficiency, transparency, encouraging domestic industry and fair and open competition. Point to those specific areas in which the Bank’s procurement framework could be better adapted 66 APPENDIX A: BANK PROCUREMENT SYSTEMS to achieving these objectives, in the perspective of its future lending patterns, as well as the overall global context, given the global evolution of procurement systems and the procurement environment. (5-10 pages) • The Bank’s own lending – efficiency? Levels of clearance, bidding methods, time taken, borrower capacity and World Bank procurement methodology. Achievement of value for money? • The Bank’s own lending – Extent to which transparency/procurement risk is taken into consideration? • The Bank’s own lending –Dealing with new areas of lending, innovation and risk? The risk-reward trade-off? 67 Appendix B: Strengthening Procurement Support Country Assistance Strategies Table B.1. Country Strategies Reviewed by Region and Fiscal Year FY AFR EAP ECA LCR MNA SAR Total 2002 6 1 7 3 1 2 20 2003 10 3 4 7 3 2 29 2004 13 2 6 6 2 1 30 2005 12 4 6 6 3 1 32 2006 3 2 6 4 4 4 23 2007 15 1 6 5 1 28 2008 15 1 7 5 2 2 32 2009 8 3 4 8 3 3 29 2010 10 2 4 4 2 22 2011 5 3 5 4 2 3 22 Total 97 22 55 52 22 19 267 Source: IEG analysis of CASs. Note: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. Table B.2. Country Strategies with Procurement References – IBRD and IDA Countries Region IBRD IDA Total AFR 7 39 46 EAP 4 9 13 ECA 16 8 24 LCR 18 5 23 MNA 7 3 10 SAR - 8 8 Total 52 72 124 Source: IEG analysis of CASs. Note: IDA/IBRD countries are taken from the July 2012 Operational Manual. Blend countries were considered as IDA countries. AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. 68 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT Table B.3. Nature of Procurement References in CASs 2002-11 Issue 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Total a) The CAS made reference to 16 27 27 27 18 23 24 18 13 10 203 an existing or planned procurement diagnostic b) The country was at the time 8 23 17 19 18 18 23 22 17 18 183 of the CAS facing issues related to procurement c) The CAS had procurement 4 11 14 18 11 10 12 15 12 9 116 outcomes and/or monitoring indicators d) The CAS envisaged support 8 17 18 19 17 15 16 17 14 10 151 for procurement reform through either investment lending, policy lending, or technical assistance. Source: IEG analysis of CASs. Note: CAS = Country Assistance Strategy. Table B.4. Types of Procurement Difficulties Mentioned in CASs – By Year Number of reports Issue 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Total 1- Does the CAS refer to 1 10 5 8 6 3 8 7 7 3 58 procurement difficulties that impede Bank lending? (for example, in portfolio review) 2- Does the CAS refer to 8 19 16 15 17 18 22 21 16 17 169 procurement difficulties that limit country development effectiveness? 3-Both 1 and 2 1 6 4 4 5 3 7 6 6 2 44 Source: IEG analysis of CASs. Note: CAS = Country Assistance Strategy. Table A.5. Types of Procurement Difficulties Mentioned in CASs – By IBRD/ IDA Number of countries Issue IBRD IDA Total 1- Does the CAS refer to procurement difficulties that impede 16 30 46 Bank lending? (for example, in portfolio review) 2- Does the CAS refer to procurement difficulties that limit 39 64 103 country development effectiveness? 3-Both 1 and 2 11 25 36 Source: IEG analysis of CASs. Note: CAS = Country Assistance Strategy. 69 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT Table B.6. Countries with Identified Legislative/Regulatory Issues and Planned Bank Support CASCR/PR mention of Country IDA/IBRD Region Fiscal year improvement 1 Bangladesh IDA SAR FY11 N/A 2 Cambodia IDA EAP FY05 YES 3 Cameroon IDA AFR FY04 YES 4 Colombia IBRD LCR FY03 YES 5 Congo, Republic of IDA AFR FY08 YES 6 Croatia IBRD ECA FY05 YES 7 Djibouti IDA MNA FY05 & FY09 YES (FY05), N/A (FY09) 8 Dominican Republic IBRD LCR FY05 YES 9 Ecuador IBRD LCR FY03 N/A 10 El Salvador IBRD LCR FY05 YES 11 Guatemala IBRD LCR FY05 NO 12 Guyana IDA LCR FY03 YES 13 Indonesia IBRD EAP FY04 & FY09 YES 14 Kosovo IDA ECA FY06 N/A 15 Lebanon IBRD MNA FY08 NO 16 Lesotho IDA AFR FY06 YES 17 Macedonia, FYR IBRD ECA FY07 NO 18 Madagascar IDA AFR FY04 YES 19 Moldova IDA ECA FY09 YES 20 Nepal IDA SAR FY04 YES 21 Niger IDA AFR FY08 N/A 22 Nigeria IDA AFR FY05 & FY10 YES (FY05), N/A (FY10) 23 Pakistan IDA SAR FY02 & FY06 YES 24 Panama IBRD LCR FY06 YES 25 Peru IBRD LCR FY07 YES 26 Senegal IDA AFR FY03 NO 27 Ukraine IBRD ECA FY08 YES 28 Uruguay IBRD LCR FY05 YES Source: IEG analysis of CASs. Note: N/A refers to the strategy reports whose completion reports were not available at the time of the review. 70 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT Table B.7. Countries with Planned Bank Support for the Procurement Institutional Framework CASCR/PR mention of Country IDA/IBRD Region Fiscal year improvement Afghanistan IDA SAR FY02, FY03, FY06 YES Albania IBRD ECA FY02 NO Angola IDA AFR FY03 & FY07 NO (FY03), N/A (FY07) Azerbaijan IBRD ECA FY07 & FY11 NO (FY07), N/A (FY11) Belize IBRD LCR FY09 NO Bhutan IDA SAR FY11 N/A Bolivia IDA LCR FY09 YES Brazil IBRD LCR FY04 NO Burkina Faso IDA AFR FY10 N/A Cameroon IDA AFR FY04 YES Cape Verde IDA AFR FY09 NO Central African Republic IDA AFR FY10 N/A Chad IDA AFR FY11 N/A China IBRD EAP FY03 YES Colombia IBRD LCR FY03 & FY08 YES Congo, Republic of IDA AFR FY08 & FY09 YES Djibouti IDA MNA FY05 YES Dominican Republic IBRD LCR FY10 N/A Gambia, The IDA AFR FY08 YES Ghana IDA AFR FY04 & FY07 YES (FY04), NO(FY07) Guinea-Bissau IDA AFR FY09 N/A Guyana IDA LCR FY03 YES Haiti IDA LCR FY09 N/A India IDA SAR FY09 YES Indonesia IBRD EAP FY09 YES Iraq IBRD MNA FY06 & FY09 YES (FY06), N/A (FY09) Kosovo IDA ECA FY10 N/A Lao PDR IDA EAP FY05 YES Lebanon IBRD MNA FY06 NO Liberia IDA AFR FY09 N/A Macedonia, FYR IBRD ECA FY07 NO Malawi IDA AFR FY03 & FY07 YES Maldives IDA SAR FY08 NO Mali IDA AFR FY08 YES Mauritania IDA AFR FY02 NO Moldova IDA ECA FY09 YES Montenegro IBRD ECA FY11 N/A Nepal IDA SAR FY04 & FY09 YES (FY04), N/A(FY09) Niger IDA AFR FY08 N/A OECS Countries IBRD LCR FY06 & FY10 YES (FY06), N/A (FY10) Pakistan IDA SAR FY02 NO Panama IBRD LCR FY06 & FY11 NO (FY06), N/A (FY11) Peru IBRD LCR FY03 & FY07 YES Philippines IBRD EAP FY02 NO Rwanda IDA AFR FY07 YES Sierra Leone IDA AFR FY10 N/A Somalia IDA AFR FY08 N/A Sri Lanka IDA SAR FY03 YES Sudan IDA AFR FY08 N/A Tajikistan IDA ECA FY10 N/A Tanzania IDA AFR FY07 YES 71 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT CASCR/PR mention of Country IDA/IBRD Region Fiscal year improvement Togo IDA AFR FY08 N/A Tunisia IBRD MNA FY10 N/A Ukraine IBRD ECA FY08 NO Yemen, Rep. of IDA MNA FY06 & FY09 YES (FY06), N/A (FY09) Source: IEG analysis of CASs. Note: N/A refers to the strategy reports whose completion reports were not available at the time of the review. Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. Table B.8. Countries with Bank Support for Efficiency of Procurement Operations and Markets CASCR/PR mention of Country IDA/IBRD Region Fiscal year improvement Argentina IBRD LCR FY04 YES Azerbaijan IBRD ECA FY07 NO Bangladesh IDA SAR FY06 & FY11 YES Brazil IBRD LCR FY04 YES Chad IDA AFR FY04 NO Colombia IBRD LCR FY03 NO Guatemala IBRD LCR FY05 YES Honduras IDA LCR FY07 & FY11 YES Kosovo IDA ECA FY10 NO Nepal IDA SAR FY04 NO Panama IBRD LCR FY08 & FY11 YES Peru IBRD LCR FY03 YES Sri Lanka IDA SAR FY08 NO Tanzania IDA AFR FY11 NO Thailand IBRD EAP FY11 NO Togo IDA AFR FY08 NO Uruguay IBRD LCR FY05 YES Vietnam IDA EAP FY07 YES Yemen, Rep. of IDA MNA FY06 NO Zambia IDA AFR FY08 NO Source: IEG analysis of CASs. Note: Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. 72 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT Table B.9. CASs—Regional Variations in the Nature of Procurement References (% positive responses) What is the CAS Does the CAS refer to Are the key issues identified in the areas of: discussion procurement difficulties based on? that: Is the Impede discussion in Bank Efficiency of Legislative Procurement Integrity and the CAS lending? Limit country procurement or institutions transparency Region based on an (such as development operations regulatory and country / fraud and existing in effectiveness? and framework? capacity? corruption? procurement portfolio markets? diagnostic? review) AFR 48 23 65 20 48 24 37 EAP 64 36 59 18 27 32 45 ECA 58 13 47 20 31 16 42 LCR 56 13 71 37 54 38 37 MNA 50 41 59 32 45 36 36 SAR 47 26 89 42 68 37 53 Overall 53 22 63 25 45 28 40 % YES Source: IEG analysis of CASs. Note: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. Table B.10. CASs - Regional Variations in Procurement Diagnostics and Indicators (% positive responses) Basis of support Is the procurement strategy or Are procurement related Are procurement related Region activities based on diagnostic monitoring Indicators outcomes included in the recommendations? included in the CAS? CAS? AFR 30 36 23 EAP 36 64 45 ECA 35 33 31 LCR 42 50 40 MNA 36 32 41 SAR 37 47 37 Overall % of YES 35 41 32 Responses Source: IEG analysis of CASs. Note: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. 73 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT Table B.11. CAS Regional Variations: Vehicles of Procurement Support – Lending and AAA (%positive responses) Vehicles of support Support though Further diagnostic Support through DPL/ Support through Region an investment work? policy based lending? TA/ grants? loan? AFR 42 36 37 8 EAP 45 32 59 14 ECA 49 16 29 4 LCR 56 27 48 2 MNA 41 5 55 0 SAR 21 42 68 0 Overall % YES 45 28 43 5 Source: IEG analysis of CASs. Note: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. Table B.12. CAS Regional Variations: World Bank Procurement Support – Area of Focus of Bank Support (%positive responses) Nature of Support Efficiency of Integrity and Institutional procurement Harmonization transparency of the framework operations and markets and UCS Legislative Public Procurement and staff/ (better delegation/ (harmonization and System (disclosure, Region management community driven with other regulatory complaints Capacity/ projects; support for donors, with framework? mechanism, dispute dissemination PPP; e-procurement; country resolution, external of information credit/ contracting systems) audit) processes AFR 27 47 5 22 9 EAP 45 55 32 32 27 ECA 20 31 9 11 18 LCR 27 42 35 31 29 MNA 23 41 18 14 32 SAR 42 58 37 26 26 Overall % of YES 28 44 17 22 19 Responses Source: IEG analysis of CASs. Note: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. 74 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT Table B.13. Results from Country Strategy Reports – By Region To what extent were the following areas strengthened? Region Legislative and Institutional Efficiency of Integrity and Harmonization regulatory framework and procurement transparency of and UCS framework? staff/ operations and the public (harmonization management markets (better procurement with other capacity/ delegation/ system donors, with dissemination of community driven (disclosure, country systems) information projects; support complaints for PPP; e- mechanism, procurement; dispute credit/ contracting resolution, processes external audit) Scores 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 AFR 52 6 25 17 54 6 17 23 88 0 12 0 69 8 21 2 96 0 4 0 EAP 36 0 50 14 64 0 14 21 79 0 14 7 50 21 29 0 79 0 14 7 ECA 45 5 26 24 71 3 21 5 84 3 11 3 74 5 16 5 76 8 16 0 LCR 61 0 21 18 61 5 24 11 53 13 18 16 55 8 24 13 71 8 16 5 MNA 77 0 15 8 46 0 46 8 85 0 8 8 85 0 8 8 69 15 15 0 SAR 25 8 25 42 25 8 42 25 58 8 33 0 75 8 17 0 92 0 8 0 Overall % 51 4 26 20 57 4 23 15 76 4 14 5 67 8 20 5 82 5 11 2 Source: IEG analysis of CASCRs, CASCRRs, and CASCR Reviews. Note: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region; UCS = use of country systems. Table B.14. Movements Over Time in the Content of CAS Discussions on Procurement and Bank Follow-Up—Three-Year Moving Averages CAS Question Key values of 3-year moving average Possible explanation Is the CAS discussion based on an Increase from 53 percent in 2004 to 61 The stock of diagnostic reports could existing procurement diagnostic? percent in 2007, then dropping be becoming less current. progressively to 43 percent in 2011. To what extent were key issues Some increase from 24 percent in 2004 to This could reflect the declining identified in legislative and 30 percent in 2008, dropping back to 21 proportion of CPARs that focus regulatory frameworks? percent in 2011. considerably on legislative and regulatory issues, in later years. To what extent were there key Initial decline, from 43 percent in 2004 to Stronger Bank attention to these issues issues in procurement institutions 33 percent in 2006. Later, scores trended and country capacity? steadily upward to 56 percent in 2011, To what extent were there key Generally upward trend, from 30 percent in Stronger Bank attention to these issues in integrity and the first two years to a peak of 51 percent issues. transparency/fraud and corruption in 2008. Some subsequent decline, to 43 percent, but results remain considerably above the baseline Are the resulting procurement Sharp decline from 45 percent in 2005 to The strategy/activities are on average strategies, or planned activities, 21 percent in 2011. considerably less based on diagnostics based on diagnostic than the CAS discussion itself (see recommendations? above). 75 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT CAS Question Key values of 3-year moving average Possible explanation To what extent was further This has become steadily less important, The Bank may consider diagnostic diagnostic work a key vehicle of from an original 60 percent to 29 percent work now to be more a thing of the follow up support? in 2011. past in many countries. To what extent were other vehicles …. The focus on DPLs and technical of support used for follow up – assistance/grants have both held quite such as support through lending? steady over the evaluation period, while support through investment lending has been given little attention although with a very modest peak of 11 percent in 2009. To what extent was there support Steady drop from a high of 35 percent in The Bank may now feel ready to move for the legislative and regulatory 2005. beyond such issues that may often be framework? seen as the first step in a reform process. To what extent has support for Slightly rising trend to a maximum of 52 Increase attention to institutional issues institutional frameworks and percent in 2011. over time staff/management capacity evolved? To what extent has there been a Quite low but with a rising trend from 13 Increase attention to market issues, change in emphasis in support for percent in 2004 to 22 percent in 2012. albeit still at low levels efficiency of procurement operations and markets: How has the nature of support for The moving average rose to a high of 30 It is not clear why there is a decline in integrity and transparency of the percent in 2006, but has since dropped to this ratio. public procurement systems around 20 percent. evolved? How has support for harmonization Support grew from eight percent to 27 This decline could be connected to the and use of country systems percent in 2008, but has since fallen back general lack of progress over this evolved? to 18 percent in 2011 period in the Bank’s support for the use of country systems. Source: IEG analysis of CAS documents. 76 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT Table B.15. Diagnostics, Difficulties and Issues Identified In Strategy Reports What is the CAS Does the CAS refer to Are the key issues identified in the areas of: discussion procurement difficulties that: based on? FY Is the impede limit country the procurement efficiency of Integrity and discussion in Bank development legislative institutions procurement transparency the CAS lending? effectiveness? or and country operations / fraud and based on an (eg in regulatory capacity? and corruption? existing portfolio framework? markets? procurement review) diagnostic? 2002 2003 2004 53 19 53 24 43 25 29 2005 57 25 55 26 37 31 31 2006 59 23 58 27 33 32 39 2007 61 21 62 28 37 23 44 2008 58 21 69 30 46 28 51 2009 55 20 68 27 53 27 50 2010 50 27 71 24 54 30 47 2011 43 23 74 21 56 30 43 Source: IEG analysis of CASs. Note: Three-year moving averages (%) Table B.16. Basis and Vehicles of World Bank Procurement Support Basis of support Vehicles of support FY Is the procurement Are Are Further Support Support Support strategy or procurement procurement diagnostic through through though an activities based on related related work? DPL/ technical investment diagnostic monitoring outcomes policy- assistance/ loan? recommendations? Indicators included in based grants? included in the CAS? lending? the CAS? 2002 2003 2004 40 33 22 60 25 38 3 2005 45 45 29 58 31 43 2 2006 40 49 35 55 35 44 2 2007 40 47 35 50 32 46 6 2008 35 39 34 46 25 45 9 2009 34 39 38 37 23 44 11 2010 27 44 38 30 27 47 8 2011 21 43 37 29 30 46 4 Source: IEG analysis of CASs. Note: Three-year moving averages. CAS = Country Assistance Strategy; DPL = development policy lending. 77 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT Table B.17. Extent of Results Achieved by World Bank Procurement Support – Three-Year Moving Averages (%) Efficiency of procurement Integrity and operations and transparency of markets (better the public delegation/ procurement Institutional community driven system Harmonization and Framework and projects; support (disclosure, UCS staff/ management for PPP; e- complaints (harmonization Legislative and Capacity/ procurement; mechanism, with other donors, regulatory dissemination of credit/ contracting dispute resolution, with country FY framework? information processes external audit) systems) 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 2002 2003 2004 46 5 25 24 62 4 20 13 77 4 16 3 68 8 20 4 85 7 6 2 2005 46 3 30 21 50 5 27 17 70 5 20 4 66 9 19 7 85 7 8 0 2006 47 3 23 27 47 4 29 20 67 7 20 6 59 8 21 12 81 3 17 0 2007 56 2 22 20 55 1 26 18 78 4 11 6 63 7 23 7 75 2 23 0 2008 55 5 22 17 57 2 22 19 79 3 9 9 67 3 22 7 72 4 21 4 2009 57 5 28 10 56 7 20 17 73 3 15 10 69 10 20 2 77 6 13 4 2010 51 4 23 23 68 7 16 8 78 3 13 6 64 8 26 2 89 5 3 4 Source: IEG analysis of CASs. Note: PPP = public-private partnership; UCS = use of country systems. World Bank Budgeting for Procurement Services The main deliverables of the World Bank’s fiduciary units relate to the fiduciary aspects of the preparation and supervision of projects, to the fiduciary inputs to sector work (including such sector work not managed by fiduciary staff), and to the general provision of fiduciary support to country-related operational work. Prior to FY08, budgeting and costing practices for fiduciary support for project preparation and supervision varied across the different regional vice presidencies. In most regions such costs were charged to existing product codes for each project while in others they were charged to the portfolio management business processes. These differences in costing practices across the Bank rendered the tracking of expenditures across the Bank’s fiduciary units difficult. In order to unify regional practices, a simplified coding structure was introduced for all regions in FY08. This new structure defined four new business processes for fiduciary services The project specific activities charged to these business processes were to be coded on a country (and not sector or project) basis using the following Internal Orders: • LENF-WB. Financial management services for project preparation. 78 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT • SPNF-WB. Financial management services for project supervision. • LENP-WB. Procurement services for project preparation. • SPNP-WB. Procurement services for project supervision. For annual budget processes, allocations are made up-front to the fiduciary units in a region for their work, which is thus not charged to individual projects. Accordingly, it is not possible from time-recording data to estimate the costs of procurement (or other fiduciary services). Tables B.18 and B.19 show the available project preparation and supervision costs for the Bank’s regions. Table B.18. Project Related Procurement Costs By Region and Year - Preparation ($) FY AFR EAP LCR MNA SAR Total 2002 116,006 116,006 2003 98,464 98,464 2004 100,238 96,190 196,428 2005 228,854 601,037 829,891 2006 281,143 381,645 662,788 2007 283,236 1,268,358 1,551,595 2008 373,376 1,001,421 1,374,797 2009 517,100 373,514 1,241,914 2,132,528 2010 1,187,747 582,862 315,920 1,306,309 3,392,838 2011 1,242,710 478,501 589,840 382,047 1,193,617 3,886,715 Source: World Bank data. Note: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. Table B.19. Project Related Procurement Costs by Region and Year - Supervision ($) FY AFR EAP ECA LCR MNA SAR Total 2002 152,298 103,915 256,213 2003 1,349,535 7,153 1,356,689 2004 1,625,385 64,001 1,689,386 2005 1,555,616 157,073 1,712,689 2006 1,564,839 31,929 1,596,768 2007 1,778,223 1,486,422 3,264,645 2008 2,082,125 2,473,916 4,556,041 2009 2,650,078 1,115,106 2,944,672 6,709,856 2010 3,024,778 2,699,443 884,019 3,158,981 9,767,221 2011 3,897,783 2,839,722 168,426 2,585,689 1,055,461 3,203,998 13,751,078 Source: World Bank data. Note: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. 79 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT Although both the East Asia and Pacific and South Asia Regions show data for the fiscal years preceding 2008, only in the EAP region were the costing and coding practices for some of the years prior to 2008 essentially the same as those established in FY08. Therefore, with the exception of East Asia and Pacific, the data for only the years starting FY08 would be an accurate representation of the procurement costs associated with the preparation and supervision of Bank projects – and for several regions the data became available even later. Such data as are available indicate broadly that nominal costs for regional procurement activities have been growing modestly over the past few years, largely in line with such increases overall for the Bank operational budgets, but perhaps with a modest shift from preparation to supervision costs. Stand-alone fiduciary work led by the operational support units themselves was to be coded in line with its content. For example, Country Procurement Assessment Reports (CPARs), Country Financial Accountability Assessments (CFAAs), PEFAs, and other integrated products where financial management/PR is the lead unit would be coded as ESW while capacity building including workshops would be coded as non-lending technical assistance or client training. Table B.20. Cost of Procurement - ESW by Region and Year ($ 000) FY AFR EAP ECA LCR MNA SAR Total 2002 233 562 756 111 - - 1,663 2003 1,450 257 741 290 115 382 3,236 2004 1,469 163 253 527 296 - 2,709 2005 236 385 158 729 455 - 1,963 2006 1,702 - 688 292 43 599 3,324 2007 1,137 436 848 603 - 678 3,703 2008 114 460 280 306 274 448 1,883 2009 - - 384 168 283 246 1,081 2010 - - 102 167 283 171 722 2011 92 82 83 43 122 - 423 Total 6,434 2,345 4,292 3,236 1,873 2,524 20,704 Source: World Bank data. Note: The cost of World Bank Procurement ESW was obtained by considering the cost of preparing report types classified in one database as CPARs, Integrative Fiduciary Assessments and those identified as "other procurement study." Regions: AFR = Africa Region; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; LCR = Latin America and the Caribbean Region; MNA = Middle East and North Africa Region; SAR = South Asia Region. Table B.21. Cost of Procurement - NLTA by Region and Year ($ 000) FY AFR EAP ECA LCR MNA OTH SAR Total 2002 - - - - - - - - 2003 - 130 - - - - - 130 2004 582 - - 4 - - - 586 2005 - - - - 131 - - 131 80 APPENDIX B: STRENGTHENING PROCUREMENT SUPPORT 2006 142 - - - 83 - - 225 2007 19 7 - - - - -20 6 2008 280 34 - - - - 110 424 2009 - - 70 - 151 - 115 336 2010 1,194 - - - 245 - 163 1,602 2011 164 15 - - 26 89 - 294 Total 2,382 186 70 4 637 89 368 3,735 Source: World Bank data. Note: To obtain the procurement NLTA projects, IEG considered all NLTA projects from the Procurement sector Board as well as those whose name contained the word “procurement.” Regions: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia; OTH = other. 81 Appendix C: Supporting Capacity Through AAA Table C.1. Template for CPAR and MAPS Evaluation QUESTIONS Comments and factors evaluated Number of Average observations scorea Scoring for all cells: 3= highly achieved, 0= negligibly met (Unless otherwise indicated) I. CONTROL DATA ON THE PROJECT 1. 1 FY of completion (year) 47 1.2 FY in which undertaken (year) NA NA 1.3 Title of Report CPAR=37; 1.4 Type (CPAR/MAPS/ PEFA/IFA) MAPS=9; CPIP=3 1.5 Region 47 1.6 Country/State 47 1.7 Small State? Standard of a population below 1.5 million people. 47 5 A fragile situation is defined as having either: (a) a composite World Bank, African Development Bank and 1.8 Fragile / Conflict state Asian Development Bank Country Policy and Institutional Assessment rating of 3.2 or less; or (b) the presence of a 47 9 United Nations and/or regional peace-keeping or peace- building mission (for example, African Union, European Union, NATO), with the exclusion of border monitoring operations, during the past three years. 1.9 IBRD/IDA/Blend 47 IBRD=11; 1.10 Is there any mention of Bank support for IDA=18; procurement reform? Blend=2 1.11 Cost of ESW ($ thousands) NA NA 1.12 Time taken (months) NA NA II. PROCESS OF PREPARATION / COUNTRY AND PARTNER OWNERSHIP 3 - FM, PROCUREMENT AND PREM SECTOR specialist all together on the team or collaborating; 2: ONLY Procurement, but both others represented as peer review or two of three sector specialists on team; 1: 2.1 To what extent does the ESW reflect internal ONLY procurement with one represented on peer review. collaboration among Procurement, PREM and FM? 34 1.9 0: only procurement specialists. Collaboration can be (1, 2, 3) (score 3-0) through sharing information, seeking mutual comment and concurrence on outputs and scheduling joint missions, producing joint outputs, and conducting joint report disseminations, as appropriate. To what extent did client participate in preparation of 2.2 To what extent did the Client Stakeholder/ ESW/AAA? All client agencies participated as expected government participate in preparation of procurement (3); Some client agencies participated effectively (2); 49 2.5 ESW/AAA (especially bidders)? (score 3-0) (Is this Only the client coordinating agencies participated evident from the report?) effectively (1); Participation was weak or ineffective at coordinating as well as at line level (0). 2.5 Substantial participation by all these groups (Score 2.3 To what extent were civil society, academia, 3), Significant participation by at least two groups (Score NGOs, and media involved? (score 3-0) 46 0.9 2), Some participation by at least one group (Score 1); The involvement was negligible (Score 0). 82 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA QUESTIONS Comments and factors evaluated Number of Average observations scorea 1. High (Harmonization of MDBs' procurement guidelines in principle, and standardization of bidding documents for the procurement of goods, works, small works, plant design supply and installation, and consultant's request for proposals; joint training, capacity 2.4 To what extent did other development partners building, diagnostic work as part of co-financed projects, 49 1.4 participate? (score 3-0) knowledge sharing and business outreach initiatives) (4) 2. Substantial (significant progress in one or two of the above areas and expectation that it will progress well) (3) 3. Modest (some progress towards harmonization but not there in any of the above) (2) 4. Negligible - No progress (0) III. QUALITY, CONTENT, COVERAGE AND STRUCTURE III.1 Quality and Coverage of Analysis (i) Are the objectives clear, comprehensive and consistent with the purpose? (ii) Are there adequate 3.1 What is the overall clarity of objectives and linkages between the objectives, analysis, 47 2.7 analysis, structure of report (score 3-0) recommendations, and action plan? (iii) Are recommendations clear and actionable? 3.2 A Country Procurement System (CPS) is the composite of several components. Does this ESW assess all these components in terms of design 3= highly achieved, 0= negligibly met, NA NA quality, of their working in isolation and of their functionality as institutional components of the CPS? (CPAR IV17-18)? III.2 Legal and Regulatory Framework 0 Does it discuss at minimum the following: (i) Level of 3.3 Does the ESW/ AAA discuss the quality of the uniformity of application; (ii) Comprehensiveness of overall legislative framework: (score 3-0) (MAPS1a coverage, (iii) Accessibility, (iv) Clarity of hierarchy of 48 2.7 and CPAR AI 6-9 and PEFA PI19i)? procurement sources clear (3 if all met, 2, if only 3 met, 1, if only 2 met, and 0, if only one met) 3.4 Does the ESW assess (i) whether the regulatory The ESW assesses the clarity of (i) Description of the framework clearly describes the admitted procurement admitted procurement methods; (ii) Conditions under methods, their applicability and the admitted 48 2.6 which each method applies; (iii) Conditions under which exceptions? (MAPS I1b. CPAR A2 q6 D4 q10)?Score exceptions are admitted (such as sole sourcing). 3-0: ESW assesses qualitatively all key steps in depth and 3.5 Does the ESW assess all applicable regulation of highlights clearly deficiencies or factors that put at risk the key steps of the admitted procurement methods the outcome of the procurement process, 3 points; ESW and their related procedures? The key steps to be assesses satisfactorily the key steps and highlights assessed cover: Procurement planning, deficiencies or factors that put at risk the outcome of the 48 2.6 prequalification, advertising, scope of communications procurement process, 2 points; ESW only partially during bidding, receipt of bids and bid-opening, bid assesses the key steps and highlights deficiencies or evaluation, contract award and effectiveness. (MAPS factors that put at risk the outcome of the procurement I1c-g; CPAR A2 q3; D6 q1-3,D8-10, PEFA ii) process, 1 point. 0=ESW has no meaningful discussion of these issues. This ESW assesses qualitatively the availability of (i) 3.6 Does the report mention the extent to which Standard bidding documents for goods, works and regulations are implemented, updated, supported by services; (ii) Guidance on process and procedures for standard bidding documents, guidelines and prequalification of bidders, bid evaluation and (iii) 48 2.6 application manuals? (MAPS2a-2f; CPAR D4 q1-5; Standard general contract conditions and as far as D5q1-6; E6, D11 q1-13) (score 3-0) possible the quality of such documents (Completeness, suitability, consistency with best practices). III.3 Institutional Framework 83 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA QUESTIONS Comments and factors evaluated Number of Average observations scorea Does the ESW discuss (i) the presence and 3.7 Does the report discuss the functionality of the effectiveness of agency responsible for overall normative and regulatory procurement bodies? procurement policy formulation? (ii) the presence and 47 2.8 (MAPS:I4; CPAR: A1-Q11-13, D1-Q4-6; PEFA iii) effectiveness of authority to exercise oversight regarding (score 3-0) proper application of the procurement rules and regulations? (i) Does it discuss whether procurement planning and associated expenditures are part of the budget formulation process and contribute to multiyear 3.8 Does the report discuss the integration of planning? (ii) Does it discuss whether the Budget law Procurement into the Budgeting, Planning and Audit and financial procedures support timely procurement, 47 2.1 Process? (MAPS 3a-d; CPAR C q6-9; PEFA PI 1,2, contract execution, and payment? (iii) Does it discuss 4,10,12,20,21,24,26,28) (score 3-0) whether procurement actions can only be initiated with existing budget appropriations? (iv) User's guides or Manuals for contracting entities? 3.9 Monitoring: Does the report discuss the existence (i) Does the ESW discuss whether the relevant data is and dissemination of monitoring and procurement being collected (electronically?)? (ii) Does it discuss 47 2.1 statistics? (MAPS 11; CPAR A2-Q12; D6-Q3) (score adequacy of the system? (iii) Does it discuss disclosure 3-0) of data? (iv) Does it discuss accessibility to the data? III.4 Capacity Levels and Capacity Development This ESW assesses (i) Skill levels, performance levels of procurement managers and officers, and the structure of 3.10 Staff Capacity: Does the report evaluate capacity their incentive system; (ii) Government strategy for levels and discuss capacity building efforts of the capacity development including training, for improving 47 2.6 government in the area of public procurement? performance of procurement officers in charge of critical (MAPS 5c-d; CPAR D3Q1)) (score 3-0) procurement functions (Prequalification, bid evaluation, contract negotiations); (iii) Strategy of implementation as part of the Action Plan. 3.11 Does it discuss capacity needs of different institutions such as Ministries, Public sector agencies, (i) Are different ministries /public sector agencies and so on? Does it discuss financial sustainability of assessed? (ii) is financial sustainability of training 47 1.6 present capacity building processes, especially if assessed? (iii) is the issue of retention of key staff donor funded? Does it discuss staff retention issues discussed? and remedies? (Score 3-0) III.5 Fraud and Corruption: Laws and Regulations and Enforcement This ESW assesses (i) The scope of provisions, the enforcement mechanisms (e.g. disqualification, debarment etc.) and their effectiveness, laws regulating 3.12 To what extent does the ESW assess legislation the responsibility of public procurement officers and their on F&C, enforcement methods and procedures, ethic effects; (ii) It also assesses the levels of public 47 2.3 codes? (MAPS IV12CPAR A2 q13&15, A3 q4, B q2, perception of public procurement integrity, reporting A3 q2, D3) (Score 3-0) channels and public-private partnership institutions; (iii) It also assesses the existence of codes of ethics in the public and in the private sector. III.6 Performance of the Public Procurement Market Does the ESW discuss: (i) the functionality of the public procurement market? (ii) whether the procurement market is known for its integrity and ethical behaviors? 3.13 To what extent does the ESW assess the (iii) whether there are effective mechanisms for efficiency of public procurement transactions and the partnerships between the public and private sector? (iv) 47 2.1 quality of the outcomes? (MAPS III 6-7. CPAR F2 whether there are any major systemic constraints (for q1&2, A2 q2, D7 q4, D12 q1-3, F1,3) (Score 3-0) example inadequate access to credit, contracting practices, and so forth) inhibiting the private sector's capacity to access the procurement market? 84 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA QUESTIONS Comments and factors evaluated Number of Average observations scorea Does the ESW discuss: (i) the existence of a fair complaints mechanism and whether the complaint review body has full authority and independence for resolution of complaints? (ii) whether decisions are deliberated on the basis of available information and is the whether the final decision is reviewed and ruled upon by a body (or 3.14 To what extent does the ESW assess the authority) with enforcement capacity under the law? (iii) complaints process and the appeal mechanism? 47 2.4 Whether decisions are published and made available to (MAPS IV 10; CPAR A3q3, D11 q5-9, D1 q5) all interested parties and to the public; (iv) whether procedures exist to enforce the outcome of the dispute resolution process? (v) whether the appeals mechanism and the complaint review body have full authority and independence for resolution of complaints? (vi) Does it discuss whether these bodies have adequate capacity? Does this ESW evaluate (i) Robustness of the contract management, administrative practices and ICT support; 3.15 To what extent does the ESW cover issues (ii) Respect of contract conditions, including the fair related to contract administration and mgmt? (MAPS: handling and resolution of disputes, (iii) Skill levels and 48 2.0 I8; CPAR D11:Q6; PEFA iv) (score 3-0) motivation of managers and administrative staff and their capacity of solving unexpected contractual problems that can disrupt contract activities. Does the ESW examine (i) How audits are conducted and its effects on procurement operations and contract project execution. (ii) Delays and inefficiencies arising 3.16 To what extent does ESW raise the question or from discrepancies between bid evaluations and award examines how internal and external audits and recommendations conducted by government agencies, 47 2.4 controls are conducted? (MAPS IV 9; CPAR D11 q and the review criteria of government and state audit and 15) control bodies; (iii) Existence of official procedures to resolve evaluative discrepancies and the skill level of auditors and controllers. III.7 Miscellaneous Questions 3.17 Does the report cover e-procurement issues? Score (1-0) 47 30 (Y/N) 3.18 Does the report cover sub-national or local Score (1-0) 42 26 government issues? (Y?N) 3.19 Does the report discuss sustainability and Score (1-0) 45 6 “green'”procurement (Y/N) 3.20 Does the report discuss public -private Score (1-0) 45 23 partnerships? (Y/N) 3.22 Does the report discuss community level Score (1-0) 24 3 procurement ? (Y/N) III.8 Use of Empirical Data to Support Findings. Rate whether the ESW includes quantitative data on Question 3.23 Is the report supported by empirical 47 1.3 results to buttress its findings. (Score 3-0) data? Does this ESW include (i)a thorough analysis of risks regulatory, institutional, organizational and ethical that III.9 Risk Assessment. Question 3.24 Does the can affect the procurement system; (ii) risk ratings for ESW conduct a risk assessment of public and of 47 1.5 country procurement and for Bank procurement explicit Bank procurement? (CPAR) (Score 3-0) and logically derived from the risk analysis? (iii) Is there a clear strategy to address the risks? (i) Are the recommendations prioritized? (ii) Is the plan III.10Action Plan Question 3.25 What is the quality time bound (or are there open ended of the Action Plan included in this ESW report? 44 2.2 recommendations)? (iii) Is there some discussion of who (CPAR)(score 3-0) I to be responsible for implementation? Note: The number in the "average score" field indicates the number of reports evaluated from small states. The number in the "average score" field indicates the number of reports evaluated from fragile/conflicted states. a. The majority of questions were scored on a 0–3 scale, with 3 being highly achieved, and 0 being negligibly met. For yes/no questions, 3.17-3.22, this field indicates the number of yeses. 85 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA Table C.2. IEG Results Analysis - Countries and Type of ESW Country CPAR MAPS Total Albania 1 1 2 Azerbaijan 2 2 Colombia 2 1 3 Ethiopia 2 2 Ghana 1 1 2 Indonesia 1 1 2 Lao PDR 1 1 2 Mauritius 1 1 2 Mexico 2 2 Morocco 1 1 2 Philippines 3 3 Rwanda 2 2 Sierra Leone 1 1 2 Tanzania 1 1 2 Turkey 1 1 2 Vietnam 1 1 2 West Bank and Gaza 2 2 Total 25 11 36 World Bank AAA Support – Non-Lending Technical Assistance IEG undertook a review of procurement related non-lending technical assistance (NLTA) undertaken by the Bank. While such NLTA is deemed to be a part of overall analytic and advisory work (AAA), parallel to the Bank’s Economic and ESW, it is usually short, specific, and often used for support to project preparation or implementation. As part of the review, IEG did a search of all NLTAs approved between FY02 and FY11 financed by Bank budget and two trust funds: the Governance Partnership Facility and the Bank-Netherlands Partnership Program. A word search for procurement (and related key words; for example, bid, tender, acquisition) in the title of the NLTA/Trust Fund gave the total sample of 60 “projects.” Of these, 30 were funded by Bank Budget, and for 9 of these, there was documentation available. Of these, 3 focused on land acquisition, leaving a sample of 6 focused on themes relevant to the public procurement framework. 86 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA Table C.3. List of Procurement-Related NLTAs Reviewed Project Name Cost of AAA ($ 000) (P102631) Modification of Procurement Documents & Processes for WB Projects in 34 India (P104074) SONATRACH Reimbursable Technical Assistance in Algeria 245 (P104899) Procurement and Service Delivery: Monitoring Procurement Outcomes in 122 India (P108846) Improving Procurement Capability of TMAs in Pakistan 12 (P110795) Updating of Model Bidding Documents for China 10 (P118446) Ethiopia Follow up on Procurement Reforms 52 Source: World Bank database. Two NLTAs dealt with procurement issues in specific sectors. The first, in Pakistan, looked at both formal and informal procurement practices in four municipal administrations, with a focus on water and sanitation services and recommended simplified procedures, training for officials and more involvement of civil society. The trust funded NLTA, delivered in 2007, was closely related with the Bank’s Second NWFP Community Infrastructure Project, completed in 2009. Although the analysis and recommendations seem of high quality, there is no evidence in the project Implementation Completion Report on whether the recommendations were carried out. The Bank remains active in the sector, so it is possible that the recommendations would be useful for subsequent work. The second NLTA in this category supported Algeria’s SONATRACH, the state owned Petroleum Company, on the production of bidding documents. There is comprehensive reporting in the documentation prepared that is of a high standard. The project is still active, but there are no aide memoires in the file since 2007 to indicate the state of progress. Four NLTAs dealt with broader procurement issues. The first NLTA helped to revise model bidding document for national competitive bidding in China. The Bank supported a review of the experience with the 1997 model bidding documents, and updated them based on the Tendering and Bidding Law, and the Procurement Law. A draft set was reviewed by a group of 25 participants from experienced tendering companies, project management offices, and some government agencies, and then by World Bank and the Asian Development Bank. The MDBs seem to be of high quality, but the eventual outcome is unknown. The second NLTA supported a 2011 workshop in Ethiopia for participants from six East African countries to agree on harmonized procurement policies to facilitate regional integration, and to agree on a common framework for basic procurement training that could be codified in certification programs. Results achieved were a set of action plans on professional development, trade and investment facilitation, 87 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA capacity development, and planning and implementation of regional integration projects. The Bank agreed to be the champion to ensure recommendations were adopted, but there is no evidence in the NLTA documentation, the CPS or elsewhere of follow through. The third NLTA supported a review and modification of bidding documents used for Bank projects in India to bring them into line with the latest practices adopted by the Bank. The work was carried out in consultation with a Task Force, constituting members from various ministries and other public organizations. The completion report states that the impact cannot be determined because of unresolved issues between the Bank and the Government, and the uncertainty as to whether Task Force members would be available for discussions in the future. The fourth NLTA supported the design of procedures for citizen monitoring in Orissa, India, including pilot testing and creation of guidance material. The NLTA was supported by a Norwegian trust fund that had funded similar work in Bangladesh, Nepal and Pakistan. The Trust Fund completion report says that the project supported the preparation of a Community Procurement Manual and community training to for oversight of local anti-poverty programs. While previous implementation by NGOs had focused on highlighting irregularities rather than suggesting solutions, the NLTA emphasized the latter, and thus helped engender cooperation from the Government. Procurement in Budget Management – PEFAs and Integrated Capacity Table C.4. Procurement-Related Issues in a PEFA Assessment—All Indicators Indicator Description Procurement issues PI-4 Stock and monitoring of (i) Stock of expenditure payment arrears (as a percentage of expenditure payment arrears actual total expenditure for the corresponding fiscal year) and any recent change in the stock. (ii) Availability of data for monitoring the stock of expenditure payment arrears. PI-10 Public access to key fiscal (i) Annual budget documentation Timely availability of information (ii) In-year budget execution reports contract awards is an important dimension of (iii) Year-end financial statements procurement system (iv) External audit reports accountability and anti- (v) Contract awards: Award of all contracts with value above corruption approx. USD 100,000 equiv. are published at least quarterly through appropriate means. (vi) Resources available to primary service units PI-12 Multi-year perspective in fiscal (i) Preparation of multi -year fiscal forecasts and functional planning, expenditure policy and allocations; budgeting (ii) Scope and frequency of debt sustainability analysis (iii) Existence of sector strategies with multi-year costing of 88 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA Indicator Description Procurement issues recurrent and investment expenditure; (iv) Linkages between investment budgets and forward expenditure estimates. PI-20 Effectiveness of internal controls (i) Effectiveness of expenditure commitment controls. for non-salary expenditure (ii) Comprehensiveness, relevance and understanding of other internal control rules/ procedures. (iii) Degree of compliance with rules for processing and recording transactions. PI-21 (i) Coverage and quality of the internal audit function. Effectiveness of internal audit (ii) Frequency and distribution of reports. (iii) Extent of management response to internal audit findings. PI-24. Quality and timeliness of in-year (i) Scope of reports in terms of coverage and compatibility with budget reports budget estimates (ii) Timeliness of the issue of reports (iii) Quality of information PI-26 Scope, nature and follow-up of (i) Scope/nature of audit performed (incl. adherence to auditing external audit standards). (ii) Timeliness of submission of audit reports to legislature. (iii) Evidence of follow up on audit recommendations. PI-28 Legislative scrutiny of external (i) Timeliness of examination of audit reports by the legislature (for audit reports reports received within the last three years). (ii) Extent of hearings on key findings undertaken by the legislature. (iii) Issuance of recommended actions by the legislature and implementation by the executive. Source: IEG analysis of PEFA instrument. Table C.5. PEFA Procurement Indicator PI-19 – Original and Reformulated Versions Original version Dimension (i) Evidence on the use of open competition for award of contracts that exceed the nationally established monetary threshold for small purchases (percentage of the number of contract awards that are above the threshold) (ii) Extent of justification for use of less competitive procurement methods (iii) Existence and operation of a procurement complaints mechanism Reformulated version Dimension Minimum requirements for dimension score (i) Transparency, comprehensiveness The legal and regulatory framework for procurement should: and competition in the legal and (i) be organized hierarchically and precedence is clearly established; regulatory framework (ii) be freely and easily accessible to the public through appropriate means; (iii) apply to all procurement undertaken using government funds11; (iv) make open competitive procurement the default method of procurement and define clearly the situations in which other methods can be used and how this is to be justified; (v) provide for public access to all of the following procurement information: government procurement plans, bidding opportunities, contract awards, and data on resolution of procurement complaints; (vi) provide for an independent administrative procurement review process for handling procurement complaints by participants prior to contract signature. 89 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA (ii) Use of competitive procurement When contracts are awarded by methods other than open competition, they are methods justified in accordance with the legal requirements: (iii) Public access to complete, Key procurement information (government procurement plans, bidding reliable and timely procurement opportunities, contract awards, and data on resolution of procurement complaints) information is made available to the public through appropriate means. (iv) Existence of an independent Complaints are reviewed by a body which: administrative procurement (i) is comprised of experienced professionals, familiar with the legal framework for complaints system. procurement, and includes members drawn from the private sector and civil society as well as government; (ii) is not involved in any capacity in procurement transactions or in the process leading to contract award decisions; (iii) does not charge fees that prohibit access by concerned parties; (iv) follows processes for submission and resolution of complaints that are clearly defined and publicly available; (v) exercises the authority to suspend the procurement process; (vi) issues decisions within the timeframe specified in the rules/regulations; and (vii) issues decisions that are binding on all parties (without precluding subsequent access to an external higher authority). Source: PEFA Secretariat. Table C.6. Country Analysis – PEFA Repeat Assessments Country Dates of assessments Sampling frame Albania Jul 06, Jan 12 ESW Review Bangladesh Jul 05, Dec 10 Field visit and ESW Review Ethiopia Oct. 07, Oct 10 Field visit and ESW Review Ghana Jun. 06. Jan 10 ESW Review Honduras April 09, Sep11 Indonesia Oct 07, Jan 11 ESW Review Moldova Apr 06, Jul 8 ESW Review Mozambique Feb 08, Mar 11 ESW Review Senegal Jan 09, Jun 11 Serbia Feb 07, Sept 10 ESW Review Sierra Leone Dec 07, Sept 10 ESW Review Tanzania May 06, May 10 Field visit and ESW Review Timor Leste Feb 07, Jun 10 ESW Review Source: IEG analysis of PEFA reports. 90 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA Table C.7. Procurement Indicators Compared Across 13 Countries % of countries with higher or % of countries % countries that could not be PEFA improving with lower or % of countries that stayed compared between the two indicator/dimension scores declining scores the same points of time PI-4 dim (i) 30 10 0 60 dim (ii) 30 10 20 40 PI-10 40 10 40 10 PI-12 dim (i) 0 20 60 20 dim (ii) 60 0 20 20 dim (iii) 10 50 30 10 dim (iv) 10 50 30 20 PI-19 dim (i) 60 20 10 10 dim (ii) 20 0 40 40 dim (iii) 10 0 60 30 PI-20 dim (i) 20 20 40 20 dim (ii) 20 20 30 30 dim (iii) 20 0 50 30 PI-21 dim (i) 10 10 50 30 dim (ii) 20 10 40 30 dim (iii) 20 40 20 20 PI-24 dim (i) 40 0 40 20 dim (ii) 50 10 20 20 dim (iii) 60 0 20 20 PI-26 dim (i) 20 20 40 20 dim (ii) 40 0 20 40 dim (iii) 50 10 20 20 PI-28 dim (i) 40 30 10 20 dim (ii) 20 20 30 20 dim (iii) 20 20 40 20 Average all 30 16 33 26 dimensions Source: IEG analysis. 91 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA Table C.8. Procurement Indicators Compared Across 10 Countries % of countries % of countries PEFA with higher or with lower or % of countries that stayed Total indicator/dimension improving declining the same scores scores PI-4 dim (i) 75 25 0 100 dim (ii) 50 17 33 100 PI-10 44 11 44 100 PI-12 dim (i) 0 25 75 100 dim (ii) 75 0 25 100 dim (iii) 11 56 33 100 dim (iv) 11 56 33 100 PI-19 dim (i) 67 22 11 100 dim (ii) 33 0 67 100 dim (iii) 14 0 86 100 PI-20 dim (i) 25 25 50 100 dim (ii) 29 29 43 100 dim (iii) 29 0 71 100 PI-21 dim (i) 14 14 71 100 dim (ii) 29 14 57 100 dim (iii) 25 50 25 100 PI-24 dim (i) 50 0 50 100 dim (ii) 63 13 25 100 dim (iii) 75 0 25 100 PI-26 dim (i) 25 25 50 100 dim (ii) 67 0 33 100 dim (iii) 63 13 25 100 PI-28 dim (i) 50 38 13 100 dim (ii) 29 29 43 100 dim (iii) 25 25 50 100 Average all 41 20 43 100 dimensions Source: IEG analysis. Note: For Ghana, some indicators were recalibrated from the 2006 assessment, and these values were used here. Such recalibration is not formally recognized by the PEFA Secretariat, but in this case seems to provide a more accurate picture. 92 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA Table C.9. Results from Reviews of Select PEFA Assessments Number Total of number reports Count of individual scores of with 0 1 2 3 reviewed evaluable Average Questions reports content score Total 1 To what extent does the ESW reflect 22 15 1.0 10 0 0 5 15 internal collaboration? (1, 2, 3) (score 3-0) 2 To what extent did the Client 22 20 2.3 0 7 0 13 20 Stakeholder participate in prepration of procurement ESW/AAA? (score 3- 0) 3 To what extent did other 22 21 1.0 2 17 2 0 21 Development Partners participate? (score 3-0) 4 What is the overall clarity of 22 22 2.0 0 1 21 0 22 objectives and analysis, structure of report (score 3-0) Are the objectives clear and consistent with the purpose of a PEFA? Is the analysis clear, and are there adequate linkages between the objectives, analysis, recommendations, and action plan? Is the structure of the report substantially consistent with the requirements in the PEFA guidelines? If not, do they explain the deviation? 5 Does the ESW/ AAA discuss the 22 22 2.1 0 4 11 7 22 quality of the overall legislative framework: (score 3-0) (MAPS1a and CPAR AI 6-9 and PEFA PI19i ) 6 Does the report discuss whether the 22 22 2.5 0 3 4 15 22 law states competitive procurement methods with exceptions clarified? (score 3-0) (MAPS1b; CPAR A2 q6; D4-10; PEFA ii) 7 Does the report discuss rules and 22 22 0.4 16 3 3 0 22 regulations for key aspects? (MAPS Indicator 1c-g; ) (CPAR A2 Q3; D6 q 1-3, D8-10; PEFA ii) (score 3-0) 8 Does the report mention the extent 22 22 0.3 17 4 1 0 22 to which regulations are implemented, updated, supported by users guides (MAPS2a-2f; CPAR D4 q1-5; D5q1-6; E6, D11 q1-13) (score 3-0) 93 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA 9 Does the report discuss the 22 22 2.4 0 4 5 13 22 functionality of the normative and regulatory procurement bodies? (MAPS:I4; CPAR: A1-Q11-13, D1- Q4-6; PEFA iii) (score 3-0) 10 Does the report discuss the 22 22 1.0 8 6 7 1 22 integration of Procurement into the Budgeting, Planning and Audit Process? (MAPS 3a-d; CPAR C q6- 9; PEFA PI 1,2, 4,10,12,20,21,24,26,28) (score 3-0) 11 Staff Capacity: Does the report 22 22 1.8 3 3 11 5 22 discuss capacity building and quality control guidance for staff performance? (MAPS 5c-d; CPAR D3Q1)) (score 3-0) Does it provide a detailed institutional analysis of key government ministries and agencies in the actual performance of public procurement responsibilities, their strengths and weaknesses, the governance environment in which they operate, and the incentives needed to improve performance? Does it discuss capacity needs of different institutions such as Ministries, Public sector agencies, CPCs, and so on? Does it discuss issues post contract? Does it discuss what has been done so far in terms of training and capacity enhancement? 12 Monitoring: Does the report discuss 22 22 1.9 2 4 10 6 22 the maintenance and dissemination of monitoring and procurement statistics? (MAPS 11; CPAR A2- Q12; D6-Q3) (score 3-0) 13 Corruption laws and enforcement: 22 22 0.5 14 4 4 0 22 To what extent does the ESW discuss ethics codes, corruption laws and enforcement? (MAPS I12; CPAR A2-Q15; A3-Q4; B-Q2; A3- Q2; A2-Q13; A#-Q1; D3) (score 3-0) 14 To what extent does the ESW 22 22 0.1 20 2 0 0 22 discuss Proc market and efficiency? (MAPS: Indicators 6 and 7; CPAR: D2-Q7, F2-Q-1, F2-Q2; A2-Q2, D7- Q4, D12-Q1, D12-Q2, D12-Q3) (score 3-0) 94 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA 15 To what extent does the ESW 22 22 2.1 1 6 5 10 22 discuss the complaints process and appeals mechanism? (MAPS: I10; CPAR: A3-Q3, D11-Q5-9, D1-Q5; PEFA iv) (score 3-0) 16 To what extent does the ESW cover 22 22 0.0 22 0 0 0 22 issues related to contract administration and mgmt? (MAPS: I8; CPAR D11:Q6; PEFA iv) (score 3-0) 17 Does the report cover e- 22 22 N/A 21 1 - - 22 procurement issues? (Y/N) 18 Does the report cover sub-national 22 22 N/A 22 0 - - 22 or local government issues? (Y?N) 19 Does the report discuss 22 22 N/A 22 0 - - 22 sustainability and 'Green' procurement (Y/N) 20 Does the report discuss community 22 22 N/A 22 0 - - 22 level procurement / Public / private partnerships? (Y/N) 21 Quantity of empirical data to support 22 0 0 11 9 2 22 findings. Source: IEG review of PEFA Assessment Reports. Note: For questions 17-20, 1 = the number of reports for which the response was Yes and 0 = number for which the response was No. For question 21 on the quantity of empirical data to support findings, 3 = High, 2 = Significant and 1 = Modest. PEFA Influence on PFM Aspects of Country Policies or Programs – Indirect Influence on Procurement In Ghana a government public financial management Action Plan adopted in 2006 was informed by the 2006 PEFA assessment; most of the proposed “quick wins” have been addressed. In the area of procurement, the Action Plan provided for strengthening of procurement processes, focusing on establishing effective procurement institutions and processes in Ministries, Departments and Agencies, Metropolitan, Municipal, and District Assemblies, and accompanying staff training. In the case of the repeat assessment, the Government coordinated and attended all meetings. A new Bank public financial management Reform project drew on PEFA in its design. The timing of the repeat assessment was critical, coming shortly after a new government was installed in January, 2009. PEFA helped the new government get an accurate picture of the public financial management situation, and thus the new Government was very committed to the PEFA process. The PEFA has been used to inform discussions during the development of the 2009-13 Public Financial Management Reform Plan (Mackie and Captio 2011). In Moldova, government commitment to the PEFA process was demonstrated by the fact that it carried out its own self-assessment using the PEFA methodology before the consultant team arrived. Once the consultants had completed their draft 95 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA assessment, the two assessments were compared, and scores adjusted as needed based of sharing of new information. In Mozambique, the government found the 2008 repeat assessment had more ownership than the first, since the tool was better understood by officials, in part as a result of training provided prior to the assessment mission. While none of the PEFA indicators are built into the Performance Assessment Framework for Public Financial Management (2007-09), the PEFA assessment is included in the Strategic Public Financial Management Vision (2010-20). However, this hasn’t yet been used as the basis for a program with actionable activities (Mackie and Caprio 2011). 2. Tanzania provides a less favorable example. The PEFA process was initially driven by the hope of Government and development partners that it could reduce transaction costs of annual PER/CFAA exercises. The PEFA Secretariat recommends repeat assessments take place on average every three years, and the average time between PEFA assessments is close to this at 2 ¾ years (PEFA Secretariat, 2011). Yet in the case of Tanzania, between 2004 and 2010, there were 8 different PEFA assessments carried out, with many published as part of other reports such as Public Expenditure and Financial Accountability Reviews (PEFARs). While some of the PEFAs had a narrow focus such as one on semi autonomous agencies and parastatals, and one on Zanzibar, most of them covered central government. The problem this creates is that there are unrealistic expectations regarding improved scores, with such a short time between assessments. In addition, changes in interpretation by assessors can become more important than actual performance changes when the interval is short. In Tanzania, these problems were further compounded by weak quality control and funding issues, capacity constraints, and staff turnover within government, and difficulties with the public financial management dialogue, all combining to paint PEFA as a donor-driven exercise, without any obvious reduction in transaction costs from previous assessment regimes. Some reports, such as the 2007-08 assessment published as part of the PEFAR, was not accepted by the government. Other related analytical work by the Bank was more highly regarded, for example Debt Sustainability Analyses carried out with the International Monetary Fund in 2007 and 2009, and World Bank, 2008 (Mackie and Caprio 2011). However, this more highly regarded work focuses on other issues besides procurement. PEFA Influence on PFM Aspects of Country Policies or Programs - Donor Assistance While there is evidence of the public financial management instrument serving as a focal point and baseline for public financial management reform, there is limited evidence of the PEFA serving as a point of harmonization for procurement reforms as a part of this process. Procurement reforms have found greater traction in 96 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA harmonization through more in-depth instruments such as the CPAR or MAPS. In Bangladesh, the 2011 PEFA assessment forms a baseline for progress on public financial management reforms and is important to enable donors to make operational decisions on budget support and use of country systems. This assessment builds on the 2009 Public Expenditure and Institutional Review, which emphasizes at many points the need for more effective linkages between procurement and other parts of the budget cycle. In Ghana, all key partners participated in the 2009 PEFA assessment process (World Bank, International Monetary Fund, DfID, Netherlands, the German Organization for Technical Cooperation, European Union) and provided comments on the draft. Development partners used the results as an input to decisions regarding budget support, and to provide a baseline for their fiduciary assessments. The Ghana Joint Assistance Strategy (G-JAS) 2007-2010 commits development partners to using the PEFA for medium-term monitoring of country financial management capacity. It was also noted that the External Review of Public Financial Management (ERPFM, incorporating the PEFA methodology since 2006) provides shared analytical foundation for partner-government dialogue, and for decisions on the use of country systems for investment projects. Selected PEFA indicators are incorporated into the Multi-donor Budget Support Performance Assessment Framework. While there has been no reduction in the number of public financial management assessments as a result of PEFA, the Government notes that missions are shorter, fiduciary assessments by France, the U.K. Department for International Development (DfID) and European Union are considered more efficient using PEFA as a baseline (Mackie and Caprio 2011). In Moldova, the Bank led the process, and other key partners participated in funding and/or participating in workshops (DfID, SIDA, International Monetary Fund, and the European Commission). Donors cite PEFA scores in their fiduciary assessments, but still field separate teams, since donors believe PEFA is not sufficient for fiduciary risk assessment. There is no evidence of a reduction in public financial management diagnostics, with broad based OECD and European Commission diagnostics circa 2011 that have not relied on the PEFA methodology. Donors have recognized that their own practices need to improve (the “D” indicators all scored D or D+) and have been addressing the issue. These indicators are indirectly linked to procurement. For example, when budget support is unpredictable (indicator D-1), it can both delay the procurement process, and increase payments in arrears on contracts already under way (Mackie and Caprio 2011). In Mozambique, the Government and development partners prepared an assessment of the Health Sector; based on an adaption of the PEFA methodology 97 APPENDIX C: SUPPORTING CAPACITY THROUGH AAA (published in April 2009). This was financed by the Swiss Development Cooperation on behalf of the Co-operation Partners and coordinated by a joint Ministry of Health/ Development Partners’ Reference Group set up for this purpose. PEFA is also used to inform decisions on the use of country systems, and evaluations of fiduciary risk. PEFA may have led to some reduction in the number of general assessments, though the International Monetary Fund’s Report on the Observance of Standards and Codes and PEFA 2008 missions took place only a few months apart, and the added value of conducting two assessments so close together is not clear. But there are still a large number of individual fiduciary assessments carried out (Mackie and Caprio 2011). 1. In Tanzania, the public financial management dialog has been problematic. In 2004, PFMRP II was developed, and supported by a group of development partners. A low disbursement rate and a midterm evaluation, which queried the level of government ownership, led to the suspension of PFMRP II a year early in June 2008, and the creation of PFMRP III which ran from 2008 to 2010 (rated moderately satisfactory in the 2009 GBS review). This may have contributed to the Government’s low regard for PEFAs and PERs; problems in the PEFA process discussed earlier might have also contributed to the problematic dialog. The PEFA Assessment may have indirectly supported the Government in public financial management areas which previously did not receive adequate attention e.g. strengthening parliamentary scrutiny, procurement and the auditor general‘s office. Development partners reported that the PEFA Assessment was not a primary source of information in the ongoing development of PFMRP IV, the current instrument for multi-donor support to public financial management. Reports of the Controller Auditor General were considered to be more useful. 98 Appendix D: Procurement-Related Lending Operations, 2002–11 Sample Construction and Select Findings DEVELOPMENT POLICY OPERATIONS Development policy operations (DPOs) were identified as relevant for procurement reform primarily on the basis of the extent to which their policy reform content was focused on procurement. Two sources were used; Business Warehouse and the Development Policy Actions Database (DPAD) maintained by OPCS. DPAD supplements Business Warehouse’s detailed project information with consolidated information on all prior actions and benchmarks associated with each DPO. Prior actions are codified by theme and sector. Combining projects that appeared in both datasets, 544 DPOs in all were identified as being approved between FY02 and FY11 (Table D.1). Table D.1. DPO Databases In DPAD Not in DPAD Total In Business Warehouse 544 38 582 Not in Business Warehouse 10 Total 554 Source: IEG analysis of World Bank data. Note: The 38 projects that are not in the DPAD database were not used because the identification and classification of the prior actions in each DPL –only available in DPAD- was needed to categorize the projects as procurement-related. 2. To identify procurement-related DPOs, a word search of prior actions that included the word “procurement” was undertaken. The total number of prior actions was also available and hence the extent to which procurement-related elements were present in each operation was determined (Table D.2). Table D.2. Procurement-Related DPOs Some procurement content Of which: High procurement content Criteria At least one prior action is procurement Share of procurement related prior actions related over total number of prior actions greater or equal than 15%a Number of DPOs 202 42b Share of total number of 37 7 DPOs (%) Source: IEG analysis of World Bank data. a. For the projects with high procurement content, the average number of prior actions was 11 of which an average of 2 prior actions were procurement related. b. The 42 projects with high procurement content represent 21% of the 202 in the procurement sample. 99 APPENDIX D: PROCUREMENT-RELATED LENDING OPERATIONS, 2002–11 Selection was further narrowed applying the country selectivity framework for the overall evaluation described in Chapter 1, which describes 21 focus countries (including 11 field visit countries). Of these, 18 focus countries had 79 DPLs with at least some procurement content, and of these, only 13 had high procurement content, that is, with at least 15 percent of all conditions referring to procurement). Table D.3. Development Policy Operations—Procurement Sample Reviewed countries Of which field visit All countries (21) countries(11) DPOs with some procurement content 202 79 39a Of which closed 199 77 37 Of which DPOs with high procurement 42 13 2b content Source: IEG analysis of World Bank data. a. In the following nine countries: Azerbaijan, Bangladesh, Ethiopia, Indonesia, Morocco, Peru, Senegal, Tanzania, and Turkey. b. These included the Bhutan Development Policy Grant, Mozambique PRSC 6, and Sierra Leone Third and Fourth Governance Reform DPOs and the Growth Credit DPO. Table D.4. Development Policy Operations with Procurement Elements (2002–11) Operations with prior actions relating to public expenditure, financial management, and Operations with prior actions referring explicitly to procurement procurement No. projects Commitments No. projects Commitments Average Average ($millions) ($millions) share of share of procurement “Theme 27” Fiscal No. Commitment prior actions prior actions year projects ($ millions) (%) (%) 2002 44 9,794 33 7,185 17 4,756 7.4 20.1 2003 45 6,000 30 2,960 15 1,651 9.5 23.5 2004 40 6,132 30 4,172 18 1,671 10.2 25.8 2005 55 6,528 41 3,571 21 2,039 10.6 24.9 2006 50 7,099 37 4,103 22 2,250 9.4 25.9 2007 57 6,280 47 4,637 22 1,680 12.7 29.1 2008 45 6,464 37 5,068 16 1,999 11.6 25.8 2009 67 17,571 48 12,735 28 5,459 12.7 31.3 2010 79 21,056 54 11,554 25 4,584 15.3 35.4 2011 62 9,615 45 5,839 18 3,506 14.4 35.4 Source: IEG analysis of World Bank data. Note: All projects with components under “Theme 27,” which consists of public expenditure, financial management, and procurement. These DPOs include the word “procurement” in the prior action 100 APPENDIX D: PROCUREMENT-RELATED LENDING OPERATIONS, 2002–11 Investment Lending Based on their content, World Bank projects are classified and assigned sector and themes codes in a variety of topics. Theme 27, that refers to “public expenditure, financial management, and procurement,” was used to narrow the search for procurement investment loans. Thus 193 projects were identified as having at least one component categorized under Theme 27, and project appraisal documents for these 193 projects were reviewed to identify the share of procurement content in each. The share of procurement content was assigned based on the nature of components and activities and their importance in relation to the overall project objectives, and not on financing commitments designated to components or activities. As a result, 101 projects with procurement elements were identified; that is, close to half of the projects were discarded from the initial sample (Table D.4). Table D.5. Investment Lending Projects, FY02-11 Number of Share of total projects number of ILs (%) All investment lending 2,497 100 Of which with components categorized under Theme 27 193 8 Of which with components with procurement elements 101 4 Source: IEG analysis of World Bank data. Note: IL = investment loan. From the 101 projects in the procurement sample, only 14 projects had high procurement content (Table D.5). 1 Of these 14, 6 pertain to the South Asia Region and half were approved between FY02 and FY05. Table D.6. Procurement-Related Investment Loans Some procurement content High procurement content Criteria Share of procurement content Share of procurement content equal greater than zero or greater than 30% Number of ILs 101 14a Share of total number of ILs 4% 0.6% Source: IEG analysis of World Bank data. a. The 14 projects with high procurement content represent 14 percent of the 101 in the procurement sample. Similar to the review of procurement DPOs, the evaluation also focused on projects from the 21 focus countries. There were 25 projects pertaining to 15 of the 21 reviewed countries. 2 Only three projects, in two countries, had high procurement content: the Bhutan Institutional Capacity Building for Procurement Project, and Bangladesh Public Procurement Reform Projects I and II. 101 APPENDIX D: PROCUREMENT-RELATED LENDING OPERATIONS, 2002–11 Table D.7. Investment Lending - Procurement Sample Reviewed Of which field visit All countries countries (21) countries (11) ILs with some procurement content 101 25 14 Of which closed 43 11 5 Of which ILs with high procurement content 14 3 2a Source: IEG analysis of World Bank data. a Bangladesh INSTITUTIONAL DEVELOPMENT FUNDS WITH A PROCUREMENT FOCUS More than 200 IDF grants over the last decade refer to public expenditure management-related themes. Around 62 bear titles that refer explicitly to procurement. Thirteen IDF projects from the 21 focus countries were reviewed in depth. Table D.8. Procurement-Related IDFs Some procurement content Criteria Projects explicitly referring to procurementa Number of IDFs 62b Share of total number of IDFs 10% Source: IEG analysis of World Bank data. a. These IDFs include the word "Procurement" in the project name. b. A total of 612 IDFs were approved during FY02-11. Table D.9. IDF - Procurement Sample Reviewed Of which field visit (11) All countries countries (21) countries IDFs with some procurement content 62 13 9 Of which closed 51 13 9 Source: IEG analysis of World Bank data. Note: The 13 IDFs from the focus countries included Azerbaijan Strengthening Implementation of Public Procurement Law; Azerbaijan e- Procurement Strengthening; Indonesia Development of Procurement Training Providers; Lao PDR Improve Efficiency of Public Procurement; Lao PDR Strengthen Implementation, Monitoring and Evaluation of the Public Procurement System; Morocco Modernization of Public Procurement in Maghreb; Philippines Institutionalization of Procurement; Philippines Strengthening Internal Audit for Procurement; Philippines Professionalization of Public Procurement Practitioners and Functions; Philippines Strengthening the Capacity of the Procurement System; Turkey Public Procurement Reform; Vietnam Public Procurement Bulletin; and Vietnam Strengthen Implementation, Monitoring and Evaluation of the Public Procurement System. Overall sample characteristics are described in Table D.10 and D.11. 102 APPENDIX D: PROCUREMENT-RELATED LENDING OPERATIONS, 2002–11 Table D.10. Sampled Projects - Descriptive Statistics by Lending Instrument and Region IBRD/IDA amount Grant amount Number of projects ($ millions) ($ millions) Instrument type DPL 68 12,225 - IL 15 393 - IDF 15 - 4 Total 98 12,618 4 Region AFR 42 3,491 0 EAP 27 4,915 2 ECA 10 2,568 1 LCR 11 1,462 0 MNA 2 120 0 SAR 6 62 0 Total 98 12,618 4 Source: IEG analysis of World Bank data. Table D.11. Sampled Projects - Descriptive Statistics: FCS and Small States Number of projects Total project amount ($ millions) Fragile state 8 114 DPL 6 85 IL 2 29 Not fragile state 90 12,508 DPL 62 12,140 IDF 15 4 IL 13 363 Total 98 12,622 Small state 4 34 DPL 2 32 IDF 1 0 IL 1 2 Not small state 94 12,588 DPL 66 12,193 IDF 14 3 IL 14 391 Total 98 12,622 Source: IEG analysis of World Bank data. 103 APPENDIX D: PROCUREMENT-RELATED LENDING OPERATIONS, 2002–11 Table D.12. Coverage of Issues Related to Procurement in Lending Operations at the Country Level – 20 Sample Countries A. Legal, Regulatory, and Institutional Legislative framework 14 Regulatory framework 18 Implementation arrangements for applicable regulation 17 Functionality of the normative and regulatory procurement 18 Integration of procurement into the budgeting, planning and audit process 2 Monitoring and procurement statistics 4 Capacity/cadres for public procurement 16 B. Fiduciary - Transparency and Corruption Transparency and disclosure 16 Complaints process and appeal mechanism/dispute settlement 6 Internal or External Audits 15 Corruption and Fraud 8 C. Procurement Efficiency and Capacity development Market for public procurement 9 E-procurement 11 Local training 16 International training 6 Training of trainers institutions 6 Sustainable and long-term training strategy 6 Improved contract administration/management 4 Subnational procurement 12 Source: IEG analysis of World Bank data. 104 Appendix E: Collaboration PREM, Financial Management, and Procurement Figure E.1. Sector Mapping of PFM Lending Source: World Bank Governance and Public Sector Management, Poverty Reduction, and Economic Management Vice Presidency/Network. Table E.1. Examples of Collaboration– PREM, Financial Management, and Procurement Date Project Type approved Collaborative features (P108787) Lao PDR Public Fin 2008 and First led by FM, second by PREM. Management and IL 2013 Senior Procurement Specialist on team found that Government P128854 Lao PDR PFM has put in place standard procurement regulations and Strengthening Project processes, but hasn't lead to improved outcomes. Each procurement has stack of paper behind it, but is not credible. Found provinces more in charge of procurement, using off budget resources, all untouched by centrally driven reforms. (P070561) Turkey DPL (first of 2001 Led by FM, good collaboration PREM and PR, all working Programmatic Financial & series of 3) together with excellent collaboration based on people wanting Public Sector Adjustment Loan to work together. (PFPSAL) (P088543) Brazil Minas Gerais DPL 2006 PREM led, with close collaboration from Partnership For Development (subnational) PR, both looking at fiduciary and developmental aspects in a swap. Monitoring systems developed for supervision. SWAP in Minas Gerais used project to strengthen UCS and used UCS to manage project. Source: IEG interviews. 105 APPENDIX E: COLLABORATION—PREM, FINANCIAL MANAGEMENT, AND PROCUREMENT WBI Support to Building Procurement Capacity Table E.2. WBI Programmatic Procurement Support Through Trust Funds (June 2012) Country Trust Finish Amount O/S Name AIS # Fund Start Date Status Fund # Date Disbursed Commitments Source Strengthening Reform and Improving Public Procurement Performance in AFR and EAP Program TE- Implementation Multiple P117984- P117984 11/10/2010 4/30/2013 ACTV $297,514 $47,705 (GPF) TRAC- TF098464 BNPP Fragility & Conflict: Procurement Reform and Capacity Building in Fragile TE- 8/2/2012 States The P117984- (Extension P128719 8/2/2010 ACTV $228,229 $68,475 Netherlands TRAC- being TF097591 processed) Multi-Stakeholder Procurement Monitoring II TE- (East Africa Quick Wins) The P128707- P128707 1/9/2012 6/9/2013 ACTV $28,335 $50,007 Netherlands TRAC- TF011487 Multi-Stakeholder Procurement Monitoring (East TE- Africa Quick Wins) The P117984- P117984 8/1/2011 6/30/2012 ACTV $49,335 $0 Netherlands TRAC- TF010323 Curriculum Development for Procurement Leadership in TE- MENA P109309- P109309 Spain 1/31/2011 6/30/2012 ACTV $14,103 $38,617 TRAC- TF098836 Source: WBI. 106 APPENDIX E: COLLABORATION—PREM, FINANCIAL MANAGEMENT, AND PROCUREMENT Table E.3. WBI Self Assessment of Contract Monitoring Outcomes (February 2013) Outcomes Description of the Outcome Significance of the Outcome Contribution of WBI 1 Steering Group/ In 2011 GTZ agreed to co- GTZ is a powerful bilateral WBI proposed the idea of the Open Advocacy (1) convene and committed development actor. Their agreement Contracting Lab to GTZ and engaged resources to the Open to co-convene the OC Lab will lend in a series of conversations with them Contracting (OC) Lab. The Lab more credibility and attract other about the Lab’s initial vision and seeks to convene leaders and organizations to join. structure. innovators who want to make contract transparency and monitoring the norm. 2 Steering Group/ In 2012, after a year-long outreach To make the OC Lab and its efforts WBI co-designed and co-sponsored a Advocacy (2) and consultation process, 5 more effective and sustainable, it must be design meeting in Leesburg in May organizations (namely CoST, led by a diverse and large group of 2012 where the 5 additional Integrity Action, the Government of organizations with resources, organizations agreed to join the Lab. Philippines, Oxfam America, and influence and expertise. WBI also engaged in conversation Transparency International) joined and consultations with these the OC Lab in addition to GTZ and organizations before the design WBI. Together these 7 meeting. organizations now form the steering group of the OC Lab, and co-design and co-fund its activities. 3 Steering Group/ In January 2013 Steering group The ultimate aim of the OC Lab is to WBI engaged in conversations and Advocacy 3 members, such as COST and make open contracting a new social convened meetings with OC Lab Integrity Action, and OC Lab norm. It is thus essential that other members to build a better members, such a Publish What global initiatives embrace and/or understanding of open contracting You Pay and the Africa Center for adopt open contracting. and to identify strategic advocacy Freedom of Information, are opportunities. advocating to include open contracting principles in the G8, the Open Government Partnership and in the Extractive Industries Transparency Initiative. 4 Uganda/ In June 2012, after a 1-year Uniting CSOs in a coalition will WBI facilitated the first meeting where Coalition process, a group of diverse actors make them a more credible these diverse actors working on Building to from the public, private and civil counterpart to government procurement issues came together. Promote OC (1) society sectors in Uganda signed a agencies. As a Partnership CSOs During and after the meeting WBI Memorandum of Understand to are able to leverage each other’s provided advice and guidance on how form the Uganda Contract resources, networks and expertise. to form a coalition. Monitoring Coalition. The aim of They can also better coordinate the Partnership is to improve advocacy and monitoring efforts. Uganda’s procurement system through monitoring and advocacy. 5 Uganda/ Over the summer of 2012, a Having a tool will enable the WBI provided seed funding, shared Coalition subgroup of the Coalition with Coalition to collect appropriate and monitoring tools from other countries Building to experience in education and with high quality contract performance and reviewed the draft tool. Promote OC 2 input from the Ministry of data that can then be used for Education developed a tool that advocacy and in policy dialogue with will enable community members to government agencies. This makes monitor the construction of the Coalition a more capable partner schools. for the Ministry of Education, and one that is interested in delivering well-built schools to citizens. 6 Uganda/ Equipped with a formal coalition This public scrutiny will encourage WBI supported the Coalition in Coalition structure and deeper knowledge of the school, constructors and local developing a united message to Building to and tools on contract monitoring, government representatives to build engage constructively with the Promote OC 3 the Coalition secured agreement better schools. If the results of this government. This included coaching, from the Ministry of Education in initial monitoring are useful to the network building support and the summer of 2012 to monitor Ministry of Education, other technical advice on procurement CSO contract performance of school Ministries might agree to more monitoring. construction. The agreement will monitoring. 107 APPENDIX E: COLLABORATION—PREM, FINANCIAL MANAGEMENT, AND PROCUREMENT empower members of the Coalition to monitor the construction of selected schools in Uganda. 7 Mongolia/ In June 2012 a group of 25 CSOs Uniting CSOs in a coalition will WBI provided guidance to several Regulatory in Mongolia working on make them a more credible individuals and CSOs who were trying Changes to procurement formed a formal counterpart to government to form the Partnership, and advised Promote OC 1 coalition, called the Public agencies. As a Partnership they can on engaging potential partnership Procurement Partnership. The aim leverage each other’s resources, members and on structuring the of the Partnership is to improve networks and expertise. They can Partnership. WBI shared examples of Mongolia’s procurement system also better coordinate advocacy and other formal procurement coalitions. through monitoring and advocacy. monitoring efforts. 8 Mongolia/ In the summer of 2012, the Public Having a supporting infrastructure, WBI supported the Partnership by Regulatory Procurement Partnership such as a strategic plan, will sharing with them strategic plans from Changes to developed a 5 year strategic plan strengthen the capacity of CSOs to other coalitions and giving feedback Promote OC 2 with a focus on shaping the new advocate for reforms and to monitor on the different iterations of the plan. procurement law implementing contracting processes in a more WBI also funded a workshop on regulations in the short term. targeted and strategic manner, thus strategic planning for board members making their efforts more effective. of the Partnership. 9 Mongolia/ Over the summer of 2012, the Coordinating CSO advocacy efforts WBI supported the Partnership in Regulatory Public Procurement Partnership will magnify their strength to engage developing a united message to Changes to engaged in a policy dialogue with with government agencies on policy engage constructively with the Promote OC 3 the Ministry of Finance on the formulation and implementation. government. This included coaching, Implementing Rules and network building support and Regulations of the newly amended technical advice on procurement CSO Public Procurement law. This was monitoring. the first time that CSOs working on procurement in Mongolia advocated for regulatory changes as a joint network with a united message. 10 Mongolia As a result of the dialogue, in the The changes in the regulations will WBI supported the Partnership in Regulatory Fall of 2012 the Ministry of Finance institutionalize citizen participation in developing a united message to Changes to changed the Implementing Rules procurement in terms of values and engage constructively with the Promote OC 4 and Regulations of the newly practicalities. government. This included coaching, amended Public Procurement law. network building support and Several of the Partnership’s asks, technical advice on procurement CSO such as allowances for monitors monitoring. and the use of specific reporting templates were incorporated in the Implementing Rules and Regulations. The Partnership will now support the Ministry in coordinating CSO procurement monitoring efforts that are required under the amended procurement law. 11 Nigeria In November 2012, the Power Outcome demonstrates the power of The Nigerian Contract Monitoring ATI Holding Company of Nigeria multi-stakeholder engagement and Coalition is a division of the West released some records and monitoring, as it contributes to African Contract Monitoring Coalition information demanded by the improving compliance with the which is being regionally coordinated Nigeria Contract Monitoring Public Procurement Act and rules, by the Ghana Anti-Corruption coalition regarding a World Bank- which in turn leads to improvements Commission (GACC) which is supported contract for the supply in budget implementation and public supported by WBI through network and installation of High Voltage service delivery. strengthening and coaching on social Distribution systems in Abuja, reporting practices. The Nigerian CM Lagos and Ibadan, in response to coalition is currently implementing a the favorable ruling of the Federal WB-IDA supported project titled High Court in Abuja on a suit “Multi-stakeholder Engagement for brought by the Coalition in Effective Public Procurement Process September 2012. in Nigeria” 108 APPENDIX E: COLLABORATION—PREM, FINANCIAL MANAGEMENT, AND PROCUREMENT 12 Nigeria In September 2012 the Federal This result is significant to the WBI facilitated the convening of the ATI High Court in Abuja ruled favorably contract monitoring process as it coalition through multi-stakeholder on a suit brought by the Public and establishes a legal precedent engagement to improve compliance Private Development Centre, supporting access to information. with the Public Procurement Act and national convener of the Nigerian This outcome demonstrates the Rules including workshops and CM Coalition in September 2012 power of diverse stakeholders trainings organized in partnership with that demanded the disclosure of working together as coalition to AFR. WBI continues to support the records and information on the promote transparency and coalition by providing coaching and WB-supported contract for the accountability in public contracting. network-building support with other supply and installation of High coalitions in West and East Africa. Voltage Distribution systems in Abuja, Lagos and Ibadan. 13 Nigeria In a July 2012 training workshop in Outcome demonstrates the power of WBI facilitated the convening of the Capacity Abuja, the Nigerian CM Coalition multi-stakeholder engagement and coalition through multi-stakeholder Building built the capacity of the members monitoring. The coalition composed engagement to improve compliance of the Nigerian Society of of the Bureau of Public Procurement with the Public Procurement Act and Engineers and of civil society (BPP), the Nigerian Society of Rules including workshops and representatives to effectively Engineers, a media representative trainings organized in partnership with monitor power sector procurement and civil society organizations. AFR. WBI continues to support the processes, from project conception CSOs include the national convener coalition by providing coaching and to contract award and through Public and Private Development network-building support with other project implementation. The Centre (PPDC), Media Rights coalitions in West and East Africa. coalition also trained the observers Agenda (MRA), Centre for on how to use both the 2007 Organizational and Professional Public Procurement Act and the Ethics (COPE-AFRICA), Initiative for 2011 Freedom of Information Act, Food, Environment and Health procurement monitoring tools Society (IFEHS). The coalition is developed by the Public and training and deploying observers, Private Development Centre in comprising engineering consultation with a wide range of professionals and representatives of stakeholders, to observe and civil society organizations, to report on the procurement process monitor procurement processes for through the Procurement Portal power projects in Nigeria. Right from Observatory. In addition, The project conception to contract award coalition, through an expert and then unto project committee set up at the Nigerian implementation, the coalition is Society of Engineers, also using monitoring tools, publications developed a standard and training materials developed “Infrastructure Rating and from different members of the Assessment Template” for project coalition. implementation monitoring after the contract award processes. The Bureau of Public Procurement, which is represented on the coalition, provided publications and training materials for all participants at the training. 14 Ghana In July 2012, the University of The GIS-based map tool is WBI created the GIS-based map as a (EI Map) Ghana took on ownership and referenced by the Oil/Gas Platform pilot, building on a need identified maintenance responsibilities for an of civil society groups as a good from consultations with local innovative GIS-based map. This example of a way to combine stakeholders and WBI new resource for the first time information for non-technical people experience/expertise with its mapping combines all publicly available such as those dealing advocacy. of all World Bank projects. This extractive industries data for The tool is presented in a digestible, involved research to compile the Ghana in one place, including intuitive and centralized way. datasets, creating the customized mining and oil field locations, Access to this information and mapping platform and uploading contracts, production and revenue collaboration among the various content, and getting feedback on the data, corporate social stakeholders will lead to more prototype, followed by supporting responsibility projects, and transparent and accountable awareness-raising on the new tool. It underlying socio-economic contracting practices in extractive was recognized as a good example of indicators. This map aids industries. The tool itself also offers putting technical capacity in touch monitoring of extractive industries great potential for application in with the oil and gas platform that use 109 APPENDIX E: COLLABORATION—PREM, FINANCIAL MANAGEMENT, AND PROCUREMENT in Ghana and is used by a range of other settings, fostering knowledge the data. Most importantly, WBI stakeholders from government exchange among countries to provided training to University of officials to parliamentarians to the support increased transparency and Ghana on how to use the system for Oil/Gas Platform of civil society accountability in public contracting. the GIS-based map to take on groups ownership and upkeep of the map and that provided a model to sustain its maintenance and use. This training also helped the University to gain web-mapping capacity and to put their GIS data online. Source: WBI. Note: GTZ = German Organization for Technical Cooperation; OC = open contracting; WBI = World Bank Institute. 110 BIBLIOGRAPHY Bibliography APEC (Asian Pacific Economic Cooperation). 1999. “Government Procurement Experts Group Non-Binding Principles on Government Procurement”. August Aberdeen Group. 2008. The E-Procurement 1999. Singapore: APEC. Benchmark Report: Driving Year-Over- ASIL (American Society of International Law). Year Superior Performance. Boston: 2013. Will the World Trade Aberdeen Group. Organization Finally Tackle Corruption ADB (Asian Development Bank). 2002. in Public Purchasing? The Revised “Understanding Public Procurement.” Agreement on Government Governance Brief, Asian Development Procurement. Bank. Manila: ADB. http//www.asil.org/insights130415.cf m. ———. 2007. Environmentally Responsible Procurement—A Reference Guide. Manila: Blue, Richard, and John Eriksson. 2011. ADB. “Evaluation of the Implementation of the Paris Declaration: United States AfDB. (African Development Bank). 2011. Government Synthesis Report”. “Evaluation of Paris Declaration Arlington, VA: The United States Implementation at the African Agency for International Development. Development Bank.” Operations Evaluation Department. Tunis: AfDB. Burger, Philippe, and I. Hawkesworth. 2011. How to Attain Value for Money: ———. 2012. “Determination of Financial Comparing PPP and Traditional Thresholds for Modes of Procurement of Infrastructure Public Procurement. OECD Goods, Works and Non-Consulting Journal on Budgeting, Volume 2011/1. Services”. African Development Bank Paris, France: OECD. Group, Procurement Policy and Fiduciary Services Department. January Casartelli, G., E. Wolfstetter. 2007. "Selection and 2012. Tunis: AfDB. Employment of Consultants: the Effectiveness of the World Bank's AfDB, IDB (Inter-American Development Bank), Policy" Working Paper. Washington, World Bank, and Curtin University of DC: World Bank. Technology. 2007. International Survey of Procurement. Perth. Commonwealth of Australia. 2008. Commonwealth Procurement Guidelines, Agaba, Edgar, and Nigel Shipman. 2007l “Public December 2008. Issued under Procurement Reform in Developing Regulation 7 of the Financial Countries: The Uganda Experience.” In Management and Accountability G. Pig and K.V. Thai (eds.) Advancing Regulations, 1997. Canberra: Public Procurement: Practices, Innovation, Commonwealth of Australia. and Knowledge Sharing (Boca Raton, FL: Pr. Academics Press), 373-91. Debevoise, E.W., and C. Yukins. 2010. “Assessing the World Bank’s Proposed Al-Azar, Rima. 2011. “A Tale of Two CDDs: Revision of Its Procurement Lessons Learned for Mitigating the Risk Guidelines,” The Government Contractor, of Fraud and Corruption in CDD 52, (21). Operations.” PowerPoint presentation, World Bank, Washington, DC. Dehn, Jan, Ritva Reinikka, and Jakob Svensson. 2003. “Survey Tools for Assessing Andrews, Matt. 2013. The Limits of Institutional Performance in Service Delivery.” In Reform in Development: Changing Rules Francois Bourguignon and Luiz Pereira for Realistic Solutions. Cambridge: da Silva (eds.), Evaluating the Poverty and Cambridge University Press 128-130. Distributional Impact of Economic Policies. 111 BIBLIOGRAPHY New York: Oxford University Press and ———. 1990. “A Review of Procurement in World Bank. Policy-Based Lending.” Report no. 8341, Washington, DC: World Bank. Dener Cem, Joanna. A Watkins and William. L. Dorotinsky. 2011. “Financial ———. 2002. India Country Assistance Evaluation. Management Information Systems—25 Washington, DC: World Bank Years of World Bank Experience on ———. 2005. The Effectiveness of World Bank What Works and What Doesn’t.” World Support for Community-Based and –Driven Bank, Washington, DC. Development: An OED Evaluation. De Silva, S. 2000. “Community-based Washington, DC: World Bank. Contracting; A Review of Stakeholder ———. 2006a. Debt Relief for the Poorest—An Experience.” Washington DC: World Evaluation Update of the HIPC Initiative. Bank. Washington, DC: World Bank. DfID (Deparment for International ———. 2006b.Engaging with Fragile States: An Development). 2011. “Approach to Evaluation of World Bank Support for Low- Value for Money.” London: Department Income Countries Under Stress. for International Development. Washington, DC: World Bank. European Union, World Bank, and Inter- ———. 2006c. Hazards of Nature, Risks to American Development Bank. 2009. Development – An IEG Evaluation of World Perú Informe de la gestión de las finanzas Bank Assistance for Natural Disasters. públicas Basado en el marco de referencia Washington, DC: World Bank. para la medición del desempeño (PEFA). Lima: European Union, World Bank and ———. 2006d. Small States: Making the Most of Inter-American Development Bank. Development Assistance – A Synthesis of World Bank Evaluation Findings. Gartner Group. 2012. “Survey Shows Why Washington, DC: World Bank. Projects Fail.” http://www.gartner.com/id=2034616. ———. 2007a. The Development Potential of Regional Programs – An Evaluation of GSDRC. 2010. Helpdesk Research Report: Public World Bank Support of Multicountry Financial Management and Frontline Operations. Washington, DC: World Service Bank. Delivery. http://www.gsdrc.org/docs/ open/HD653.pdf,. ———. 2007b. Water Management in Agriculture—Ten Years of World Bank IDB (Inter-American Development Bank). 2010. Assistance, 1994-2004. Washington, DC: “Guide for Acceptance of the Use of World Bank. Country Procurement Systems.” June 2010. Washington DC: IDB. ———. 2008a. Country Financial Accountability Assessments and Country Procurement IAEA (International Atomic Energy Agency). Assessment Reports: How Effective Are 2011. Agency for Financial Regulations World Bank Fiduciary Diagnostics? and Rules AM VI/1: Procurement of Washington, DC: World Bank. Goods and Services, February 16. Mimeo. ———. 2008b. Public Sector Reform: What Works IEG (Independent Evaluation Group). 1977. and Why? An IEG Evaluation of World “The Role and Use of Consultants in Bank Support. Washington, DC: World Bank Group Projects.” Report no. 1824, Bank. Washington, DC: World Bank. ———. 2008c. The World Bank in Indonesia 1999- ———. 1981. “An Interim Report on 2006--Country Assistance Evaluation. Procurement Issues in Bank-Financed Washington, DC: World Bank Projects.” Report no. 3557, Washington, DC: World Bank. 112 BIBLIOGRAPHY ———. 2009a. Bangladesh Country Assistance ———. 2011c. Trust Fund Support for Evaluation 2001-2008. Washington, DC: Development – An Evaluation of the World World Bank Bank’s Trust Fund Portfolio. Washington, DC: World Bank. ———. 2009b. Improving Effectiveness and Outcomes for the Poor in Health, Nutrition ———. 2011d. World Bank Country-Level and Population--An Evaluation of World Engagement on Governance and Bank Group Support since 1997. Anticorruption: An Evaluation of the 2007 Washington, DC: World Bank Strategy & Implementation Plan. Washington, DC: World Bank. ———. 2009c. Review of IDA Internal Controls: An Evaluation of Management’s Assessment ———. 2011e. World Bank Progress in and the IAD Review. Washington, DC: Harmonization and Alignment in Low- World Bank. Income Countries. Washington, DC: World Bank. ———. 2009d. The World Bank Group Guarantee Instruments 1990-2007 –An Independent ———. 2012a. The Matrix System at Work: An Evaluation. Washington, DC: World Evaluation of the World Bank’s Bank Organizational Effectiveness. Washington, DC: World Bank. ———. 2010a. Cambodia – An IEG Country Assistance Evaluation, 1999-2006. ———. 2012b. “The World Bank and Public Washington, DC: World Bank Procurement. An Independent Evaluation.” Approach Paper. ———. 2010b. Climate Change and the World Bank Washington, DC: World Bank. Group. Phase II: The Challenge of Low- Carbon Development. Washington, DC: ———. 2012c. The World Bank Group’s Response to World Bank. the Global Economic Crisis—Phase II. Washington, DC: World Bank. ———. 2010d. IDA Internal Controls: Evaluation of Management’s Remediation Program-- ———. 2013. “Comparison of the International Report on Management’s Implementation Instruments on Public Procurement.” and IAD’s Review of the Five-Point Action Background paper. Washington, DC: Plan. Washington, DC: World Bank. World Bank. ———. 2010d. Mozambique Country Program IMF (International Monetary Fund). 2012. Evaluation. Washington, DC: World Update on the Bank’s Business Bank. Modernization: Results, Openness, and ———. 2010e. Poverty Reduction Support Credits: Accountability Spring 2012.”IMF World An Evaluation of World Bank Support. Bank Development Committee. Washington, DC: World Bank. Kivistö, T., V.M. Virolainen, and E. Tella. 2003. ———. 2010f. The World Bank in Nigeria 1998- Consortia purchasing and logistics in 2007—Country Assistance Evaluation. Kuopio area (Finland)--Lessons learned Washington, DC: World Bank from a 4-year project. IRSPP conference, Budapest, and in KV Thai et al. (eds.), ———. 2011a. “Aid Effectiveness 2005-10: (2005): Benchmarking municipal public Progress in Implementing the Paris procurement activities in Finland. In Declaration.” World Bank, Washington, Challenges in Public Procurement: an DC. International Perspective. ———. 2011b. Capturing Technology for Korpela, J., and M. Tuominen. 1994. Development – An IEG Evaluation of World Benchmarking Logistics Performance with Bank Group Activities in Information and an Application of the Analytical Hierarchy Communication Technologies. Process. Lappeenranta, Finland: Washington, DC: World Bank. Lappeenranta University of Technology. 113 BIBLIOGRAPHY Kruger. K, 2009. “Ban on negotiations in tender ———. 2011c. Strengthening Country Procurement procedures – Undermining the best Systems, prepared for the 4th High Level value for money.” Journal of Public Forum on Aide Effectiveness, Procurement, 4. November 29 – December 1, 2011, Busan, Korea. OECD/DAC Working Mackie, Andrew, and Giovanni Caprio. 2011. Party on Aid Effectiveness Task Force Assessing the Impact of the FEFA on Procurement. Framework. Washington, DC: PEFA Secretariat. OECD-DAC and World Bank. 2004. “Mainstreaming and Strengthening Ministry of Foreign Affairs of Denmark. 2008. Public Procurement: A Strategic “Evaluation of the Implementation of Framework.” Draft report of the 3rd Joint the Paris Declaration: Phase One Round Table on Strengthening Synthesis Report.” Denmark: Ministry Procurement Capacities in Developing of Foreign Affairs. Countries. Washington, DC, and Paris. OECD (Organization for Economic Cooperation Pallas, Christopher L., and Jonathan Wood. and Development). 2003. Mainstreaming 2009. “The World Bank’s Use of the Procurement Function into the Public Expenditure Policy and Effectiveness Country Systems for Procurement: A Dialogue. Paris: OECD. Good Idea Gone Bad?” Development Policy Review 27 (2): 215-230. ———. 2005a. “Paris Declaration on Aid Effectiveness--Ownership, PEFA Secretariat. 2011. Independent Evaluation of the Public Expenditure and Financial Harmonization, Alignment, Results and Accountability Program. Washington, DC: Mutual Accountability.” Paris, France: PEFA. OECD. Pollit, C., and G. Bouckaert. 2004. Public ———. 2005b. “Strengthening Procurement Management Reform: A Comparative Capacities in Developing Countries.” Analysis. New York: Oxford University Volume 3 of the OECD-DAC Guidelines Press. and Reference Series, Harmonising Donor Practices for Effective Aid Delivery. Paris: Sanchez, A. 1997. The World Bank Procurement OECD. Function – Adjusting to Emerging Needs. Washington, DC. World Bank. ———. 2006a. “The Challenge of Capacity Retrieved Development: Working Towards Good from: http://intresources.worldbank.or Practice.” DAC Reference Document, g/INTPROCUREMENT/Resources/Pro Paris. curement-Strategy.pdf. ———. 2006b. “Methodology for Assessment of Singh, J., Dilip R. LImaye, Brian Henderson, and National Procurement Systems.” Paris: Xiaoyu Shi. 2012. “Public Procurement OECD. Of Energy Efficiency Services: Lessons ———. 2009. “Compendium of Country from International Experience.” Examples and Lessons Learned from Washington, DC: World Bank. Applying the Methodology for Tonkin, Christine. 2012. “Procurement Assessment of National Procurement Governance Reform at the International Systems: Sharing Experiences.” OECD Atomic Energy Agency.” Mimeo. Journal on Development 9(4). Thai, K.V., et al. 2005. “Benchmarking Municipal ———. 2011a. International Engagement in Fragile Public Procurement Activities in States: Can’t We Do Better? Paris: OECD. Finland.” In Challenges in Public ———. 2011b. A Practical Guide for Transforming Procurement: An International Perspective. Procurement Systems. Paris: OECD. Academic Press. 114 BIBLIOGRAPHY UNCITRAL (United Nations Commission on ———. 2007a. “Detailed Implementation International Trade Law) Secretariat. Review. India Health Sector. 2006-2007.” (1994, 2011) “UNCITRAL Model Law on INT, Washington, DC: World Bank. Public Procurement.” UNCITRAL, ———. 2007b. “Mandatory Provisions for NCB Vienna, Austria. (for Bank/IDA financed contracts).” Volcker, Paul, Gustavo Gaviria, John Githongo, Washington, DC: World Bank. Ben W. Heineman Jr., Walter Van ———. 2007c. “Operational Approaches and Gerven, and Sir John Vereker. 2007. Financing in Fragile States.” Independent Panel Review of the World Washington, DC. Bank Group Department of Institutional Integrity. Washington, DC: World Bank. ———. 2007d. “Strengthening World Bank Group Engagement on Governance and World Bank. 1997. “The World Bank Anti-Corruption.” Washington, DC: Procurement Function- Adjusting to World Bank. Emerging Needs.” SecM97-854, Washington, DC: World Bank. ———. 2008a. “Enhanced Support to Staff Working in Fragile and Conflict- ———. 2002a. “Fiduciary Management for Affected States.” Washington, DC: Community-Driven Development World Bank. Projects: A Reference Guide.” Washington, DC: World Bank. ———. 2008b. “Use of Country Procurement Systems in Bank-Supported Operations: ———. 2002b. Guidelines for Country Procurement Proposed Piloting Program.” R008- Assessment Reviews. Washington, DC: 0036/5. Washington, DC: World Bank. World Bank. ———. 2009a. “Guidance to Staff for Using the ———. 2003. “Standard Bidding Documents: Company Risk Profile Database Supply and Installation of Information (CRPD).” World Bank, Washington, DC. Systems.” Washington, DC. ———. 2009b. “Handling Suspected and ———. 2004. “Country Procurement Issues Alleged Fraud and Corruption in Paper—Rwanda.” Washington, DC: Procurement: OPCS and INT Protocol.” World Bank. Washington, DC: World Bank. ———. 2005a. “Africa Capacity Building Task ———.2009c. “Piloting Program in Use of Force Report: Forging Effective States Country Systems: First Progress and Engaged Societies.” Washington, Report.” Operations Policy and DC: World Bank. Country Services. Washington, DC: ———.2005b. “Expanding the Use of Country World Bank. Systems in Bank-Supported Operations: ———. 2010a. Financial Management Sector Issues and Proposals.” Operations Strategy FY11-13. 6, 9. Washington, DC: Policy and Country Services, Financial Management Sector Board. Washington, DC. ———. 2010b. “Guidance Note on the ———. 2005c. “Meeting the Challenge of Operational Risk Assessment Africa’s Development: A World Bank Framework (ORAF) – Risks to Group Action Plan.” Africa Region Achieving Results.” Operations Policy Department, Washington, DC: World and Country Services, World Bank, Bank. Washington, DC. ———. 2006. “Quality of Information and ———. 2010c. “Guidance Note: Procurement Communication Technology Arrangements Applicable to Public- Components in Bank Projects.” QAG Private Partnership Contracts Financed Assessment, Washington, DC: World Bnak. 115 BIBLIOGRAPHY under World Bank Projects.” Services under IBRD Loans and IDA Washington, DC: World Bank. Credits and Grants by World Bank Borrowers.” Washington, DC: World ———. 2010d. Monitoring Report 2010. PEFA Bank. Secretariat Report. World Bank, Washington, ———. 2011g. “Strengthening Country DC. http://pefa.org/sites/pefa.org/file Procurement Systems.” Prepared for the s/attachments/ENG-%20MR-10- 4th High Level Forum on Aide FinalReport-May192011.pdf. Effectiveness, November 29 – December 1, Busan, Korea. Washington, DC: ———. 2010e. “Piloting Program in the Use of World Bank. Country Procurement Systems—Second Progress Report.” Washington, DC: ———. 2011h. “Strengthening World Bank World Bank. Engagement on Governance and Anti- Corruption: Procurement Companion ———. 2010f. “Procurement Risk Assessment Piece.” Washington, DC: World Bank. and Management System User Guide.” OPSOR, May 2010, Updated July 2011, ———. 2011i. World Development Report: Conflict, November 2011, January 2012. Security, and Development. Washington, Washington, DC: World Bank. DC: World Bank. ———2010g. “Public Procurement of Energy ———. 2012a. “Analysis of World Bank Efficiency Services: Lessons from Completed Cases of Fraud and International Experience.” Washington, Corruption from the Perspective of DC: World Bank. Procurement.” Integrity Vice Presidency, Washington, DC: World ———. 2010h. “Republic of Moldova Country Bank. Procurement Assessment Report.” Washington, DC: World Bank. ———. 2012b. “Applying the Principles for Good International Engagement in ———. 2011a. “Community-Driven Fragile States and Situations to Development Projects: A Review of Strengthen and Transform Public Governance Issues.” OPCS Document, Procurement Systems.” Background World Bank, Washington, DC. note, OPSOR. Washington, DC: World ———. 2011b. Country Partnership Strategy Bank. Progress Report for the Republic of Moldova ———. 2012c. “The Bank's Procurement Policies for the Period FY09-FY13. Washington, and Procedures: Initiating Discussion DC: World Bank. and Approach Papers--Proposed ———. 2011c. “Diversity Benchmarking Study Review.” Report submitted to the Audit in Selected Countries.” OPSOR, Mimeo. Committee and to CODE, Washington, Washington, DC: World Bank. DC: World Bank. ———. 2011d. “Financial Management ———. 2012d. “Fiduciary Management for Information Systems—25 Years of Community-Driven Development World Bank Experience on What Works Projects: A Reference Guide.” and What Doesn’t.” Washington, DC: Washington, DC: World Bank. World Bank. ———. 2012e. Financial Management and ———. 2011e. “Green Procurement Policy Procurement in World Bank Operations: Framework in Selected Countries.” Annual Report for FY11. Washington, OPSOR, Mimeo. Washington, DC: DC: World Bank. World Bank. ———. 2012f. “Green Procurement in Selected ———. 2011f. “Guidelines: Procurement of Environmental Policy Framework: Goods, Works, and Non-Consulting Review of the World Bank‘s 116 BIBLIOGRAPHY Procurement Policies and Procedures.” ———. 2012p. “Update on the Bank’s Business Background Paper. Washington, DC: Modernization: Results, Openness, and World Bank. Accountability Spring 2012.” DC2012- 0005. Washington, DC: World Bank. ———. 2012g. “Guidance Note for Design and Management of Procurement ———. 2012q. “Virtual VP Review of the Responsibilities in Community-Driven Procurement Policy Review Proposal-- Development Projects.” Washington, An Approach Paper.” Washington, DC: DC: World Bank. World Bank. ———. 2012h. “Investment Lending Reform: ———.2013a. “Approach for Private Sector Modernizing and Consolidating Procurement in Projects financed by Operational Policies and Procedures.” IBRD/IDA, including Public Private OPCS, Washington, DC: World Bank. Partnerships.” Washington, DC: World Bank. ———. 2012i. “OPCPR Procurement Risk Assessment and Management System ———. 2013b. “Audit of the Bank’s Operational User Guide.” Washington, DC: World Framework for Using Investigation Bank. Results in Bank-Funded Projects.” Report No. IBRD FY12-16. Washington, ———. 2012j. “Procurement and PPP DC: World Bank. Transactions Guidance for MDB Public- Sector Engagements and Procurement ———. 2013c. “Procurement in World Bank Principals Applicable to Private Sector Investment Operations. Phase I: A Transactions—Guidance for MDBs.” Proposed New Framework.” OPCS. Washington, DC: World Bank. Washington, DC: World Bank. ———. 2012k. “Procurement Risk Assessment ———. 2013d. “Red Flags Learning Tool.” and Management System User Guide.” Integrity Vice Presidency. Washington, DC: World Bank. http://go.worldbank.org/D1K227ET81. ———. 2012l. “Project Appraisal Document on a ———. 2013e. “Situations of Urgent Need of Proposed Credit in the Amount of SDR Assistance or Capacity Constraints: 26.1 million ($40.0 million Equivalent to Simplified Procurement Procedures and the Republic of Moldova for the Guiding Principles.” Moldova Education Reform Project).” ———. 2013f. “Supporting Countries in Washington, DC: World Bank. Strengthening their Public Procurement ———.2012m. “Public Procurement of Energy Systems: Second Generation Reforms” Efficient Products: Lessons from (draft). Operations Risk Management Around the World.” Energy Sector Department. Washington, DC: World Management Program. Washington, Bank. DC: World Bank. ———.2013g. “Use of Country Procurement ———.2012n. “Review of the World Bank’s Systems.” Background Paper, Procurement Policies and Procedures: Washington, DC, World Bank. Achieving Social Diversity Through the World Bank Institute. 2012. “Public-Private use of Supplier Preferences in the Partnership Reference Guide, Version Procurement Process.” Background 1.0.” Washington, DC: World Bank. paper. Washington, DC: World Bank. World Trade Organization. 1996. “Agreement ———.2012o. “Review of the World Bank’s on Government Procurement (GPA).” Procurement Policies and Procedures, Geneva: World Trade Organization. The Use and Impact of the Bank’s Policy of Domestic Preferences.” Background Paper. Washington, DC: World Bank. 117 Endnotes 1 The 14 projects are: Afghanistan Emergency Public Administration Project, Afghanistan Public Administration Capacity Building Project, Afghanistan Public Financial Management Reform, Argentina La Rioja Public Sector Strengthening Program, Bangladesh Public Procurement Reform Project I and II, Bhutan Institutional Capacity Building for Procurement Project, Cameroon Transparency and Accountability Capacity Building Project, Cape Verde Small and Medium-Size Enterprise Capacity Building and Economic Governance, Haiti Governance Technical Assistance Grant, Kenya Public Sector Management TAL, Panama Enhanced Public Sector Efficiency TAL, Panama Public Policy Reform TA Project, and Ukraine e-Development Project. 2 These investment loans correspond to the following 15 countries: Albania, Bangladesh, Bhutan, Colombia, Ethiopia, Ghana, Indonesia, Lao PDR, Mexico, Mozambique, Rwanda, Senegal, Sierra Leone, Tanzania, and Turkey. 118